| testing.com > Testing Craft > Techniques (Test Automation) > Test Drivers for Code > Komet |
While the test format proposed in this document cannot meet
all testing needs, a broad range of tests can be developed which
use it. Some of the advantages of the Komet test format are:
The Komet tests are reusable. Each test program can be used to
check many test cases, since the tested call parameters are
specified as command line arguments. This allows the same test
program to be invoked many times with different parameters in
order to verify the various behaviors of the tested call. This
means that less test code will have to be generated in order to
accomplish the required testing.
Komet tests are compatible with the Darkstar parallel test driver
and can be used for both functional and stress testing.
Standardization of test format will help reduce the learning
curve when developers start to use the tests.
Simple tests are needed during initial development. The Komet
tests are simple and easy to understand.
Much of the test code can be generated by modifying a simple
template or the code can be generated by a shell script driven by
a description of the system call and it's parameters. This means
that many of the Komet tests can be developed in a short time.
The simplicity and standard format of these tests will results in
less training for test support personnel and lower test
maintenance costs.
The Komet tests are implemented at two levels. At the lower
level, a Komet test support library contains wrappers for many of
the system calls and library routines.
These wrappers provide a common interface to the system call/library
routine which executes the system call with the provided
parameters and compares the return value of the system call/library
routine with the expected return value which is provided by a
wrapper parameter.
If an error is detected during the system call/library routine
execution, the error string produced is compared to the expected
error string provided by a wrapper parameter.
Any mismatch between expected and actual values causes the
execution of an error handling routine which is specified by a
function pointer parameter provided to the test support routine.
This error handling routine can implement anything from a simple
report error and return sequence to a report, cleanup, and exit
sequence. A number of error handlers are provided in the Komet
test support library, which provide error handling for routines
returning various data types. The name of each routine starts
with 'iferr_c_' or 'iferr_x_'. The 'iferr_c_' routines report the
error and allow the test to continue while the iferr_x_' routines
report the error and exit.
At a higher level there are Komet test commands, which are simple
command interfaces to a particular system call or library routine.
The arguments to the tested call are passed in on the command
line. Numeric arguments are translated by the strtol library
routine, which allows the numbers to be specified as hexadecimal
(proceeded by a 0x), octal (proceeded by a 0), or decimal (the
default base). Character strings are passed in unchanged. More
complex tested call arguments (pointers to malloced memory areas,
etc.) must be handled on a case by base basis during the cleanup
and customization phase of the test generation.
A Komet test command reports it's pass/fail status to the outside
world via an exit string (null for pass, an ascii representation
of errflg for failure).
Also, if the test fails, appropriate diagnostic messages are
available to help localize the failure.
Each Komet test support routine and test command has general
parameters which are the same for every test as well as test
specific parameters. These general parameters come first.
tag
exprrstr
expret
perrflg
errhdlr
perrdata
The name of a Komet library routine is the name of the tested
call appended with a "_kl".
#include <u.h>
#include <libc.h>
#include <stdio.h>
#include <prototype.h>
/*
This is the access_kl library module, which is a test support wrapper for
the access routine.
It executes the access routine with the supplied parameters and compares the
resulting error string and return value with the expected values provided by
the caller.
If either comparison fails, the error handler designated by the function pointer
'errhdlr' will be called, with the "access" name, a tag string used for tracing
which call this is the expected and actual error strings, the expected and
actual return values and the error flags which show the type of mismatch that
occured.
*/
int access_kl(char *tag, char *access_experrorstr, int access_expret, ulong *perrflg,
void (*errhdlr)(char *, char *, char *, char *, int, int, ulong *, void *),
void *perrdata, char *name, int mode)
{
int access_ret;
char errorstr[ERRLEN];
ulong lerrflg=0L;
errorstr[0] = '\0';
/* Execute access routine */
access_ret = (int)access(name, mode);
/* Was an error detected */
if(access_ret == (int)-1)
{
errstr(errorstr);
/* Compare error against expected error */
if(strcmp(access_experrorstr, errorstr) != 0)
{
lerrflg = lerrflg | 02L; /* set error string comparison mismatch bit */
}
}
/* Did access routine return the expected value */
if(access_ret != access_expret)
{
lerrflg = lerrflg | 04L; /* set return value comparison mismatch bit */
}
/* Call error handler if an error was detected */
if(lerrflg)
{
errhdlr("access", tag, access_experrorstr, errorstr, access_expret, access_ret, &lerrflg, (void *)perrdata);
}
*perrflg = *perrflg | lerrflg;
return(access_ret);
}
An error handler is a function which handles any cleanup and
reporting that is required when an error is detected.
An error handler can also be used to back out an error detected
by the Komet library routine, by clearing the appropriate error
bit in the error flag. This provides greater control over the
error checking performed by the Komet test.
Although you can write your own error handlers, the following
standard error handlers should be sufficient in most cases.
iferr_c_charstar
iferr_c_int
iferr_c_long
iferr_c_ulong
iferr_c_voidstar
iferr_c_zeroptr
iferr_x_charstar
iferr_x_int
iferr_x_long
iferr_x_ulong
iferr_x_voidstar
iferr_x_zeroptr
name
tag
experrorstr
errorstr
expret
ret
perrflg
perrdata
Here is an example of how these parameters are used.
#include <u.h>
#include <libc.h>
#include <stdio.h>
#include <prototype.h>
void
iferr_x_int(char *name, char *tag, char *experrorstr, char *errorstr, int expret, int ret, ulong *perrflg, void *perrdata)
{
if((*perrflg != 0) && (perrdata != (void *)0))
{
fprintf(stderr, "INFO: %s(%s), major_cycle=%ld, minor_cycle=%ld\n",
name, tag, ((struct iferr_1 *)perrdata)->major_cycle,
((struct iferr_1 *)perrdata)->minor_cycle);
}
if(*perrflg & 02L)
{
fprintf(stderr, "ERROR: %s(%s) expected '%s' error, got '%s' error\n",
name, tag, experrorstr, errorstr);
}
if(*perrflg & 04L)
{
fprintf(stderr, "ERROR: %s(%s) expected %d return, got %d return\n",
name, tag, expret, ret);
}
if(*perrflg)
{
exittest(*perrflg);
}
return;
}
The name of a Komet test command is the name of the tested call appended with a "_k".
All Komet test commands take these parameters:
cycles
experrorstr
expret
In the access_k test the user can specify the file and access mode arguments for the access system call. If there is a read only file available then the lack of write access to it could be tested by the command 'access_k 1 "no access" -1 read_only_file 02' which will pass only if the access system call returns -1 and sets errstr to ""no access"
#include <u.h>
#include <libc.h>
#include <stdio.h>
#include <prototype.h>
main(int argc, char *argv[])
{
char *access_name;
int access_mode;
char *access_experrorstr;
int access_ret;
int access_expret;
long cycles;
long cycle;
int argindx=1;
ulong errflg=0;
if(argc != 6)
{
fprintf(stderr, "USAGE: access_k cycles access_experrorstr access_expret name mode\n");
exittest(1);
}
cycles = (long)strtol(argv[argindx++], (char **)NULL, 0);
access_experrorstr = argv[argindx++];
access_expret = (int)strtol(argv[argindx++], (char **)NULL, 0);
access_name = argv[argindx++];
access_mode = (int)strtol(argv[argindx++], (char **)NULL, 0);
for(cycle = 1 ; cycle <= cycles ; cycle++)
{
access_ret = (int)access_kl("loop", access_experrorstr, access_expret, &errflg, iferr_c_int, (void *)0, access_name,
access_mode);
}
exittest(errflg);
}
Some system calls or library routines must be used in
combination with other system calls executed in the same process
in order to fully exercise their functionality (i.e. the write
system call requires that a previous open system call has been
executed). A test combining these system calls can be produced
relatively simply by manually merging the appropriate simple (i.e.
noncomposite) Komet tests.
The name of a composite Komet test is the name of the most
dependent system system call appended with a "_K". The
most dependent system call is the one that requires the other
system calls for providing necessary input (i.e. the write system
call requires that the file be opened first) and is generally the
last system call used in the test.
For example a write_K test can be written by combining elements
of the open_k, lseek_k, and write_k tests. A brief description of
such a test's functionality is shown below.
In the write_K test the user can specify the error strings
expected during the execution of the open, lseek, and write
system calls in the test. The test creates a buffer and fills it
by repeating the supplied text string a specified number of times.
The test then opens the file (using the specified oflag and mode).
If desired, the test will lseek to a specified offset in the file
and the text buffer will then be written the specified number of
times. After each system call is executed the errstr is compared
to the expected value and any deviation causes the test to fail.
By combining this test with the read_K test which would read
files and verify their contents, a variety of tests to check the
behavior of the filesystem could be developed.
In cases where you want to test the interaction of various system calls executed in different processes a simple shell script to execute the Komet tests in the correct order and pass some coordinating information (such as a common key based on the shell's process id) can be used.
If you have specific comments or questions about this page, contact Mark Wiley (markw@ncube.com)