Linux Test Project

Previous: Using LTP Home: Testing LTP Home Next: The Future of LTP

Developing Tests for LTP

The Linux Test Project was designed to be flexible enough to allow test cases to be added to it without requiring the use of any cumbersome test driver specific features. The LTP does provide a small set of functions that can be used to help with the consistency of test cases and to act as a convenience for the developer, but the driver does not require their use. Tests written to be executed under the LTP should be self-contained so that it can be executed under pan, or separately. They should be able to detect within the test itself whether or not the test passed. If the test passes, it should return 0 or anything else if the test fails. The exact nature of return codes other than 0 may be different from one test to another. Most of the tests in LTP have been written in C, but they may be written in perl, shell scripting languages, or anything else as long as appropriate return values are preserved. This flexibility allows developers to take any quick test they have written to test something, make sure it returns 0 if it passes or anything else if it doesn't, and submit it for inclusion in the LTP.

Some of the functions in LTP make use of global variables that define various aspects of the test case. Even if it is unknown whether or not these functions will be used, it is a good idea to define these variables in order to be consistent with other test cases in the LTP.

char *TCID="test01";

The TCID variable should be defined in a way similar to the example above. The convention that has been used in other test cases in the LTP is the system call name, or some other name representing the test followed by a two digit number. The TCID should be different from any other LTP test case or results may be confusing when executing all the tests in the test suite. It is also a good idea to make the TCID be the same as the name of the source code file for the test. In this example, the file name should be something like test01.c.

The global variable TST_TOTAL is of type int and should be used to specify the number of individual test cases within the test program. Each test should be associated with an output line declaring the outcome of the test case.

extern int Tst_count;

The Tst_count variable is used as a test case counter in the main test loop. The output functions provided by LTP use this variable to get the number of the test case currently being executed. This should be automatically incremented each pass through the test loop.

for (lc=0; TEST_LOOPING(lc); lc++) {

The main test loop is just a for loop, but it implements a macro called TEST_LOOPING() to control the number of iterations through the loop. Standard command line options for LTP test cases allow the user to set a certain number of iterations or an amount of time to run each test. TEST_LOOPING() handles making sure that the test is executed for the correct number of iterations, or for the correct amount of time.

The actual test itself should be wrapped in the TEST() macro. The TEST() macro starts by resetting errno to 0 to ensure that the correct errno is detected after the test is complete. After executing the system call passed to it, TEST() sets two global variables. TEST_RETURN is set to the return code and TEST_ERRNO is set to the value of errno upon return. There is also a variation of the TEST() macro called TEST_VOID() that should be used for testing system calls that return void.

Tests that require little or no manual setup are preferred. Usually setup can be performed within the test itself, or with command line options that can be passed from the execution script. If manual setup is required, the test may be left out of automated execution scripts, or grouped with other tests that have similar setup requirements such as the network tests.

Many tests require a temporary directory to store files and directories created during the test. This is especially true of filesystem tests, and tests of system calls that operate on files and directories. The tst_tmpdir() and tst_rmdir() functions provide a convenient method of creating and cleaning up a temporary area for the test to use.

The tst_tmpdir() function creates a unique, temporary directory based on the first three characters of the TCID global variable. Once the directory is created, it makes it the current working directory and returns to continue execution of the test. The name of the directory created will be saved in an extern char* variable called TESTDIR in case it is needed by the test case, and for later removal by the tst_rmdir() function. If it is unable to create a unique name, unable to create the directory, or unable to change directory to the new location tst_tmpdir() will use tst_brk() to output a BROK message for all test cases in the test and exit via the tst_exit() function. Since no cleanup function will be automatically performed in this situation, tst_tmpdir() should only be used at the beginning of the test before any resources have been created that would require a cleanup function.

The tst_rmdir() function will remove the temporary directory created by a call to tst_tmpdir() along with any other files or directories created under the temporary directory. The system() function is used by tst_rmdir() so the test case should not perform unexpected signal handling on the SIGCHLD signal.

One of the biggest conveniences provided by using the LTP API is parse_opts(). The parse_opts() function provides a consistent set of useful command line options for test cases, and allows the developer to easily add more options.

#include "test.h"
#include "usctest.h"

char *parse_opts(int argc, char *argv[],
                 option_t option_array[],
                 void (*user_help_func)());

typedef struct {
	char *option;
	int  *flag;
	char **arg;
} option_t;

Option_array must be created by the developer to contain the desired options in addition to the default ones. User_help_func() is a pointer to a function that will be called when the user passes -h to the test case. This function should display usage information for the additional options added only. If you do not wish to specify any addition command line options, parse_opts() should be called with NULL for option_array and user_help_func().

The default options provided by parse_opts are:

Another useful feature of the LTP API is that it provides functions to output results and give test status in a consistent manner, and exit the test with an exit code consistent with the results from that output. This paper will not cover all of the functions to do this but will only briefly discuss the most common ones.

All of these functions need to be passed a ttype that specifies the type of message that is being sent. The available values for ttype are:

The first result output function is tst_resm().

void tst_resm(int ttype, char *tmesg,
	[arg ...])

This function will output tmesg to STDOUT. The tmesg string and associated args can be given to tst_resm() and the other functions listed here in the same fashion as strings with args can be passed to printf(). After outputting the message, the test case will resume.

void tst_brkm(int ttype, void (*func)(),
	char *tmesg, [arg ...])

The tst_brkm() function prints the message specified by tmesg, calls the function pointed to by func, and exits the test breaking any remaining test cases.

void tst_exit()

The tst_exit() function exits the test with status depending on ttypes passed to previous calls to functions such as tst_brkm() and tst_resm(). For TPASS, TRETR, TINFO, and TCONF the exit status is unaffected and will be 0 indicating that the test pass. TFAIL, TBROK, and TWARN all indicate that something went wrong during the test, or that the test failed and will cause tst_exit() to exit the test with a non-zero status.

When a test cases receives an unexpected signal, it is useful to provide a means of making it exit gracefully. The LTP provides a convenient way of doing this through the tst_sig() function.

#include "test.h"

void tst_sig(fork_flag, handler, cleanup)
char *fork_flag;
int (*handler)();
void (*cleanup)();

If the test case is creating child processes through functions such as fork() or system(), then tst_sig needs to know to ignore SIGCHLD. This can be accomplished by setting fork_flag to FORK. If the test case is not creating child processes, fork_flag should be set to NOFORK. Keep in mind that if the test uses tst_tmpdir() and tst_rmdir(), the fork_flag should be set to FORK because tst_rmdir() uses the system() library call.

The handler parameter of tst_sig() represents the function that will be called when an unexpected signal is intercepted. The developer may provide a custom signal handler function here that returns int, or the default signal handler may be used. To use the default signal handler for tst_sig(), pass DEF_HANDLER as the handler parameter to tst_sig(). If the default handler is used, then the TCID and Tst_count variables must be defined. The default handler will use tst_res() to output messages for all remaining tests that were incomplete when the signal was received.

The cleanup parameter is used to specify a cleanup function. After the handler has been executed, tst_sig() will execute the cleanup function. The cleanup function should take care of removing any resource used by the test such as files or directories that were created to facilitate testing. If nothing is required for cleanup, NULL can be passed to tst_sig() in place of a cleanup function.

Paul Larson 2002-09-11  Last modified on: June 15, 2006 - 16:35:59 UTC.