Linux Test Project

LTP HowTo


Table Of Contents

  1. Introduction
    1. Purpose of this document
    2. Feedback and corrections
  2. Revision History
  3. Before You Begin
    1. Downloading the test suite
    2. Contents of the LTP test suite
    3. Compiling the test suite
    4. Executing the test suite
    5. Contents of output to screen and logfile
  4. Developing tests in ANSI-C using LTP test harness APIs
    1. Basic template
    2. Understanding the template
    3. Recommendation on style
    4. More help
  5. Developing tests in Shell using LTP command line harness interface
    1. Basic template
    2. Understanding the template
    3. Recommendation on style
    4. More Help
  6. Adding new test to LTP
  7. Other Resources
  8. Acknowledgments
  9. Disclaimer
  1. Introduction

    1. Purpose of this document

      Linux Test Project (LTP) is an open source project with a goal to deliver test suites to the open source community that validate the reliability, robustness, and stability of Linux. The LTP test suite is a collection of automated and semi-automated tests for testing various aspects of the Linux operating system. The dominant programming language in LTP is ANSI-C (94%) followed by Bash (5%) and Perl (0.62%).

      This document has several goals. First, it provides a cursory view of the contents of LTP test suite. Second, it introduces a template for writing test-cases in ANSI-C and Shell. Finally, it enumerates the steps that must to be followed inorder to add a new test cases to LTP. The document reflects the authors views on code layout and coding practices.

    2. Feedback and corrections

      In an effort to keep this document current and useful, any form of feedback is greatly appreciated. Feel free to point out any mistakes in this document or contribute additional information. Please e-mail shubham@linux.vnet.ibm.com so that these changes can be made available in the next version.

  2. Revision History

    Version Author Date Comments
    1.0 Manoj Iyer
    IBM Austin TX
    manjo @ austin.ibm.com
    Jan 17 2003 Initial Draft
    1.1 Shubham Goyal
    IBM India, Bangalore
    shubham@linux.vnet.ibm.com
    Apr 28 2012 Updated Various URL's and other documentation
  3. Before You Begin

    This section will provide a cursory view of the contents of the LTP test suite, and also provide instructions on downloading, compiling and executing the test suite.

    1. Downloading the LTP test suite

      The stable version of the LTP test suite source, ltp-yyyymmdd.tgz is released on a monthly basis. The test suite may be used to verify stable and development release of the Linux kernel. The test suite is also available in binary and source RPM format. The package ltp-yyyymmdd.tgz can be downloaded from the LTP web-site using a web browser or using the 'wget' command.
      wget -c http://prdownloads.sourceforge.net/ltp/ltp-full-yyyymmdd.tgz?download (Replace yyyymmdd with appropriate values)

      You can download the latest CVS snapshot of the LTP test suite source via anonymous CVS access as follows.
      cvs -d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp login
      cvs -z3 -d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp co ltp
      Use the following to download without the CVS (directory) getting created in each subdirectory
      cvs -d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp login
      cvs -z3 -d:pserver:anonymous@ltp.cvs.sourceforge.net:/cvsroot/ltp export -D "MONTH DAY, YEAR HH:MMam/pm" ltp

    2. Contents of the LTP test suite

      The top level directory or LTP rootdirectory ./ltp-yyyymmdd contains SHELL scripts and various source directories. These scripts may be used to execute the test cases, the scripts execute the test cases using the test driver called pan.

      Scripts
      IDcheck.sh
      The test suite requires that certain users and groups (For example User:nobody Group:nobody.) exist on the system. This script will create these users and groups. This script is called from the top level Makefile located in the LTP root directory.
      runltp
      This driver script runs a majority of tests in the LTP. It can also be easily modified to run a custom list of tests if required.
      By default, this script will execute:
      • filesystem stress tests
      • disk I/O tests
      • memory management stress tests
      • ipc stress
      • scheduler tests
      • commands functional varification tests
      • system call functional varification tests
      The script is named runalltest.sh but it does not run all the tests in the LTP.
      networktests.sh
      This script executes the networking related tests. Before executing this script, two environment variables must be set RHOST (Remote Host Name) and PASSWD (root password of the remote host).
      diskio.sh
      This script executes tests related to floppy and CD-ROM drives. These tests require that a floppy disk and CD-ROM media be present in the respective drives.
      Directories
      testcases
      This directory contains the source code for all the test cases. The test cases are located under directories named after the subsystem or service they are intended to test. For example, memory management tests are located under testcases/mem etc.
      runtest
      This directory contains command files that are used by pan to run the tests. Each file contains the name of the test case "tag" and the "command" with arguments, if any, that must be executed to perform the test. For example, the file "syscalls" contains entries as follows.
       # Name of test        Command
       access01              access01
       alarm04               cp -f $LTPROOT/test-cases/bin/sig_rev $TMP;alarm04
       chdir01A              symlink01 -T chdir01; chdir01
      
      lib
      This directory contains source code for the LTP-harness APIs (libltp.a). The libltp.a library gets linked with any test-case that uses these API functions to log test results and status.
      include
      This directory contains header files that are included in any test-case that uses the LTP API-harness functions.
      /opt/ltp/testcases/bin
      This directory contains the test executables, the test cases get installed in this directory by typing "make install".
    3. Compiling the test suite

      To compile the test suite change directory to ltp-yyyymmdd and type "./configure" followed by "make". To install the test suite, type "make install" as root user. This will create symbolic links of the executable to corresponding files in ltp-yyyymmdd/ test-cases/bin directory. The test scripts (runltp for example) execute the binaries present in this directory based on the tests listed in its command file.
      Note that, "make install" can be run as non-root, but IDcheck.sh must be run separately before executing any test cases as root.

    4. Executing the test suite

      Currently there are three scripts that can be used to execute test cases in the LTP test suite, namely, runltp , networktests.sh and diskio.sh. The test cases and the test driver "pan", require certain environment variables to be set and certain directories to be present in order function correctly. These scripts take care of such pre-requisites. The scripts invoke the test driver "pan" with certain parameters and pan executes the tests and logs test results. The parameters that are be passed to pan can be controlled by user provided options to the scripts.

      The most frequently used script, runltp , currently provides the following options. If no options are specified while executing this script, the default behaviour is as follows:

      • Run a single iteration of all the default set of tests
      • Verbose output of of test-case status on screen
      • Does not produce any log file or test results by default
      	
          -a EMAIL_TO     EMAIL all your Reports to this E-mail Address
          -c NUM_PROCS    Run LTP under additional background CPU load
                          [NUM_PROCS = no. of processes creating the CPU Load by spinning over sqrt()
                                       (Defaults to 1 when value)]
          -C FAILCMDFILE  Command file with all failed test cases.
          -d TMPDIR       Directory where temporary files will be created.
          -D NUM_PROCS,NUM_FILES,NUM_BYTES,CLEAN_FLAG
                          Run LTP under additional background Load on Secondary Storage (Seperate by comma)
                          [NUM_PROCS   = no. of processes creating Storage Load by spinning over write()]
                          [NUM_FILES   = Write() to these many files (Defaults to 1 when value 0 or undefined)]
                          [NUM_BYTES   = write these many bytes (defaults to 1GB, when value 0 or undefined)]
                          [CLEAN_FLAG  = unlink file to which random data written, when value 1]
          -e              Prints the date of the current LTP release
          -f CMDFILES     Execute user defined list of testcases (separate with ',')
          -g HTMLFILE     Create an additional HTML output format
          -h              Help. Prints all available options.
          -i NUM_PROCS    Run LTP under additional background Load on IO Bus
                          [NUM_PROCS   = no. of processes creating IO Bus Load by spinning over sync()]
          -l LOGFILE      Log results of test in a logfile.
          -m NUM_PROCS,CHUNKS,BYTES,HANGUP_FLAG
                          Run LTP under additional background Load on Main memory (Seperate by comma)
                          [NUM_PROCS   = no. of processes creating main Memory Load by spinning over malloc()]
                          [CHUNKS      = malloc these many chunks (default is 1 when value 0 or undefined)]
                          [BYTES       = malloc CHUNKS of BYTES bytes (default is 256MB when value 0 or undefined) ]
                          [HANGUP_FLAG = hang in a sleep loop after memory allocated, when value 1]
          -N              Run all the networking tests. 
          -n              Run LTP with network traffic in background.
          -o OUTPUTFILE   Redirect test output to a file.
          -p              Human readable format logfiles. 
          -q              Print less verbose output to screen.
          -r LTPROOT      Fully qualified path where testsuite is installed.
          -s PATTERN      Only run test cases which match PATTERN.
          -t DURATION     Execute the testsuite for given duration. Examples:
                            -t 60s = 60 seconds
                            -t 45m = 45 minutes
                            -t 24h = 24 hours
                            -t 2d  = 2 days
          -T REPETITION   Execute the testsuite for REPETITION no. of times
          -v              Print more verbose output to screen.                   
          -w CMDFILEADDR  Uses wget to get the user's list of testcases.
          -x INSTANCES    Run multiple instances of this testsuite.
                                                                

      Execuing default set of tests
      If you wish to execute the default set of test case, you can use the following commands.
      • Change directory to /opt/ltp/
        cd /opt/ltp/
      • Execute runltp script with the following options, or chose the options that are best suited for you from the list above. ./runltp -p -l result.yyyymmdd.log

      Executing user defined set of tests
      If you wish to run a select set of test cases, you have to do the following.
      • Create a command file. The command file contains the name of the test case "tag" and the "test case" with arguments, if any, that must be executed to perform the test. For example.

        #Tag       Test case
        #---------------------------------------
        mtest01 mtest01 -p 10
        mmstress mmstress -x 100
        fork01 fork01
        chdir01 symlink01 -T chdir01
        #----------------------------------------
      • Change to the directory where LTP is installed i.e. /opt/ltp/
        cd /opt/ltp/
      • Execute tests in command file using runltp . You can execute runltp with the -f command file option to execute these tests.
        ./runltp -p -l result.02.log -f my_command_file
    5. Contents of output to screen and logfile

      Output to screen
      The test cases, test driver, and scripts print various informational messages to the screen. The test cases use the USCTEST functions to print information, the test driver "pan", prints tags that separate output from one test to another. Depending on the options that you may choose while running the scripts, the verbosity of these messages can be adjusted. For example.

      • Screen output of the command runltp -p -l results.03.log -f my_command_file

      	If some fields are empty or look unusual you may have an old version.
      	Compare to the current minimal requirements in Documentation/Changes.
      
      	/etc/issue:
      	Debian GNU/\s testing/unstable \n \l
      
      	Linux lazy 2.4.19 #1 Mon Aug 26 14:25:41 CDT 2002 i686 unknown
      
      	Gnu C                  2.95.4
      	Gnu make               3.80
      	util-linux             2.11x
      	mount                  2.11x
      	modutils               2.4.21
      	e2fsprogs              1.32
      	Linux C Library        2.3.1
      	Dynamic linker (ldd)   2.3.1
      	Procps                 3.1.1
      	Net-tools              1.60
      	Kbd                    66:
      	Sh-utils               2.0.11
      	Modules Loaded         soundcore uhci usbcore vfat fat
      	free -m reports:
      	             total       used       free     shared    buffers     cached
      	Mem:           249        211         37          0          4        135
      	-/+ buffers/cache:         71        178
      	Swap:          258         49        208
      
      	/proc/cpuinfo
      	processor       : 0
      	vendor_id       : GenuineIntel
      	cpu family      : 6
      	model           : 8
      	model name      : Pentium III (Coppermine)
      	stepping        : 10
      	cpu MHz         : 863.881
      	cache size      : 256 KB
      	fdiv_bug        : no
      	hlt_bug         : no
      	f00f_bug        : no
      	coma_bug        : no
      	fpu             : yes
      	fpu_exception   : yes
      	cpuid level     : 2
      	wp              : yes
      	flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
      	pat pse36 mmx fxsr sse
      	bogomips        : 1723.59
      
      	t_start>>>
      	tag=mtest01 stime=1043946397
      	cmdline="    mtest01"
      	contacts=""
      	analysis=exit
      	initiation_status="ok"
      	(((test_output)))
      	stopped at 3076521984 bytes
      	(((execution_status)))
      	duration=0 termination_type=exited termination_id=1 corefile=no
      	cutime=1 cstime=9
      	(((test_end)))
      
      	INFO: run mmstress -h for all options
      
      	test1: Test case tests the race condition between
      	simultaneous read faults in the same address space.
      	(((test_start)))
      	tag=mmstress stime=1043946397
      	cmdline="   mmstress "
      	contacts=""
      	analysis=exit
      	initiation_status="ok"
      	(((test_output)))
      	test2: Test case tests the race condition between
      	simultaneous write faults in the same address space.
      	test3: Test case tests the race condition between
      	simultaneous COW faults in the same address space.
      	test4: Test case tests the race condition between
      	simultaneous READ faults in the same address space.
      	The file maped is /dev/zero
      	test5: Test case tests the race condition between
      	simultaneous fork - exit faults in the same address space.
      
      	
      		more messages here skipped.
      	
      	(((execution_status)))
      	duration=41 termination_type=exited termination_id=0 corefile=no
      	cutime=605 cstime=3443
      	(((test_end)))
      	(((test_start)))
      	tag=fork01 stime=1043946438
      	cmdline="     fork01"
      	contacts=""
      	analysis=exit
      	initiation_status="ok"
      	(((test_output)))
      	fork01      1  PASS  :  fork() returned 11825
      	fork01      2  PASS  :  child pid and fork() return agree: 11825
      	(((execution_status)))
      	duration=0 termination_type=exited termination_id=0 corefile=no
      	cutime=0 cstime=0
      	(((test_end)))
      	(((test_start)))
      	tag=chdir01A stime=1043946438
      	cmdline="   symlink01 -T chdir01"
      	contacts=""
      	analysis=exit
      	initiation_status="ok"
      	(((test_output)))
      	incrementing stop
      	chdir01     1  PASS  :  chdir(2) to object file location through symbolic link
      	file is ok
      	chdir01     2  PASS  :  chdir(2) to non-existent object file location through
      	symbolic link file failed as expected
      	chdir01     3  PASS  :  Nested symbolic link access condition caught.  ELOOP is
      	returned
      	(((execution_status)))
      	duration=0 termination_type=exited termination_id=0 corefile=no
      	cutime=0 cstime=1
      	(((test_end)))
      	pan reported FAIL
      
      	###############################################################
      
      	result log is in the /home/manjo/cvs_tree/ltp/results directory
      
      	###############################################################
      							

      • Screen output of the command runltp -q -p -l result.04.log -f my_command_file (Providing the -q option to runlt runltp will supress certain messages).

      	If some fields are empty or look unusual you may have an old version.
      	Compare to the current minimal requirements in Documentation/Changes.
      
      	/etc/issue:
      	Debian GNU/\s testing/unstable \n \l
      
      	Linux lazy 2.4.19 #1 Mon Aug 26 14:25:41 CDT 2002 i686 unknown
      
      	Gnu C                  2.95.4
      	Gnu make               3.80
      	util-linux             2.11x
      	mount                  2.11x
      	modutils               2.4.21
      	e2fsprogs              1.32
      	Linux C Library        2.3.1
      	Dynamic linker (ldd)   2.3.1
      	Procps                 3.1.1
      	Net-tools              1.60
      	Kbd                    66:
      	Sh-utils               2.0.11
      	Modules Loaded         soundcore uhci usbcore vfat fat
      
      	free -m reports:
      	             total       used       free     shared    buffers     cached
      	Mem:           249        212         37          0          4        134
      	-/+ buffers/cache:         73        175
      	Swap:          258         49        208
      
      	
      		more messages here skipped.
      	
      
      	test1: Test case tests the race condition between
      	simultaneous read faults in the same address space.
      	test2: Test case tests the race condition between
      	simultaneous write faults in the same address space.
      	test3: Test case tests the race condition between
      	simultaneous COW faults in the same address space.
      	test4: Test case tests the race condition between
      	simultaneous READ faults in the same address space.
      	The file maped is /dev/zero
      	test5: Test case tests the race condition between
      	simultaneous fork - exit faults in the same address space.
      	test6: Test case tests the race condition between
      	simultaneous fork -exec - exit faults in the same
      	address space.
      	fork01      1  PASS  :  fork() returned 12156
      	fork01      2  PASS  :  child pid and fork() return agree: 12156
      	chdir01     1  PASS  :  chdir(2) to object file location through symbolic link
      	file is ok
      	chdir01     2  PASS  :  chdir(2) to non-existent object file location through
      	symbolic link file failed as expected
      	chdir01     3  PASS  :  Nested symbolic link access condition caught.  ELOOP is
      	returned
      	pan reported FAIL
      
      	###############################################################
      
      	result log is in the /home/manjo/cvs_tree/ltp/results directory
      
      	###############################################################
      							
      Output to log file
      The test driver pan uses the exit value of the test case to decide success or failure of a test. If the test case exits with a non-zero value pan records this as FAIL, if the test case exits with a value zero pan records this as PASS. The test driver "pan" generates log files, these logfiles contain results from executing the test case. You can provide the -l logfile_name option to runalltest.sh script in order to log results. If the use does not provide an absolute path, the file is created under the results directory. Currently, the result log file is generated in two formats.
      • Human readable format
        You can generate a logfile in this format by providing runltp with the -p option. This generates a log file in ASCII format, for example.

        	Test Start Time: Thu Jan 30 11:06:37 2003
        	-----------------------------------------
        	Testcase                       Result     Exit Value
        	--------                       ------     ----------
        	mtest01                        FAIL       1
        	mmstress                       PASS       0
        	fork01                         PASS       0
        	chdir01A                       PASS       0
        	-----------------------------------------------
        	Total Tests: 4
        	Total Failures: 1
        	Kernel Version: 2.4.19
        	Machine Architecture: i686
        	Hostname: lazy
      • Script readable format
        This is the default format and may be used by scripts that are designed to generate test result analysis reports. The file generated in ASCII format. For example.

        	tag=mtest01 stime=1043947511 dur=0 exit=exited stat=1 core=no cu=0 cs=11
        	tag=mmstress stime=1043947511 dur=41 exit=exited stat=0 core=no cu=671 cs=3383
        	tag=fork01 stime=1043947552 dur=0 exit=exited stat=0 core=no cu=0 cs=1
        	tag=chdir01A stime=1043947552 dur=0 exit=exited stat=0 core=no cu=0 cs=0

  4. Developing tests in ANSI-C using LTP test harness API's

    At the time of this writing, LTP has 156,775 physical source lines of code. The most dominant programming language is ANSI-C (94.43%) followed by shell (4.60%) and Perl (0.62%). This section provides an introduction to developing new test cases in ANSI-C using LTP test harness APIs.

    1. Basic template

      This section provides a basic template that may be used to develop test-cases in ANSI-C using LTP harness APIs. Please note that the following code will not compile as is.

      /******************************************************************************/
      /*                                                                            */
      /* Copyright (c) International Business Machines  Corp., 2001                 */
      /*                                                                            */
      /* This program is free software;  you can redistribute it and/or modify      */
      /* it under the terms of the GNU General Public License as published by       */
      /* the Free Software Foundation; either version 2 of the License, or          */
      /* (at your option) any later version.                                        */
      /*                                                                            */
      /* This program is distributed in the hope that it will be useful,            */
      /* but WITHOUT ANY WARRANTY;  without even the implied warranty of            */
      /* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See                  */
      /* the GNU General Public License for more details.                           */
      /*                                                                            */
      /* You should have received a copy of the GNU General Public License          */
      /* along with this program;  if not, write to the Free Software               */
      /* Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA    */
      /*                                                                            */
      /******************************************************************************/
      
      /******************************************************************************/
      /*                                                                            */
      /* File:        template.c                                                    */
      /*                                                                            */
      /* Description: This is a template that can be used to develop new test-cases */
      /*              using the LTP-harness APIs. As an example a dummy_syscall()   */
      /*              system call will be used as an example.                       */
      /*                                                                            */
      /* Total Tests: 1                                                             */
      /*                                                                            */
      /* Test Name:   dummy_syscall_test01                                          */
      /*                                                                            */
      /* Test Assertion                                                             */
      /* amp; Strategy:  A brief description of the test Assertion and Strategy     */
      /*                                                                            */
      /* Author:      Manoj Iyer manjo@austin.ibm.com                               */
      /*                                                                            */
      /* History:     Created - Jan 19 2003 - Manoj Iyer manjo@austin.ibm.com       */
      /*              Added   - Jan 20 2003 - Manoj Iyer, comments describing the   */
      /*                                      the functions used by the test-case.  */
      /*                                                                            */
      /******************************************************************************/
      
      /* Standard Include Files */
      #include <stdio.h>
      #include <stdlib.h>
      #include <errno.h>
      
      	/* Harness Specific Include Files. */
      	#include "test.h"
      	#include "usctest.h"
      
      	/* Local Defines */
      	#if !defined(TRUE) amp;amp; !defined(FALSE)
      	#define TRUE  1
      	#define FALSE 0
      	#endif
      
      	/* Extern Global Variables */
      	extern int  Tst_count;               /* counter for tst_xxx routines.         */
      	extern char *TESTDIR;                /* temporary dir created by tst_tmpdir() */
      
      	/* Global Variables */
      	char *TCID     = "dummy_syscall_test01"; /* test program identifier.          */
      	int  TST_TOTAL = 1;                  /* total number of tests in this file.   */
      
      	/* Extern Global Functions */
      	/******************************************************************************/
      	/*                                                                            */
      	/* Function:    cleanup                                                       */
      	/*                                                                            */
      	/* Description: Performs all one time clean up for this test on successful    */
      	/*              completion,  premature exit or  failure. Closes all temporary */
      	/*              files, removes all temporary directories exits the test with  */
      	/*              appropriate return code by calling tst_exit() function.       */
      	/*                                                                            */
      	/* Input:       None.                                                         */
      	/*                                                                            */
      	/* Output:      None.                                                         */
      	/*                                                                            */
      	/* Return:      On failure - Exits calling tst_exit(). Non '0' return code.   */
      	/*              On success - Exits calling tst_exit(). With '0' return code.  */
      	/*                                                                            */
      	/******************************************************************************/
      	extern void
      	cleanup()
      	{
      	/* Close all open file descriptors. */
      	
      		Insert code here. In case an unexpected failure occurs report it
      		and exit setup(), the following lines of code will demonstrate
      		this.
      	
      	if (close(fd) == -1)
      	{
      	tst_resm(TWARN, "close(%s) Failed, errno=%d : %s",
      	fname, errno, strerror(errno));
      	}
      
      	/* Remove all temporary directories used by this test. */
      
      	Insert real code here 
      
      	/* kill child processes if any. */
      
      	Insert code here
      
      	/* Exit with appropriate return code. */
      
      	tst_exit();
      	}
      
      
      	/* Local  Functions */
      	/******************************************************************************/
      	/*                                                                            */
      	/* Function:    setup                                                         */
      	/*                                                                            */
      	/* Description: Performs all one time setup for this test. This function is   */
      	/*              typically used to capture signals, create temporary dirs      */
      	/*              and temporary files that may be used in the course of this    */
      	/*              test.                                                         */
      	/*                                                                            */
      	/* Input:       None.                                                         */
      	/*                                                                            */
      	/* Output:      None.                                                         */
      	/*                                                                            */
      	/* Return:      On failure - Exits by calling cleanup().                      */
      	/*              On success - returns 0.                                       */
      	/*                                                                            */
      	/******************************************************************************/
      	void
      	setup()
      	{
      	/* Capture signals. */
      
      	Insert code here
      
      	/* Create temporary directories */
      
      	Insert code here
      
      	/* Create temporary files. */
      	
      		Insert real code here. In case an unexpected failure occurs
      		report it and exit setup(). Follow the code below for example.
      	
      	if ((fd = open(fname, O_RDWR|O_CREAT, 0700)) == -1 )
      	{
      	tst_brkm(TBROK, cleanup,
      	"Unable to open %s for read/write.  Error:%d, %s\n",
      	fname, errno, strerror(errno));
      	}
      
      	return;
      	}
      
      
      	/******************************************************************************/
      	/*                                                                            */
      	/* Function:    main                                                          */
      	/*                                                                            */
      	/* Description: Entry point to this test-case. It parses all the command line */
      	/*              inputs, calls the global setup and executes the test. It logs */
      	/*              the test status and results appropriately using the LTP API's */
      	/*              On successful completion or premature failure, cleanup() func */
      	/*              is called and test exits with an appropriate return code.     */
      	/*                                                                            */
      	/* Input:       Describe input arguments to this test-case                    */
      	/*               -l - Number of iteration                                     */
      	/*               -v - Prints verbose output                                   */
      	/*               -V - Prints the version number                               */
      	/*                                                                            */
      	/* Exit:       On failure - Exits by calling cleanup().                       */
      	/*             On success - exits with 0 exit value.                          */
      	/*                                                                            */
      	/******************************************************************************/
      	int
      	main(int   argc,    /* number of command line parameters                      */
               char **argv)   /* pointer to the array of the command line parameters.   */
      	{
      
      	/* parse options. */
      	
      		LTP test harness provides a function called parse_opts() that
      		may be used to parse standard options. For a list of standard
      		option that are available refer on-line man page at the LTP
      		web-site
      	
      	/* perform global test setup, call setup() function. */
      	setup();
      
      	/* Print test Assertion using tst_resm() function with argument TINFO. */
      	tst_resm(TINFO, "Testing if dummy_syscall() function works\n");
      
      	/* Test Case Body. */
      	
      		Insert real code goes here. In case of unexpected failure, or
      		failure not directly related to the entity being tested report
      		using tst_brk() or tst_brkm() functions.
      	
      	if (something bad happened)
      	{
      	tst_brkm(TBROK, cleanup, "Unable to open test file %s\n", fname);
      	}
      
      	/* print results and exit test-case */
      	if(results are favorable)
      	tst_resm(TPASS, "dummy_syscall worked as designed\n");
      	else
      	tst_resm(TFAIL, "dummy_syscall failed to work as designed\n");
      
      	tst_exit()
      	}
    2. Understanding the template

      Copyrights and license statement
      Each test-case starts with a copyright and license statement. All test-cases in LTP are covered under the GNU public license . Copyrights usually belong to the author or corporation that developed the tests. Test case developed by IBM will have IBM copyrights, and test cases developed by SGI will have SGI copyrights and so on.
      File header
      The file header contains information contained in the test case file under five main categories, these are as follows.
      • File: Name of the test case file. For example test01.c
      • Description: A brief description of the contents of this file, explains what this file contains, namely, a test case to test functionality of certain system call etc.
      • Total Tests: Total number of test cases in this file.
      • Test Name: Name of the test case
      • Test Assertion amp; Strategy: A brief description of the test Assertion and Strategy.
      • History: Date of creation or modification, Author and email address. Followed by comments that describes the change.
        For example: Created - Jan 19 2003 - Manoj Iyer manjo@austin.ibm.com

      Standard include and header files
      These include files must be based on recommendations made in the man pages of standard C library (libc) APIs that are used by the test case. Standard include files are followed by LTP harness specific header files and any other local header file that the test might use.

      Global variable definitions
      Include any global definitions, external global variables, and variables that are global to this test case. The variables Tst_count, TST_TOTAL and TCID are used by the harness APIs for reporting status and test results. These are the minimum set of variables that must be defined in order to use the basic functions (tst_brk, tst_brkm, tst_res, tst_resm and tst_exit) to report test status.

      Global function definitions
      The variable definition is followed by global function definitions, it is recommended that the developer define at least one function, namely, cleanup() and optionally a setup() as global extern function. The cleanup() function must be supplied to tst_brk() routine, this routine is used to report premature test failure. The cleanup() function will call the tst_exit() function which will exit with appropriate exit values.

      main()
      Function definitions are followed by main(), the entry point to the test case. The activities expected from main() can be summarized as follows:
      • parse input options.
      • call setup function or perform tasks that are a pre-requisite to perform tests.
      • print the test assertion.
      • perform the test.
      • print test results.
      • clean up and exit with appropriate exit code.
      The LTP harness library provides certain macros and functions in an attempt to simplify some of the above activities or help manage results and test status easily. For instance, The USCTEST macros and functions might be helpful for testing system calls. Below is an example of such a function.

      The parse_opt() function, for instance, may be used to parse standard (or built-in) options or user defined options to the test case. It supports a set of pre-defined standard options with the flexibility of adding more user defined options.

      	char *
      	parse_opts(int      argc,  char   *argv[], option_t option_array[], void (*user_help_func)());
      	
      	where option_t is:
      	
      	typedef struct {
      	char    *option;
      	int     *flag;
      	char    **arg;
      	} option_t;

      User defined options may be specified in the option_array argument. A help() or usage() function may be specified in the user_help_func argument.

      Although using these macros and functions are left to the developer's discretion, the use of tst_resm/tst_res, tst_brk/tst_brkm, tst_brkloop/tst_brkloopm, and tst_exit functions is highly recommended. For details on usage and syntax please refer man pages on (http://ltp.sourceforge.net) the LTP website.
      The tst_brk/tst_brkm functions are typically used in a situation where a test case breaks as a result of something other than the entity that is being tested fails. The tst_resm/tst_res functions may be used to print various message depending on the result type passed to it as argument. The result type may be one of the following.
      TINFO
      Print informational messages that report test execution status, test assertion , or other informational message. For example.
      test01    0  INFO  :  This test will test the uname command.
      TPASS
      Print the test case result. The tst_resm/tst_res function prints related messages to the screen indicating a test success. The test case will exit with a zero exit value when tst_exit function is called.
      test01    0  PASS  :  uname command works as expected.
      TFAIL
      Print the test case result. The tst_resm/tst_res function prints related messages to the screen indicating a test failure. The test will exit with a non-zero exit value when tst_exit function is called.
      test01    0  PASS  :  uname command exited with error code = 255
      TWARN
      Print warning messages. For example, when system resources are low.
      test01    0  PASS  :  System low on memory, test case might fail.
    3. Recommendation on coding style

      This section includes recommendations on code layout and style. They are optional, it reflects the authors personal coding style and practice and may be applicable in general to any program written in ANSI-C.

      • Use a file header to describe the contents of a file and maintain a change history. CVS provides the option to maintain a change log while checking in code into the CVS tree, but often times the developer need not be responsible for maintaining the code in CVS. The change log or history maintained in the file can help both the developer and the maintainer of the code to track changes made to the file.
      • Use meaningful variable names. Avoid variable names like "i"; "j", instead use variable names like "index", "array_ndx" etc. Also initialize and comment the variables appropriately. For example a for loop counter may be defined as follows:
        int loop_cntr = 0; /* counter to keep track of number of iterations */
        instead of defining the same as:
        int j;
      • Comment code appropriately. Code might be handled by seasoned programmers as well as newbies. It is a good practice to add comments in the code where ever appropriate for maintainance reasons. Comments can be stripped off from the code using AWK or PERL scripts if required.
      • Use "/* --- */" style for comments, avoid using the "//" C++ style comments.
      • Use ANSI-C style function names instead of C++ style function names, for example
        create_file()
        instead of
        CreateFile()
        Declare functions local to the file as "static". Start the function names in a new line, and each input or output variable in separate lines with appropriate comments. For example:
        	static int
        	create_tmpfile(char    *fname, /* name of temporary file created by test01  */
        	int     flags,  /* describes file as read, write only etc    */
        	mode_t  mode)   /* file open modes, S_IRWXO etc.             */
        	{ 
        	}
      • Use a function prologue. The prologue should describe the function, its input and output variables if any, and exit or return values if any. Using a function prologue is a good practice that makes code readable and maintainable. Refer to the template for an example.
      • Check with man pages for #include <include file=""/> recommendation when using standard C library (libc) functions. Avoid using unnecessary header files.
      • Check for return values from system calls. Report reason appropriately using the tst_brkm() API. The recommended style is as follows:
        	if ((ret = open(filename, O_RDWR|O_CREAT, S_IRWXO|S_IRWXU|S_IRWXG)) == -1)
        	{
        		tst_brkm(TBROK, cleanup, "open(): Failed %s\n",  strerror(errno));
        	}
        	else{}
      • Use tst_resm(TINFO, ...) to print informational messages as appropriate, alternately, a verbose option or a 'quiet' option may be provided to suppress or activate such messages.
      • Use spaces instead of tab characters. Typically four spaces may be used instead of a tab character.
      • Compile code using the "-Wall" option. Using this option, the compiler will generate more warning than when compiled without "-Wall". Treat all warnings generated by the compiler as errors and try to minimize them.
    4. More help

      LTP test suite has over a thousand test cases that can be studied in order to understand the use of the LTP APIs. For more information on syntax and usage follow the "LTP Man Pages" link on the online man page located at http://ltp.sourceforge.net under "Documentation" section. You can find man pages for the LTP APIs, USCTEST macros and test driver "pan", in the ltp-yyyymmdd/doc/ directory.

  5. Developing tests in Bash using the LTP command line harness interface

    Limited number of test in the LTP test suite are written in Shell scripts (4.6%). Currently, the focus of the LTP test suite is the reliablity and stability of the Linux kernel. However, there is increasing interest in writing tests for other parts of the Linux operating system (for example deamons, services, etc.). It might be required to develop these tests in a language other than ANSI-C.(for example BASH script or AWK script). This section will provide a template for developing new test cases in SHELL using the command line harness APIs.

    A minimum set of LTP harness APIs are available in the form of commands. These commands help developers log test status and test results in a format similar to that done by tests in ANSI-C. The source code for these commands are located in the ltp-yyyymmdd/tools/apicmds/ directory. Once the commands are compiled and installed the test-cases reside under ltp-yyyymmdd/testcases/bin directory.

    1. Basic template

      This section will introduces a basic template that can be used to develop test cases in SHELL. Please note that the follwing code will fail to execute as is.

      	################################################################################
      	##                                                                            ##
      	## Copyright (c) International Business Machines  Corp., 2001                 ##
      	##                                                                            ##
      	## This program is free software;  you can redistribute it and#or modify      ##
      	## it under the terms of the GNU General Public License as published by       ##
      	## the Free Software Foundation; either version 2 of the License, or          ##
      	## (at your option) any later version.                                        ##
      	##                                                                            ##
      	## This program is distributed in the hope that it will be useful, but        ##
      	## WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY ##
      	## or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License   ##
      	## for more details.                                                          ##
      	##                                                                            ##
      	## You should have received a copy of the GNU General Public License          ##
      	## along with this program;  if not, write to the Free Software               ##
      	## Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA    ##
      	##                                                                            ##
      	################################################################################
      	#
      	# File :        template.sh
      	#
      	# Description:  This is a template that can be used to develop new test cases
      	#               using the LTP-harness command APIs. As an example a dummy_cmd
      	#               is used in this file.
      	#
      	# Author:       Manoj Iyer, manjo@mail.utexas.edu
      	#
      	# History:      Jan 19 2003 - Created - Manoj Iyer.
      	#               Jan 20 2003 - Added - Manoj Iyer, Test #1
      	#
      	#! /bin/sh
      
      
      	# Function:     setup
      	#
      	# Description:  - Check if required commands exits
      	#               - Export global variables
      	#               - Check if required config files exits
      	#               - Create temporary files and directories
      	#
      	# Return        - zero on success
      	#               - non zero on failure. return value from commands ($RC)
      	setup()
      	{
      
      	
      		This variable is required by the LTP command line harness APIs
      		to keep track of the number of testcases in this file. This
      		variable is not local to this file.
      	
      	export TST_TOTAL=1  # Total number of test cases in this file.
      
      	# Set up LTPTMP (temporary directory used by the tests).
      	
      		Export PATH variable to point to the ltp-yyyymmdd/testcases/bin
      		directory before running the tests manually, path is set up
      		automatically when you use the runltp script.
      		LTPTMP is user defined variables used only in this test case.
      		These variables are local to this file.
      		The TMP variable is exported by the runltp script.
      	
      	LTPTMP=${TMP}       # Temporary directory to create files, etc.
      
      	
      		The TCID and TST_COUNT variables are required by the LTP
      		command line harness APIs, these variables are not local
      		to this program.
      	
      	export TCID="setup" # Test case identifier
      	export TST_COUNT=0  # Set up is initialized as test 0
      
      	# Initialize cleanup function to execute on program exit.
      	# This function will be called before the test program exits.
      	trap "cleanup" 0
      
      	# Initialize return code to zero.
      	
      		RC is used to store the return code from commands that are
      		used in this test. This variable is local to this file.
      	
      	RC=0                # Exit values of system commands used
      
      	# Check if awk command exists
      	which awk amp;>$LTPTMP/tst_template.out || RC=$?
      	if [ $RC -ne 0 ]
      	then
      	tst_brkm TBROK NULL "INIT: Command awk not found"
      	return $RC
      	fi
      
      	# Check if temp_user exists. If not add one.
      	RC=$(awk '/^temp_user/ {print 1}' /etc/passwd)
      	if [ -z $RC ]
      	then
      	RC=0
      	tst_resm TINFO "INIT: Adding temporary user temp_user"
      	useradd -m -s /bin/bash temp_user amp;>$LTPTMP/tst_template.out || RC=$?
      	if [ $RC -ne 0 ]
      	then
      	tst_brk TBROK $LTPTMP/tst_template.out NULL \
      	"Test INIT: Failed adding user temp_user Reason:"
      	return $RC
      	fi
      	fi
      	return $RC
      	}
      
      
      	# Function:     cleanup
      	#
      	# Description   - remove temporary files and directories.
      	#
      	# Return        - zero on success
      	#               - non zero on failure. return value from commands ($RC)
      	cleanup()
      	{
      	Clean up code goes here
      	}
      
      
      	# Function:     test01
      	# Description   - Test if command dummy_cmd -f tst_file.in will print
      	#                 contents of tst_file.in
      	#               - Create input file tst_file.in
      	#               -
      	test01()
      	{
      	TCID="test01"    # Identifier of this testcase.
      	TST_COUNT=1      # Test case number.
      	RC=0             # return code from commands.
      
      	
      		In case of failure your testcase MUST exit with a
      		non-zero value. This exit value will be used by pan,
      		the test driver, to report FAIL. If the testcase exits with a
      		value zero, pan reports PASS. This variable is local to this
      		file.
      	
      
      	# Print test assertion.
      	tst_resm TINFO \
      	"Test #1: dummy_cmd -f tst_file.in will print contents of tst_file.in"
      
      	# Create input file tst_file.in
      	cat amp;gt $LTPTMP/tst_file.in amp;ltamp;lt -EOF || RC=$?
      	This is a test file
      	EOF
      	if [ $RC -ne 0 ]
      	then
      	tst_brkm TBROK NULL "Test #1: Unable to create input file"
      	return $RC
      	fi
      
      	# Perform test by calling command dummy_cmd with -f option
      	# and tst_file.in as input.
      	
      		Insert real code here. In case something wicked happenes use
      		tst_brk or tst_brkm functions to report this for example,
      		follow the lines of code below.
      	
      
      	# Print test results and exit with appropriate value
      	# Return code RC contains the value returned by dummy_cmd
      	# zero on success non-zero on failure.
      	if [ $RC -ne 0 ]
      	then
      	tst_res TFAIL $LTPTMP/tst_file.out \
      	"Test #1: dummy_cmd failed to print input file. Reason:"
      	return $RC
      	else
      	tst_resm TPASS \
      	"Test #1: succeeded in printing the contents of input file."
      	fi
      	return $RC
      	}
      
      
      	# Function:     main
      	#
      	# Description:  - Execute all tests, exit with test status.
      	#
      	# Exit:         - zero on success
      	#               - non-zero on failure.
      	#
      	RC=0    # Return value from setup, and test functions.
      
      	setup  || exit $RC
      
      	test01 || exit $RC

    2. Understanding the template

      Each test-case starts with a copyright and license statement. All test-cases in LTP are covered under the GNU public license . Copyrights usually belong to the author or corporation that developed the tests. Test case developed by IBM will have IBM copyrights, and test cases developed by SGI will have SGI copyrights and so on.

      The structure of the template is similar to tests written in ANSI-C. There are certain important points to note while writing tests in SHELL using the LTP command-line APIs. These are summarized below.
      • export TST_TOTAL="total number of tests in this file". This variable must be exported. It is used by the LTP command level APIs to display status or results.
      • export TCID="name of the test case". This variable must be exported. It is used by the LTP command-line APIs to report the name of the test while printing test status or results.
      • TST_COUNT="current test case number" If the test file has more than one test, each test must set this to an appropriate value. For instance, third test (test #3) should set this variable to 3, (TST_COUNT=3). The variable is used by the command level APIs for reporting status or results.
      • The arguments to command APIs are in conformance with ANSI-C APIs. The FUNC argument to the commands is ignored, but a value must be provided for conformance with ANSI-C APIs (use NULL). The arguments to commands are as follows:
        	export TCID=test name
        	export TST_TOTAL=Total Number of Tests
        	export TST_COUNT=Test case number
        
        	tst_brk TTYPE FNAME FUNC STRING
        	TTYPE  - Test Result Type; one of TFAIL, TBROK, TCONF, and  TRETR.
        	FNAME  - Print contents of this file after the message
        	FUNC   - Cleanup function (ignored), but MUST be provided. Use NULL.
        	STRING - Message explaining the test result
        
        	tst_brkm TTYPE FUNC STRING
        	TTYPE  - Test Result Type; one of TFAIL, TBROK, TCONF, and  TRETR.
        	FUNC   - Cleanup function (ignored), but MUST be provided. Use NULL.
        	STRING - Message explaining the test result
        
        	tst_res TTYPE FNAME STRING
        	TTYPE  - Test Result Type; one of TFAIL, TBROK, TCONF, and  TRETR.
        	FNAME  - Print contents of this file after the message
        	STRING - Message explaining the test result
        
        	tst_resm TTYPE STRING
        	TTYPE  - Test Result Type; one of TFAIL, TBROK, TCONF, and  TRETR.
        	STRING - Message explaining the test result
        
        	These messages are displayed if commands are called with no arguments.
        						
      • The USCTEST functions and macros are not available in the form of commands. These functions and macros may be added at a later time. Check with the latest release of LTP test suite for recent updates.
    3. Recommendation on coding style

      This section makes certain recommendation on code layout and style, but they are optional, it reflects the authors personal coding style and practice and may be applicable in general to any program written in SHELL.
      • Use a file header to describe the contents of file and maintain a change history. It may be argued that CVS provides the option to maintain such a change log while checking in code to the CVS tree, but often times the developer need not be responsible for maintaining the code in CVS. The change log or history maintained in the file can help both developer and the maintainer of the code to track changes made to the file.
      • Use meaningful variable names, and capitalize all variables. Avoid variable like 'i', 'j' etc instead use variable names like 'INDEX', 'ARRAY_NDX' etc. Initialize and comment the variables appropriately.
      • Comment code appropriately. Code may by handled by seasoned programmers as well as newbies, it is generally a good practice to add comments in the code where ever appropriate for maintainability reasons. Comments can be stripped off from the code using AWK or PERL scipts if required.
      • Use functions if possible, the template does not follow this rule for reasons of convenience.
      • Use tst_resm TINFO ... to print informational messages as appropriate, alternately, a verbose option or a 'quiet' option may be provided to suppress or activate such messages.
      • Use 'spaces' instead of 'tab' characters. Typically four spaces may be used instead of a tab character.
    4. More help

      At the time of this writing, only tst_brk, tst_brkm, tst_res and tst_resm functions are available for use as commands. The syntax for using these commands conforms with the ANSI-C APIs. Man pages for these ANSI-C APIs may be helpful in understanding the arguments to these commands. Alternately executing these commands without any arguments will print usage. Testcases using these commands are located under ltp-yyyymmdd/testcases/commands/ directory.

  6. Adding a new test to LTP

    You can add a new test to the LTP test suite can be done in two ways.

    If the developers does not have developer/maintainer access to the the LTP CVS tree.
    • Email the test case to the LTP mailing list and a developer or maintainer of LTP will add the test case after reviewing the code.
    If the test case develper has access to the LTP CVS tree.
    • Get the test case reviewed by peers. Test it throughly under different Linux distributions and kernel versions. This step is important in order to maintain a test suite that is of highest quality, and code free from bugs.
    • Create a sub-directory under the appropriate directory. For example, when adding a new system call test case, create a new directory with an appropriate name under the ltp-yyyymmmdd/testcases/kernel/syscalls/ directory.
    • Add the source code (written in ANSI-C) to this directory, use the generic Makefile, research existing test case for Makefile syntax. Generic template for test-cases in ANSI-C may be as follows:
      CFLAGS+=    -I../../../../include
      
      LOADLIBES+= -L../../../../lib -lltp
      
      SRCS=$(wildcard *.c)
      TARGETS=$(patsubst %.c,%,$(SRCS))
      all: $(TARGETS)
      
      install:
      @for i in $(TARGETS); do ln -f $$i ../../../bin/$$i ; done
      clean:
      rm -f $(TARGETS)
      
    • If the test case is written in shell, then the Makefile might require only an "install:" section. Please make sure that there are empty tags for "all:" and "clean:". For example:
      all:
      install:
      @ln -f mail_tests.sh ../../bin/mail_tests.sh
      
      clean:
    • Add a description of the test cases to the 00_Descriptions.txt file located in the ltp-yyyymmmdd/testcases/kernel/syscalls/ directory. The following template may be used:
      test01
      test does not exist, just an example.
      test02
      test does not exist, just an example.
      test03
      test does not exist, just an example
      test04
      test does not exist, just an example.
    • Add the new sub-directory to the top level Makefile, in this case add to Makefile under ltp-yyyymmmdd/testcases/kernel/syscalls/ to the SUBDIR variable.
    • Add the test "tag" and "command" to execute to the appropriate command file under ltp-yyyymmdd/runtest/ directory, to the syscall file. Follow the syntax used in this file. This file will be used as input to the test suite driver PAN, pan will use the contents of this file to execute the test case. For example, adding a hypothetical system call test01 to the command file syscall:
      syslog11 syslog11
      syslog12 syslog12
      time01 time01
      time02 time02
      times01 times01
      times02 times02
      times03 times03
      truncate01 truncate01
      truncate02 truncate02
      truncate03 truncate03
      truncate04 truncate04
      test01 test01
    • Send an email to the mailing list notifying the subscribers the addition of a a new test. ltp-list@lists.sourceforge.net

    Test case might be included in the subsequent monthly release of the LTP test suite, and due credit will be given to the author of the test case.

  7. Other Resources

    Please refer Linux Test Project website for updates, documentation, project details, email address, and to download the latest version of the test suite itself.
    Please refer Linux Test Project Kernel Code Coverage website for LTP's kernel code coverage tools.
    The Technical papers section contains a wealth of information pertaining to testing Linux and also the LTP test suite.

  8. Acknowledgements

    A special acknowledgement to Kristin Thomas for reviewing and editing this document. I also thank everyone who provided me with valuable comments and suggestions.

  9. Disclaimer

    IBM, the IBM logo, AIX are trademarks of the IBM Corporation.
    Pentium is a trademark of Intel Corporation in the United States, other countries, or both.

    Linux is a registered trademark of Linus Torvalds.

    All other trademarks and registered trademarks are the property of their respective owners.

    This publication reflects the views of the author, and not the IBM Corporation. This publication may include typographical errors and technical inaccuracies and may be changed or withdrawn at any time. The content is provided AS IS, without warranties of any kind, either express or implied, including the implied warranties of merchantability and fitness for a particular purpose.

    This publication may contain links to third party sites that are not under the control of or maintained by IBM. Access to any such third party site is at the user's own risk and IBM is not responsible for the accuracy or reliability of any information, data, opinions, advice or statements made on these sites. IBM provides these links merely as a convenience and the inclusion of such links does not imply an endorsement.

    The test results reported in this document represent results obtained from running tests under specific conditions at IBM's laboratory facility in Austin, Texas. Users may achieve different results depending on their particular installations, configurations, workloads, or other factors. The information in this document is provided solely for the information of the user. The information is provided on an "AS IS" basis, without liability or warranty. Users use such information at their own risk. Users are, thus, advised to evaluate the code referenced in this document as is appropriate for their specific installations.

    Document Author:
    Manoj Iyer


Sourceforge.net  Last modified on: April 28, 2012 - 14:25:35 UTC.
Theme: