Test Plan for 2.5 Stabilization

Table of Contents

• Legal Notices
• Document Control
• Overview
• Assumptions and Dependencies
• Test Goals and Objectives
• Problem Reporting and Tracking
• Entry and Exit Criteria
• Test case Descriptions
• Appendix A: Execution Plan

Legal Notices

The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: IBM, eServer, xSeries, pSeries, DB2, WebSphere. A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml.

Linux is a registered trademark of Linus Torvalds.

Other company, product, and service names may be trademarks or service marks of others.  

Document Control

• Change Summary

This section reflects the changes made to the document.



Date	Author	Description of Changes
02/03/03	Stephanie Glass	Initial version of test plan
02/21/03	Stephanie Glass	Updated plan with comments from department team leads
02/27/03	Stephanie Glass	Updated plan with comments from other people
03/25/2003	Joy Y. Goodreau	General edit. Inserted copyright and legal notices
03/28/2003	Stephanie Glass	General edit for additional comments

Overview

• Purpose

This test plan overviews testing the LTC Test Department is doing to help stabilize the Linux® 2.5 kernel. The perception was that when the Linux community was moving from the 2.3 to the 2.4 kernel, the time required for the 2.4 kernel to become stable was too long. The LTC Test Department is working with the Open Source community, the IBM brands, and different distributions to shorten this cycle for the 2.6 kernel.

By testing the 2.5 kernel while the kernel is still under development, the LTC Test team expects to help shorten the time required for the 2.6 kernel to stabilize.



• Scope

The objective for the 2.5 testing is to use variety of test tools and environments in order to find defects in the kernel. Once the defects are reported, the community can create fixes for those defects, which enables the kernel to stabilize quickly.

As part of this effort, the LTC Test team is participating with the OSDL on 2.5 kernel stabilization Open Source project.

Although the LTC Test team will be using many different test tools and scripts, the team will not be running any formal functional test unless stated otherwise in this document.



• Environment

For IBM eServer™ xSeries™ hardware a combination of UP, 2-way, 4-way and 8-way will be used running the following distributions:


• Red Hat 7.3
• Red Hat 8.0
• SuSE 8.0
• SuSE 8.1
• SuSE SLES8


For IBM eServer pSeries™ hardware the LTC Test team has access to three B80 machines, of which two are 2-way PPC64 machines and the third is a 4-way PPC64 machine. On these systems the team will run the following distributions:


• Red Hat 7.1
• SLES 8

Assumptions and Dependencies

• Assumptions

The following assumptions are necessary to start testing:


• All testing will be done on a 2.5.x kernel
• All defects found will be reported on bugme.osdl.org. for xSeries machines.
• All defects found will be reported on bugzilla.linux.ibm.com in the PPC64 Development Family for pSeries machines.




• Dependencies

The following list shows the dependencies for this project:


• For PPC64 testing the team must have available a weekly build from Oz Lab.

Test Goals and Objectives

• Test Approach and Methodology

The goal of this testing project is to use different test tools and scenarios to find as many defects as possible in the 2.5 kernel. Emphasis will be placed on using the IBM brands, both hardware and software, to make sure the 2.5 kernel functions properly with those brands.

Results information will be kept for all test runs. At minimum, the information gathered will be either a log of TOP or SAR information collected during each run. If the individual test suite creates any additional logs for a test run, those files will also be kept. This information will be stored in a central repository that is backed-up weekly.



• Schedule

Below is the schedule by test type:

Type Of Test	Plan Start	Actual Start	Plan End	Actual End
Focus Test on xSeries	01/06/2003	01/06/2003	03/03/2002
Integration Test on xSeries	02/17/2003	02/17/2003	06/01/2003 or when 2.6 is created
Stress Test on xSeries	02/14/2003	02/24/2003	06/01/2003 or when 2.6 is created
Focus Test on pSeries	02/03/2003	02/03/2003	03/24/2003
Integration Test on pSeries	03/03/2003	03/17/2003	06/01/2003 or when 2.6 is created
Stress Test on pSeries	03/03/2003	03/17/2003	06/01/2003 or when 2.6 is created

• Installation/Configuration Test

At this time there is no installation test scheduled. The LTC Test team is investigating whether to perform any configuration testing for the 2.5 kernel.



• Focus Test

The goal of the focus test is to ensure product stability prior to integration and stress tests. The following tests will be run as part of the focus test:


1. The LTP runall test
2. All tests in the LTP even if they are not part of the runall script
3. All test tools on the test tool matrix
4. Database and Web servers tests




• Integration Test

Integration tests attempt to create customer scenarios that view the system as a whole. There will be two different types of workloads for the integration test: automated and manual. Automated tests will be run every week for 120 hours in an automated fashion, including setup. The setup for the manual tests have not been automated and are therefore considered manual tests. Manual tests will run at least 96 hours. The duration of these tests will increase to fourteen days prior to the completion of all 2.5 kernel testing.

There will be a minimal of six integration workloads. The workloads can be divided into three tests that are manual and three that are automated. The tests are as follows:
• Automated
• LTP with filesystem testing
• dbgrinder with MySQL database
• pagepoker running with an Apache server
• Manual
• Trade3/akstress going to WebSphere® with a DB2® backend. DOTs will be used to add additional load to the DB2 server




• Stress Test

Stress tests verify the robustness of the product during high system usage. In addition to the integration tests, which were designed to stress the kernel, a test scenario called ltp_stress was written. This test will be used on two 4-way xSeries machines in a rotation. The plan is for each test to run for 30 days. The two machines will have staggered start dates appropriately 14 days apart. This will enable the team to covered a large number of 2.5 kernels.

Problem Reporting and Tracking

Entry and Exit Criteria

This section details the conditions that preclude entrance and exit from the test cycle.

• Entry Criteria

• 2.5 kernels are available



• Exit Criteria

• A 2.6 kernel is released
• All planned tests have been executed
• All defects found during the test cycle have been opened in bugme.osdl.org for xSeries and bugzilla.linux.ibm.com for pSeries
• Verification Summary Report available to include overall completed success rate

Test Case Information

• Test Tools

• STAF/STAX
• lcov/gcov
• pan



• Test Software


In addition to the test cases listed in this document, several other software products will be used in testing on 2.5 kernels:
• Databases
  • DB2
  • Oracle
  • Sybase
  • PostgreSQL
  • MySQL
• Web Servers
  • WebSphere
  • Apache
  • JBOSS




• Test Case Name, Description, and Location


All of the following test cases will be run at least once. After running the following test cases, the LTC Test team will create a chart detailing if the test cases ran, if the test cases exposed any problems with the 2.5 kernel, and if the team will add the test cases to the standard 2.5 integration scenarios. This information will be documented in the Execution Plan found at IIOSB under Execution Plan for the 2.5 Stabilization Effort.



Name	Description	Location
DOTS	Open source database tool that works with DB2, Oracle, Sybase, PostgreSQL, and MySQL	http://ltp.sourceforge.net/
OSDL Database Test Suite	Open Source database tool	http://www.osdl.org/projects/performance/osdldbt.html
Bonnie++	A test suite that performs several hard drive/filesystem tests	http://www.coker.com.au/bonnie++/
Cerberus	The Cereberus Test Control System is a free test suite for use by developers and others to test hardware. It generates good filesystem stress in the process. It is used by Red Hat as part of their test environment for new distributions.	http://sourceforge.net/projects/va-ctcs
dbench	Filesystem benchmark that generates good filesystem load	ftp://samba.org/pub/tridge/dbench/
fs_iode	This test creates several subdirectories and files off of two parent directories and removes directories and files as part of the test	http://ltp.sourceforge.net/
dbgrinder	This test is a perl script that inflict stress on a mysql server	http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ltp/utils/database/dbgrinder/
fs_maim	A set of scripts to test and stress filesystem and storage management utilities	http://ltp.sourceforge.net/
IOZone	Filesystem benchmark tool that includes the following commands: (read,write,re-read,re-write, readbackwards, read strided, fread, fwrite, random read, pread, aio_read, aio_write)	http://www.iozone.org/
lftest	A tool/test designed to create large files and lseek from the beginning of the file to the end of the file after each block write	http://ltp.sourceforge.net/
LTP (Linux Test Project)	The collection of test cases for testing the Linux kernel and related features	http://ltp.sourceforge.net/
PostMark	Filesystem benchmark that simutlates load generated by enterprise applications such as email, news and web-based commerce	http://www.netapp.com/tech_library/postmark.html
Stress	Puts the system under a specified amount of load	http://weather.ou.edu/~apw/projects/stress/
Connectathon	An NFS interoperability and conformance test suite	http://www.connectathon.org/
ISIC	A suite of utilities to exercise the stability of an IP Stack and its component stacks	http://www.packetfactory.net/Projects/ISIC/
NetPerf	A benchmark that can be used to measure the performance of many different types of networking	http://www.netperf.org/
NetPipe	Variable time bench mark (for example, it measures network performance using variable sized communication transfers)	http://www.scl.ameslab.gov/netpipe/
TAHI	Provides interoperability and conformance tests for IPv6	http://www.tahi.org/
VolanoMark	A java chatroom benchmark/stress	http://www.volano.com/benchmarks.html
OpenCryptoki	Some unit test level test cases for openCryptoki	http://www-124.ibm.com/developerworks/projects/openCryptoki
cpuburn	A program designed to heavily load CPU chips	http://users.ev1.net/~redelm/
Nesssus	Remote security scanner	http://www.nessus.org/
SRI (SBLIM Reference Implementation)	A component of the SBLIM project. Its purposes are to easily setup, run, and test system management scenarios based on CIM/CIMOM technology and to test CIM Providers.	http://oss.software.ibm.com/developerworks/projects/sblim/
Hammerhead	A Web server stress tool that can simulate multiple connects and users	http://hammerhead.sourceforge.net/
MemTest86	A thorough real-mode memory tester	http://www.memtest86.com/
httperf	A popular web server benchmark tool for measuring web server performance	http://www.hpl.hp.com/personal/David_Mosberger/httperf.html
vmregress	Benchmark tool for memory management	http://www.csn.ul.ie/~mel/projects/vmregress/
Seige	An http regression testing and benchmarking utility	http://www.joedog.org/siege/index.shtml
Pagepoker	A popular web tester	test case location
Trade2/Trade3	A J2EE, real-world WebSphere performance benchmark application	http://www-3.ibm.com/software/webservers/
VSTHlite	An LSB test suite for posix threads	http://www.opengroup.org/testing/downloads/vsthlite.html
Chat	Used for file_struct scalability testing	http://lbs.sourceforge.net/chat/
Contest	Test system responsiveness by running kernel compilation under a number of different load conditions	http://members.optusnet.com.au/ckolivas/contest/
glibench/clibench	Benchmarking tool to check your computer CPU and hard disk performance	http://clibench.daemonware.ch/
lmbench	Suite of simple, portable benchmarks	http://www.bitmover.com/lmbench/
AIM Benchmarks	Performance benchmarks	http://sourceforge.net/projects/aimbench/

Appendix A

Execution Plan