Version: 0.05
Maintainer: Stephanie Glass (sglass@us.ibm.com)
(512) 838-9284 T/L 678-9284
908/1C007
IBM Linux Technology Center - TEST
11501 Burnet Road
Austin, TX 78758
© Copyright 2003, IBM
It is the responsibility of the user of this document to ensure that he or she is using the current version of this document. To validate that your copy of this document is the most current, view the latest version of this document at /documentation/test_plans/25_testplan.php.
The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: IBM, eServer, xSeries, pSeries, DB2, WebSphere. A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml.
Linux is a registered trademark of Linus Torvalds.
Other company, product, and service names may be trademarks or service marks of others.
This section reflects the changes made to the document.
| Date | Author | Description of Changes |
|---|---|---|
| 2003/02/03 | Stephanie Glass | Initial version of test plan. |
| 2003/02/21 | Stephanie Glass | Updated plan with comments from department team leads. |
| 2003/02/27 | Stephanie Glass | Updated plan with comments from other people. |
| 2003/03/25 | Joy Y. Goodreau | General edit. Inserted copyright and legal notices. |
| 2003/03/28 | Stephanie Glass | General edit for additional comments. |
This test plan overviews testing the LTC Test Department is doing to help stabilize the Linux® 2.5 kernel. The perception was that when the Linux community was moving from the 2.3 to the 2.4 kernel, the time required for the 2.4 kernel to become stable was too long. The LTC Test Department is working with the Open Source community, the IBM brands, and different distributions to shorten this cycle for the 2.6 kernel.
By testing the 2.5 kernel while the kernel is still under development, the LTC Test team expects to help shorten the time required for the 2.6 kernel to stabilize.
The objective for the 2.5 testing is to use variety of test tools and environments in order to find defects in the kernel. Once the defects are reported, the community can create fixes for those defects, which enables the kernel to stabilize quickly.
As part of this effort, the LTC Test team is participating with the OSDL on 2.5 kernel stabilization Open Source project.
Although the LTC Test team will be using many different test tools and scripts, the team will not be running any formal functional test unless stated otherwise in this document.
For IBM eServer™ xSeries™ hardware a combination of UP, 2-way, 4-way and 8-way will be used running the following distributions:
For IBM eServer pSeries™ hardware the LTC Test team has access to three B80 machines, of which two are 2-way PPC64 machines and the third is a 4-way PPC64 machine. On these systems the team will run the following distributions:
The following assumptions are necessary to start testing:
The following list shows the dependencies for this project:
The goal of this testing project is to use different test tools and scenarios to find as many defects as possible in the 2.5 kernel. Emphasis will be placed on using the IBM brands, both hardware and software, to make sure the 2.5 kernel functions properly with those brands.
Results information will be kept for all test runs. At minimum, the information gathered will be either a log of TOP or SAR information collected during each run. If the individual test suite creates any additional logs for a test run, those files will also be kept. This information will be stored in a central repository that is backed-up weekly.
Below is the schedule by test type:
| Type Of Test | Plan Start | Actual Start | Plan End | Actual End |
|---|---|---|---|---|
| Focus Test on xSeries | 2003/01/06 | 2003/01/06 | 2002/03/03 | |
| Integration Test on xSeries | 2003/02/17 | 2003/02/17 | 2003/06/01 or when 2.6 is created | |
| Stress Test on xSeries | 2003/02/14 | 2003/02/24 | 2003/06/01 or when 2.6 is created | |
| Focus Test on pSeries | 2003/02/03 | 2003/02/03 | 2003/03/24 | |
| Integration Test on pSeries | 2003/03/03 | 2003/03/17 | 2003/06/01 or when 2.6 is created | |
| Stress Test on pSeries | 2003/03/03 | 2003/03/17 | 2003/06/01 or when 2.6 is created |
At this time there is no installation test scheduled. The LTC Test team is investigating whether to perform any configuration testing for the 2.5 kernel.
The goal of the focus test is to ensure product stability prior to integration and stress tests. The following tests will be run as part of the focus test:
Integration tests attempt to create customer scenarios that view the system as a whole. There will be two different types of workloads for the integration test: automated and manual. Automated tests will be run every week for 120 hours in an automated fashion, including setup. The setup for the manual tests have not been automated and are therefore considered manual tests. Manual tests will run at least 96 hours. The duration of these tests will increase to fourteen days prior to the completion of all 2.5 kernel testing.
There will be a minimal of six integration workloads. The workloads can be divided into three tests that are manual and three that are automated. The tests are as follows:
Stress tests verify the robustness of the product during high system usage. In addition to the integration tests, which were designed to stress the kernel, a test scenario called ltp_stress was written. This test will be used on two 4-way xSeries machines in a rotation. The plan is for each test to run for 30 days. The two machines will have staggered start dates appropriately 14 days apart. This will enable the team to covered a large number of 2.5 kernels.
All problems, issues and defects will be opened in http://bugme.osdl.org/. for xSeries machines and http://bugzilla.linux.ibm.com/ in the PPC64 Development Family for pSeries machines.
This section details the conditions that preclude entrance and exit from the test cycle.
In addition to the test cases listed in this document, several other software products will be used in testing on 2.5 kernels:
All of the following test cases will be run at least once. After running the following test cases, the LTC Test team will create a chart detailing if the test cases ran, if the test cases exposed any problems with the 2.5 kernel, and if the team will add the test cases to the standard 2.5 integration scenarios. This information will be documented in the Execution Plan found at IIOSB under Execution Plan for the 2.5 Stabilization Effort.
| Name | Description | Location |
|---|---|---|
| DOTS | Open source database tool that works with DB2, Oracle, Sybase, PostgreSQL, and MySQL | http://ltp.sourceforge.net/ |
| OSDL Database Test Suite | Open Source database tool | http://www.osdl.org/projects/performance/osdldbt.html |
| Bonnie++ | A test suite that performs several hard drive/filesystem tests | http://www.coker.com.au/bonnie++/ |
| Cerberus | The Cereberus Test Control System is a free test suite for use by developers and others to test hardware. It generates good filesystem stress in the process. It is used by Red Hat as part of their test environment for new distributions. | http://sourceforge.net/projects/va-ctcs/ |
| dbench | Filesystem benchmark that generates good filesystem load | ftp://samba.org/pub/tridge/dbench/ |
| fs_iode | This test creates several subdirectories and files off of two parent directories and removes directories and files as part of the test | http://ltp.sourceforge.net/ |
| dbgrinder | This test is a perl script that inflict stress on a mysql server | http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ltp/utils/database/dbgrinder/ |
| fs_maim | A set of scripts to test and stress filesystem and storage management utilities | http://ltp.sourceforge.net/ |
| IOZone | Filesystem benchmark tool that includes the following commands: (read,write,re-read,re-write, readbackwards, read strided, fread, fwrite, random read, pread, aio_read, aio_write) | http://www.iozone.org/ |
| lftest | A tool/test designed to create large files and lseek from the beginning of the file to the end of the file after each block write | http://ltp.sourceforge.net/ |
| LTP (Linux Test Project) | The collection of test cases for testing the Linux kernel and related features | http://ltp.sourceforge.net/ |
| PostMark | Filesystem benchmark that simutlates load generated by enterprise applications such as email, news and web-based commerce | http://www.netapp.com/tech_library/postmark.html |
| Stress | Puts the system under a specified amount of load | http://weather.ou.edu/~apw/projects/stress/ |
| Connectathon | An NFS interoperability and conformance test suite | http://www.connectathon.org/ |
| ISIC | A suite of utilities to exercise the stability of an IP Stack and its component stacks | http://www.packetfactory.net/Projects/ISIC/ |
| NetPerf | A benchmark that can be used to measure the performance of many different types of networking | http://www.netperf.org/ |
| NetPipe | Variable time bench mark (for example, it measures network performance using variable sized communication transfers) | http://www.scl.ameslab.gov/netpipe/ |
| TAHI | Provides interoperability and conformance tests for IPv6 | http://www.tahi.org/ |
| VolanoMark | A java chatroom benchmark/stress | http://www.volano.com/benchmarks.html |
| OpenCryptoki | Some unit test level test cases for openCryptoki | http://www-124.ibm.com/developerworks/projects/openCryptoki/ |
| cpuburn | A program designed to heavily load CPU chips | http://users.ev1.net/~redelm/ |
| Nesssus | Remote security scanner | http://www.nessus.org/ |
| SRI (SBLIM Reference Implementation) | A component of the SBLIM project. Its purposes are to easily setup, run, and test system management scenarios based on CIM/CIMOM technology and to test CIM Providers. | http://oss.software.ibm.com/developerworks/projects/sblim/ |
| Hammerhead | A Web server stress tool that can simulate multiple connects and users | http://hammerhead.sourceforge.net/ |
| MemTest86 | A thorough real-mode memory tester | http://www.memtest86.com/ |
| httperf | A popular web server benchmark tool for measuring web server performance | http://www.hpl.hp.com/personal/David_Mosberger/httperf.html |
| vmregress | Benchmark tool for memory management | http://www.csn.ul.ie/~mel/projects/vmregress/ |
| Seige | An http regression testing and benchmarking utility | http://www.joedog.org/siege/index.shtml |
| Pagepoker | A popular web tester | test case location |
| Trade2/Trade3 | A J2EE, real-world WebSphere performance benchmark application | http://www-3.ibm.com/software/webservers/ |
| VSTHlite | An LSB test suite for posix threads | http://www.opengroup.org/testing/downloads/vsthlite.html |
| Chat | Used for file_struct scalability testing | http://lbs.sourceforge.net/chat/ |
| Contest | Test system responsiveness by running kernel compilation under a number of different load conditions | http://members.optusnet.com.au/ckolivas/contest/ |
| glibench/clibench | Benchmarking tool to check your computer CPU and hard disk performance | http://clibench.daemonware.ch/ |
| lmbench | Suite of simple, portable benchmarks | http://www.bitmover.com/lmbench/ |
| AIM Benchmarks | Performance benchmarks | http://sourceforge.net/projects/aimbench/ |
The latest Execution Plan for 2.5 Stabilization can be found at Sourceforge under 2.5 Execution Matrix.
| Last modified on: August 02, 2006 - 17:13:58 UTC. |