08:01 AM PST
April 18, 2014
LTP logo LTP logo
(LTP Menu) o Home Page
o Project Page
o News
o Lists & IRC
o Kernel Errata

Code Coverage
o LTP's Kernel Code Coverage
o LTP's Gcov Extension(lcov)
o LTP's Gcov-kernel Extension
o Coverage "Galaxy" Map

Documentation
o LTP HowTo
o Inside LTP Testsuite
o 2.5 Execution Matrix
o 2.5 Test Plan
o 2.4 Test Plan
o STAX LTP Driver HowTo
o DOTS HowTo
o Linux Test Tools
o LTP man pages

Test Results
o Expected errors for LTP
o Kernel Test Results
o 2002 Test Results
o 2001 Test Results

Technical Papers

Source
[CVS|Web]
Download

Related Projects
o Kernel Fix
o Open *POSIX Test Suite
o OpenHPI Test Suite
o (CTCS) Cerberus
o (STAF) Software Testing Automation Framework
o EAL2+ Certification Test Suite
o SuSE EAL3+ Certification Test Suite
o Red Hat EAL3+ Certification Test Suite
SourceForge Logo
2.5 kernel stabilization test plan  

Test Plan for 2.5 Stabilization

 

 

Version: 0.05

Maintainer: Stephanie Glass
sglass@us.ibm.com
(512) 838-9284 T/L 678-9284
908/1C007
IBM Linux Technology Center - TEST
11501 Burnet Road
Austin, TX 78758
© Copyright 2003, IBM
 
 
It is the responsibility of the user of this document to ensure that he or she is using the current version of this document. To validate that your copy of this document is the mose current, view the latest version of this document at Sourceforge under 2.5 Test Plan.
 
 

Table of Contents

 

Legal Notices

The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: IBM, eServer, xSeries, pSeries, DB2, WebSphere. A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml.

Linux is a registered trademark of Linus Torvalds.

Other company, product, and service names may be trademarks or service marks of others.  

Document Control

  • Change Summary

  • This section reflects the changes made to the document.

    Date Author Description of Changes
    02/03/03 Stephanie Glass Initial version of test plan
    02/21/03 Stephanie Glass Updated plan with comments from department team leads
    02/27/03 Stephanie Glass Updated plan with comments from other people
    03/25/2003 Joy Y. Goodreau General edit. Inserted copyright and legal notices
    03/28/2003 Stephanie Glass General edit for additional comments





Overview

  • Purpose

  • This test plan overviews testing the LTC Test Department is doing to help stabilize the Linux® 2.5 kernel. The perception was that when the Linux community was moving from the 2.3 to the 2.4 kernel, the time required for the 2.4 kernel to become stable was too long. The LTC Test Department is working with the Open Source community, the IBM brands, and different distributions to shorten this cycle for the 2.6 kernel.

    By testing the 2.5 kernel while the kernel is still under development, the LTC Test team expects to help shorten the time required for the 2.6 kernel to stabilize.

  • Scope

  • The objective for the 2.5 testing is to use variety of test tools and environments in order to find defects in the kernel. Once the defects are reported, the community can create fixes for those defects, which enables the kernel to stabilize quickly.

    As part of this effort, the LTC Test team is participating with the OSDL on 2.5 kernel stabilization Open Source project.

    Although the LTC Test team will be using many different test tools and scripts, the team will not be running any formal functional test unless stated otherwise in this document.

  • Environment

  • For IBM eServer™ xSeries™ hardware a combination of UP, 2-way, 4-way and 8-way will be used running the following distributions:

    • Red Hat 7.3
    • Red Hat 8.0
    • SuSE 8.0
    • SuSE 8.1
    • SuSE SLES8


    For IBM eServer pSeries™ hardware the LTC Test team has access to three B80 machines, of which two are 2-way PPC64 machines and the third is a 4-way PPC64 machine. On these systems the team will run the following distributions:

    • Red Hat 7.1
    • SLES 8

Assumptions and Dependencies

  • Assumptions

  • The following assumptions are necessary to start testing:

    • All testing will be done on a 2.5.x kernel
    • All defects found will be reported on bugme.osdl.org. for xSeries machines.
    • All defects found will be reported on bugzilla.linux.ibm.com in the PPC64 Development Family for pSeries machines.


  • Dependencies

  • The following list shows the dependencies for this project:

    • For PPC64 testing the team must have available a weekly build from Oz Lab.


Test Goals and Objectives

  • Test Approach and Methodology

  • The goal of this testing project is to use different test tools and scenarios to find as many defects as possible in the 2.5 kernel. Emphasis will be placed on using the IBM brands, both hardware and software, to make sure the 2.5 kernel functions properly with those brands.

    Results information will be kept for all test runs. At minimum, the information gathered will be either a log of TOP or SAR information collected during each run. If the individual test suite creates any additional logs for a test run, those files will also be kept. This information will be stored in a central repository that is backed-up weekly.

  • Schedule

  • Below is the schedule by test type:
    Type Of Test Plan Start Actual Start Plan End Actual End
    Focus Test on xSeries 01/06/2003 01/06/2003 03/03/2002
    Integration Test on xSeries 02/17/2003 02/17/2003 06/01/2003 or when 2.6 is created
    Stress Test on xSeries 02/14/2003 02/24/2003 06/01/2003 or when 2.6 is created
    Focus Test on pSeries 02/03/2003 02/03/2003 03/24/2003
    Integration Test on pSeries 03/03/2003 03/17/2003 06/01/2003 or when 2.6 is created
    Stress Test on pSeries 03/03/2003 03/17/2003 06/01/2003 or when 2.6 is created




  • Installation/Configuration Test

  • At this time there is no installation test scheduled. The LTC Test team is investigating whether to perform any configuration testing for the 2.5 kernel.

  • Focus Test

  • The goal of the focus test is to ensure product stability prior to integration and stress tests. The following tests will be run as part of the focus test:

    1. The LTP runall test
    2. All tests in the LTP even if they are not part of the runall script
    3. All test tools on the test tool matrix
    4. Database and Web servers tests


  • Integration Test

  • Integration tests attempt to create customer scenarios that view the system as a whole. There will be two different types of workloads for the integration test: automated and manual. Automated tests will be run every week for 120 hours in an automated fashion, including setup. The setup for the manual tests have not been automated and are therefore considered manual tests. Manual tests will run at least 96 hours. The duration of these tests will increase to fourteen days prior to the completion of all 2.5 kernel testing.

    There will be a minimal of six integration workloads. The workloads can be divided into three tests that are manual and three that are automated. The tests are as follows:
    • Automated
      • LTP with filesystem testing
      • dbgrinder with MySQL database
      • pagepoker running with an Apache server
    • Manual
      • Trade3/akstress going to WebSphere® with a DB2® backend. DOTs will be used to add additional load to the DB2 server


  • Stress Test

  • Stress tests verify the robustness of the product during high system usage. In addition to the integration tests, which were designed to stress the kernel, a test scenario called ltp_stress was written. This test will be used on two 4-way xSeries machines in a rotation. The plan is for each test to run for 30 days. The two machines will have staggered start dates appropriately 14 days apart. This will enable the team to covered a large number of 2.5 kernels.

 

Problem Reporting and Tracking

All problems, issues and defects will be opened in bugme.osdl.org. for xSeries machines and bugzilla.linux.ibm.com in the PPC64 Development Family for pSeries machines.

 

Entry and Exit Criteria

This section details the conditions that preclude entrance and exit from the test cycle.

  • Entry Criteria

    • 2.5 kernels are available

  • Exit Criteria

    • A 2.6 kernel is released
    • All planned tests have been executed
    • All defects found during the test cycle have been opened in bugme.osdl.org for xSeries and bugzilla.linux.ibm.com for pSeries
    • Verification Summary Report available to include overall completed success rate

Test Case Information

  • Test Tools

    • STAF/STAX
    • lcov/gcov
    • pan

  • Test Software


  • In addition to the test cases listed in this document, several other software products will be used in testing on 2.5 kernels:
    • Databases
      • DB2
      • Oracle
      • Sybase
      • PostgreSQL
      • MySQL
    • Web Servers
      • WebSphere
      • Apache
      • JBOSS


  • Test Case Name, Description, and Location


  • All of the following test cases will be run at least once. After running the following test cases, the LTC Test team will create a chart detailing if the test cases ran, if the test cases exposed any problems with the 2.5 kernel, and if the team will add the test cases to the standard 2.5 integration scenarios. This information will be documented in the Execution Plan found at IIOSB under Execution Plan for the 2.5 Stabilization Effort.

    Name Description Location
    DOTS Open source database tool that works with DB2, Oracle, Sybase, PostgreSQL, and MySQL http://ltp.sourceforge.net/
    OSDL Database Test Suite Open Source database tool http://www.osdl.org/projects/performance/osdldbt.html
    Bonnie++ A test suite that performs several hard drive/filesystem tests http://www.coker.com.au/bonnie++/
    Cerberus The Cereberus Test Control System is a free test suite for use by developers and others to test hardware. It generates good filesystem stress in the process. It is used by Red Hat as part of their test environment for new distributions. http://sourceforge.net/projects/va-ctcs
    dbench Filesystem benchmark that generates good filesystem load ftp://samba.org/pub/tridge/dbench/
    fs_iode This test creates several subdirectories and files off of two parent directories and removes directories and files as part of the test http://ltp.sourceforge.net/
    dbgrinder This test is a perl script that inflict stress on a mysql server http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ltp/utils/database/dbgrinder/
    fs_maim A set of scripts to test and stress filesystem and storage management utilities http://ltp.sourceforge.net/
    IOZone Filesystem benchmark tool that includes the following commands: (read,write,re-read,re-write, readbackwards, read strided, fread, fwrite, random read, pread, aio_read, aio_write) http://www.iozone.org/
    lftest A tool/test designed to create large files and lseek from the beginning of the file to the end of the file after each block write http://ltp.sourceforge.net/
    LTP
    (Linux Test Project)
    The collection of test cases for testing the Linux kernel and related features http://ltp.sourceforge.net/
    PostMark Filesystem benchmark that simutlates load generated by enterprise applications such as email, news and web-based commerce http://www.netapp.com/tech_library/postmark.html
    Stress Puts the system under a specified amount of load http://weather.ou.edu/~apw/projects/stress/
    Connectathon An NFS interoperability and conformance test suite http://www.connectathon.org/
    ISIC A suite of utilities to exercise the stability of an IP Stack and its component stacks http://www.packetfactory.net/Projects/ISIC/
    NetPerf A benchmark that can be used to measure the performance of many different types of networking http://www.netperf.org/
    NetPipe Variable time bench mark (for example, it measures network performance using variable sized communication transfers) http://www.scl.ameslab.gov/netpipe/
    TAHI Provides interoperability and conformance tests for IPv6 http://www.tahi.org/
    VolanoMark A java chatroom benchmark/stress http://www.volano.com/benchmarks.html
    OpenCryptoki Some unit test level test cases for openCryptoki http://www-124.ibm.com/developerworks/projects/openCryptoki
    cpuburn A program designed to heavily load CPU chips http://users.ev1.net/~redelm/
    Nesssus Remote security scanner http://www.nessus.org/
    SRI
    (SBLIM Reference Implementation)
    A component of the SBLIM project. Its purposes are to easily setup, run, and test system management scenarios based on CIM/CIMOM technology and to test CIM Providers. http://oss.software.ibm.com/developerworks/projects/sblim/
    Hammerhead A Web server stress tool that can simulate multiple connects and users http://hammerhead.sourceforge.net/
    MemTest86 A thorough real-mode memory tester http://www.memtest86.com/
    httperf A popular web server benchmark tool for measuring web server performance http://www.hpl.hp.com/personal/David_Mosberger/httperf.html
    vmregress Benchmark tool for memory management http://www.csn.ul.ie/~mel/projects/vmregress/
    Seige An http regression testing and benchmarking utility http://www.joedog.org/siege/index.shtml
    Pagepoker A popular web tester test case location
    Trade2/Trade3 A J2EE, real-world WebSphere performance benchmark application http://www-3.ibm.com/software/webservers/
    VSTHlite An LSB test suite for posix threads http://www.opengroup.org/testing/downloads/vsthlite.html
    Chat Used for file_struct scalability testing http://lbs.sourceforge.net/chat/
    Contest Test system responsiveness by running kernel compilation under a number of different load conditions http://members.optusnet.com.au/ckolivas/contest/
    glibench/clibench Benchmarking tool to check your computer CPU and hard disk performance http://clibench.daemonware.ch/
    lmbench Suite of simple, portable benchmarks http://www.bitmover.com/lmbench/
    AIM Benchmarks Performance benchmarks http://sourceforge.net/projects/aimbench/




Appendix A

Execution Plan

The latest Execution Plan for 2.5 Stabilization can be found at Sourceforge under 2.5 Execution Matrix.