Linux Test Project

Test Plan for 2.5 Stabilization

Version: 0.05
Maintainer: Stephanie Glass (sglass@us.ibm.com)
(512) 838-9284 T/L 678-9284
908/1C007
IBM Linux Technology Center - TEST
11501 Burnet Road
Austin, TX 78758
© Copyright 2003, IBM

It is the responsibility of the user of this document to ensure that he or she is using the current version of this document. To validate that your copy of this document is the most current, view the latest version of this document at /documentation/test_plans/25_testplan.php.

Table of Contents

  1. Legal Notices
  2. Document Control
  3. Overview
  4. Assumptions and Dependencies
  5. Test Goals and Objectives
  6. Problem Reporting and Tracking
  7. Entry and Exit Criteria
  8. Test case Descriptions
  9. Appendix A: Execution Plan
  1. The following terms are registered trademarks of International Business Machines Corporation in the United States and/or other countries: IBM, eServer, xSeries, pSeries, DB2, WebSphere. A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml.

    Linux is a registered trademark of Linus Torvalds.

    Other company, product, and service names may be trademarks or service marks of others.

  2. Document Control

    1. Change Summary

      This section reflects the changes made to the document.

      Date Author Description of Changes
      2003/02/03 Stephanie Glass Initial version of test plan.
      2003/02/21 Stephanie Glass Updated plan with comments from department team leads.
      2003/02/27 Stephanie Glass Updated plan with comments from other people.
      2003/03/25 Joy Y. Goodreau General edit. Inserted copyright and legal notices.
      2003/03/28 Stephanie Glass General edit for additional comments.
  3. Overview

    1. Purpose

      This test plan overviews testing the LTC Test Department is doing to help stabilize the Linux® 2.5 kernel. The perception was that when the Linux community was moving from the 2.3 to the 2.4 kernel, the time required for the 2.4 kernel to become stable was too long. The LTC Test Department is working with the Open Source community, the IBM brands, and different distributions to shorten this cycle for the 2.6 kernel.

      By testing the 2.5 kernel while the kernel is still under development, the LTC Test team expects to help shorten the time required for the 2.6 kernel to stabilize.

    2. Scope

      The objective for the 2.5 testing is to use variety of test tools and environments in order to find defects in the kernel. Once the defects are reported, the community can create fixes for those defects, which enables the kernel to stabilize quickly.

      As part of this effort, the LTC Test team is participating with the OSDL on 2.5 kernel stabilization Open Source project.

      Although the LTC Test team will be using many different test tools and scripts, the team will not be running any formal functional test unless stated otherwise in this document.

    3. Environment

      For IBM eServer™ xSeries™ hardware a combination of UP, 2-way, 4-way and 8-way will be used running the following distributions:

      • Red Hat 7.3
      • Red Hat 8.0
      • SuSE 8.0
      • SuSE 8.1
      • SuSE SLES8

      For IBM eServer pSeries™ hardware the LTC Test team has access to three B80 machines, of which two are 2-way PPC64 machines and the third is a 4-way PPC64 machine. On these systems the team will run the following distributions:

      • Red Hat 7.1
      • SLES 8
  4. Assumptions and Dependencies

    1. Assumptions

      The following assumptions are necessary to start testing:

      1. All testing will be done on a 2.5.x kernel.
      2. All defects found will be reported on http://bugme.osdl.org/ for xSeries machines.
      3. All defects found will be reported on http://bugzilla.linux.ibm.com/ in the PPC64 Development Family for pSeries machines.
    2. Dependencies

      The following list shows the dependencies for this project:

      1. For PPC64 testing the team must have available a weekly build from Oz Lab.
  5. Test Goals and Objectives

    1. Test Approach and Methodology

      The goal of this testing project is to use different test tools and scenarios to find as many defects as possible in the 2.5 kernel. Emphasis will be placed on using the IBM brands, both hardware and software, to make sure the 2.5 kernel functions properly with those brands.

      Results information will be kept for all test runs. At minimum, the information gathered will be either a log of TOP or SAR information collected during each run. If the individual test suite creates any additional logs for a test run, those files will also be kept. This information will be stored in a central repository that is backed-up weekly.

    2. Schedule

      Below is the schedule by test type:

      Type Of Test Plan Start Actual Start Plan End Actual End
      Focus Test on xSeries 2003/01/06 2003/01/06 2002/03/03
      Integration Test on xSeries 2003/02/17 2003/02/17 2003/06/01 or when 2.6 is created
      Stress Test on xSeries 2003/02/14 2003/02/24 2003/06/01 or when 2.6 is created
      Focus Test on pSeries 2003/02/03 2003/02/03 2003/03/24
      Integration Test on pSeries 2003/03/03 2003/03/17 2003/06/01 or when 2.6 is created
      Stress Test on pSeries 2003/03/03 2003/03/17 2003/06/01 or when 2.6 is created
    1. Installation/Configuration Test

      At this time there is no installation test scheduled. The LTC Test team is investigating whether to perform any configuration testing for the 2.5 kernel.

    2. Focus Test

      The goal of the focus test is to ensure product stability prior to integration and stress tests. The following tests will be run as part of the focus test:

      1. The LTP runall test.
      2. All tests in the LTP even if they are not part of the runall script.
      3. All test tools on the test tool matrix.
      4. Database and Web servers tests.
    3. Integration Test

      Integration tests attempt to create customer scenarios that view the system as a whole. There will be two different types of workloads for the integration test: automated and manual. Automated tests will be run every week for 120 hours in an automated fashion, including setup. The setup for the manual tests have not been automated and are therefore considered manual tests. Manual tests will run at least 96 hours. The duration of these tests will increase to fourteen days prior to the completion of all 2.5 kernel testing.

      There will be a minimal of six integration workloads. The workloads can be divided into three tests that are manual and three that are automated. The tests are as follows:

      1. Automated

        1. LTP with filesystem testing.
        2. dbgrinder with MySQL database.
        3. pagepoker running with an Apache server.
      2. Manual

        1. Trade3/akstress going to WebSphere® with a DB2® backend. DOTs will be used to add additional load to the DB2 server
    4. Stress Test

      Stress tests verify the robustness of the product during high system usage. In addition to the integration tests, which were designed to stress the kernel, a test scenario called ltp_stress was written. This test will be used on two 4-way xSeries machines in a rotation. The plan is for each test to run for 30 days. The two machines will have staggered start dates appropriately 14 days apart. This will enable the team to covered a large number of 2.5 kernels.

  6. Problem Reporting and Tracking

    All problems, issues and defects will be opened in http://bugme.osdl.org/. for xSeries machines and http://bugzilla.linux.ibm.com/ in the PPC64 Development Family for pSeries machines.

  7. Entry and Exit Criteria

    This section details the conditions that preclude entrance and exit from the test cycle.

    1. Entry Criteria

      1. 2.5 kernels are available.
    2. Exit Criteria

      1. A 2.6 kernel is released.
      2. All planned tests have been executed.
      3. All defects found during the test cycle have been opened in bugme.osdl.org for xSeries and bugzilla.linux.ibm.com for pSeries.
      4. Verification Summary Report available to include overall completed success rate.
  8. Test Case Information

    1. Test Tools

      • STAF/STAX
      • lcov/gcov
      • pan
    2. Test Software

      In addition to the test cases listed in this document, several other software products will be used in testing on 2.5 kernels:

      • Databases:
        • DB2
        • Oracle
        • Sybase
        • PostgreSQL
        • MySQL
      • Web Servers:
        • WebSphere
        • Apache
        • JBOSS
    3. Test Case Name, Description, and Location

      All of the following test cases will be run at least once. After running the following test cases, the LTC Test team will create a chart detailing if the test cases ran, if the test cases exposed any problems with the 2.5 kernel, and if the team will add the test cases to the standard 2.5 integration scenarios. This information will be documented in the Execution Plan found at IIOSB under Execution Plan for the 2.5 Stabilization Effort.

      Name Description Location
      DOTS Open source database tool that works with DB2, Oracle, Sybase, PostgreSQL, and MySQL http://ltp.sourceforge.net/
      OSDL Database Test Suite Open Source database tool http://www.osdl.org/projects/performance/osdldbt.html
      Bonnie++ A test suite that performs several hard drive/filesystem tests http://www.coker.com.au/bonnie++/
      Cerberus The Cereberus Test Control System is a free test suite for use by developers and others to test hardware. It generates good filesystem stress in the process. It is used by Red Hat as part of their test environment for new distributions. http://sourceforge.net/projects/va-ctcs/
      dbench Filesystem benchmark that generates good filesystem load ftp://samba.org/pub/tridge/dbench/
      fs_iode This test creates several subdirectories and files off of two parent directories and removes directories and files as part of the test http://ltp.sourceforge.net/
      dbgrinder This test is a perl script that inflict stress on a mysql server http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/ltp/utils/database/dbgrinder/
      fs_maim A set of scripts to test and stress filesystem and storage management utilities http://ltp.sourceforge.net/
      IOZone Filesystem benchmark tool that includes the following commands: (read,write,re-read,re-write, readbackwards, read strided, fread, fwrite, random read, pread, aio_read, aio_write) http://www.iozone.org/
      lftest A tool/test designed to create large files and lseek from the beginning of the file to the end of the file after each block write http://ltp.sourceforge.net/
      LTP (Linux Test Project) The collection of test cases for testing the Linux kernel and related features http://ltp.sourceforge.net/
      PostMark Filesystem benchmark that simutlates load generated by enterprise applications such as email, news and web-based commerce http://www.netapp.com/tech_library/postmark.html
      Stress Puts the system under a specified amount of load http://weather.ou.edu/~apw/projects/stress/
      Connectathon An NFS interoperability and conformance test suite http://www.connectathon.org/
      ISIC A suite of utilities to exercise the stability of an IP Stack and its component stacks http://www.packetfactory.net/Projects/ISIC/
      NetPerf A benchmark that can be used to measure the performance of many different types of networking http://www.netperf.org/
      NetPipe Variable time bench mark (for example, it measures network performance using variable sized communication transfers) http://www.scl.ameslab.gov/netpipe/
      TAHI Provides interoperability and conformance tests for IPv6 http://www.tahi.org/
      VolanoMark A java chatroom benchmark/stress http://www.volano.com/benchmarks.html
      OpenCryptoki Some unit test level test cases for openCryptoki http://www-124.ibm.com/developerworks/projects/openCryptoki/
      cpuburn A program designed to heavily load CPU chips http://users.ev1.net/~redelm/
      Nesssus Remote security scanner http://www.nessus.org/
      SRI (SBLIM Reference Implementation) A component of the SBLIM project. Its purposes are to easily setup, run, and test system management scenarios based on CIM/CIMOM technology and to test CIM Providers. http://oss.software.ibm.com/developerworks/projects/sblim/
      Hammerhead A Web server stress tool that can simulate multiple connects and users http://hammerhead.sourceforge.net/
      MemTest86 A thorough real-mode memory tester http://www.memtest86.com/
      httperf A popular web server benchmark tool for measuring web server performance http://www.hpl.hp.com/personal/David_Mosberger/httperf.html
      vmregress Benchmark tool for memory management http://www.csn.ul.ie/~mel/projects/vmregress/
      Seige An http regression testing and benchmarking utility http://www.joedog.org/siege/index.shtml
      Pagepoker A popular web tester test case location
      Trade2/Trade3 A J2EE, real-world WebSphere performance benchmark application http://www-3.ibm.com/software/webservers/
      VSTHlite An LSB test suite for posix threads http://www.opengroup.org/testing/downloads/vsthlite.html
      Chat Used for file_struct scalability testing http://lbs.sourceforge.net/chat/
      Contest Test system responsiveness by running kernel compilation under a number of different load conditions http://members.optusnet.com.au/ckolivas/contest/
      glibench/clibench Benchmarking tool to check your computer CPU and hard disk performance http://clibench.daemonware.ch/
      lmbench Suite of simple, portable benchmarks http://www.bitmover.com/lmbench/
      AIM Benchmarks Performance benchmarks http://sourceforge.net/projects/aimbench/
  9. Appendix A

    1. Execution Plan

      The latest Execution Plan for 2.5 Stabilization can be found at Sourceforge under 2.5 Execution Matrix.


Sourceforge.net  Last modified on: August 02, 2006 - 17:13:58 UTC.
Theme: