Re: [LTP] New README

[Janet Clegg <janetc@sgi.com>]

> This brings us to the topic of a runtime-driver; something responsible
> for retrieving command lines and running them.  It might also be
> responsible configuration analysis and result reporting.  Configuration
> analysis refers to the runtime-driver being able to report whether a
> test program or even a specific test case within a test program failed. 

Hi Aaron,

You meant "Result Reporting" in the last sentence above.

You also hit one of my pet peeves.  A test succeeds when it finds a
failure.  A test fails when it gives wrong results, whether reporting
a PASS or a FAIL (test case failure).   Results Reporting (or Results 
Analysis) will summarize which, if any, of the test cases or test 
programs reported a failure of the software under test.

As long as I'm on pet peeves, here's another instance, taken from the 
FAQ "How do I analyze the results":

$ ./dup02
dup02       1  PASS  :  dup(-1) Failed, errno=9 : Bad file descriptor
dup02       2  PASS  :  dup(1500) Failed, errno=9 : Bad file descriptor

The message that the dup call "Failed" can be misleading.  The dup call 
didn't fail -- it correctly reported an error for bad input.  The test
would communicate much better if the word "Failed" were simply deleted
from the output.  I use the term "negative test case" if I want to 
indicate that a test item is "failing in a correct and expected manner".

janet

-- 
________________________________________________________________________
Janet Clegg                                         janet.clegg@sgi.com
Silicon Graphics, Inc                               651) 683-5207