(automake-1.16.info)Log files generation and test results recording

---

Next: Testsuite progress output Prev: Command-line arguments for test drivers Up: API for Custom Test Drivers

---

Enter node , (file) or (file)node

15.3.3.2 Log files generation and test results recording
........................................................

The test driver must correctly generate the files specified by the
‘--log-file’ and ‘--trs-file’ option (even when the tested program fails
or crashes).

   The ‘.log’ file should ideally contain all the output produced by the
tested program, plus optionally other information that might facilitate
debugging or analysis of bug reports.  Apart from that, its format is
basically free.

   The ‘.trs’ file is used to register some metadata through the use of
custom reStructuredText fields.  This metadata is expected to be
employed in various ways by the parallel test harness; for example, to
count the test results when printing the testsuite summary, or to decide
which tests to re-run upon ‘make recheck’.  Unrecognized metadata in a
‘.trs’ file is currently ignored by the harness, but this might change
in the future.  The list of currently recognized metadata follows.

‘:test-result:’
     The test driver must use this field to register the results of
     _each_ test case run by a test script file.  Several
     ‘:test-result:’ fields can be present in the same ‘.trs’ file; this
     is done in order to support test protocols that allow a single test
     script to run more test cases.

     The only recognized test results are currently ‘PASS’, ‘XFAIL’,
     ‘SKIP’, ‘FAIL’, ‘XPASS’ and ‘ERROR’.  These results, when declared
     with ‘:test-result:’, can be optionally followed by text holding
     the name and/or a brief description of the corresponding test; the
     harness will ignore such extra text when generating
     ‘test-suite.log’ and preparing the testsuite summary.

‘:recheck:’
     If this field is present and defined to ‘no’, then the
     corresponding test script will _not_ be run upon a ‘make recheck’.
     What happens when two or more ‘:recheck:’ fields are present in the
     same ‘.trs’ file is undefined behaviour.

‘:copy-in-global-log:’
     If this field is present and defined to ‘no’, then the content of
     the ‘.log’ file will _not_ be copied into the global
     ‘test-suite.log’.  We allow to forsake such copying because, while
     it can be useful in debugging and analysis of bug report, it can
     also be just a waste of space in normal situations, e.g., when a
     test script is successful.  What happens when two or more
     ‘:copy-in-global-log:’ fields are present in the same ‘.trs’ file
     is undefined behaviour.

‘:test-global-result:’
     This is used to declare the "global result" of the script.
     Currently, the value of this field is needed only to be reported
     (more or less verbatim) in the generated global log file
     ‘$(TEST_SUITE_LOG)’, so it’s quite free-form.  For example, a test
     script which run 10 test cases, 6 of which pass and 4 of which are
     skipped, could reasonably have a ‘PASS/SKIP’ value for this field,
     while a test script which run 19 successful tests and one failed
     test could have an ‘ALMOST PASSED’ value.  What happens when two or
     more ‘:test-global-result:’ fields are present in the same ‘.trs’
     file is undefined behaviour.

Let’s see a small example.  Assume a ‘.trs’ file contains the following
lines:

     :test-result: PASS server starts
     :global-log-copy: no
     :test-result: PASS HTTP/1.1 request
     :test-result: FAIL HTTP/1.0 request
     :recheck: yes
     :test-result: SKIP HTTPS request (TLS library wasn't available)
     :test-result: PASS server stops

Then the corresponding test script will be re-run by ‘make check’, will
contribute with _five_ test results to the testsuite summary (three of
these tests being successful, one failed, and one skipped), and the
content of the corresponding ‘.log’ file will _not_ be copied in the
global log file ‘test-suite.log’.