(automake-1.16.info)Overview of Custom Test Drivers Support

---

Next: Declaring Custom Test Drivers Up: Custom Test Drivers

---

Enter node , (file) or (file)node

15.3.1 Overview of Custom Test Drivers Support
----------------------------------------------

Starting from Automake version 1.12, the parallel test harness allows
the package authors to use third-party custom test drivers, in case the
default ones are inadequate for their purposes, or do not support their
testing protocol of choice.

   A custom test driver is expected to properly run the test programs
passed to it (including the command-line arguments passed to those
programs, if any), to analyze their execution and outcome, to create the
‘.log’ and ‘.trs’ files associated to these test runs, and to display
the test results on the console.  It is responsibility of the author of
the test driver to ensure that it implements all the above steps
meaningfully and correctly; Automake isn’t and can’t be of any help
here.  On the other hand, the Automake-provided code for testsuite
summary generation offers support for test drivers allowing several test
results per test script, if they take care to register such results
properly (Note: Log files generation and test results recording).

   The exact details of how test scripts’ results are to be determined
and analyzed is left to the individual drivers.  Some drivers might only
consider the test script exit status (this is done for example by the
default test driver used by the parallel test harness, described in the
previous section).  Other drivers might implement more complex and
advanced test protocols, which might require them to parse and
interpreter the output emitted by the test script they’re running
(examples of such protocols are TAP and SubUnit).

   It’s very important to note that, even when using custom test
drivers, most of the infrastructure described in the previous section
about the parallel harness remains in place; this includes:

   • list of test scripts defined in ‘TESTS’, and overridable at runtime
     through the redefinition of ‘TESTS’ or ‘TEST_LOGS’;
   • concurrency through the use of ‘make’’s option ‘-j’;
   • per-test ‘.log’ and ‘.trs’ files, and generation of a summary
     ‘.log’ file from them;
   • ‘recheck’ target, ‘RECHECK_LOGS’ variable, and lazy reruns of
     tests;
   • inter-test dependencies;
   • support for ‘check_*’ variables (‘check_PROGRAMS’,
     ‘check_LIBRARIES’, ...);
   • use of ‘VERBOSE’ environment variable to get verbose output on
     testsuite failures;
   • definition and honoring of ‘TESTS_ENVIRONMENT’,
     ‘AM_TESTS_ENVIRONMENT’ and ‘AM_TESTS_FD_REDIRECT’ variables;
   • definition of generic and extension-specific ‘LOG_COMPILER’ and
     ‘LOG_FLAGS’ variables.

On the other hand, the exact semantics of how (and if) testsuite output
colorization, ‘XFAIL_TESTS’, and hard errors are supported and handled
is left to the individual test drivers.