runtests

(Harness for running TAP-compliant test scripts)

SYNOPSIS

runtests [-hv] [-b builddir] [-s srcdir] test ...

runtests [-hv] [-b builddir] [-s srcdir] -l test-list

runtests [-h] -o [-b builddir] [-s srcdir] test

runtests is a pure C implementation of a harness for test scripts using TAP (the Test Anything Protocol). This is the text-based protocol used by Perl's test suite and implemented in Perl by Test::Harness. runtests is a pure C reimplementation of Test::Harness that supports some additional features useful for C-based software packages that use Automake.

By default, the arguments to runtests are taken as tests to run. The tests to run can instead be listed in a file, one per line, passed to runtests with the -l option. (Blank lines and lines starting with # are ignored.) Either way, each test name should be a path to an executable, possibly with a trailing -t or .t removed. The path should normally be given as relative to the source or build directory; see below for source and build directory handling. runtests will run each test program in the order listed and summarize the results.

If runtests is given the -o option, it instead runs a single test program given on the command line. This is equivalent to running the program directly except that the program will be searched for in the C_TAP_SOURCE and C_TAP_BUILD directories and the environment will be set up as described below.

When runtests is built, if the C preprocessor symbols C_TAP_SOURCE and C_TAP_BUILD are defined, they're taken to be paths to the source and build directory for the test suite (normally a subdirectory of the main source or build directory for the package). These paths can also be set with the -s and -b options, respectively. If these paths are set, via either means, they're exported in the C_TAP_SOURCE and C_TAP_BUILD environment variables to all test programs. SOURCE and BUILD are also set in the environment for backward compatibility, but please switch to C_TAP_SOURCE and C_TAP_BUILD.

If set, the C_TAP_SOURCE and C_TAP_BUILD directories are also used to find test programs. If a test program named in test-list or given on the command line with -o is not found relative to the current directory, it is looked for in the C_TAP_BUILD directory and then in the C_TAP_SOURCE directory (if either is set). This only applies to tests that are not absolute paths.

OPTIONS

-b builddir

Sets the build directory, overriding a C_TAP_BUILD preprocessor directive set at build time. This directory will be put into the C_TAP_BUILD (and BUILD) environment variable, and test programs not found relative to the current working directory of runtests will be searched for relative to this directory next.

-h

Display a usage message and exit, doing nothing else.

-l test-list

Rather than taking the list of tests to run from the command line, read the list of tests from the provided test-list file. Each line of the file should be the name of the test, without any trailing -t or .t. Blank lines and lines starting with # are ignored, as is leading whitespace before the name of the test. Test names must not contain whitespace.

After each test name, a whitespace-separated list of options may be given. The following options are currently supported:

libtool: If running under valgrind (see the valgrind option below), use libtool to invoke valgrind. This avoids running valgrind on the wrapper shell script generated by libtool. If this option is set, the C_TAP_LIBTOOL environment variable must be set to the full path to the libtool program to use to run valgrind and thus the test. Ignored if the test isn't being run under valgrind.
valgrind: Run the test under valgrind if C_TAP_VALGRIND is set. The contents of that environment variable are taken as the valgrind command (with options) to run. The command is parsed with a simple split on whitespace, and no quoting is supported.

-o

Rather than running all tests listed in a test list, run the single test program named on the command line. This program will be searched for using the same algorithm used for test programs listed in a test list and will have the same environment setup, but all of its output will be displayed and the exit status will match its exit status.

-s srcdir

Sets the source directory, overriding a C_TAP_SOURCE preprocessor directive set at build time. This directory will be put into the C_TAP_SOURCE (and SOURCE) environment variable and test programs not found relative to the current working directory of runtests or the C_TAP_BUILD directory will be searched for relative to this directory.

-v

Rather than showing a summary of the results of each test, display the full output of each test as it runs. This can also be requested by setting the C_TAP_VERBOSE environment variable.

TEST PROTOCOL

The canonical documentation for TAP is TAP::Parser::Grammar or, in older versions of Perl, Test::Harness::TAP. This is a brief summary.

Every individual test done by a test script must get a unique number. The output of each test executable should contain:

    1..<last>

where <last> is the number of the last test. 1.. may be omitted. Normally, this line is given first, but runtests allows it to be given anywhere in the test output (in case the number of tests are determined dynamically for some reason).

For each test, the script must output a line to standard output in one of these three formats:

    ok <number>
    not ok <number>
    ok <number> # skip <reason>
    not ok <number> # todo <reason>

where <number> is the test number. The <number> may be followed by an optional comment except in the last two forms. The last two forms both indicate a skipped test. <reason> should be some brief reason for why the test was skipped, but is optional.

As a special case, the first line of the output may be in the form:

    1..0 # skip some reason

which indicates that this entire test case should be skipped and gives a reason.

All other output lines are ignored, although for compliance with the TAP protocol all other output should go to standard error or begin with a # sign.

ENVIRONMENT

BUILD

Set to the value of the C preprocessor symbol C_TAP_BUILD when runtests was built or the -b option, if either was set. Use C_TAP_BUILD instead of this environment variable.

C_TAP_BUILD

Set to the value of the C preprocessor symbol C_TAP_BUILD when runtests was built or the -b option, if either was set.

C_TAP_LIBTOOL

If set, runtests will use this as the full path to the libtool program to use to run tests under valgrind when both the valgrind and libtool options are set for that test in a test list. This avoids running valgrind on the wrapper shell script generated by libtool.

C_TAP_SOURCE

Set to the value of the C preprocessor symbol C_TAP_SOURCE when runtests was built or the -s option, if either was set.

C_TAP_VALGRIND

If set, runtests will run supported tests under valgrind using the command (with options) in this environment variable. Supported tests are those with the valgrind option set in a test list file. (valgrind is not supported with tests run from the runtests command line.)

The value of this environment variable must be the full command to run valgrind, starting with the full path to the valgrind binary and then any space-separated valgrind options to use. Spaces and quoting in the options are not supported; each space-separated "word" will be passed to valgrind as a separate option.

Also see C_TAP_LIBTOOL for packages that use libtool to build binaries.

C_TAP_VERBOSE

If set, runtests will act as if the -v option was given and show the complete output of each test, instead of only the summary of total and failing tests. This can be useful when debugging a test failure, since it can be set in the environment of the build without changing anything else about the build.

SOURCE

Set to the value of the C preprocessor symbol C_TAP_SOURCE when runtests was built or the -s option, if either was set. Use C_TAP_SOURCE instead of this environment variable.

NOTES

runtests does not abort the entire test suite when one of the tests bails out (via the bail() C API or the equivalent Perl BAIL_OUT call). It only aborts the rest of that test program and then moves to the next one. This is contrary to the Perl Test::More documentation, but the Perl prove behavior seems less useful. Separate test cases should be independent, so the harness should be able to continue with other testing despite one test program failing.

While not required, arranging for the test programs to end in -t (preferred since it doesn't cause confusion with extensions) or .t is recommended because it keeps the test executable name distinct from any executables that are part of the main purpose of the package. This will also allow a future version of runtests to find all test programs by recursively searching a directory.

Consider using prove(1) instead of this test suite harness if you're testing Perl packages. It has more features for handling Perl test scripts. It does not, however, have the C_TAP_BUILD and C_TAP_SOURCE search and environment handling, and it assumes all test cases are written in Perl unless run with special options.

TAP version directives in the TAP output are currently just ignored. These are part of the official grammar for future compatibility, but the version has not yet been used for anything, so ignoring them seems to be the best approach for now.

BUGS

The following features of TAP are not yet implemented by this harness:

The text after # skip should be reported as the reason for the skipped test.

AUTHOR

Russ Allbery <eagle@eyrie.org>

COPYRIGHT AND LICENSE

Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.

SPDX-License-Identifier: FSFAP