< C TAP Harness README Russ Allbery > Software > C TAP Harness Writing TAP Tests >

User-Visible C TAP Harness Changes

C TAP Harness 4.8 (2022-05-08)

Add missing rm for an old bstrndup symlink, which would cause make install to fail if C TAP Harness had previously been installed. Thanks, Horst H. von Brand. (Fixes #9)

Mark allocation functions in C support libraries with their corresponding deallocation functions so that GCC 11 and later can diagnose memory deallocation bugs.

Update to rra-c-util 10.2:

C TAP Harness 4.7 (2020-05-16)

Allow the package test suite to be run with C_TAP_VERBOSE without breaking test results via inheritance of that setting.

Fix warnings with GCC 10.

C TAP Harness 4.6 (2020-01-07)

Reformat all C source using clang-format 10 and the formatting rules specified in .clang-format.

Update to rra-c-util 8.1:

C TAP Harness 4.5 (2019-08-31)

Add new bcalloc_type and breallocarray_type macros that take a type instead of a size as their third argument and cast the return value to a pointer to that type. These are more friendly to C++ code than C-style allocation functions returning void *.

The test harness and libtap C library can now alternatively be compiled with a C++ compiler, making it easier to incorporate them into a C++ project. Thanks to Peter Paris for the initial report.

C TAP Harness 4.4 (2018-12-25)

Add test_cleanup_register_with_data to the C TAP library. This is the same as test_cleanup_register except takes a generic pointer, which is then passed to the cleanup function as a third argument. This should have been the only API from the beginning, but test_cleanup_register is preserved for backward compatibility. Patch from Marc Dionne.

Fix error checking for malloc failure in bstrndup function (caught by cppcheck).

Update to rra-c-util 8.0:

C TAP Harness 4.3 (2018-05-06)

Add support for valgrind and libtool test options in test lists. Test lists now take a space-separated set of options after the test name. If the valgrind option is present and C_TAP_VALGRIND is set in the environment, the test will be run by passing it as an option to the command given in C_TAP_VALGRIND. If the libtool option is also set, valgrind will be run via the libtool script set in C_TAP_LIBTOOL, using --mode=execute, so that valgrind will run on the underlying binary and not the libtool shell wrapper.

On test failures, report the values as left and right instead of wanted and seen. This idea is stolen from the Rust assert framework. It avoids having to care about the order in which values are passed into the test functions.

Fix is_string comparisons involving NULL pointers so that the string "(null)" will no longer compare equal to NULL (although the diagnostic output on test failure is still mildly confusing).

Add new check-cppcheck target that runs cppcheck on all source code, and fix one unnecessary check for NULL that it uncovered.

Add SPDX-License-Identifier headers to all substantial source files, and add a test to check for them. This imports more supporting test machinery whose canonical home is in rra-c-util. If you want to use files in tests/tap/perl, copy them from rra-c-util instead of this package.

C TAP Harness now imports the Perl test modules from rra-c-util to support some checks, so tests/docs/pod-spelling-t and tests/docs/pod-t have been updated to the versions from rra-c-util. Projects that were previously copying those tests should be able to continue to use them, but will now need the modules in tests/tap/perl (the canonical version of which are maintained in rra-c-util).

C TAP Harness 4.2 (2017-12-30)

Add is_blob function to the C test library. This tests whether two regions of memory are identical, similar to ok(memcmp(...)) but reporting where the regions differ. Patch from Daniel Collins.

Avoid zero-length realloc allocations in breallocarray.

Fix new fall-through warning in GCC 7.

Switch to the compiler warning Autoconf macros and warning set from rra-c-util with the addition of -ansi -pedantic for GCC and -pedantic-errors for Clang. Add some casts to fix warnings from -Wconversion, and suppress some spurious warnings from Clang about tests/tap/float.c.

C TAP Harness 4.1 (2016-12-23)

Add is_bool function to the C test library. This compares its two arguments only for their truthfulness. is_bool(true, arg) is the same as ok(arg), but there are times (such as when testing for a false value) where this allows for clearer code or clearer output.

C TAP Harness 4.0 (2016-05-07)

When building runtests, one must now set the C_TAP_SOURCE and C_TAP_BUILD C preprocessor symbols to the source and build directories, instead of SOURCE and BUILD. An updated Makefile.am recipe is documented in README.

runtests now sets C_TAP_SOURCE and C_TAP_BUILD in the environment in addition to SOURCE and BUILD. All test programs using this harness should switch to the new C_TAP_SOURCE and C_TAP_BUILD environment variables. SOURCE and BUILD are very generic and may conflict with other programs and uses, and setting them will be removed in a later version.

The TAP test libraries (C and shell) now use C_TAP_SOURCE and C_TAP_BUILD environment variables instead of SOURCE and BUILD for the test_file_path() and test_tmpdir() functions. If you were using these libraries with another test harness, you will need to set the new environment variables.

Fix missing va_end() call in is_double(), which would have caused compilation failures or other problems on some platforms. Thanks, Julien ÉLIE.

Rename the script to bootstrap from a Git checkout to bootstrap, matching the emerging consensus in the Autoconf world.

C TAP Harness 3.4 (2015-08-18)

Fix segfault in runtests when given a test list containing only comments and blank lines. Thanks, aherbert.

C TAP Harness 3.3 (2015-04-26)

If runtests is given the -v option, or if the environment variable C_TAP_VERBOSE is set, the complete output of each test program will be shown instead of the summary of total and failing tests. Based on work by D. Brashear.

C TAP Harness now compiles cleanly with Clang with -Weverything -Wno-padded -pedantic-errors, and automatically detects Clang and switches warning flags for make warnings.

C TAP Harness 3.2 (2014-12-25)

The runtests harness now supports ignoring comments and blank lines in the test list specified with -l. Leading whitespace before the test name is also ignored.

C TAP Harness 3.1 (2014-07-02)

ok, okv, and all is_* functions now return true if the test succeeds and false if it fails, matching the return status of the corresponding Perl Test::More functions. This allows more succinct code when the actions of a test program should vary based on the success or failure of previous tests. Based on a patch by Peter Pöschl.

diag and sysdiag now always return 1, making it easier to insert calls into compound statements when debugging tests. Based on a patch by Peter Pöschl.

Add new breallocarray API that does the same as realloc but takes calloc-style arguments to specify the size and checks internally for integer overflow. Inspired by the OpenBSD reallocarray function.

Check for integer overflows on memory allocation. All the possible issues for this code are rather theoretical, but one may as well strive for correctness.

Replace all uses of sprintf with a simpler string concatenation function that checks for allocation overflow. (The standards assumptions for this package don't permit assuming asprintf or a sufficiently non-broken snprintf to simulate asprintf.)

C TAP Harness 3.0 (2014-01-28)

The test_cleanup_register API has changed in this release. Cleanup functions must now take two parameters, not one, and are called from all test processes, not just the primary one. The new second argument indicates whether the cleanup function was called in the primary process (the one in which plan or plan_lazy was called). External resources, such as files, should generally only be freed when the cleanup function is called in the primary process, but tests may want to free internal resources, like memory, in all processes to ease analysis with tools like valgrind.

When running test programs from a list, reopen standard input for each program to /dev/null, and be more careful about closing all duplicates of file descriptors left behind after setting up standard output and standard error so that extraneous file descriptors aren't leaked to the child process.

C TAP Harness 2.4 (2013-12-25)

Add new diag_file_add and diag_file_remove APIs to the basic C TAP library. These functions manage a list of registered file that contains supplemental diagnostic information. Each registered file is checked before each output function for any new lines, and any lines are displayed as if they'd been passed to diag(). This can be useful if, for example, the test involves a background daemon whose output can be logged to a disk file.

C TAP Harness 2.3 (2013-11-13)

Add new test_cleanup_register API to the basic C TAP library. This registers a C callback function that's called during exit from the test and passed in a boolean argument indicating whether the test succeeded or failed.

Suppress lazy plans and the summary of tests at the end of a test program if the program aborted with bail or sysbail.

Add warn_unused_result gcc attributes to the C TAP library functions where ignoring the return value is almost certainly a bug (such as all the malloc functions).

Add portability for warn_unsed_result attributes to tap/macros.h.

C TAP Harness 2.2 (2013-08-14)

bail and sysbail now exit with status 255 to match the behavior of BAIL_OUT in Perl's Test::More.

Document that runtests handling of test cases that bail out differs from the documented behavior of BAIL_OUT in Perl's Test::More and the behavior of prove, and document why.

C TAP Harness 2.1 (2013-03-15)

When locating test programs, try a suffix (-t, .t, or no suffix) with all bases before moving on to the next suffix. The behavior in the previous release was not backward-compatible: it would find the unsuffixed helper program in the build directory instead of the actual test in the source directory for some rra-c-util tests when the build directory and the source directory weren't the same.

Document that TAP version directives in the TAP output are ignored.

C TAP Harness 2.0 (2013-03-14)

The default behavior of tests/runtests has changed to make it act more like other test drivers. Now, to provide a file containing a list of tests, use the -l option. Existing users should add -l to the command line in front of the test list. Otherwise, the command-line arguments are taken as tests to run, as if they were all listed in a test list file.

runtests now tries the test name without any extension if the test name with either -t or .t was not found. It also does not consider a directory to be a valid test when searching for the executable, even if the directory is executable.

Flush the harness output after each test. This effectively implements line buffering even when standard output is not a terminal and therefore output isn't flushed after each line of test output.

When displaying partial status of tests with a deferred plan, show the total number of tests as ? rather than the number of tests seen so far. This matches the behavior of Perl's prove utility.

More correctly handle backspacing over the test count in several abort cases when reporting status to a terminal.

Add GCC annotations to some internal functions in runtests to help clang --analyze better understand code correctness. Remove a dead store caught by clang --analyze.

C TAP Harness 1.12 (2012-05-11)

Fix additional uses of local in the shell TAP library for portability to Solaris /bin/sh and document a bad interaction between backquotes and double quotes for the puts, test_file_path, and test_tmpdir functions.

In the C TAP library, if bail is called after plan_lazy but before any tests are run, don't output the plan or the summary of test results.

Add the UNUSED macro (to mark possibly-usused variables for gcc) to tests/tap/macros.h for use in the C TAP library.

Make the __attribute__ handling in tests/tap/macros.h more aware of different versions and compilers. All attributes are now suppressed for GCC older than 2.96 (instead of 2.7), since __malloc__ became available then. __alloc_size__ is suppressed for GCC older than 4.3. Warnings about unknown diagnostics are suppressed for LLVM and Clang, which pretend to be GCC but don't support all the same attributes.

C TAP Harness 1.11 (2012-04-25)

Reorganize the source package so that runtests.c is in the tests directory and the C and shell TAP libraries are in tests/tap. Change the include directives in the C TAP library source files accordingly. This is somewhat confusing for the standalone package, but means that the files are located in the same locations here as they would be in packages into which they're copied, which simplifies instructions and the build system. This also removes the need to build the C TAP library with -I flags at the cost of requiring that it be located in a tests subdirectory, allowing easier use of additional supporting headers without requiring that all tests be built with a -I directive.

Do not attempt to use feature-test macros unless compiled with gcc's -ansi flag or PEDANTIC is defined. Defining _XOPEN_SOURCE 600 on Solaris 10, which is required for isinf() if feature-test macros are in use, causes compilation to fail unless the compiler is run in C99 mode due to an ABI change. The basic TAP library doesn't care about this distinction and doesn't want to force specific compiler flags, so avoiding feature-test macros is a better fix. Thanks to Bob Friesenhahn for the bug report.

Do not use local in the shell TAP library. Solaris /bin/sh doesn't support it, and the shell library is otherwise portable to that shell. Instead, use a prefix of tap_ on all non-public variables.

Move the __attribute__ and BEGIN_DECLS/END_DECLS macros from tests/tap/basic.h to a new tests/tap/macros.h header file so that they can be more easily reused by any other TAP add-on. Be sure to copy tests/tap/macros.h into your source package and add it to the source files for the libtap library when upgrading.

Move the is_double C TAP test function into a separate source file. Including this function may require linking with libm on some platforms, which is undesirable if the package otherwise doesn't use math functions. The new tests/tap/float.c and tests/tap/float.h files need only be included in a package that wants to do floating point tests. Users of is_double will now need to include tests/tap/float.h in addition to tests/tap/basic.h (and link all clients of the libtap library with -lm).

C TAP Harness 1.10 (2012-02-29)

Add bstrndup to the basic C TAP library. This is akin to strndup (copying at most n characters of the string), but calls sysbail if memory allocation fails.

C TAP Harness 1.9 (2011-12-24)

Add test_tmpdir and test_tmpdir_free functions to the basic C TAP library and a test_tmpdir function to the shell TAP library. These functions create a temporary directory under the build tests directory for tests to use for transient files.

runtests now frees all allocated resources on exit so that it can be used to run test cases with valgrind memory allocation checking without adding false positives.

C TAP Harness 1.8 (2011-09-02)

Add bmalloc, bcalloc, brealloc, and bstrdup functions to the basic C TAP library. These functions call sysbail if memory allocation fails.

If BUILD were not set when building runtests and the -b option was not present, SOURCE and the -s option would be ignored when searching for tests. Fix runtests to still honor SOURCE and -s when searching for tests even in that case.

C TAP Harness 1.7 (2011-04-28)

Add a more complete usage message to runtests and add support for a -h command-line flag to display the usage message.

C TAP Harness 1.6 (2010-12-29)

is_double() now takes a third argument, an epsilon. Two numbers are considered equal if their absolute difference is less than epsilon. is_double() also now treats wanted and seen values of NaN (not a number) as equal. Thanks to PICCA Frédéric-Emmanuel for the proposed changes.

The ok_program function in the shell libtap.sh library no longer strips text after a colon and a space from the program output if the expected status is non-zero. Instead, if program output may contain system-specific error messages after a colon and a space, put the new function strip_colon_error before the program to do this stripping. Thanks to Carsten Hey for the idea.

strip_colon_error is now smarter about preserving an initial word ending in a colon (which is generally the program name) while still stripping error messages later in the line.

The test_file_path function in the shell libtap.sh library now always returns the empty string, rather than possible absolute paths starting at /, if $BUILD and $SOURCE are not set.

Flush standard error in the C TAP library before printing results for more deterministic output. Thanks to Carsten Hey for the idea.

All of C TAP Harness now compiles with gcc -ansi -pedantic and should be fully C89-compatible. Note that either C99 or SUSv3 is required to build C TAP Harness. (This should not be a problem on any modern platform.) Based on work by Carsten Hey.

Simplify and improve output formatting in the summary of failing tests in some edge cases.

Add explicit license statements to the files meant to be copied into other packages rather than referring to LICENSE.

C TAP Harness 1.5 (2010-08-25)

In ok_program in the shell TAP library, strip error output after the second colon, not after the first. This code helps comparing error output that includes system error messages, but needs to skip past the first colon, since it separates the program name from the error message.

runtests.c now builds without warnings with gcc -Wswitch-enum.

C TAP Harness 1.4 (2010-07-07)

Add a test_file_path() function to the basic C and shell TAP libraries, which searches the build and source directories for a particular file and returns the full path. This is a utility function that can be used to find test data files.

C TAP Harness 1.3 (2010-06-09)

Add API documentation for the functions provided by the basic TAP C library. The documentation is in POD and *roff format in docs/api and installed as man pages in the (generally unused) case of make install.

Add docs/writing-tests, which provides some basic instructions for how to write TAP tests in Perl, C, and shell using the facilities provided by C TAP Harness. This document is suitable for inclusion in packages that use C TAP Harness as instructions for contributors.

Add an okv() function to the C TAP library. This is the same as ok() but takes the test description as a va_list to make it easier to reuse the reporting infrastructure in other tests.

C TAP Harness 1.2 (2010-04-13)

Support giving the plan at any point in the test output rather than requiring it be first. The TAP protocol permits giving the plan last instead of first, but since it's just as easy to support providing the plan in the middle of the test output, allow that as well.

Add support for lazy planning to the basic TAP library. Instead of calling plan at the start of the test, call plan_lazy instead, which doesn't require specifying a test count. Then, the plan will automatically be printed based on the last test number at the end of the program. This is not the recommended approach for writing tests since, should the test program exit in the middle for some reason, everything will appear to succeed. But it can save time in writing tests quickly and allows for dynamic numbers of tests.

Add diag and sysdiag functions to the basic TAP library, which output diagnostics to standard error preceded by the protocol-required # character.

Clean up data types in the basic C TAP library. Test numbers and counts are now unsigned long, and ok_int compares longs.

Add the GCC nonnull attribute to the bail functions in the TAP library.

C TAP Harness 1.1 (2009-08-15)

Set an exit handler in libtap that prints a summary of the test results at the end of a test script, similar to Perl's Test::More.

When looking for 0..1 # skip plan lines indicating the whole test case should be skipped, correctly compare "skip" case-insensitively. This fixes interoperability with Perl's Test::More.

Consume all output from the test case before closing its file descriptor, even if the case is aborted due to bailing out or sending an invalid test number. This prevents tests from spuriously dying with SIGPIPE.

Enable Automake silent rules. For a quieter build, pass the --enable-silent-rules option to configure or build with make V=0.

C TAP Harness 1.0 (2009-05-21)

Initial public release based on the version of runtests.c and the libtest library that shipped with remctl 2.13. The changes below are relative to that version.

Set the SOURCE and BUILD environment variables when running tests to the roots of the source and build test directories if set at compile time or via the -s and -b command-line options to runtests.

Search for test programs first in the current directory and then in the build and source directories, in that order, as specified by SOURCE and BUILD or by the -s and -b options. This allows some tests to be compiled and some shipped with the package and merged together at run time by runtests, even if the build directory is not the source directory.

Support running a single test program with the -o command-line option. This is mostly interesting to take advantage of the environment variables and test searching algorithm described above.

Add support for 0..1 # skip <reason> plan lines, indicating that the entire test case should be skipped.

Add support for aborting a test case with "Bail out!" and reporting the reason given on that line.

Add support for omitting the test number from an "ok" or "not ok" line, in which case the number is assumed to be one more than the previous number.

Add support for todo tests, which are recorded as skipped if failing and failed if they succeed unexpectedly.

Omitting the space after "ok" or "not ok" line is now supported, and a test number of 0 now results in an error.

Recognize skip directives regardless of case.

Change the reporting of reasons to abort the test case to use ABORTED and the abort reason in parentheses for more consistent output.

Rename test_init() to plan() and ok_int(), ok_double(), and ok_string() to is_int(), is_double(), and is_string() in the C TAP library to more closely match the function names in Perl's test modules.

Add is_hex() C TAP library function to compare two unsigned longs and print mismatches in hex.

Change all C library functions to take a test description (printf-style) and to no longer take the test number. Instead, keep a static count of the current test number.

Add bail() and sysbail() C TAP library functions to bail out with the right magic output string.

Add skip_all() C TAP library function to skip all tests and exit.

Stop exporting the printcount interface in the shell version of the TAP library. Instead, provide plan, skip_all, ok, skip, ok_block, skip_block, ok_program, and bail shell functions, similar to the C library interface, with the ok* functions taking a test description as the first argument.

Show mismatch output from the is_*() functions before the "not ok" line instead of after it.

Add the initial "1.." to plan() output for better TAP compliance.

Support either "-t" or ".t" as the ending of test cases.

Remove the dependency on both config.h and the general portability framework and instead include the necessary C headers directly. This requires ANSI C.

Remove the functions from libtap that assume libutil from rra-c-util. Those library functions will be provided by that package.

< C TAP Harness README Russ Allbery > Software > C TAP Harness Writing TAP Tests >