New tool added - API Sanity AutoTest

API Sanity AutoTest (ASAT) is an automatic generator of basic unit tests for shared C/C++ libraries. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files. The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code and program hanging. It may be considered as a tool for out-of-the-box low-cost sanity checking (fuzzing) of the library API or as a test development framework for initial generation of templates for advanced tests. Also it supports universal T2C format of tests, random test generation mode, specialized data types and other useful features.

  See also: Upstream Tracker for C/C++ libraries.

  We are looking for sponsorship to continue development of this tool. Please send your proposals to api-sanity-autotest@linuxtesting.org.

if (window.showTocToggle) { var tocShowText = "show"; var tocHideText = "hide"; showTocToggle(); }

The latest release can be downloaded from this page.

This program is free software. You may use, redistribute and/or modify it under the terms of either the GNU GPL or LGPL.

GNU/Linux, FreeBSD, Haiku (BeOS), Mac OS X, MS Windows (Xp, Vista, 7).

Linux: GCC (3.0-4.5.2, recommended 4.0 or newer) binutils (c++filt, readelf, objdump) Perl (base)

WARNING: if you are using ccache program (i.e. gcc points to /usr/lib/ccache/gcc) then it should be newer than 3.1.2 or disabled.

MS Windows: MinGW (gcc.exe, c++filt.exe) Active Perl MS Visual C++ (dumpbin.exe, undname.exe, cl.exe) add gcc.exe path (C:\MinGW\bin\) to your system PATH variable run vsvars32.bat script (C:\Microsoft Visual Studio 9.0\Common7\Tools\) Library Version Number of Tests Problems Found Crash (segfault, signal SEGV) Abort (signal ABRT) All emitted signals: FPE, BUS, ILL and others Non-zero exit code Program hanging Requirement failure (if specified)

For generating, building and running tests you should provide the XML descriptor for your library version. It is a simple XML-file that specifies version number, paths to header files and shared libraries and optionally some other information. An example of the descriptor is the following (0.3.4.xml):

0.3.4 /usr/local/libssh/0.3.4/include/ /usr/local/libssh/0.3.4/lib/

Command for generating a test suite:
 perl api-sanity-autotest.pl -l -d -gen 

You can view generated tests using the index file:
 tests///view_tests.html 
or manually in the directory:
 tests///groups/ 

Command for building tests:
 perl api-sanity-autotest.pl -l -d -build 

Command for running tests:
 perl api-sanity-autotest.pl -l -d -run 

After a time will be generated test report:
 test_results///test_results.html 

To improve generated tests quality, you can provide the collection of specialized types for the library.

The tool has many useful options for manipulating with the test generation and execution processes. See the list of all options on this page.

Generate, build and execute the tests for the libssh library:
 perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -build -run 

Tests will be generated to:
 tests/libssh/0.3.4/groups/ 
 tests/libssh/0.3.4/view_tests.html 

The report will be generated to:
 test_results/libssh/0.3.4/test_results.html 

Generate tests in the Template2Code (T2C) format:
 perl api-sanity-autotest.pl -l libssh -d 0.3.4.xml -gen -t2c 

The package with T2C tests will be generated to:
 tests_t2c/libssh/0.3.4/t2c-libssh-tests-0.3.4.tar.gz 

Execute the tests using the Xvfb server:
 perl api-sanity-autotest.pl -l allegro -d 4.9.21.xml -run -xvfb 

Generate tests using the Splint specifications (annotations) in the headers:
 perl api-sanity-autotest.pl -l popt -d 1.16.xml -gen -splint-specs 

Trinity - The Linux system call fuzzer, iknowthis - A system call fuzzer for UNIX-like systems,

The basic idea of the test data generation algorithm is to recursively initialize parameters of a function using the values returned (or returned through the out-parameter) by other functions for structured data types (class, struct, union) or by some simple values for intrinsic data types (int, float, enum, ...). The recursion step includes the heuristic selection of the appropriate function, that should be called to initialize complex parameters for other functions. If some parameter of a function cannot be initialized, then the algorithm tries to select other function.

Let's see the example test for FT_Activate_Size ( FT_Size size ) function from the FreeType2 library:

#include int main(int argc, char *argv[]) { FT_Library alibrary = 0; FT_Init_FreeType(&alibrary);//initialize "alibrary" FT_Face face = 0; FT_New_Face( alibrary, "sample.ttf", 0, &face);//initialize "face" FT_Size size = 0; FT_New_Size(face, &size);//initialize "size" FT_Activate_Size(size);//target call return 0; }

In this test case the parameter "size" of target function FT_Activate_Size is initialized through the call of FT_New_Size function using its 2nd out-parameter. The first parameter "face" of FT_New_Size function is recursively initialized by the use of FT_New_Face function's 4th out-parameter. And finally the first parameter "alibrary" of FT_New_Face is initialized by the call of FT_Init_FreeType function on the 3rd recursion step. Other parameters of FT_New_Face function are initialized by intrinsic values.

Please send your bug reports, feature requests and questions directly to api-sanity-autotest@linuxtesting.org or post to the issue tracker at forge.ispras.ru

LSB 4.0 certification test suites for Qt3 (9792 interfaces tested), Qt4 (10803 interfaces tested) and libxml2 (1284 interfaces tested) libraries were developed with the help of this tool, also known as "AZOV Framework" in the past (2007-2009).

Add CUnit format support. See 2011 GSoC LSB projects. Extending support of Splint and ACSL annotations. Making better distinction between pointer arguments (*x) and arrays (x[]). More kinds of generated test data files (images, fonts, ...). Option for generating bound values for parameters.

The tool was developed by the Russian Linux Verification Center at ISPRAS. Andrey Ponomarenko is the leader of this project.

We would like to thank everyone who has contributed to the success of this project!

wikipedia.org, “Sanity testing in software development” wikipedia.org, “Fuzz testing” R. S. Zybin , V. V. Kuliamin , A. V. Ponomarenko , V. V. Rubanov and E. S. Chernov, “Automation of broad sanity test generation”