Introduction

(The “Introduction” text is adapted from the Wikipedia article on Unit testing, for a stronger focus on SMASH and C++.)

Unit testing is a software testing method by which individual units of source code are tested to determine if they are fit for use. View a unit as the smallest testable part of an application. Thus, a unit is typically an entire class.

Ideally, each test case is independent from the others. Substitutes such as method stubs and mock objects can be used to assist testing a module in isolation. Unit tests are written and run by software developers to ensure that code meets its design and behaves as intended.

Benefits

The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. A unit test provides a strict, written contract that the piece of code must satisfy. As a result, it affords several benefits.

Finds problems early

Unit testing finds problems early in the development cycle.

In test-driven development, unit tests are created before the code itself is written. When the tests pass, that code is considered complete. The same unit tests are run against that function frequently as the larger code base is developed either as the code is changed or via an automated process with the build. If the unit tests fail, it is considered to be a bug either in the changed code or the tests themselves. The unit tests then allow the location of the fault or failure to be easily traced. Since the unit tests alert the development team of the problem before handing the code off to testers or clients, it is still early in the development process.

Facilitates change

Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (e.g., in regression testing). The procedure is to write test cases for all functions and methods so that whenever a change causes a fault, it can be quickly identified.

Readily available unit tests make it easy for the programmer to check whether a piece of code is still working properly.

Simplifies integration

Unit testing may reduce uncertainty in the units themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts, integration testing becomes much easier.

Design

When software is developed using a test-driven approach, the combination of writing the unit test to specify the interface plus the refactoring activities performed after the test is passing, may take the place of formal design. Each unit test can be seen as a design element specifying classes, methods, and observable behaviour.

SMASH Specific Hints and Rules

The typical unit test in SMASH is centered around one C++ class. But many of the classes in SMASH rely on specific data to do any useful operations. The obvious candidates are

Each of these are interfaces to data that most classes in SMASH read, modify, or create. For example, consider testing smash::DecayAction. The class is created with a const-ref to a smash::ParticleData object. This class in turn requires a smash::ParticleType object for its constructor. To make things worse, the smash::DecayAction::perform function requires a pointer to smash::Particles (which contains a map of all existing smash::ParticleData objects). The perform function further calls smash::Action::choose_channel which requires a std::vector of smash::ProcessBranch to determine the the final state particles.

Compromise

We see that testing DecayAction in isolation will be hard. If we'd want to follow the purist rules for unit testing we'd have to mock all those classes. Up to now we have not used mocking, as it would create even more work when the design of SMASH changes. We should consider mocking on a case by case basis, but feel free to just use the real thing for now.

Instead of creating complete mock classes we can use the BUILD_TESTS macro in actual SMASH classes to easily construct mock objects

See Mock functions and classes.

Good Example

While implementing the initial conditions (see src/test/initial_conditions.cc at the very end), the test to verify momentum conservation was written first and then the code has been adjusted, until the test passed successfully. This can serve as a positive example of test-driven development.

Running tests & Test-driven development

Tests are built with cmake. They can be disabled via the BUILD_TESTING option of cmake; per default tests are enabled.

Once a test is built you will find an executable in the top-level build directory. Every test has a name that is used for the .cc file, the executable name, and the make target names. Take for example the pdgcode test:

The source for the test is in src/tests/pdgcode.cc.
The executable will be in ${CMAKE_BINARY_DIR}/pdgcode.
The make file will have two targets: pdgcode and run_pdgcode. The former will only build the executable, the latter will build and run it.

For test-driven development, the run target can be quite handy, since it requires a single command to compile and run a unit test. A vim user, for example, will be able to call :make run_pdgcode (possibly mapped to a key, such as F10) and get compiler output and test output into the Quickfix buffer.

Finally, if you want to run all tests in the test suite you can first build all tests with

make -j4 all

and then run all tests with

ctest -j4

. Using ctest instead of make test allows you to run tests in parallel with the -j flag. (Use a number that corresponds to the number of cores on the machine where you're working on, instead of -j4.) If you run ctest with the -V flag you will see more output from the tests. The test output is always captured in a file, so that you don't necessarily have to rerun a failed test to see what happened. You can find it in the Testing directory in the build dir.

Get started

In SMASH we use a unit testing framework that was originally developed for the Vc library. It simplifies test creation to the bare minimum. The following code suffices to run a test:

#include "unittest.h"
 
TEST(test_name) {
  int test = 1 + 1;
  COMPARE(test, 2) << "more details";
  VERIFY(1 > 0);
}

This creates one test function (called "test_name"). This function is called without any further code and executes two checks. If, for some reason, the compiler would determine that test needs to have the value 3, then the output would be:

   FAIL: ┍ at /home/mkretz/src/smash/src/tests/testfile.cc:5 (0x40451f):
   FAIL: │ test (3) == 2 (2) -> false more details
   FAIL: ┕ test_name

   Testing done. 0 tests passed. 1 tests failed.

Let's take a look at what this tells us.

The test macro that failed was in testfile.cc in line 5.
If you want to look at the disassembly, the failure was at 0x40451f.
The COMPARE macro compared the expression test against the expression 2. It shows that test had a value of 3 while 2 had a value of 2 (what a surprise). Since the values are not equal test == 2 returns false.
The COMPARE, FUZZY_COMPARE, VERIFY, and FAIL macros can be used as streams. The output will only appear on failure and will be printed right after the normal output of the macro.
Finally the name of the failed test (the name specified inside the TEST() macro) is printed.
At the end of the run, a summary of the test results is shown. This may be important when there are many TEST functions.

If the test passed you'll see:

   PASS: test_name

   Testing done. 1 tests passed. 0 tests failed.

You can compile tests with the smash_add_unittest macro. You only need to pass it the name of the .cc file (without the file extension). So, if your test code above was saved in tests/testfile.cc, then you'd add the line

smash_add_unittest(testfile)

to the CMakeLists.txt . You will then get two new targets that you can build with make: testfile and run_testtest . The latter can be used to build and run a test quickly in "code - compile - test" cycles in test-driven development.

Table of Contents