Contents[Hide]

1.  Building Cgreen Test Suites

Cgreen is a tool for building unit tests in the C language. These are usually written alongside the production code by the programmer to prevent bugs. Even though the test suites are created by software developers, they are intended to be human readable C code, as part of their function is an executable specification. Used in this way, the test harness delivers constant quality assurance.

In other words you’ll get less bugs.

1.1.  Writing basic tests

Cgreen tests are simply C functions with no parameters and no return value. To signal that they actually are tests we mark them with the Ensure macro. An example might be…

Ensure(strlen_of_hello_is_five) {
    assert_that(strlen("Hello"), is_equal_to(5));
}

The test name can be anything you want as long as it fullfills the rules for an identifier in C.

The assert_that() call is the primary part of an assertion, which is complemented with a constraint, in this case is_equal_to(). This makes a very fluent interface to the asserts, that actually reads like english.

Assertions send messages to Cgreen, which in turn outputs the results.

Here are the standard constraints…

Constraint Passes if actual value/expression…
is_true evaluates to true
is_false evaluates to false
is_equal_to(value) == value
is_not_equal_to(value) != value
is_greater_than(value) > value
is_less_than(value) < value
is_equal_to_contents_of(pointer, size) matches the data pointed to by pointer to a size of size bytes
is_not_equal_to_contents_of(pointer, size) does not match the data pointed to by pointer to a size of size bytes
is_equal_to_string(value) are equal when compared using strcmp()
is_not_equal_to_string(value) are not equal when compared using strcmp()
contains_string(value) contains value when evaluated using strstr()
does_contain_string(value) does not contain value when evaluated using strstr()
begins_with_string(value) starts with the string value
is_equal_to_double(value) are equal to value within the number of significant digits (you can set significant_figures_for_assert_double_are(int figures))
is_not_equal_to_double(value) are not equal to value within the number of significant digits

The boolean assertion macros accept an int value. The equality assertions accept anything that can be cast to intptr_t and simply perform an == operation. The string comparisons are slightly different in that they use the <string.h> library function strcmp(). If is_equal_to() is used on char * pointers then the pointers have to point at the same string to pass.

A cautionary note about the constraints is that you cannot use C/C++ string literal concatenation (like "don’t" "use" "string" "concatenation") in the parameters to the constraints. If you do, you will get weird error messages about missing arguments to the constraint macros. This is caused by the macros using argument strings to produce nice failure messages.

1.2.  Legacy style assertions

Cgreen have been around for a while, developed and matured. There is another style of assertions that was the initial version, a style that we now call the legacy style. If you are not interested in historical artifacts, I recommend that you skip this section.

But for completness, here are the legacy style assertion macros:

Assertion Description
assert_true(boolean) Passes if boolean evaluates true
assert_false(boolean) Fails if boolean evaluates true
assert_equal(first, second) Passes if first == second
assert_not_equal(first, second) Passes if first != second
assert_string_equal(char *, char *) Uses strcmp() and passes if the strings are equal
assert_string_not_equal(char *, char *) Uses strcmp() and fails if the strings are equal

Each assertion has a default message comparing the two values. If you want to substitute your own failure messages, then you must use the *_with_message() counterparts…

Assertion
assert_true_with_message(boolean, message, …)
assert_false_with_message(boolean, message, …)
assert_equal_with_message(tried, expected, message, …)
assert_not_equal_with_message(tried, unexpected, message, …)
assert_string_equal_with_message(char *, char *, message, …)
assert_string_not_equal_with_message(char *, char *, message, …)

All these assertions have an additional char * message parameter, which is the message you wished to display on failure. If this is set to NULL, then the default message is shown instead. The most useful assertion from this group is assert_true_with_message() as you can use that to create your own assertion functions with your own messages.

Actually the assertion macros have variable argument lists. The failure message acts like the template in printf(). We could change the test above to be…

Ensure(strlen_of_hello_is_five) {
    const char *greeting = "Hello";
    int length = strlen(greeting);
    assert_equal_with_message(length, 5, "[%s] should be 5, but was %d", greeting, length);
}

A slightly more user friendly message when things go wrong. But, actually, Cgreens default messages are so good that you are encouraged to skip the legacy style and go for the more modern constaints style assertions.

1.3.  A runner

For the tests to actually be run there needs to be a running test suite. (But see also Automatic Test Discovery.) We can create one especially for this test like so…

TestSuite *our_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_is_five);
    return suite;
}

In case you have spotted that strlen_of_hello_is_five() should have an ampersand in front of it, add_test() is actually a macro. The & is added automatically. Further more, the Ensure-macro actually mangles the tests name, so it is not actually a function name. (This also might make them a bit difficult to find in the debugger….)

To run the test suite, we call run_test_suite() on it. This function cleans up the test suite after running it, so we can just write…

run_test_suite(our_tests(), create_text_reporter());

The results of assertions are ultimately delivered as passes and failures to a collection of callbacks defined in a TestReporter structure. The only predefined TestReporter in Cgreen is the TextReporter that delivers messages in plain text like we have already seen.

The complete test code now looks like…

#include "cgreen/cgreen.h"
#include <string.h>

Ensure(strlen_of_hello_is_five) {
    assert_that(strlen("Hello"), is_equal_to(5));
}

TestSuite *our_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_is_five);
    return suite;
}

int main(int argc, char **argv) {
    return run_test_suite(our_tests(), create_text_reporter());
}

The return value of run_test_suite() is a Unix exit code.

Compiling and running gives…

gcc -c strlen_test.c
gcc strlen_test.o -lcgreen -o strlen_test
./strlen_test
Running "our_tests" (1 tests)...
Completed "our_tests": 1 pass, 0 failures, 0 exceptions.

The test messages are only shown on failure. If we break our test to see it…

Ensure(strlen_of_hello_is_five) {
    assert_that(strlen("Hiya", is_equal_to(5));
}

…we’ll get the helpful message…

Running "our_tests" (1 tests)...
strlen_test.c:5: Failure: -> strlen_of_hello_is_five
        Expected [strlen("Hiya")] to [equal] [5]
                actual value:   [4]
                expected value: [5]
Completed "our_tests": 0 passes, 1 failure, 0 exceptions.

Cgreen starts every message with the location of the test failure so that the usual error message identifying tools (like emacs next-error) will work out of the box.

Once we have a basic test scaffold up, it’s pretty easy to add more tests. Adding a test of strlen() with an empty string for example…

...
Ensure(strlen_of_empty_string_is_zero) {
    assert_equal(strlen("\0"), 0);
}

TestSuite *our_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_is_five);
    add_test(suite, strlen_of_empty_string_is_zero);
    return suite;
}
...

And so on.

1.4.  Set up and Tear down

It’s common for test suites to have a lot of duplicate code, especially when setting up similar tests. Take this database code for example…

#include "cgreen/cgreen.h"
#include <stdlib.h>
#include <mysql/mysql.h>
#include "person.h"

static void create_schema() {
    MYSQL *connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
    mysql_query(connection, "create table people (name, varchar(255) unique)");
    mysql_close(connection);
}

static void drop_schema() {
    MYSQL *connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
    mysql_query(connection, "drop table people");
    mysql_close(connection);
}

Ensure(can_add_person_to_database) {
    create_schema();
    Person *person = create_person();
    set_person_name(person, "Fred");
    save_person(person);
    Person *found = find_person_by_name("Fred");
    assert_that(get_person_name(person), is_equal_to_string("Fred"));
    drop_schema();
}

Ensure(cannot_add_duplicate_person) {
    create_schema();
    Person *person = create_person();
    set_person_name(person, "Fred");
    assert_that(save_person(person), is_true);
    Person *duplicate = create_person();
    set_person_name(duplicate, "Fred");
    assert_that(save_person(duplicate), is_false);
    drop_schema();
}

TestSuite *person_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, can_add_person_to_database);
    add_test(suite, cannot_add_duplicate_person);
    return suite;
}

int main(int argc, char **argv) {
    return run_test_suite(person_tests(), create_text_reporter());
}

We have already factored out the duplicate code into its own functions create_scheme() and drop_schema(), so things are not so bad. At least not yet. What happens when we get dozens of tests? For a test subject as complicated as a database ActiveRecord, having dozens of tests is very likely.

We can get Cgreen to do some of the work for us by declaring these methods as setup and teardown functions in the test suite.

Here is the new version…

...
static void create_schema() { ... }

static void drop_schema() { ... }

Ensure(can_add_person_to_database) {
    Person *person = create_person();
    set_person_name(person, "Fred");
    save_person(person);
    Person *found = find_person_by_name("Fred");
    assert_that(get_person_name(person), is_equal_to_string("Fred"));
}

Ensure(cannot_add_duplicate_person) {
    Person *person = create_person();
    set_person_name(person, "Fred");
    assert_that(save_person(person), is_true);
    Person *duplicate = create_person();
    set_person_name(duplicate, "Fred");
    assert_that(save_person(duplicate), is_false);
}

TestSuite *person_tests() {
    TestSuite *suite = create_test_suite();
    set_setup(suite, create_schema);
    set_teardown(suite, drop_schema);
    add_test(suite, can_add_person_to_database);
    add_test(suite, cannot_add_duplicate_person);
    return suite;
}
...

With this new arrangement Cgreen runs the create_schema() function before each test, and the drop_schema() function after each test. This saves some repetitive typing and reduces the chance of accidents. It also makes the tests more focused.

The reason we try so hard to strip everything out of the test functions is the fact that the test suite acts as documentation. In our person.h example we can easily see that Person has some kind of name property, and that this value must be unique. For the tests to act like a readable specification we have to remove as much mechanical clutter as we can.

A couple of details. You can have only one setup and one teardown in each TestSuite as indicated by the names set_setup() and set_teardown(). Also the teardown function may not be run if the test crashes, causing some test interference. This brings us nicely onto the next section…

1.5.  Each test in its own process

Consider this test method…

Ensure(will_seg_fault) {
    int *p = NULL;
    (*p)++;
}

Crashes are not something you would normally want to have in a test run. Not least because it will stop you receiving the very test output you need to tackle the problem.

To prevent segmentation faults and other problems bringing down the test suites, Cgreen runs every test in its own process.

Just before calling the setup function, Cgreen fork()'s. The main process waits for the test to complete normally or die. This includes the calling the teardown function, if any. If the test process dies, an exception is reported and the main test process carries on.

For example…

#include "cgreen/cgreen.h"
#include <stdlib.h>

Ensure(will_seg_fault) {
    int *p = NULL;
    (*p)++;
}

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test(suite, will_seg_fault);
    run_test_suite(suite, create_text_reporter());
}

When built and run, this gives…

Running "main" (1 tests)...
crash_test.c:4: Exception: -> will_seg_fault
        Test exited unexpectedly, likely from a non-standard exception, SIGSEGV, or other signal
Completed "main": 0 passes, 0 failures, 1 exception.

The obvious thing to do now is to fire up the debugger. Unfortunately, the constant fork()'ing of Cgreen can be an extra complication too many when debugging. It’s enough of a problem to find the bug.

To get around this, and also to allow the running of one test at a time, Cgreen has the run_single_test() function. The signatures of the two run methods are…

  • int run_test_suite(TestSuite *suite, TestReporter *reporter);

  • int run_single_test(TestSuite *suite, char *test, TestReporter *reporter);

The extra parameter of run_single_test(), the test string, is the name of the test to select. This could be any test, even in nested test suites (see below). Here is how we would use it to debug our crashing test…

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test(suite, will_seg_fault);
    run_single_test(suite, "will_seg_fault", create_text_reporter());
}

When run in this way, Cgreen will not fork().

This deals with the segmentation fault case, but what about a process that fails to complete by getting stuck in a loop?

Well, Cgreen will wait forever too. Using the C signal handlers, we can place a time limit on the process by sending it an interrupt. To save us writing this ourselves, Cgreen includes the die_in() function to help us out.

Here is an example of time limiting a test…

...
Ensure(will_seg_fault) { ... }

Ensure(this_would_stall) {
    die_in(1);
    while(0 == 0) { }
}

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_test(suite, will_seg_fault);
    add_test(suite, this_would_stall);
    run_test_suite(suite, create_text_reporter());
}

When executed, the code will slow for a second, and then finish with…

Running "main" (2 tests)...
crash_test.c:4: Exception: -> will_seg_fault
        Test exited in unexpectedly, likely from a non-standard exception, SIGSEGV, or other signal
crash_test.c:9: Exception: -> will_stall
        Test exited in unexpectedly, likely from a non-standard exception, SIGSEGV, or other signal
Completed "main": 0 passes, 0 failures, 2 exceptions.

Note that you see the test results as they come in. Cgreen streams the results as they happen, making it easier to figure out where the test suite has problems.

Of course, if you want to set a general time limit on all your tests, then you can add a die_in() to a setup() function. Cgreen will then apply the limit to all of them.

1.6.  Building composite test suites

The TestSuite is a composite structure. This means test suites can be added to test suites, building a tree structure that will be executed in order.

Let’s combine the strlen() tests with the Person tests above. Firstly we need to remove the main() calls. E.g…

#include "cgreen/cgreen.h"
#include <string.h>

Ensure(strlen_of_hello_is_five) { ... }
Ensure(strlen_of_empty_string_is_zero) { ... }

TestSuite *our_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, strlen_of_hello_is_five);
    add_test(suite, strlen_of_empty_string_is_zero);
    return suite;
}

Then we can write a small runner script with a new main() function…

#include "strlen_tests.c"
#include "person_tests.c"

TestSuite *our_tests();
TestSuite *person_tests();

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_suite(suite, our_tests());
    add_suite(suite, person_tests());
    if (argc > 1) {
        return run_single_test(suite, argv[1], create_text_reporter());
    }
    return run_test_suite(suite, create_text_reporter());
}

It’s usually easier to place the TestSuite prototypes in the runner scripts, rather than have lot’s of header files. This is the same reasoning that let us drop the prototypes for the test functions in the actual test scripts. We can get away with this, because the tests are more about documentation than encapsulation.

It’s sometimes handy to be able to run just a single test from the command line, so we added a simple if block to take the test name as an optional argument. The entire test suite will be searched for the named test. This trick also saves us a recompile when we debug.

We’ve placed each test suite in its own file, but that is not necessary. We could build several test suites in the same file, even nesting them. We can even add mixtures of test functions and test suites to the same parent test suite. Loops will give trouble, however.

If we do place several suites in the same file, then all the suites will be named the same in the breadcrumb trail in the test message. They will all be named after the function the create call sits in. If you want to get around this, or you just like to name your test suites, you can use create_named_test_suite() instead of create_test_suite(). This takes a single string parameter. In fact create_test_suite() is just a macro that inserts the func constant into create_named_test_suite().

What happens to setup and teardown functions in a TestSuite that contains other 'TestSuite’s?

Well firstly, Cgreen does not fork() when running a suite. It leaves it up to the child suite to fork() the individual tests. This means that a setup and teardown will run in the main process. They will be run once for each child suite.

We can use this to speed up our Person tests above. Remember we were creating a new connection and closing it again in the fixtures. This means opening and closing a lot of connections. At the slight risk of some test interference, we could reuse the connection accross tests…

...
static MYSQL *connection;

static void create_schema() {
    mysql_query(connection, "create table people (name, varchar(255) unique)");
}

static void drop_schema() {
    mysql_query(connection, "drop table people");
}

Ensure(can_add_person_to_database) { ... }
Ensure(cannot_add_duplicate_person) { ... }

void open_connection() {
    connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
}

void close_connection() {
    mysql_close(connection);
}

TestSuite *person_tests() {
    TestSuite *suite = create_test_suite();
    set_setup(suite, create_schema);
    set_teardown(suite, drop_schema);
    add_test(suite, can_add_person_to_database);
    add_test(suite, cannot_add_duplicate_person);

    TestSuite *fixture = create_named_test_suite("Mysql fixture");
    add_suite(fixture, suite);
    set_setup(fixture, open_connection);
    set_teardown(fixture, close_connection);
    return fixture;
}

The trick here is creating a test suite as a wrapper whose sole purpose to wrap the main test suite in the fixture. This is our fixture pointer. This code is a little confusing, because we have two sets of fixtures in the same test script.

We have the MySQL connection fixture. This is runs open_connection() and close_connection() just once at the beginning and end of the person tests. This is because the suite pointer is the only member of fixture.

We also have the schema fixture, the create_schema() and drop_schema(), which is run before and after every test. Those are still attached to the inner suite.

In the real world we would probably place the connection fixture in its own file…

static MYSQL *connection;

MYSQL *get_connection() {
    return connection;
}

static void open_connection() {
    connection = mysql_init(NULL);
    mysql_real_connect(connection, "localhost", "me", "secret", "test", 0, NULL, 0);
}

static void close_connection() {
    mysql_close(connection);
}

TestSuite *connection_fixture(TestSuite *suite) {
    TestSuite *fixture = create_named_test_suite("Mysql fixture");
    add_suite(fixture, suite);
    set_setup(fixture, open_connection);
    set_teardown(fixture, close_connection);
    return fixture;
}

This allows the reuse of common fixtures across projects.