Contents[Hide]

1.  Mocking functions with Cgreen

When testing you want certainty above all else. Random events destroy confidence in your test suite and force needless extra runs "to be sure". A good test places the subject under test into a tightly controlled environment. A test chamber if you like. This makes the tests fast, repeatable and reliable.

To create a test chamber for testing code, we have to control any outgoing calls from the code under test. We won’t believe our test failure if our code is making calls to the internet for example. The internet can fail all by itself. Not only do we not have total control, but it means we have to get dependent components working before we can test the higher level code. This makes it difficult to code top down.

The solution to this dilemma is to write stub code for the components whilst the higher level code is written. This pollutes the code base with temporary code, and the test isolation disappears when the system is eventually fleshed out.

The ideal is to have minimal stubs written for each individual test. Cgreen encourages this approach by making such tests easier to write.

1.1.  The problem with streams

How do we test this code…?

char *read_paragraph(int (*read)(void *), void *stream) {
    int buffer_size = 0, length = 0;
    char *buffer = NULL;
    int ch;
    while ((ch = (*read)(stream)) != EOF) {
        if (++length > buffer_size) {
            buffer_size += 100;
            buffer = (char *)realloc(buffer, buffer_size + 1);
        }
        if ((buffer[length] = ch) == '\n') {
            break;
        }
        buffer[length + 1] = '\0';
    }
    return buffer;
}

This is a fairly generic stream filter that turns the incoming characters into C string paragraphs. Each call creates one paragraph, returning a pointer to it or returning NULL if there is no paragraph. The paragraph has memory allocated to it and the stream is advanced ready for the next call. That’s quite a bit of functionality, and there are plenty of nasty boundary conditions. I really want this code tested before I deploy it.

The problem is the stream dependency. We could use a real stream, but that will cause all sorts of headaches. It makes the test of our paragraph formatter dependent on a working stream. It means we have to write the stream first, bottom up coding rather than top down. It means we will have to simulate stream failures - not easy. It will also mean setting up external resources. This is more work, will run slower, and could lead to spurious test failures.

By contrast we could write a simulation of the stream for each test, called a "server stub".

For example, when the stream is empty nothing should happen. We hopefully get NULL from read_paragraph when the stream is exhausted. That is, it just returns a steady stream of 'EOF’s.

static int empty_stream(void *stream) {
    return EOF;
}

Ensure(reading_lines_from_empty_stream_gives_null) {
    assert_that(read_paragraph(&empty_stream, NULL), is_null);
}

TestSuite *stream_tests() {
    TestSuite *suite = create_test_suite();
    add_test(suite, reading_lines_from_empty_stream_gives_null);
    return suite;
}

Our simulation is easy here, because our fake stream returns only one value. Things are harder when the function result changes from call to call as a real stream would. Simulating this would mean messing around with static variables and counters that are reset for each test. And of course, we will be writing quite a few stubs. Often a different one for each test. That’s a lot of clutter.

Cgreen handles this clutter for us by letting us write a single programmable function for all our tests.

1.2.  Record and playback

We can redo our example by creating a stub_stream() function (any name will do)…

static int stub_stream(void *stream) {
    return (int)mock();
}

Hardly longer that our trivial server stub above, it is just a macro to generate a return value, but we can reuse this in test after test.

For our simple example above we just tell it to always return EOF

#include <cgreen/mocks.h>

static int stub_stream(void *stream) {
    return (int)mock(stream);
}

Ensure(reading_lines_from_empty_stream_gives_null) {
    always_expect(stub_stream, will_return(EOF));
    assert_that(read_paragraph(stub_stream, NULL), is_null);
}

The always_expect() macro takes as arguments the function name and the will_return() defines the return value. This is an expectation of a call to the stub, and we have told stub_stream() to always return EOF when called.

Let’s see if our production code actually works…

Running "stream_tests" (1 tests)...
Completed "stream_tests": 1 pass, 0 failures, 0 exceptions.

So far, so good. On to the next test.

If we want to test a one character line, we have to send the terminating EOF or "\n" as well as the single character. Otherwise our code will loop forever, giving an infinite line of that character.

Here is how we can do this…

Ensure(one_character_stream_gives_one_character_line) {
    expect(stub_stream, will_return('a'));
    expect(stub_stream, will_return(EOF));
    char *line = read_paragraph(&stub_stream, NULL);
    assert_that(line, is_equal_to_string("a"));
    free(line);
}

Unlike the always_expect() instruction, expect() sets up an expectation of a single call and specifying will_return() sets the single return value for just that call. It acts like a record and playback model. Successive expectations map out the return sequence that will be given back once the test proper starts.

We’ll add this test to the suite and run it…

Running "stream_tests" (2 tests)...
stream_test.c:19: Failure: -> one_character_stream_gives_one_character_line
        Expected [line] to [equal string] ["a"]
                actual value:   ["8a"]
                expected value: ["a"]
Completed "stream_tests": 1 pass, 1 failure, 0 exceptions.

Oops. Our code under test doesn’t work. Already we need a fix…

char *read_paragraph(int (*read)(void *), void *stream) {
    int buffer_size = 0, length = 0;
    char *buffer = NULL;
    int ch;
    while ((ch = (*read)(stream)) != EOF) {
        if (++length > buffer_size) {
            buffer_size += 100;
            buffer = (char *)realloc(buffer, buffer_size + 1);
        }
        if ((buffer[length - 1] = ch) == '\n') {
            break;
        }
        buffer[length] = '\0';
    }
    return buffer;
}

After which everything is fine…

Running "stream_tests" (2 tests)...
Completed "stream_tests": 2 passes, 0 failures, 0 exceptions.

How do the Cgreen stubs work? Each expect() describes one call to the stub and the will_return() calls build up a static list of return values which are used and returned in order as those calls arrive. The return values are cleared between tests.

The mock() macro captures the parameter names and the func property (the name of the stub function). Cgreen can then use these to look up entries in the return list, and also to generate more helpful messages.

We can crank out our tests quite quickly now…

Ensure(one_word_stream_gives_one_word_line) {
    expect(stub_stream, will_return('t'));
    expect(stub_stream, will_return('h'));
    expect(stub_stream, will_return('e'));
    always_expect(stub_stream, will_return(EOF));
    assert_that(read_paragraph(&stub_stream, NULL), is_equal_to_string("the"));
}

I’ve been a bit naughty. As each test runs in its own process, I haven’t bothered to free the pointers to the paragraphs. I’ve just let the operating system do it. Purists may want to add the extra clean up code.

I’ve also used always_expect() for the last instruction. Without this, if the stub is given an instruction it does not expect, it will throw a test failure. This is overly restrictive, as our read_paragraph() function could quite legitimately call the stream after it had run off of the end. OK, that would be odd behaviour, but that’s not what we are testing here. If we were, it would be placed in a test of its own. The always_expect() call tells Cgreen to keep going after the first three letters, allowing extra calls.

As we build more and more tests, they start to look like a specification of the wanted behaviour…

Ensure(drops_line_ending_from_word_and_stops) {
    expect(stub_stream, will_return('t'));
    expect(stub_stream, will_return('h'));
    expect(stub_stream, will_return('e'));
    expect(stub_stream, will_return('\n'));
    assert_that(read_paragraph(&stub_stream, NULL), is_equal_to_string("the"));
}

…and just for luck…

Ensure(single_line_ending_gives_empty_line) {
    expect(stub_stream, will_return('\n'));
    assert_that(read_paragraph(&stub_stream, NULL), is_equal_to_string(""));
}

This time we musn’t use always_return(). We want to leave the stream where it is, ready for the next call to read_paragraph(). If we call the stream beyond the line ending, we want to fail.

Oops, that was a little too fast. Turns out we are failing anyway…

Running "stream_tests" (5 tests)...
stream_test.c:36: Failure: -> drops_line_ending_from_word_and_stops
        Expected [read_paragraph(&stub_stream, NULL)] to [equal string] ["the"]
                actual value:   ["the
"]
                expected value: ["the"]
stream_test.c:41: Failure: -> single_line_ending_gives_empty_line
        Expected [read_paragraph(&stub_stream, NULL)] to [equal string] [""]
                actual value:   ["
öjj"]
                expected value: [""]
Completed "stream_tests": 3 passes, 2 failures, 0 exceptions.

Clearly we are passing through the line ending. Another fix later…

char *read_paragraph(int (*read)(void *), void *stream) {
    int buffer_size = 0, length = 0;
    char *buffer = NULL;
    int ch;
    while ((ch = (*read)(stream)) != EOF) {
        if (++length > buffer_size) {
            buffer_size += 100;
            buffer = (char *)realloc(buffer, buffer_size + 1);
        }
        if ((buffer[length - 1] = ch) == '\n') {
            buffer[--length] = '\0';
            break;
        }
        buffer[length] = '\0';
    }
    return buffer;
}

And we are passing again…

Running "stream_tests" (5 tests)...
Completed "stream_tests": 5 passes, 0 failures, 0 exceptions.

There are no limits to the number of stubbed methods within a test, only that two stubs cannot have the same name. So the following will cause problems…

static int stub_stream(void *stream) {
    return (int)mock();
}

Ensure(bad_test) {
    expect(stub_stream, will_return('a'));
    do_stuff(&stub_stream, &stub_stream);
}

It will be necessary to have two stubs to make this test behave…

static int first_stream(void *stream) {
    return (int)mock();
}

static int second_stream(void *stream) {
    return (int)mock();
}

Ensure(good_test) {
    expect(first_stream, will_return('a'));
    expect(second_stream, will_return('a');
    do_stuff(&first_stream, &second_stream);
}

We now have a way of writing fast, clear tests with no external dependencies. The information flow is still one way though, from stub to the code under test. When our code calls complex procedures, we won’t want to pick apart the effects to infer what happened. That’s too much like detective work. And why should we? We just want to know that we dispatched the correct information down the line.

Things get more interesting when we thing of the traffic going the other way, from code to stub. This gets us into the same territory as mock objects.

1.3.  Setting expectations on mock functions

To swap the traffic flow, we’ll look at an outgoing example instead. Here is the prewritten production code…

void by_paragraph(int (*read)(void *), void *in, void (*write)(void *, char *), void *out) {
    while (1) {
        char *line = read_paragraph(read, in);
        if (line == NULL) {
            return;
        }
        (*write)(out, line);
        free(line);
    }
}

This is the start of a formatter utility. Later filters will probably break the paragaphs up into justified text, but right now that is all abstracted behind the void write(void *, char *) interface. Our current interests are: does it loop through the paragraphs, and does it crash?

We could test correct paragraph formation by writing a stub that collects the paragraphs into a struct. We could then pick apart that struct and test each piece with assertions. This approach is extremely clumsy in C. The language is just not suited to building and tearing down complex edifices, never mind navigating them with assertions. We would badly clutter our tests.

Instead we’ll test the output as soon as possible, right in the called function…

...
void expect_one_letter_paragraph(char *paragraph, void *stream) {
    assert_string_equal(paragraph, "a", NULL);
}

Ensure(one_character_is_made_into_a_one_letter_paragraph) {
    by_paragraph(
            &one_character_stream,
            NULL,
            &expect_one_letter_paragraph,
            NULL);
}
...

By placing the assertions into the mocked function, we keep the tests minimal. The catch with this method is that we are back to writing individual functions for each test. We have the same problem as we had with hand coded stubs.

Again, Cgreen has a way to automate this. Here is the rewritten test…

static int reader(void *stream) {
    return (int)mock(stream);
}

static void writer(void *stream, char *paragraph) {
    mock(stream, paragraph);
}

Ensure(one_character_is_made_into_a_one_letter_paragraph) {
    expect(reader, will_return('a'));
    always_expect(reader, will_return(EOF));
    expect(writer, when(paragraph, is_equal_to_string("a")));
    by_paragraph(&reader, NULL, &writer, NULL);
}

Where are the assertions?

Unlike our earlier stub, reader() can now check its parameters. In object oriented circles, an object that checks its parameters as well as simulating behaviour is called a mock object. By analogy reader() is a mock function, or mock callback.

Using the expect macro, we have set up the expectation that writer() will be called just once. That call must have the string "a" for the paragraph parameter. If the actual value of that parameter does not match, the mock function will issue a failure straight to the test suite. This is what saves us writing a lot of assertions.

When specifying behavior of mocks there are three parts. First, how often the specified behaviour or expectation will be executed:

Macro

Description

expect(function, …)

Expected once, in order

always_expect(function, …)

Expect this behavior from here onwards

never_expect(function)

This mock is expected to never be called

You can specify constraints and behaviours for each expectation (except for never_expect() naturally). A constraint places restrictions on the parameters (and will tell you if the expected restriction was not met), and a behaviour specifies what the mock should do if the parameter constraints are met.

A parameter constraint is defined using the when(parameter, constraint) macro. It takes two parameters:

Parameter

Description

parameter

The name of the parameter to the mock function

constraint

A constraint placed on that parameter

There is a multitude of constraints available (actually, exactly the same as for the assertions we saw earlier):

Constraint

Type

is_equal_to(value)

Integers

is_not_equal_to(value)

Integers

is_greater_than(value)

Integers

is_less_than(value)

Integers

 

 

is_equal_to_contents_of(pointer, size_of_contents)

Bytes/Structures

is_not_equal_to_contents_of(pointer, size_of_contents)

Bytes/Structures

 

 

is_equal_to_string(value)

String

is_not_equal_to_string(value)

String

contains_string(value)

String

does_not_contain_string(value)

String

begins_with_string(value)

String

 

 

is_equal_to_double(value)

Double

is_not_equal_to_double(value)

Double

Then there are two ways to return results:

Macro

Description

will_return(value)

Return the value from the mock function (which needs to be declared returning that type

will_set_contents_of_parameter(parameter_name, value, size)

Writes the value in the referenced parameter

You can combine these in various ways:

  expect(mocked_file_writer,
        when(data, is_equal_to(42)),
        will_return(EOF));
  expect(mocked_file_reader,
        when(file, is_equal_to_contents_of(&FD, sizeof(FD))),
        when(input, is_equal_to_string("Hello world!"),
        will_set_contents_of_parameter(status, FD_CLOSED, sizeof(bool))));

If multiple when() are specified they all need to be fullfilled. You can of course only have one for each of the parameters of your mock function.

You can also have multiple will_set_contents_of_parameter() in an expectation, one for each reference parameter, but naturally only one will_return().

It’s about time we actually ran our test…

Running "stream_tests" (6 tests)...
Completed "stream_tests": 6 passes, 0 failures, 0 exceptions.

Confident that a single character works, we can further specify the behaviour. Firstly an input sequence…

Ensure(no_line_endings_makes_one_paragraph) {
    expect(reader, will_return('a'));
    expect(reader, will_return(' '));
    expect(reader, will_return('b'));
    expect(reader, will_return(' '));
    expect(reader, will_return('c'));
    always_expect(reader, will_return(EOF));
    expect(writer, when(paragraph, is_equal_to_string("a b c")));
    by_paragraph(&reader, NULL, &writer, NULL);
}

A more intelligent programmer than me would place all these calls in a loop. Next, checking an output sequence…

Ensure(line_endings_generate_separate_paragraphs) {
    expect(reader, will_return('a'));
    expect(reader, will_return('\n'));
    expect(reader, will_return('b'));
    expect(reader, will_return('\n'));
    expect(reader, will_return('c'));
    always_expect(reader, will_return(EOF));
    expect(writer, when(paragraph, is_equal_to_string("a")));
    expect(writer, when(paragraph, is_equal_to_string("b")));
    expect(writer, when(paragraph, is_equal_to_string("c")));
    by_paragraph(&reader, NULL, &writer, NULL);
}

Again we can se that the expect() calls follow a record and playback model. Each one tests a successive call. This sequence confirms that we get "a", "b" and "c" in order.

Then we’ll make sure the correct stream pointers are passed to the correct functions. This is a more realistic parameter check…

Ensure(resources_are_paired_with_the_functions) {
    expect(reader, when(stream, is_equal_to(1)), will_return('a'));
    always_expect(reader, when(stream, is_equal_to(1)), will_return(EOF));
    expect(writer, when(stream, is_equal_to(2)));
    by_paragraph(&reader, (void *)1, &writer, (void *)2);
}

And finally we’ll specify that the writer is not called if there is no paragraph.

Ensure(empty_paragraphs_are_ignored) {
    expect(reader, will_return('\n'));
    always_expect(reader, will_return(EOF));
    never_expect(writer);
    by_paragraph(&reader, NULL, &writer, NULL);
}

This last test is our undoing…

Running "stream_tests" (10 tests)...
streams_tests.c:93: Test Failure: -> empty_paragraphs_are_ignored
        Function [writer] has an expectation that it will never be called, but it was
Completed "stream_tests": 14 passes, 1 failure, 0 exceptions.

Obviously blank lines are still being dispatched to the writer(). Once this is pointed out, the fix is obvious…

void by_paragraph(int (*read)(void *), void *in, void (*write)(void *, char *), void *out) {
    while (1) {
        char *line = read_paragraph(read, in);
        if ((line == NULL) || (strlen(line) == 0)) {
            return;
        }
        (*write)(out, line);
        free(line);
    }
}

Tests with never_expect() can be very effective at uncovering subtle bugs.

Running "stream_tests"...
Completed "stream_tests": 14 passes, 0 failures, 0 exceptions.

All done.

1.4.  Mocks Are…

Using mocks is a very handy way to isolate a unit and catch and control calls to external units. Depending on your style of coding two schools of thinking have emerged. And of course Cgreen supports both!

Strict or Loose Mocks

The two schools are thinking a bit differently about what mock expectations means. Does it mean that all external calls must be declared and expected? What happens if a call was made to a mock that wasn’t expected? And vice versa, if an expected call was not made?

Actually, the thinking is not only a school of thought, but you might want to switch from one to the other. So Cgreen allows for that too.

By default Cgreen mocks are 'strict', which means that a call to an non-expected mock will be considered a failure. So will an expected call that was not fullfilled. You might consider this a way to define a unit through all its exact behaviours towards its neighbours.

On the other hand, 'loose' mocks are looser. They allow both unfullfilled expectations and try to handle unexpected calls in a reasonable way.

You can use both with in the same suite of tests using the call cgreen_mocks_are(strict_mocks); and cgreen_mocks_are(loose_mocks); respectively.

Learning Mocks

Working with legacy code and trying to apply TDD, BDD or even simply add some unit tests is not easy. You’re working with unknown code that does unknown things with unknown counterparts.

So the first step would be to isolate the unit. We won’t go into details on how to do that here, but basically you would replace the interface to other units with mocks. This is a somewhat tedious manual labor, but will result in an isolated unit where you can start applying your unit tests.

Once you have your unit isolated in a harness of mocks, we need to figure out which calls it does to other units, now mocks, in the specific case we are trying to test.

This might be complicated, so Cgreen makes that a bit simpler. There is a third 'mode' of the Cgreen mocks, the learning mocks.

If you temporarily add the call cgreen_mocks_are(learning_mocks); at the beginning of your unit test, the mocks will record all calls and present a list of those calls in order, including the actual parameter values, on the standard output.

So let’s look at the following example from the Cgreen unit tests. It’s a bit contorted since the test actually call the mocked functions directly, but I believe it will serve as an example.

static char *string_out(int p1) {
    return (char *)mock(p1);
}

static int integer_out() {
    return (int)mock();
}

Ensure(Mocks, learning_mocks_emit_pastable_code) {
    cgreen_mocks_are(learning_mocks);
    string_out(1);
    string_out(2);
    integer_out();
    integer_out();
    string_out(3);
    integer_out();
}

We can see the call to cgreen_mocks_are() starting the test and setting the mocks into learning mode.

If we run this, just as we usually run tests, the following will show up in our terminal:

learning_mocks_emit_pastable_code: learned mocks:
        expect(string_out, when(p1, is_equal_to(1)));
        expect(string_out, when(p1, is_equal_to(2)));
        expect(integer_out);
        expect(integer_out);
        expect(string_out, when(p1, is_equal_to(3)));
        expect(integer_out);

If this were a real test we could just copy this and paste it in place of the call to cgreen_mocks_are() and we’re done.