Contents[Hide]

 

1.  Changing Cgreen Reporting

1.1.  Replacing the reporter

In every test suite so far, we have run the tests with this line…

return run_test_suite(our_tests(), create_text_reporter());

We can change the reporting mechanism just by changing this method. Here is the code for create_text_reporter()

TestReporter *create_text_reporter(void) {
    TestReporter *reporter = create_reporter();
    if (reporter == NULL) {
        return NULL;
    }
    reporter->start_suite = &text_reporter_start_suite;
    reporter->start_test = &text_reporter_start_test;
    reporter->show_fail = &show_fail;
    reporter->show_incomplete = &show_incomplete;
    reporter->finish_test = &text_reporter_finish;
    reporter->finish_suite = &text_reporter_finish;
    return reporter;
}

The TestReporter structure contains function pointers that control the reporting. When called from create_reporter() constructor, these pointers are set up with functions that display nothing. The text reporter code replaces these with something more dramatic, and then returns a pointer to this new object. Thus the create_text_reporter() function effectively extends the object from create_reporter().

The text reporter only outputs content at the start of the first test, at the end of the test run to display the results, when a failure occours, and when a test fails to complete. A quick look at the text_reporter.c file in Cgreen reveals that the overrides just output a message and chain to the versions in reporter.h.

To change the reporting mechanism ourselves, we just have to know a little about the methods in the TestReporter structure.

1.2.  The TestReporter structure

The Cgreen TestReporter is a pseudo class that looks something like…

typedef struct _TestReporter TestReporter;
struct _TestReporter {
    void (*destroy)(TestReporter *reporter);
    void (*start_suite)(TestReporter *reporter, const char *name, const int count);
    void (*start_test)(TestReporter *reporter, const char *name);
    void (*show_pass)(TestReporter *reporter, const char *file, int line,
                                   const char *message, va_list arguments);
    void (*show_fail)(TestReporter *reporter, const char *file, int line,
                                   const char *message, va_list arguments);
    void (*show_incomplete)(TestReporter *reporter, const char *file, int line,
                                   const char *message, va_list arguments);
    void (*assert_true)(TestReporter *reporter, const char *file, int line, int result,
                                   const char * message, ...);
    void (*finish_test)(TestReporter *reporter, const char *file, int line);
    void (*finish_suite)(TestReporter *reporter, const char *file, int line);
    int passes;
    int failures;
    int exceptions;
    void *breadcrumb;
    int ipc;
    void *memo;
};

The first block are the methods that can be overridden.

void (*destroy)(TestReporter *reporter)

This is the destructor for the default structure. If this is overridden, then the overriding function must call destroy_reporter(TestReporter *reporter) to finish the clean up.

void (*start_suite)(TestReporter *reporter, const char *name, const int count)

This is the first of the callbacks. At the start of each test suite Cgreen will call this method on the reporter with the name of the suite being entered and the number of tests in that suite. The default version keeps track of the stack of tests in the breadcrumb pointer of TestReporter. If you make use of the breadcrumb functions, as the defaults do, then you will need to call reporter_start() to keep the book keeping in sync.

void (*start_test)(TestReporter *reporter, const char *name)

At the start of each test Cgreen will call this method on the reporter with the name of the test being entered. Again, the default version keeps track of the stack of tests in the breadcrumb pointer of TestReporter. If you make use of the breadcrumb functions, as the defaults do, then you will need to call reporter_start() to keep the book keeping in sync.

void (*show_pass)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)

This method is initially empty as there most reporters see little point in reporting passing tests (but you might do), so there is no need to chain the call to any other function. Besides the pointer to the reporter structure, Cgreen also passes the file name of the test, the line number of failed assertion, the message to show and any additional parameters to substitute into the message. The message comes in as printf() style format string, and so the variable argument list should match the substitutions.

void (*show_fail)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)

The partner of show_pass(), and the one you’ll likely overload first.

void (*show_incomplete)(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments)

When a test fails to complete, this is the handler that is called. As it’s an unexpected outcome, no message is received, but we do get the name of the test. The text reporter combines this with the breadcrumb to produce the exception report.

void (*assert_true)(TestReporter *reporter, const char *file, int line, int result, const char * message, …)

This is not normally overridden and is really internal. It is the raw entry point for the test messages from the test suite. By default it dispatches the call to either show_pass() or show_fail().

void (*finish_test)(TestReporter *reporter, const char *file, int line)

The counterpart to the (*start_test)() call. It is called on leaving the test. It needs to be chained to the reporter_finish() to keep track of the breadcrumb book keeping.

void (*finish_suite)(TestReporter *reporter, const char *file, int line)

The counterpart to the (*start_suite)() call called on leaving the test suite, and similar to the (*finish_test)() if your reporter needs a handle on that event too. The default text reporter chains both this and (*finish_test)() to the same function where it figures out if it is the end of the top level suite. If so, it prints the familiar summary of passes and fails.

The second block is simply resources and book keeping that the reporter can use to liven up the messages…

passes

The number of passes so far.

failures

The number of failures generated so far.

exceptions

The number of test functions that have failed to complete so far.

breadcrumb

This is a pointer to the list of test names in the stack.

The breadcrumb pointer is different and needs a little explanation. Basically it is a stack, analogous to the breadcrumb trail you see on websites. Everytime a start() handler is invoked, the name is placed in this stack. When a finish() message handler is invoked, a name is popped off.

There are a bunch of utility functions in cgreen/breadcrumb.h that can read the state of this stack. Most useful are get_current_from_breadcrumb() which takes the breadcrumb pointer and returns the curent test name, and get_breadcrumb_depth() which gives the current depth of the stack. A depth of zero means that the test run has finished.

If you need to traverse all the names in the breadcrumb, then you can call walk_breadcrumb(). Here is the full signature…

void walk_breadcrumb(Breadcrumb *breadcrumb, void (*walker)(const char *, void *), void *memo);

The void (*walker)(const char *, void *) is a callback that will be passed the name of the test suite for each level of nesting. It is also passed the memo pointer that was passed to the walk_breadcrumb() call. You can use this pointer for anything you want, as all Cgreen does is pass it from call to call. This is so aggregate information can be kept track of whilst still being reentrant.

The last parts of the TestReporter structure are…

ipc

This is an internal structure for handling the messaging between reporter and test suite. You shouldn’t touch this.

memo

By contrast, this is a spare pointer for your own expansion.

1.3.  An example XML reporter

Let’s make things real with an example. Suppose we want to send the output from Cgreen in XML format, say for storing in a repository or for sending across the network.

Suppose also that we have come up with the following format…

<?xml?>
<suite name="Top Level">
    <suite name="A Group">
        <test name="a_test_that_passes">
        </test>
        <test name="a_test_that_fails">
            <fail>
                <message>A failure</message>
                <location file="test_as_xml.c" line="8"/>
            </fail>
        </test>
    </suite>
</suite>

In other words a simple nesting of tests with only failures encoded. The absence of failure is a pass.

Here is a test script, test_in_xml.c that we can use to construct the above output…

#include "cgreen/cgreen.h"

Ensure(this_test_passes) {
    assert_true(1);
}

Ensure(this_test_fails) {
    assert_true_with_message(0, "A failure");
}

TestSuite *test_group() {
    TestSuite *suite = create_test_suite();
    add_test(suite, this_test_passes);
    add_test(suite, this_test_fails);
    return suite;
}

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_suite(suite, test_group());
    return run_test_suite(suite, create_text_reporter());
}

The text reporter is used just to confirm that everything is working. So far it is.

Running "main" (2 tests)...
test_in_xml.c:8: Test Failure: -> test_group -> this_test_fails
        A failure
Running "main" (2 tests)...
Completed "main": 1 pass, 1 failure, 0 exceptions.

Our first move is to switch the reporter from text, to our not yet written XML version…

#include "cgreen/cgreen.h
#include "xml_reporter.h"

...

int main(int argc, char **argv) {
    TestSuite *suite = create_test_suite();
    add_suite(suite, test_group());
    return run_test_suite(suite, create_xml_reporter());
}

We’ll start the ball rolling with the xml_reporter.h header file…

#ifndef _XML_REPORTER_HEADER_
#define _XML_REPORTER_HEADER_

#include "cgreen/reporter.h"

TestReporter *create_xml_reporter();

#endif

…and the simplest possible reporter in reporter.c.

#include "xml_reporter.h"
#include "cgreen/reporter.h"

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    return reporter;
}

One that outputs nothing.

gcc -c test_as_xml.c
gcc -c xml_reporter.c
gcc xml_reporter.o test_as_xml.o -lcgreen -o xml
./xml

Yep, nothing.

Let’s add the outer test tags first, so that we can see Cgreen navigating the test suite…

#include "xml_reporter.h"
#include "cgreen/reporter.h"
#include <stdio.h>

static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count);
static void xml_reporter_start_test(TestReporter *reporter, const char *name);
static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line);
static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line);

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start_suite = &xml_reporter_start_suite;
    reporter->start_test = &xml_reporter_start_test;
    reporter->finish_test = &xml_reporter_finish_test;
    reporter->finish_suite = &xml_reporter_finish_suite;
    return reporter;
}

static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count) {
    printf("<suite name=\"%s\">\n", name);
    reporter_start(reporter, name);
}

static void xml_reporter_start_test(TestReporter *reporter, const char *name) {
    printf("<test name=\"%s\">\n", name);
    reporter_start(reporter, name);
}

static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line) {
    reporter_finish(reporter, filename, line);
    printf("</test>\n");
}

static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line) {
    reporter_finish(reporter, filename, line);
    printf("</suite>\n");
}

Although chaining to the underlying reporter_start() and reporter_finish() functions is optional, I want to make use of some of the facilities later.

Our output meanwhile, is making its first tentative steps…

<suite name="main">
<suite name="test_group">
<test name="this_test_passes">
</test>
<test name="this_test_fails">
</test>
</suite>
</suite>

We don’t want a passing message, so the show_fail() function is all we need…

...
static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments);

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start_suite = &xml_reporter_start_suite;
    reporter->start_test = &xml_reporter_start_test;
    reporter->show_fail = &xml_show_fail;
    reporter->finish_test = &xml_reporter_finish_test;
    reporter->finish_suite = &xml_reporter_finish_suite;
    return reporter;
}

...

static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    printf("<fail>\n");
    printf("\t<message>\"");
    vprintf(message, arguments);
    printf("\"</message>\n");
    printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
    printf("</fail>\n");
}

We have to use vprintf() to handle the variable argument list passed to us. This will probably mean including the stdarg.h header as well as stdio.h.

This gets us pretty close to what we want…

<suite name="main">
<suite name="test_group">
<test name="this_test_passes">
</test>
<test name="this_test_fails">
<fail>
        <message>A failure</message>
        <location file="test_in_xml.c" line="9"/>
</fail>
</test>
</suite>
</suite>

For completeness we should add a tag for an incomplete test. We’ll output this as a failure, athough we don’t get a location this time…

#include "xml_reporter.h"
#include "cgreen/reporter.h"
#include "cgreen/breadcrumb.h"

...

static void xml_show_incomplete(TestReporter *reporter, const char *name) {
    printf("<fail>\n");
    printf("\t<message>Failed to complete</message>\n");
    printf("</fail>\n");
}

All that’s left then is the XML declaration and the thorny issue of indenting. Although the indenting is not strictly necessary, it would make the output a lot more readable.

The test depth is kept track of for us with the breadcrumb object in the TestReporter structure. We’ll add an indent() function that outputs the correct number of tabs…

static indent(TestReporter *reporter) {
    int depth = get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb);
    while (depth-- > 0) {
        printf("\t");
    }
}

The get_breadcrumb_depth() function just gives the current test depth as recorded in the reporters breadcrumb (from cgreen/breadcrumb.h). As that is just the number of tabs to output, the implementation is trivial.

We can then use this function in the rest of the code. Here is the complete listing…

#include "xml_reporter.h"
#include "cgreen/reporter.h"
#include "cgreen/breadcrumb.h"

#include <stdio.h>

static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count);
static void xml_reporter_start_test(TestReporter *reporter, const char *name);
static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line);
static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line);
static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments);

TestReporter *create_xml_reporter() {
    TestReporter *reporter = create_reporter();
    reporter->start_suite = &xml_reporter_start_suite;
    reporter->start_test = &xml_reporter_start_test;
    reporter->show_fail = &xml_show_fail;
    reporter->finish_test = &xml_reporter_finish_test;
    reporter->finish_suite = &xml_reporter_finish_suite;
    return reporter;
}

static indent(TestReporter *reporter) {
    int depth = get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb);
    while (depth-- > 0) {
        printf("\t");
    }
}

static void xml_reporter_start_suite(TestReporter *reporter, const char *name, int count) {
    if (get_breadcrumb_depth((CgreenBreadcrumb *)reporter->breadcrumb) == 0) {
        printf("<?xml?>\n");
    }
    indent(reporter);
    printf("<suite name=\"%s\">\n", name);
    reporter_start(reporter, name);
}

static void xml_reporter_start_test(TestReporter *reporter, const char *name) {
    indent(reporter);
    printf("<test name=\"%s\">\n", name);
    reporter_start(reporter, name);
}

static void xml_show_fail(TestReporter *reporter, const char *file, int line, const char *message, va_list arguments) {
    indent(reporter);
    printf("<fail>\n");
    indent(reporter);
    printf("\t<message>");
    vprintf(message, arguments);
    printf("</message>\n");
    indent(reporter);
    printf("\t<location file=\"%s\" line=\"%d\"/>\n", file, line);
    indent(reporter);
    printf("</fail>\n");
}

static void xml_show_incomplete(TestReporter *reporter, const char *name) {
    indent(reporter);
    printf("<fail>\n");
    indent(reporter);
    printf("\t<message>Failed to complete]]></message>\n");
    indent(reporter);
    printf("</fail>\n");
}

static void xml_reporter_finish_test(TestReporter *reporter, const char *filename, int line) {
    reporter_finish(reporter, filename, line);
    indent(reporter);
    printf("</test>\n");
}

static void xml_reporter_finish_suite(TestReporter *reporter, const char *filename, int line) {
    reporter_finish(reporter, filename, line);
    indent(reporter);
    printf("</suite>\n");
}

And finally the desired output…

<?xml?>
<suite name="main">
    <suite name="test_group">
        <test name="this_test_passes">
        </test>
        <test name="this_test_fails">
            <fail>
                <message>A failure</message>
                <location file="test_in_xml.c" line="9"/>
            </fail>
        </test>
    </suite>
</suite>

Job done.

Possible other extensions include reporters that write to syslog, talk to IDE plug-ins, paint pretty printed documents or just return a boolean for monitoring purposes.