Unit-testing (embedded) C applications with Ceedling

Just like a lot of other embedded software engineers, I used to ship my embedded applications to production without testing them properly. Only some manual tests were done. I was under the impression that there's no real way to test them: you know, embedded applications run in a custom hardware and interact with this specific hardware heavily, which makes them not so easy to test automatically.

But the more I worked in this field, the more I thought that there should be some way to make my applications more reliable. Of course, I strive to develop right application design and write good implementation, but mistakes (sometimes very silly ones) happen. And one day, I've come across the awesome book: Test Driven Development for Embedded C by James W. Grenning. It is a very good book, and it explains the topic very thoroughly; highly recommended.

Although this article is based heavily on what I've learned from this book, it is not just a very short version of it: while trying to put my knowledge to practice, I found some new tools that simplify the process even more. I hope that the article will help you to start quickly.

What we will test

Embedded development differs from other software engineering fields in that embedded application has to interact with the hardware, and the hardware might be very different from application to application.

Unfortunately, I've seen a great deal of code which performs some application logic while interacting with the hardware directly. It is sometimes done in the name of efficiency (on the very low-end chips, even function calls might be considered as expensive), but eventually it might lead to one's habit, which, of course, is to be avoided.

The main idea (which is very good in itself, not only for unit testing) is to separate the hardware interaction and the application logic as much as possible. Then, we end up with a bunch of separate modules, which can be tested outside of the hardware.

And when we write our application with the testability in mind, we literally have to separate things. So, one “side effect” of writing a testable code is the modularity, which is a good thing to have. Testable code is all good!

So, in this article, we will not test the hardware layer. It has to be as thin as possible, and I still test it manually. What we will test is everything else: the modules that use hardware layer, and the modules that use other modules.

While there are techniques to run tests on the target hardware (and the book by James W. Grenning touches upon them, among other things), this article focuses on testing on the host machine instead.

And it's probably worth mentioning that unit tests are not the silver bullet that will magically turn your projects in completely bug-free ones. This is not true even for desktop programming, where we use the same compiler for tests and for production; and in the embedded world, it's even worse, since:

  • We can't test for compiler bugs (and, unfortunately, embedded compilers tend to be… not very good);
  • There may be subtle issues in software-hardware interaction;
  • etc

Still, from my personal experience, the final outcome is a way better on carefully tested projects than on untested ones. At least, well-written tests will save you from your own silly mistakes. And if you're like me, you definitely want that.

Tools that we'll be using

The aforementioned book takes advantage of several awesome tools:

Unity is curiously powerful Unit Testing in C for C. It aims to support most embedded compilers, from 8-bit tiny processors to 64-bit behemoths. Unity is designed to be small, yet still provide you rich expressive assertion set.
CMock is a utility for automagical generation of stubs and mocks for Unity Tests. It's a collection of scripts in Ruby which investigate your C headers and create reusable fake versions for your tests.

These tools are surely life-changing for me, but there is something that is still missing: the test build system. Probably it didn't yet exist when the book was written, so it isn't mentioned in the book at all. Here it is:

Ceedling is a build system for C projects that is something of an extension around Ruby’s Rake (make-ish) build system. Ceedling is primarily targeted at Test-Driven Development in C and is designed to pull together CMock, Unity, and CException – three other awesome open-source projects you can’t live without if you're creating awesomeness in the C language.

Armed with these great tools, let's move on.

Sample project

No surprise, we'll be learning how to write unit tests by testing example project. Let it be rather simple device, which I have actually developed firmware for: the indicator for battery charger that runs on 8-bit PIC16 MCU with 16 kB flash and 1 kB RAM. As you see, resources are quite limited.

The repository of this example project can be found here: https://github.com/dimonomid/test_ceedling_example.

What does the project look like

In order to bring the project to its initial state, checkout to the tag v0.02:

  $ git checkout v0.02

This device should merely measure some voltages (which are converted to an integer via ADC), and display them properly. It should also be available to communicate with the checker device, which will calibrate the electric circuits.

Initial project tree looks as follows:

.
└── src
    ├── appl
    │   ├── appl_adc_channels.h
    │   └── appl.c
    ├── bsp
    │   ├── bsp_adc.c
    │   └── bsp_adc.h
    └── util
        ├── adc_handler.c
        ├── adc_handler.h
        ├── itoae.c
        ├── itoae.h
        └── util_macros.h

There are three directories:

  • appl: main application code;
  • bsp: board- and MCU-dependent code;
  • util: various utilities.

Since the resources are very limited, we don't use a RTOS: just a super-loop. We also can't use any printf-like functions, as they are too expensive. And we can't use floats: we use integers only.

So, the main() looks like this:

int main(void)
{
    //TODO: MCU-specific init
 
    //-- init MCU-specific ADC stuff
    bsp_adc__init();
 
    //-- enter endless loop
    for (;;){
        bsp_adc__proceed();
    }
}

This is a quite common scheme for non-RTOS solutions: in the super-loop, we just call each module's proceed function, which might do some job. Currently, we have just one module there: bsp_adc.

The bsp_adc module is an MCU-dependent routines to retrieve raw ADC values. Its API looks like this:

/**
 * Type that is used for ADC raw counts.
 */
typedef uint16_t T_BspAdcCounts;
 
 
 
/**
 * Perform module initialization, including the hardware initialization
 */
void bsp_adc__init(void);
 
/**
 * Returns raw ADC counts for the specified channel
 */
T_BspAdcCounts bsp_adc__value__get(enum E_ApplAdcChannel channel_num);
 
/**
 * To be repeatedly called from the application's super loop: stores current
 * measurement when it's ready and switches between different channels as
 * appropriate
 */
void bsp_adc__proceed(void);

As you see, it's very easy. When we call bsp_adc__value__get(), it returns the raw ADC value for specified channel, which is not particularly useful. We'd like to convert it to something more human-readable, right? That is, to the value in Volts.

There is an adc_handler module for that. Basically, it just applies a summand and a multiplier to the raw value returned by bsp_adc__value__get(), and we get an integer value in Volts, multiplied by some scale factor (in this example, the factor 100 is used; so, the value 1250 means 12.5 Volts.

For each instance of ADC handler (struct S_ADCHandler), we call its constructor function:

T_ADCHandler_Res adc_handler__ctor(
        T_ADCHandler *me, const T_ADCHandler_CtorParams *p_params
        );

Passing a pointer to parameters:

/**
 * Constructor params for ADC handler
 */
typedef struct S_ADCHandler_CtorParams {
 
    /**
     * Maximum value that ADC could return. Say, for 10-bit ADC it should be
     * 0x3ff.
     */
    T_ADCHandler_CountsValue   max_counts;
 
    /**
     * The board-dependent maximum voltage that could be measured, it
     * corresponds to the max_counts.
     *
     * It is only needed for calculation of nominal multiplier.
     */
    T_ADCHandler_Voltage      bsp_max_voltage;
 
    /**
     * Calibration data: summand and multiplier.
     *
     * Set just all zeros to use nominal. Nominal mul will be calculated by
     * max_counts and bsp_max_voltage. add will be copied from
     * nominal_add (see below)
     */
    T_ADCHandler_Clb  clb;
 
    /**
     * Nominal summand, in volts
     */
    T_ADCHandler_Voltage nominal_add_volts;
 
} T_ADCHandler_CtorParams;

And then, when we want to convert some raw ADC value my_raw_adc_value to Volts, it is as easy as:

T_ADCHandler_Voltage my_voltage = adc_handler__voltage__get_by_counts_value(
        &my_instance, my_raw_adc_value
        );

As you see, the ADC handler module is completely self-contained: it doesn't have any dependencies. Modules like this are the easiest ones to test, so, let's start our test journey from ADC handler.

Install testing tools

Before we can go any further, we need to install the aforementioned Ceedling and all accompanying tools. Thanks to guys from ThrowTheSwitch.org, installation process is very simple. You need ruby for this. Once you have ruby installed, install Ceedling by typing in your terminal:

$ gem install ceedling

And that's it! Now you have all tools ready to work, and among other things, there is a ceedling binary, which we're going to take advantage of.

Adding test stuff to project

The ceedling binary allows us to create new project tree by executing the command ceedling new my_project, including even main source directory. Since I usually create my projects in some other way, I need to move things around after ceedling has created new project.

Let's move on: cd to the project's directory (where you have the src directory), and execute the following:

$ ceedling new test_ceedling

This will create the new directory test_ceedling with the following contents:

  • test: directory for our test C files;
  • build: directory in which our built test will be located;
  • vendor: ceedling-provided binaries and all accompanying stuff. It includes CMock, Unity, and other tools. Since we're just users of Ceedling, we have little interest of actual contents of this directory, except the documentation:
  • vendor/ceedling/docs: documentation of Ceedling and components. Very useful;
  • src: the directory in which Ceedling assumed to have source files, but we're going to store files not here, but in our own src directory;
  • rakefile.rb: it is needed to run tests; you don't need to understand it;
  • project.yml: the actual project file, which we need to adjust for our needs.

The heart of the test build system is the project file: project.yml. Among other things, it contains paths to your application source files. Since our source files are contained not where ceedling assumed by default, we need to change the project file a bit: open the file project.yml and find the paths section:

:paths:
  :test:
    - +:test/**
    - -:test/support
  :source:
    - src/**
  :support:
    - test/support

As you might have already guessed, we need to change src/** to the path to our actual source files, relative to the location of the project file. Ok, that's easy: let's add all source paths that currently exist. We end up with three paths there:

  :source:
    - ../src/appl
    - ../src/bsp
    - ../src/util

Now, in order to avoid confusion, let's delete the auto-created test_ceedling/src directory, since we're not going to use it. We also want to add some .gitkeep files in our empty directories, so that git will keep them in repository. From the root of the repository, type:

$ touch test_ceedling/build/.gitkeep
$ touch test_ceedling/test/support/.gitkeep

Add files to the repository and commit:

$ git add .
$ git commit

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.03.

Actually, our test build system, though empty, is ready to run! Try it: make sure you're in the test_ceedling directory, and type in the terminal:

$ rake test:all

You should see the following output:

--------------------
OVERALL TEST SUMMARY
--------------------

No tests executed.

It works, and it predictably reports that we have no tests. So, let's add some meat to the bones!

Testing standalone modules

Writing test for ADC handler

As mentioned above, we'll start by writing tests for our ADC handler, since it is one of the easiest things to test: it has no application-specific dependencies. The job of ADC handler is to convert from raw ADC counts to voltage, and vice versa. We'll test this functionality.

First of all, let's create blank test file. I have a template for this:

/*******************************************************************************
 *    INCLUDED FILES
 ******************************************************************************/
 
//-- unity: unit test framework
#include "unity.h"
 
//-- module being tested
//   TODO
 
 
/*******************************************************************************
 *    DEFINITIONS
 ******************************************************************************/
 
/*******************************************************************************
 *    PRIVATE TYPES
 ******************************************************************************/
 
/*******************************************************************************
 *    PRIVATE DATA
 ******************************************************************************/
 
 
/*******************************************************************************
 *    PRIVATE FUNCTIONS
 ******************************************************************************/
 
 
/*******************************************************************************
 *    SETUP, TEARDOWN
 ******************************************************************************/
 
void setUp(void)
{
}
 
void tearDown(void)
{
}
 
/*******************************************************************************
 *    TESTS
 ******************************************************************************/
 
void test_first(void)
{
    //TODO
}

We can use this template whenever we need to write new test. Our tests are just functions with names that start with test_. In the example above, we have just empty test_first() function.

We also have two special functions: setUp() and tearDown(). The setUp() is called before running each test, and tearDown() is called after running each test. We'll take advantage of them soon. Now, let's add ADC handler test here.

We begin by adding the header of the module being tested:

//-- module being tested: ADC handler
#include "adc_handler.h"

Then, add an instance that we'll run tests against, together with the result code returned from constructor:

/*******************************************************************************
 *    PRIVATE DATA
 ******************************************************************************/
 
static T_ADCHandler     _adc_handler;
static T_ADCHandler_Res _ctor_result;

And then, construct/destruct it in setUp() / tearDown(), respectively.

void setUp(void)
{
    T_ADCHandler_CtorParams params = {};
 
    //-- 10-bit ADC
    params.max_counts = 0x3ff;
 
    //-- board-dependent maximum measured voltage: 10 Volts
    params.bsp_max_voltage = 10/*V*/ * ADC_HANDLER__SCALE_FACTOR__U;
 
    //-- the offset is 0 Volts
    params.nominal_add_volts = 0/*V*/;
 
 
    //-- construct the ADC handler, saving the result to _ctor_result
    _ctor_result = adc_handler__ctor(&_adc_handler, &params);
}
 
void tearDown(void)
{
    adc_handler__dtor(&_adc_handler);
}

Now, the easiest test we can come up with is to check that constructor has returned successful status, i.e. ADC_HANDLER_RES__OK. So, rename our dummy test_first to test_ctor_ok, and add our first assert:

void test_ctor_ok(void)
{
    //-- check that constructor returned OK
    TEST_ASSERT_EQUAL_INT(ADC_HANDLER_RES__OK, _ctor_result);
}

We're ready to run our first test!

$ rake test:all

Test 'test_adc_handler.c'
-------------------------
Generating runner for test_adc_handler.c...
Compiling test_adc_handler_runner.c...
Compiling test_adc_handler.c...
Compiling unity.c...
Compiling adc_handler.c...
Compiling cmock.c...
Linking test_adc_handler.out...
Running test_adc_handler.out...

-----------
TEST OUTPUT
-----------
[test_adc_handler.c]
  - ""

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  1
PASSED:  1
FAILED:  0
IGNORED: 0

It works, and our test has passed. Good!

Among other things, you see that Ceedling has figured out that it needs to build adc_handler.c. How does it work? Ceedling examines the headers we include in our test file, and looks for the appropriate source file. Since we have adc_handler.h included, Ceedling looks for adc_handler.c file in all source paths that we've specified in our project.yml, and compiles it. Pretty easy.

If you used to apply TDD practices, you know that it's good to make sure our tests fail if code behaves in wrong way. We don't do TDD here, since we already have some code before we write tests, but we still can make sure that our tests can fail. If we change adc_handler__ctor() so that it returns, for example, ADC_HANDLER_RES__CLB_ERR__WRONG_PARAMS, our test will fail as follows:

-----------
TEST OUTPUT
-----------
[test_adc_handler.c]
  - ""

-------------------
FAILED TEST SUMMARY
-------------------
[test_adc_handler.c]
  Test: test_ctor_ok
  At line (72): "Expected 1 Was 6"

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  1
PASSED:  0
FAILED:  1
IGNORED: 0

---------------------
BUILD FAILURE SUMMARY
---------------------
Unit test failures.

That's it. Let's change adc_handler back to correct state, and commit.

We definitely don't want to include build output in the repository, so, add the test_ceedling/build directory to ignore list. In the root of the repo, create .gitignore file with the following contents:

test_ceedling/build

And then, commit changes:

$ git add .
$ git commit -m"added test_adc_handler"

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.04.

Now let's check that ADC handler is actually able to convert from ADC counts to voltage. We'll test the function adc_handler__voltage__get_by_counts_value(). With the parameters we've given in setUp() function, we need to make sure that 0 counts are converted to 0.0 Volts, 0x3ff counts are converted to 10.0 Volts, and, for example, 0x3ff / 3 counts are converted to 3.33 Volts.

Here we go:

void test_counts_to_voltage(void)
{
    T_ADCHandler_Voltage voltage;
 
 
    //------------------------------------------------------------------
    voltage = adc_handler__voltage__get_by_counts_value(
            &_adc_handler, 0
            );
    TEST_ASSERT_EQUAL_INT(0/*V*/ * ADC_HANDLER__SCALE_FACTOR__U, voltage);
 
 
    //------------------------------------------------------------------
    voltage = adc_handler__voltage__get_by_counts_value(
            &_adc_handler, 0x3ff
            );
    TEST_ASSERT_EQUAL_INT(10/*V*/ * ADC_HANDLER__SCALE_FACTOR__U, voltage);
 
 
    //------------------------------------------------------------------
    voltage = adc_handler__voltage__get_by_counts_value(
            &_adc_handler, 0x3ff / 3
            );
    TEST_ASSERT_EQUAL_INT((3.33/*V*/ * ADC_HANDLER__SCALE_FACTOR__U), voltage);
}

If we run this test, it should pass:

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  2
PASSED:  2
FAILED:  0
IGNORED: 0

So, we can be sure now that ADC handler performs its basic job.

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.05.

Writing test for integer-to-string conversion

As was mentioned before, we can't use floats since they are too expensive for this cheap MCU, so, we store voltage as integer in Volts, multiplied by the factor 100. Of course, we need to convert the voltage integer value like 1250 to the string like “12.5”. That's what the awfully named module itoae is for. The name “itoae” stands for “integer-to-array-extended”.

Its API looks like this:

/**
 * Itoa a bit extended: allows to set minimal length of the string
 * (effectively allowing us to align text by right edge),
 * and allows to put decimal point at some fixed position.
 *
 * @param p_buf
 *      where to save string data
 * @param value
 *      value to convert to string
 * @param dpp
 *      decimal point position. If 0, then no decimal point is put.
 *      if 1, then it is put one digit from the right side, etc.
 * @param min_len
 *      minimum length of the string. If actual string is shorter than
 *      specified length, then leftmost characters are filled with
 *      fill_char.
 * @param fill_char
 *      character to fill "extra" space
 */
void itoae(uint8_t *p_buf, int value, int dpp, int min_len, uint8_t fill_char);

You can find the source in the file src/util/itoae.c. The implementation is rather dumb: first, we call “regular” itoa that doesn't know anything about decimal points, it just converts integer to string. Then, if we need for decimal point, then move some characters to the right, and insert the ..

As you see, this function is very straightforward to test as well. Let's create new file test_ceedling/test/test_itoae.c from the template above, and include the header of module being tested:

//-- module being tested
#include "itoae.h"

We're going to have a buffer for generated string:

#define _BUF_LEN    20
 
/**
 * Buffer to store generated string data
 */
static uint8_t _buf[ _BUF_LEN ];

As well as the function that wills the buffer with 0xff:

void _fill_with_0xff(void)
{
    int i;
    for (i = 0; i < sizeof(_buf); i++){
        _buf[i] = 0xff;
    }
}

We will call this function before each assert, so that the buffer is reinitialized every time. And a couple of simple tests:

void test_basic( void )
{
    _fill_with_0xff();
    itoae(_buf, 123, 0, 0, '0');
    TEST_ASSERT_EQUAL_STRING("123", _buf);
 
    _fill_with_0xff();
    itoae(_buf, -123, 0, 0, '0');
    TEST_ASSERT_EQUAL_STRING("-123", _buf);
}
void test_dpp( void )
{
    _fill_with_0xff();
    itoae(_buf, 123, 1, 0, '0');
    TEST_ASSERT_EQUAL_STRING("12.3", _buf);
 
    _fill_with_0xff();
    itoae(_buf, 123, 2, 0, '0');
    TEST_ASSERT_EQUAL_STRING("1.23", _buf);
 
    _fill_with_0xff();
    itoae(_buf, 123, 3, 0, '0');
    TEST_ASSERT_EQUAL_STRING("0.123", _buf);
 
    _fill_with_0xff();
    itoae(_buf, 123, 4, 0, '0');
    TEST_ASSERT_EQUAL_STRING("0.0123", _buf);
}

Run the tests:

-------------------
FAILED TEST SUMMARY
-------------------
[test_itoae.c]
  Test: test_dpp
  At line (127): "Expected '12.3' Was '12.3\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF\0xFF'"

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  4
PASSED:  3
FAILED:  1
IGNORED: 0

Oops! Something went wrong. It seems that although the dot was inserted to the string correctly, the terminating 0x00 wasn't moved to the right by 1 char.

After examining the source, I see that here's the offending piece of code:

        int i;
        for (i = 0; i < (dpp + 1/*null-terminate*/); i++){
            p_buf[len - i] = p_buf[len - i - 1];
        }
        p_buf[len - dpp] = '.';

Although I provided the handling of terminating null char, there is a traditional off-by-one error here. The correct code looks as follows:

        int i;
        for (i = 0; i < (dpp + 1/*null-terminate*/); i++){
            p_buf[len - i + 1] = p_buf[len - i];
        }
        p_buf[len - dpp] = '.';

Save, switch to the terminal, run the tests again:

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  4
PASSED:  4
FAILED:  0
IGNORED: 0

Cool! These very simple tests already saved me from the very silly mistake. Believe it or not, this mistake with itoae has actually happened to me, and tests helped to reveal it. This episode quickly encouraged me to invest my time in tests even more.

Note: not all tests code is included in the article, since it is very repetitive and straightforward. You can get everything done by using the prepared repository. Type there: git checkout v0.06.

Testing modules with dependencies

The application ADC module

Our application needs for some resourceful way to get current voltage on some particular channel, in Volts. It would be unwise to use our bsp_adc + adc_handler modules across the whole application.

Now, let's add the module appl_adc, which will have at least the function to get current voltage on some channel, in Volts.

The header src/appl/appl_adc.h should contain at least the following:

//-- for T_ADCHandler_Voltage
#include "adc_handler.h"
 
//-- for enum E_ApplAdcChannel
#include "appl_adc_channels.h"
 
/**
 * Initialize module
 */
void appl_adc__init(void);
 
/**
 * Get current voltage of the given channel.
 */
T_ADCHandler_Voltage appl_adc__voltage__get(enum E_ApplAdcChannel channel_num);

This function should call bsp_adc__value__get() for the given channel_num, then feed returned value to the appropriate adc_handler, and return resulting value in Volts.

For now, let's just fake the implementation (src/appl/appl_adc.c):

#include "appl_adc.h"
 
void appl_adc__init(void)
{
    //TODO
}
 
T_ADCHandler_Voltage appl_adc__voltage__get(enum E_ApplAdcChannel channel_num)
{
    //TODO
    return 0;
}

And get to tests: create new file test_appl_adc.c, and put the tests template in it. As usual, include the header of the module being tested:

//-- module being tested
#include "appl_adc.h"

Our setUp() and tearDown() are very simple:

void setUp(void)
{
    //-- before each test, re-initialize appl_adc module
    appl_adc__init();
}
 
void tearDown(void)
{
    //-- nothing to do here
}

And get to the test for appl_adc__voltage__get():

void test_voltage_get(void)
{
    // .....
}

As was mentioned before, appl_adc__voltage__get() first of all should call bsp_adc__value__get() for the appropriate channel. How do we test it?

The answer is - with CMock. And Ceedling will help us here as well: if we need to “mock” some module, all we need to do is to include the header of the module to mock, with the mock_ prefix (well, actually, the prefix is customizable in project.yml, and by default it is mock_).

Go on then: add the following include directive:

//-- mocked modules
#include "mock_bsp_adc.h"

Now, the module being tested will use mocked versions of all functions from bsp_adc.h, and we are provided with the “expect” functions. Let's see them in action:

void test_voltage_get(void)
{
    //-- We expect bsp_adc__value__get() to be called:
    bsp_adc__value__get_ExpectAndReturn(
            //-- the argument that is expected to be given to
            //   bsp_adc__value__get()
            APPL_ADC_CH__I_SETT,
 
            //-- and the value that bsp_adc__value__get() should return
            (0x3ff / 2)
            );
 
    //-- actually call the function being tested, that should perform
    //   all pending expected calls
    T_ADCHandler_Voltage voltage = appl_adc__voltage__get(
            APPL_ADC_CH__I_SETT
            );
 
    //-- check the voltage returned (we assume that adc_handler is initialized
    //   with the same params, where 0x3ff is the maximum ADC value, and
    //   it corresponds to the value (10 * ADC_HANDLER__SCALE_FACTOR__U))
    TEST_ASSERT_EQUAL_INT((5 * ADC_HANDLER__SCALE_FACTOR__U), voltage);
}

If we run the tests, we get the following result:

-------------------
FAILED TEST SUMMARY
-------------------
[test_appl_adc.c]
  Test: test_voltage_get
  At line (77): "Expected 500 Was 0"

So it complains that returned value was wrong. Okay, let's fake our dummy appl_adc__voltage__get() even more: make it return 500 instead of 0:

T_ADCHandler_Voltage appl_adc__voltage__get(enum E_ApplAdcChannel channel_num)
{
    //TODO
    return 500;
}

And run tests again:

-------------------
FAILED TEST SUMMARY
-------------------
[test_appl_adc.c]
  Test: test_voltage_get
  At line (56): "Function 'bsp_adc__value__get' called less times than expected."

Oh, cool! It reports that the function bsp_adc__value__get() was called less times than expected (effectively, it wasn't called at all). CMock in action!

Well, it's time to implement appl_adc__voltage__get() more or less completely: we're going to use both bsp_adc and adc_handler there.

appl_adc.c
/*******************************************************************************
 *    INCLUDED FILES
 ******************************************************************************/
 
#include "appl_adc.h"
#include "appl_adc_channels.h"
#include "bsp_adc.h"
 
 
/*******************************************************************************
 *    PRIVATE DATA
 ******************************************************************************/
 
static T_ADCHandler _adc_handlers[ APPL_ADC_CH_CNT ];
 
 
/*******************************************************************************
 *    PUBLIC FUNCTIONS
 ******************************************************************************/
 
void appl_adc__init(void)
{
    enum E_ApplAdcChannel channel;
 
    //-- init all ADC channels
    for (channel = 0; channel < APPL_ADC_CH_CNT; channel++){
        T_ADCHandler_CtorParams params = {};
 
        //-- here, we initialize all channels with the same params,
        //   but in real life different ADC channels may, of course,
        //   have different parameters.
        params.max_counts = 0x3ff;
        params.bsp_max_voltage = 10/*V*/ * ADC_HANDLER__SCALE_FACTOR__U;
        params.nominal_add_volts = 0/*V*/;
 
        //-- construct the ADC handler, saving the result to _ctor_result
        adc_handler__ctor(&_adc_handlers[ channel ], &params);
    }
}
 
T_ADCHandler_Voltage appl_adc__voltage__get(enum E_ApplAdcChannel channel_num)
{
    return adc_handler__voltage__get_by_counts_value(
            &_adc_handlers[ channel_num ],
            bsp_adc__value__get(channel_num)
            );
}

Ok, it seems, everything is right. Try to run tests:

Linking test_appl_adc.out...
build/test/out/appl_adc.o: In function 0097ppl_adc__init':
/home/dimon/projects/indicator_git/test_ceedling/../src/appl/appl_adc.c:70: undefined reference to 0097dc_handler__ctor'
build/test/out/appl_adc.o: In function 0097ppl_adc__voltage__get':
/home/dimon/projects/indicator_git/test_ceedling/../src/appl/appl_adc.c:76: undefined reference to 0097dc_handler__voltage__get_by_counts_value'
collect2: error: ld returned 1 exit status

....

NOTICE: If the linker reports missing symbols, the following may be to blame:
  1. Test lacks #include statements corresponding to needed source files.
  2. Project search paths do not contain source files corresponding to #include statements in the test.
  3. Test does not #include needed mocks.

Oh, dear. Linker complains about undefined reference to ADC handler functions. And Ceedling is being very kind here by providing us with useful notice: as it suggests, one of the possible reasons is that test lacks #include statements corresponding to needed source files. Do you remember that Ceedling examines the headers we include in our test file, and looks for the appropriate source file? So, to make it work, we should include the header adc_handler.h to our test_appl_adc.c:

//-- other modules that need to be compiled
#include "adc_handler.h"

Now, the tests should finally work:

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  7
PASSED:  7
FAILED:  0
IGNORED: 0

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.07.

More isolated test of the application ADC module

As you might have noticed, in the previous section we in fact ended up testing two modules at once: appl_adc.c and adc_handler.c. It doesn't seem right: at least, we already have dedicated tests for adc_handler.c, right? And, after all, even the term “unit test” suggests that we should test one unit at a time. Although sometimes I allow myself to break this rule, let's try to isolate tests for appl_adc.c as much as possible.

You might have already guessed that we're going to mock ADC handler as well, instead of using real code. So, first of all, let's remove #include "adc_handler.h", and include mocked version instead:

//-- mocked modules
#include "mock_bsp_adc.h"
#include "mock_adc_handler.h"

And now, we in fact have several options.

Ignore arguments given to ADC handler

The easiest option is to ignore arguments given to adc_handler__voltage__get_by_counts_value() whatsoever. That's what …_IgnoreAndReturn() functions are for (generated by CMock):

void test_voltage_get(void)
{
    //-- We expect bsp_adc__value__get() to be called:
    bsp_adc__value__get_ExpectAndReturn(
            //-- the argument that is expected to be given to
            //   bsp_adc__value__get()
            APPL_ADC_CH__I_SETT,
 
            //-- and the value that bsp_adc__value__get() should return
            123
            );
 
    //-- Expect call to adc_handler__voltage__get_by_counts_value(),
    //   ignoring arguments. The mocked version just returns 456.
    adc_handler__voltage__get_by_counts_value_IgnoreAndReturn(456);
 
    //-- actually call the function being tested, that should perform
    //   all pending expected calls
    T_ADCHandler_Voltage voltage = appl_adc__voltage__get(
            APPL_ADC_CH__I_SETT
            );
 
    //-- check the voltage returned (it should be 456 from the mock above)
    TEST_ASSERT_EQUAL_INT(456, voltage);
}

Run tests:

-------------------
FAILED TEST SUMMARY
-------------------
[test_appl_adc.c]
  Test: test_voltage_get
  At line (57): "Function 'adc_handler__ctor' called more times than expected."

Oh yes, we forgot that now we have to mock not only adc_handler__voltage__get_by_counts_value(), but the constructor as well, which is called from setUp(). We're going to ignore its arguments too, so, the modified setUp() looks as follows:

void setUp(void)
{
    //-- ADC handler constructor is going to be called for each channel.
    //   Mocked constructors all return ADC_HANDLER_RES__OK.
    enum E_ApplAdcChannel channel;
    for (channel = 0; channel < APPL_ADC_CH_CNT; channel++){
        adc_handler__ctor_IgnoreAndReturn(ADC_HANDLER_RES__OK);
    }
 
    //-- before each test, re-initialize appl_adc module
    appl_adc__init();
}

Now tests pass.

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.08.

Check arguments given to ADC handler

The arguments given to adc_handler__voltage__get_by_counts_value() are:

  • Pointer to an instance of T_ADCHandler;
  • Counts value to convert to Volts.

So, if we want to check arguments, we need to access T_ADCHandler for each particular channel, which is private to appl_adc.c. For this to be done, we have to create test-accompanying “protected” function in appl_adc that will return appropriate pointer:

/**
 * For usage in tests only!
 */
T_ADCHandler *_appl_adc__adc_handler__get(enum E_ApplAdcChannel channel)
{
    return &_adc_handlers[channel];
}

We won't include this function prototype in the header file appl_adc.h; instead, we will include it as the external function prototype to test_appl_adc.c:

/*******************************************************************************
 *    EXTERNAL FUNCTION PROTOTYPES
 ******************************************************************************/
 
extern T_ADCHandler *_appl_adc__adc_handler__get(enum E_ApplAdcChannel channel);

And now, instead of ignoring arguments given to adc_handler__voltage__get_by_counts_value(), we can use …_ExpectAndReturn() mock function:

    //-- Expect call to adc_handler__voltage__get_by_counts_value(),
    //   ignoring arguments. The mocked version just returns 456.
    adc_handler__voltage__get_by_counts_value_ExpectAndReturn(
            //-- pointer to the appropriate ADC handler instance
            _appl_adc__adc_handler__get(APPL_ADC_CH__I_SETT),
            //-- value returned from bsp_adc__value__get()
            123,
            //-- returned value in Volts
            456
            );

Notice that the value returned from mocked bsp_adc__value__get() (i.e. 123) matches the value given to adc_handler__voltage__get_by_counts_value(). And tests pass. If the value mismatches, tests will fail.

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.09.

Use mock with callback

Apart from easy-to-use helpers, CMock provides us with the very flexible callback helper. The callback should have the same signature as the mocked function, but it takes one additional argument: num_calls. When the function called first time, num_calls is 0, and it will be incremented by 1 with each subsequent call.

In the callback, we might check whatever we want, and if something goes wrong, we can call Unity macro TEST_FAIL_MESSAGE().

Let's implement such a callback:

/*******************************************************************************
 *    PRIVATE FUNCTIONS
 ******************************************************************************/
 
static T_ADCHandler_Voltage _get_by_counts_value_Callback(
        T_ADCHandler *me,
        T_ADCHandler_CountsValue counts_value,
        int num_calls
        )
{
    T_ADCHandler_Voltage ret = 0;
 
    switch (num_calls){
        case 0:
            if (counts_value != 123){
 
                //-- We can check whatever we want here. For example, we may
                //   check the data pointed to by "me", but NOTE that currently
                //   it is just zeros, since we have mocked adc_handler__ctor()
                //   as well, so the original constructor isn't called, and
                //   instances are left unitialized.
 
                TEST_FAIL_MESSAGE(
                        "adc_handler__voltage__get_by_counts_value() was called "
                        "with wrong counts_value"
                        );
            }
 
            ret = 456;
            break;
 
        default:
            TEST_FAIL_MESSAGE(
                    "adc_handler__voltage__get_by_counts_value() was called "
                    "too many times"
                    );
            break;
    }
 
    return ret;
}

And in our test_voltage_get(), we use it as follows:

    //-- Expect call to adc_handler__voltage__get_by_counts_value()
    adc_handler__voltage__get_by_counts_value_StubWithCallback(
            _get_by_counts_value_Callback
            );

Although callbacks like this don't look quite elegant, and for this particular example it is an unnecessary overkill, they are extremely flexible. So, keep it in your toolbox, and use when appropriate.

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.10.

Dealing with compiler-specific stuff

Compilers often have some useful non-standard built-in things. For example, the XC8 Microchip compiler has the function __builtin_software_breakpoint(), which, how its name suggests, puts software breakpoint. If the MCU runs in it with debugger attached, debugger halts execution. This function becomes available if we include the "xc.h" header.

I often use it for some conditions that should never happen. For example, our appl_adc__voltage__get() should never be called with wrong channel_num. Let's add a check for this:

#include "xc.h"
 
// .....
 
T_ADCHandler_Voltage appl_adc__voltage__get(enum E_ApplAdcChannel channel_num)
{
    T_ADCHandler_Voltage ret = 0;
 
    if (channel_num < APPL_ADC_CH_CNT){
        ret = adc_handler__voltage__get_by_counts_value(
                &_adc_handlers[ channel_num ],
                bsp_adc__value__get(channel_num)
                );
    } else {
        //-- illegal channel_num given: should never be here
        __builtin_software_breakpoint();
    }
 
    return ret;
}

Checks like this are a must have in any application, but if we try to run tests, we'll end up with the following error:

Compiling appl_adc.c...
../src/appl/appl_adc.c:15:16: fatal error: xc.h: No such file or directory
 #include "xc.h"

Obviously, GCC (which is used for tests by default) have neither such a built-in function, nor the “xc.h” header file.

We can address this problem by using the Ceedling “support” directory, which is located by default at test/support. Let's create the “xc.h” file in it, and put the following contents there:

xc.h
#ifndef _MY_XC_DUMMY
#define _MY_XC_DUMMY
 
void __builtin_software_breakpoint(void);
 
#endif // _MY_XC_DUMMY

If we run tests now, we'll have different error:

Linking test_appl_adc.out...
build/test/out/appl_adc.o: In function 0097ppl_adc__voltage__get':
/home/dimon/projects/indicator_git/test_ceedling/../src/appl/appl_adc.c:101: undefined reference to 0095_builtin_software_breakpoint'
collect2: error: ld returned 1 exit status

Nice: at least, our “xc.h” file is clearly used by appl_adc.c file, and now we need to provide actual implementation of __builtin_software_breakpoint(). The easiest way to do that is to mock it. So, add the following line to our test_appl_adc.c file:

#include "mock_xc.h"

Now, run tests, and they pass!

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  7
PASSED:  7
FAILED:  0
IGNORED: 0

And we can write one more test: let's check that __builtin_software_breakpoint() is called if we pass illegal channel number:

void test_voltage_get_wrong_channel_number(void)
{
    //-- we expect __builtin_software_breakpoint() to be called ...
    __builtin_software_breakpoint_Expect();
 
    //-- ... when we call appl_adc__voltage__get() with illegal
    //   channel number.
    appl_adc__voltage__get(
            APPL_ADC_CH_CNT
            );
}

And tests pass again:

--------------------
OVERALL TEST SUMMARY
--------------------
TESTED:  8
PASSED:  8
FAILED:  0
IGNORED: 0

You're encouraged to verify that if we remove a call to __builtin_software_breakpoint() in case of illegal channel number, tests will fail.

Note: you can get everything done by using the prepared repository. Type there: git checkout v0.12.

Some more notes about testing on the host machine

Testing on host machine is quite convenient: running tests is just a matter of a few keystrokes, tests run fast, and we get results almost immediately. But we should take some care, since the architectures are different.

As already discussed above, different compilers have some built-in functions. Apart from this, the memory alignment is often different: at 8-bit MCUs, the alignment is 1 byte, but on your host machine it's usually 4 or 8 bytes (depending on your architecture). So, we have to multi-target our applications.

I often find myself creating a file like my_crosscompiler.h, in which I declare some things depending on the compiler being used. For example, if I need some structure to be packed, I have to use compiler-specific attribute. So it may look like this:

#if defined(__XC8__)
//-- no need for "packed" attr on 8-bit MCU
#   define __MY_CROSS_ATTR_PACKED
#elif defined(__GNUC__)
#   define __MY_CROSS_ATTR_PACKED   __attribute__((packed))
#endif

And in the application code, I use the macro __MY_CROSS_ATTR_PACKED.

This way, we can write code that works both on target MCU as well as on the host machine. Of course, it takes additional effort and time, but so do tests in general. I spend a lot of time writing tests these days. It pays off very well.

Other testing ideas

Writing tests code is often considered as a tedious process, and I can't entirely disagree. However, I always encourage myself to find some new ways to test things, instead of repeatedly test this and that.

As an example, consider the EEPROM module (Electrically Erasable Programmable Read-Only Memory). We will most likely end up with MCU- or board-specific module bsp_eeprom which can just read plain data to and from specified addresses. As it is heavily hardware-dependent, we can't test it on the host machine.

In addition to bsp_eeprom, it's convenient to have application-dependent module like appl_eeprom, which should have functions to write or read some application entities to and from EEPROM. Of course, these application-dependent functions call bsp_eeprom__... functions underneath. For example, we might have the following functions to read/write the multiplier of each particular ADC channel:

int16_t appl_eeprom__adc_clb_mul__get(enum E_ApplAdcChannel channel);
void appl_eeprom__adc_clb_mul__set(enum E_ApplAdcChannel channel, int16_t mul);

If the application is rather large, there may be tons of functions like that. It is very tedious to test them all separately.

Instead, we may think about the most easy mistake to make. For things like the appl_eeprom module, it is the copy-paste mistake: when we have lots of similar functions (in fact, they all call the same bsp_eeprom functions, but for different addresses), it is easy to copy and paste from one function to another, and it is equally easy to forget to adjust the pasted code properly.

So, I often use the technique like this: define stub callbacks for bsp_eeprom functions, which just check if the given area is “clean”, and if it is, then fill this area with some predefined data (make it “dirty”). If the area is already “dirty”, then error is generated.

Then, perform “write” test: I call every “write” function from appl_eeprom module with all allowed arguments, and after that, the whole working region of EEPROM should be “dirty”, without holes. And, as I said before, each callback also checks whether the region it is going to write is clean. This way, we can easily eliminate these “copy-paste” problems: if some function writes to wrong area, we will end up with overwritten data and “holes”, which will be caught by our tests.

And, of course, exactly the same test should be done for “read” functions.

It is much more fun (and fast) than test each and every function separately, and in the end we'll have tests that are reliable enough.

Conclusion

The tools by guys from ThrowTheSwitch.org allow us to test our C code almost painlessly. Thank you, guys!

I hope this article helps you to get somewhat big picture about Ceedling and its companions, and I encourage you to examine the documentation, which is quite concise.

The easiest way to get documentation of all components in one place is to create new ceedling project by executing:

$ ceedling new my_project

And navigate to my_project/vendor/ceedling/docs directory, which contains several pdf files.

And again, if you feel serious about investing a great deal of time into testing your embedded designs (which is probably a good idea), consider reading the book Test Driven Development for Embedded C by James W. Grenning, which explains various testing methods, approaches and tools very thoroughly.

Let's write C code that doesn't suck!


You might as well be interested in:

Discuss on reddit:

Discussion

Dmitry Ulanov, 2015/10/06 13:00

For me a little bit complex way to start.

In our projects we using very simple approarch based on asserts ('ASSERT(clause)', 'ASSERT_EQL(el1, el2)'…) and integrate them directly with embedded code, which is called in Debug after start-up procedures in Microcontroller automatically.

For example a module with very simple switch-logic - https://goo.gl/0IZVXQ.

Dmitry Frank, 2015/10/06 16:21

Asserts like this and unit tests that I'm talking about in the article are entirely different things, and they are not mutually exclusive. Of course, I use asserts, and it doesn't prevent me from unit-testing my code as well.

derlo, 2016/02/07 00:10

Dmitry, code on your page https://goo.gl/0IZVXQ needs header file for those ASSERTS definition - where can I find them ?

Marek Radecki, 2015/11/24 10:52

Hey! This is a great tutorial. Just one thing that came to my mind while reading it. Don't you think that in itoae tests you should have used setUp to call _fill_with_0xff(). Repetition is a bad thing here. My eyes hurt.

Dmitry Frank, 2015/11/24 10:59

Thanks for the comment.

Well, I was thinking about it, but then, we'd end up with lots of 2-line test functions, and each one should be named. This is even worse I believe, so I implemented it as it is now.

Andre, 2016/01/24 09:35

Nice article, Dmitry.

But I agree with Marek, repetion smells bad. Therefore, I'd recommend the following:

static void test_itoae_dpp_expect(int dpp, const char *expected)
{
  _fill_with_0xff();
  itoae(_buf, 123, dpp, 0, '0');
  TEST_ASSERT_EQUAL_STRING(expected, _buf);
}
void test_dpp( void )
{
  test_itoae_dpp_expect(1, "12.3");
  test_itoae_dpp_expect(2, "1.23");
  test_itoae_dpp_expect(3, "0.123");
  test_itoae_dpp_expect(4, "0.0123");
} 

For _fill_with_0xff() I'd just use memset(_buf, 0xff, sizeof(_buf)); (from string.h)

One more (important) remark: leading underscores followed by uppercase letter are reserved for the compiler (_BUF), the same applies for all symbols with two underscores (appl_adc__init)

http://stackoverflow.com/questions/228783/what-are-the-rules-about-using-an-underscore-in-a-c-identifier

Best regards, Andre

Dmitry Frank, 2016/02/06 17:44

Hi Andre,

Thank you very much for your quite valuble remark about the underscores and uppercase letters! Learned something new today.

As to the avoiding repetition, yes, I admit that your solution is much better than mine. I'll try to find time and fix the article. Thanks again!

kek, 2015/12/09 11:14

may I propose that you get rid of all the above and create an SIMPLE (!!!) code with say one (2 the most!) SIMPLE functions in just a FEW source / header files. The way it is now is too complex to illustrate how that works - I quit ~ 15% down the track

Dmitry Frank, 2015/12/09 12:14

I'm afraid I won't get rid of all the above. I'm not sure I understand what's so hard in testing, for example, itoae module: no dependencies, just clear input and clear output.

And btw I've read some of “simple” tutorials before, and I was actually disappointed since I wanted more than just the plain basics. So eventually I figured it myself, and have written my own tutorial. If you want some tutorial that only touches basics, just get another one.

Matt Chernosky, 2015/12/15 05:09

Hi Dmitry,

Nice to see such a thorough tutorial! I've used Ceedling too, and I think it's great for testing embedded software (not that there are a whole lot of options). The documentation isn't always very helpful though.

I'm still not seeing unit testing used a lot in the embedded world. It's work like this that is going to help get embedded software out of the dark ages.

Matt

Dmitry Frank, 2015/12/15 17:15

Hi Matt, thanks for your feedback!

derlo, 2016/02/06 04:44

I tried it and the amount of time needed to setup and design tests is greater than the savings in later testing. All is very complex and often confusing. And because of that there is a high chance that the 'tests themselves' need to be verified/tested as well. So I would stay with the old way of testing.

Dmitry Frank, 2016/02/07 09:36
I tried it and the amount of time needed to setup and design tests is greater than the savings in later testing.

Hmm, we must have different notions of what is large amount of time, especially about setup. Setup basically is:

$ gem install ceedling
$ ceedling new test_ceedling

And in project.yml, adjust paths → source.

Large amount of time, huh? :)

Anyway, it's not at all a surprise that the people that got used to the “old way of testing” find learning formal tests an unnecessary effort. Up to you, of course.

Matt Chernosky, 2016/02/29 05:00

Ceedling makes it so easy to add new tests! Just add functions starting with test_ to the module test file… and the tests are automatically discovered and run.

And, auto-generate mocks from .h files with a single #include “mock_ “!

If it's difficult to write the tests, I'd recommend trying the test-driven (TDD) approach. It's a different mindset, but it has so many advantages. It really forces you to think about how each independent unit is going to be used.

embedded beginer, 2016/03/13 09:32

Hi Dmitry,

Thanks for the great tutorial. do you have any idea how to setup the project with eclipse. I tried to build it in eclipse but i've got the following error: Cannot run program “rake”: Launching failed

Dmitry Frank, 2016/03/13 09:35

Hi, thanks for the comment.

I never tried to run it from Eclipse, so I'm afraid I won't be able to help. However, the error message suggests that you don't have rake binary installed. Do you?

Kirill, 2016/03/19 19:16

If you are using Windows, then “rake” command is in fact a bat-file “rake.bat”. So, try in your eclipse settings to change “rake” into “rake.bat”.

hmijail, 2016/04/12 19:38

I agree that some streamlining would be good, but this is the only acceptable “docs” I have found for Ceedling & friends. The disaster that is the official documentation has managed to send me a couple of times to reassess CMocka and Cpputest…

… which is a pity, given how powerful the whole set looks like once one manages to get the full picture.

Nitin Sinha, 2016/12/09 09:50

I started with Unity and was successful in covering the basics. Now when i jumped onto cmock i had a load of promlems on my door.

1. Everytime i mocked my header file(Func.h) with default settings… i was able to generate mock files. With only Func_Expectandreturn(int, int). In case to generate other function with :Cexception, :Ignore, :Array, etc i was unable to do so. i didn't much relevant data in the internet as well.

2. from the cmock doc folder, i tried doing some thing like this: ruby cmock.rb –mock_path=“[myPath]\mocks” –plugin=:Cexception,:Ignore Func.h This command in the prompt generated mocks but only with default configuration. i.e all i had was Func_Expectandreturn(int, int) thats it. Not Func_ExpectAndThrow.

3. where have i missed, am i suppose to include Cexception/src .h file somewhere? if yes, then where.

4. Also tried rakefile: — load 'cmock.rb' cmock = CMock.new(:plugins ⇒ [:cexception, :ignore], :mock_path ⇒ '[PATH]\mocks') — this also didn't work. ended up with default mocking as above.

Please Help me.

Nitin Sinha India

T, 2017/01/17 13:00

Thanks Dmitry, this post has helped me numerous times - I keep returning to it after my Unity unit test knowledge has advanced further and it always helps fill in a gap or two! T

Dmitry Frank, 2017/01/17 14:15

Hi, thanks for you comment! I'm happy to help! :)

Alexandru Balmus, 2017/02/23 16:26

Hi. The Test-Driven Development for Embedded C ebook by James Grenning can be bought DRM free(!) from The Pragmatic Bookshelf website: https://pragprog.com/book/jgade/test-driven-development-for-embedded-c. They also send updates to the books into your Dropbox and/or your email account. Thanks for this great article.

Dmitry Frank, 2017/02/28 18:42

Cool, thanks for the tip!

Rohan Majumdar, 2017/06/22 12:12

Is there a way to do data driven testing with this framework.

And after comparing this whole setup with Cunit I found ceedling very nice.

The modular structure of this framework is awesome and something unheard of in the crass and crude embedded domain.

Kudos!!!

Enter your comment (please, English only). Wiki syntax is allowed:
     __  _____  ____  __  __   __ __
 __ / / / ___/ /_  / / / / /  / //_/
/ // / / /__    / /_/ /_/ /  / ,<   
\___/  \___/   /___/\____/  /_/|_|
 
articles/unit_testing_embedded_c_applications.txt · Last modified: 2015/10/05 22:16 by dfrank
Driven by DokuWiki Recent changes RSS feed Valid CSS Valid XHTML 1.0