Related
I have an app that generates a dependencies.dot file which I then want to convert to an SVG image.
When I do that from a simple application like so:
int main(int argc, char * argv[])
{
system("dot -Tsvg ../BUILD/dependencies.dot > ../BUILD/dependencies.svg");
return 0;
}
It works great. The SVG image is present and working.
When I instead run it from my Qt application, the SVG file is created (by the shell), but it remains empty.
Any idea what could prevent dot from outputting data to its stdout?
Just in case, I also tried a cat to send the input through stdin instead of a filename:
system("cat ../BUILD/dependencies.dot | dot -Tsvg > ../BUILD/dependencies.svg");
And that didn't make any differences.
Also using the full path (/usr/bin/dot) did not help either.
Another test, I tried to use popen() and the first fread() immediately returns 0 (i.e. the mark of EOF).
It may not be Qt, but something is interacting with dot's ability to do anything. Any pointers on why that is would be wonderful.
Maybe an important note? I start my app. from a console, so stdin, stdout and stderr should all work as expected. I actually can see debug logs appearing there and other apps seem to work just as expected (i.e. My Qt app. can successfully run make, for example).
Here is an example of the resulting SVG (when I don't run it from within my Qt app):
For reference, the source code can be found on github. This is part of the snapbuilder. A tool that I use to run a build on launchpad. It's still incomplete, but it's getting there.
https://github.com/m2osw/snapcpp/tree/master/snapbuilder
The specific function to look for: project::generate_svg().
I still have no clue what side effect Qt has on system() that the dot command would fail. However, if using my own fork() + execve(), then it works.
I wanted a new process class for my environment, so I implemented that. This newer version is using FIFOs or directly opening closing files that it passes to the process.
...
// write the script in `std::stringstream dot` then:
//
std::string script(dot.str());
g_dot_process = std::make_shared<cppprocess::process>("dependencies");
g_dot_process->set_command("/usr/bin/dot");
g_dot_process->add_argument("-Tsvg");
g_dot_process->add_input(cppprocess::buffer_t(script.data(),
script.data() + script.length()));
g_dot_process->set_capture_output();
g_dot_process->set_output_capture_done(output_captured);
g_dot_process->start(); // TODO: check return value for errors
...
// and in output_captured()
//
void snap_builder::svg_ready(std::string const & svg)
{
std::string const svg_filename(...);
{
std::ofstream out;
out.open(svg_filename);
if(!out.is_open())
{
std::cerr << "error: \n";
return;
}
out.write(svg.c_str(), svg.size());
}
dependency_tree->load(QString::fromUtf8(svg_filename.c_str()));
}
Now the dot file is generated and displayed as expected.
This is rather strange since most everything else I've done with a simple system() call works as expected. There must be something about stdin or stdout that makes dot not do its work.
I am using IAR EWARM's cspybat to run some unit tests for my embedded code using Unity. I would like an easy way for my build server to determine if the unit tests passed or failed. Is there a way for CSPY to return a nonzero error code if my unit tests fail? I have tried changing the return value in main() with no change. Is there a function I can call to force an error to be returned?
My cspybat batch file looks like this:
"C:\Program Files (x86)\IAR Systems\Embedded Workbench 7.4\common\bin\cspybat" -f "C:\Work\Sandbox\ST\stmicroeval\_iar_ewarm_project\settings\Project.UnitTest.general.xcl" --backend -f "C:\Work\Sandbox\ST\stmicroeval\_iar_ewarm_project\settings\Project.UnitTest.driver.xcl"
Unfortunately, no.
I've solved this by replacing "exit" with a function that prints a specific pattern, plus the exit code. I then wrapped the call to cspybat into a script that 1) strips the output of the extra output and 2) exits with the desired exit code.
It's late 2020 and they still don't offer a mechanism to do this.
We solved it by including a macro file with the contents:
execUserExit()
{
__message "program exited with __exit_value = ", __exit_value:%d ;
}
And having our own exit variable in the code:
extern "C" int __exit_value=0xff;
That we set prior to calling exit() (though you could just write your own version of exit())
This makes the debugger always print SOMETHING, even if the program crashes on startup.
Then we parse with a python wrapper:
pattern = "__exit_value =\s([\-|0-9|a-f|A-F|x]*)"
retvalue = int(re.findall(pattern,process.stdout)[0])
I am able to use cmocka and getting default results on the screen. I want to get the results for the unit test in JUnit format.
CMocka supports JUnit format by using the env variable CMOCKA_MESSAGE_OUTPUT or using API cmocka_set_message_output(CM_OUTPUT_XML);
But still no xml file gets generated. Can anyone help out in obtaining results in JUnit format?
The most credible source is actually the cmocka.c source file that contains entire implementation of the framework. This file is not too large, so I will cite something from the source file version 1.0.1.
There are two conditions to generate XML output by cmocka and the third condition is needed to store output in file.
1. XML output can be generated only if tests are called by cmocka_run_group_tests()
The customizable output format can be obtained only from the test runner cmocka_run_group_tests() or from its full variation cmocka_run_group_tests_name().
There is no other route that can lead to XML output. If a singe test is started by run_test() the output cannot be XML.
The summary format
[ PASSED ] 0 test(s).
[ FAILED ] 1 test(s), listed below:
can be generated in one of the following possible cases:
the test is started by one of the deprecated test runners: run_tests(), _run_tests() or run_group_tests(), _run_group_tests(); in that case it is even possible to see compilation warning about usage of a deprecated function;
the test is started by cmocka_run_group_tests() and the output format is CM_OUTPUT_STDOUT.
2. cmocka message output should be set to CM_OUTPUT_XML
The default output format can be set by calling cmocka_set_message_output(CM_OUTPUT_XML) before running tests. However, even if such default is set in the test source it can be overwritten by the environment variable CMOCKA_MESSAGE_OUTPUT. That variable has higher priority than the default set by cmocka_set_message_output().
The value of CMOCKA_MESSAGE_OUTPUT is case insensitive. The variable is taken into account if it is equal to one of the following values: stdout, subunit, tab or xml.
So, if the environment variable has value stdout the function mocka_set_message_output() has no effect.
That variable can be used to force different output formats of already compiled binary:
CMOCKA_MESSAGE_OUTPUT=stdout ./nulltest
CMOCKA_MESSAGE_OUTPUT=subunit ./nulltest
CMOCKA_MESSAGE_OUTPUT=tap ./nulltest
CMOCKA_MESSAGE_OUTPUT=xml ./nulltest
Thus, if the test is started by cmocka_run_group_tests() but the output is not affected by mocka_set_message_output() it means that there is set variable CMOCKA_MESSAGE_OUTPUT=stdout in the shell.
3. It should be possible to create a new file by cmocka to write its XML output directly to that file
If both previous conditions are satisfied, it is possible to ask cmocka to write its XML output directly to a file. If the environment variable CMOCKA_XML_FILE is set, then cmocka will try to write XML to file with name of that variable value.
Usage example:
CMOCKA_XML_FILE='./out.xml' CMOCKA_MESSAGE_OUTPUT=xml ./nulltest
The file is written if:
file with such name does not exist;
such file can be created.
Thus if there are more than one test runners in one compiled binary test application, only the first runner can write its output to that file.
The output is written to the shell even if CMOCKA_XML_FILE is set, but the file already exists or it cannot be created.
Of course, it is possible just to redirect shell output to a file overwriting existent file or appending to existent file if such file exists.
The example bellow can be used to check different options for output. It can be built by the command
gcc -g nulltest.c -o nulltest -Ipath_to_cmocka_headers -Lpath_to_cmocka_library_binary -lcmocka
nulltest.c
#include <stdarg.h>
#include <stddef.h>
#include <setjmp.h>
#include <cmocka.h>
/* A test case that fails. */
static void null_test_failed(void **state) {
(void) state; /* unused */
assert_int_equal(0, 1);
}
int main(void) {
const struct CMUnitTest tests[] = {
cmocka_unit_test(null_test_failed),
};
const struct UnitTest tests_deprecated[] = {
unit_test(null_test_failed),
};
cmocka_set_message_output(CM_OUTPUT_XML);
/* group test functions that use customizable output format */
cmocka_run_group_tests(tests, NULL, NULL);
cmocka_run_group_tests_name("custom group name", tests, NULL, NULL);
/* run single test with standard output */
run_test(null_test_failed);
/* DEPRECATED TEST RUNNER functions that can give only standard output */
run_tests(tests_deprecated);
_run_tests(tests_deprecated, 1);
run_group_tests(tests_deprecated);
_run_group_tests(tests_deprecated, 1);
return 0;
}
The XML is printed to stdout, you need to redirect it to a file ...
I'd like to be able to unit test my Arduino code. Ideally, I would be able to run any tests without having to upload the code to the Arduino. What tools or libraries can help me with this?
There is an Arduino emulator in development which could be useful, but it doesn't yet seem to be ready for use.
AVR Studio from Atmel contains a chip simulator which could be useful, but I can't see how I would use it in conjunction with the Arduino IDE.
Don't Run Unit Tests on the Arduino Device or Emulator
The case against microcontroller Device/Emulator/Sim-based tests
There's a lot of discussion about what unit test means and I'm not
really trying to make an argument about that here. This post is not
telling you to avoid all practical testing on your ultimate target
hardware. I am trying to make a point about optimizing your
development feedback cycle by eliminating your target hardware from
your most mundane and frequent tests. The units under test are assumed
to be much smaller than the whole project.
The purpose of unit testing is to test the quality of your own code. Unit tests should generally never test the functionality of factors outside of your control.
Think about it this way: Even if you were to test functionality of the Arduino library, the microcontroller hardware, or an emulator, it is absolutely impossible for such test results to tell you anything about the quality of your own work. Hence, it is far more valuable and efficient to write unit tests that do not run on the target device (or emulator).
Frequent testing on your target hardware has a painfully slow cycle:
Tweak your code
Compile and upload to Arduino device
Observe behavior and guess whether your code is doing what you expect
Repeat
Step 3 is particularly nasty if you expect to get diagnostic messages via serial port but your project itself needs to use your Arduino's only hardware serial port. If you were thinking that the SoftwareSerial library might help, you should know that doing so is likely to disrupt any functionality that requires accurate timing like generating other signals at the same time. This problem has happened to me.
Again, if you were to test your sketch using an emulator and your time-critical routines ran perfectly until you uploaded to the actual Arduino, then the only lesson you're going to learn is that the emulator is flawed--and knowing this still reveals nothing about the quality of your own work.
If it's silly to test on the device or emulator, what should I do?
You're probably using a computer to work on your Arduino project. That computer is orders of magnitudes faster than the microcontroller. Write the tests to build and run on your computer.
Remember, the behavior of the Arduino library and microcontroller should be assumed to be either correct or at least consistently incorrect.
When your tests produce output contrary to your expectations, then you likely have a flaw in your code that was tested. If your test output matches your expectations, but the program does not behave correctly when you upload it to the Arduino, then you know that your tests were based on incorrect assumptions and you likely have a flawed test. In either case, you will have been given real insights on what your next code changes should be. The quality of your feedback is improved from "something is broken" to "this specific code is broken".
How to Build and Run Tests on Your PC
The first thing you need to do is identify your testing goals. Think about what parts of your own code you want to test and then make sure to construct your program in such a way that you can isolate discrete parts for testing.
If the parts that you want to test call any Arduino functions, you will need to provide mock-up replacements in your test program. This is much less work than it seems. Your mock-ups don't have to actually do anything but providing predictable input and output for your tests.
Any of your own code that you intend to test needs to exist in source files other than the .pde sketch. Don't worry, your sketch will still compile even with some source code outside of the sketch. When you really get down to it, little more than your program's normal entry point should be defined in the sketch file.
All that remains is to write the actual tests and then compile it using your favorite C++ compiler! This is probably best illustrated with a real world example.
An actual working example
One of my pet projects found here has some simple tests that run on the PC. For this answer submission, I'll just go over how I mocked-up some of Arduino library functions and the tests I wrote to test those mock-ups. This is not contrary to what I said before about not testing other people's code because I was the one who wrote the mock-ups. I wanted to be very certain that my mock-ups were correct.
Source of mock_arduino.cpp, which contains code that duplicates some support functionality provided by the Arduino library:
#include <sys/timeb.h>
#include "mock_arduino.h"
timeb t_start;
unsigned long millis() {
timeb t_now;
ftime(&t_now);
return (t_now.time - t_start.time) * 1000 + (t_now.millitm - t_start.millitm);
}
void delay( unsigned long ms ) {
unsigned long start = millis();
while(millis() - start < ms){}
}
void initialize_mock_arduino() {
ftime(&t_start);
}
I use the following mock-up to produce readable output when my code writes binary data to the hardware serial device.
fake_serial.h
#include <iostream>
class FakeSerial {
public:
void begin(unsigned long);
void end();
size_t write(const unsigned char*, size_t);
};
extern FakeSerial Serial;
fake_serial.cpp
#include <cstring>
#include <iostream>
#include <iomanip>
#include "fake_serial.h"
void FakeSerial::begin(unsigned long speed) {
return;
}
void FakeSerial::end() {
return;
}
size_t FakeSerial::write( const unsigned char buf[], size_t size ) {
using namespace std;
ios_base::fmtflags oldFlags = cout.flags();
streamsize oldPrec = cout.precision();
char oldFill = cout.fill();
cout << "Serial::write: ";
cout << internal << setfill('0');
for( unsigned int i = 0; i < size; i++ ){
cout << setw(2) << hex << (unsigned int)buf[i] << " ";
}
cout << endl;
cout.flags(oldFlags);
cout.precision(oldPrec);
cout.fill(oldFill);
return size;
}
FakeSerial Serial;
and finally, the actual test program:
#include "mock_arduino.h"
using namespace std;
void millis_test() {
unsigned long start = millis();
cout << "millis() test start: " << start << endl;
while( millis() - start < 10000 ) {
cout << millis() << endl;
sleep(1);
}
unsigned long end = millis();
cout << "End of test - duration: " << end - start << "ms" << endl;
}
void delay_test() {
unsigned long start = millis();
cout << "delay() test start: " << start << endl;
while( millis() - start < 10000 ) {
cout << millis() << endl;
delay(250);
}
unsigned long end = millis();
cout << "End of test - duration: " << end - start << "ms" << endl;
}
void run_tests() {
millis_test();
delay_test();
}
int main(int argc, char **argv){
initialize_mock_arduino();
run_tests();
}
This post is long enough, so please refer to my project on GitHub to see some more test cases in action. I keep my works-in-progress in branches other than master, so check those branches for extra tests, too.
I chose to write my own lightweight test routines, but more robust unit-test frameworks like CppUnit are also available.
In the absence of any pre-existing unit test frameworks for Arduino, I have created ArduinoUnit. Here's a simple Arduino sketch demonstrating its use:
#include <ArduinoUnit.h>
// Create test suite
TestSuite suite;
void setup() {
Serial.begin(9600);
}
// Create a test called 'addition' in the test suite
test(addition) {
assertEquals(3, 1 + 2);
}
void loop() {
// Run test suite, printing results to the serial port
suite.run();
}
I have considerable success unit testing my PIC code by abstracting out the hardware access and mocking it in my tests.
For example, I abstract PORTA with
#define SetPortA(v) {PORTA = v;}
Then SetPortA can easily be mocked, without adding overhead code in the PIC version.
Once the hardware abstraction has been tested a while I soon find that generally code goes from the test rig to the PIC and works first time.
Update:
I use a #include seam for the unit code, #including the unit code in a C++ file for the test rig, and a C file for the target code.
As an example I want to multiplex four 7 segment displays, one port driving the segments and a second selecting the display. The display code interfaces with the displays via SetSegmentData(char) and SetDisplay(char). I can mock these in my C++ test rig and check that I get the data I expect. For the target I use #define so that I get a direct assignment without the overhead of a function call
#define SetSegmentData(x) {PORTA = x;}
It seems that emulino would do the job perfectly.
Emulino is an emulator for the Arduino platform by Greg Hewgill. (Source)
GitHub repository
simavr is an AVR simulator using avr-gcc.
It already supports a few ATTiny and ATMega microcontrollers, and - according to the author - it's easy to add some more.
In the examples lies simduino, an Arduino emulator. It supports running the Arduino bootloader and can be programmed with avrdude through Socat (a modified Netcat).
You can unit test in Python with my project, PySimAVR. Arscons is used for building and simavr for simulation.
Example:
from pysimavr.sim import ArduinoSim
def test_atmega88():
mcu = 'atmega88'
snippet = 'Serial.print("hello");'
output = ArduinoSim(snippet=snippet, mcu=mcu, timespan=0.01).get_serial()
assert output == 'hello'
Start test:
$ nosetests pysimavr/examples/test_example.py
pysimavr.examples.test_example.test_atmega88 ... ok
I built arduino_ci for this purpose. Although it's limited to testing Arduino libraries (and not standalone sketches), it enables unit tests to be run either locally or on a CI system (like Travis CI or Appveyor).
Consider a very simple library in your Arduino Library directory, called DoSomething, with do-something.cpp:
#include <Arduino.h>
#include "do-something.h"
int doSomething(void) {
return 4;
};
You'd unit test it as follows (with a test file called test/is_four.cpp or some such):
#include <ArduinoUnitTests.h>
#include "../do-something.h"
unittest(library_does_something)
{
assertEqual(4, doSomething());
}
unittest_main() // this is a macro for main(). just go with it.
That's all. If that assertEqual syntax and test structure looks familiar, it's because I adopted some of Matthew Murdoch's ArduinoUnit library
that he referred to in his answer.
See Reference.md for more information about unit testing I/O pins, the clock, Serial ports, etc.
These unit tests are compiled and run using a script contained in a ruby gem. For examples of how to set that up, see the README.md or just copy from one of these examples:
A practical example, testing a Queue implementation
Another set of tests on another Queue project
A complex example, simulating a library that controls an interactive device over a SoftwareSerial connection as part of the Adafruit FONA library
The DoSomething example library shown above, used to test arduino_ci itself
I am not aware of any platform which can test Arduino code.
However, there is the Fritzing platform, which you can use to model the hardware and later on export PCB diagrams and stuff.
Worth checking.
We are using Arduino boards for data acquisition in a large scientific experiment. Subsequently, we have to support several Arduino boards with different implementations. I wrote Python utilities to dynamically load Arduino hex images during unit testing. The code found on the link below supports Windows and Mac OS X via a configuration file. To find out where your hex images are placed by the Arduino IDE, hit the shift key before you hit the build (play) button. Hit the shift key while hitting upload to find out where your avrdude (command line upload utility) is located on your system / version of Arduino. Alternatively, you can look at the included configuration files and use your install location (currently on Arduino 0020).
http://github.com/toddstavish/Python-Arduino-Unit-Testing
This program allows automated running of several Arduino unit tests. The testing process is started on the PC but the tests run on the actual Arduino hardware. One set of unit tests is typically used to test one Arduino library.
(this
Arduino Forum: http://arduino.cc/forum/index.php?topic=140027.0
GitHub project page: http://jeroendoggen.github.com/Arduino-TestSuite
Page in the Python Package Index: http://pypi.python.org/pypi/arduino_testsuite
The unit tests are written with the "Arduino Unit Testing Library": http://code.google.com/p/arduinounit
The following steps are performed for each set of unit tests:
Read the config file to find out which tests to run
The script compiles and uploads an Arduino sketch that contains the unit testing code.
The unit tests are run on the Arduino board.
The results of the test are printed over the serial port and analyzed by the Python script.
The script starts the next test, repeating the above steps for all test that are requested in the configuration file.
The script prints a summary showing an overview of all the failed/passed tests in the complete testsuite.
Keep hardware-specific code separate or abstracted away from the rest so you can test and debug that bigger "rest" on any platform for which you have good tools and with which you're familiar most.
Basically, try to build as much of the final code from as many known-to-work building blocks as possible. The remaining hardware-specific work will then be much easier and faster. You may finish it by using existing emulators and/or emulating devices on your own. And then, of course, you'll need to test the real thing somehow. Depending on circumstances, that may or may not be very well automatable (i.e. who or what will press buttons and provide other inputs? who or what will observe and interpret various indicators and outputs?).
James W. Grenning writes great books and this one is about unit testing embedded C code Test Driven Development for Embedded C.
I am using Searduino when writing Arduino code. Searduino is an Arduino simulator and a development environment (Makefiles, C code ...) that makes it easy to hack in C/C++ using your favorite editor. You can import Arduino sketches and run them in the simulator.
Screenshot of Searduino 0.8: http://searduino.files.wordpress.com/2014/01/jearduino-0-8.png
Searduino 0.9 will be released and a video will be recorded as soon as the lasts tests are done .... in a day or two.
Testing on the simulator is not to be considered as real tests, but it certainly have helped me a lot in finding stupid/logical mistakes (forgetting to do pinMode(xx, OUTPUT), etc.).
BTW: I am one of the people developing Searduino.
There is a project called ncore, which provides native core for Arduino. And allows you to write tests for Arduino code.
From the project description
The native core allows you to compile and run Arduino sketches on the
PC, generally with no modification. It provides native versions of
standard Arduino functions, and a command-line interepreter to give
inputs to your sketch that would normally come from the hardware
itself.
Also on the "what do I need to use it" section
If you want to build the tests, you'll need cxxtest from
http://cxxtest.tigris.org. NCORE has been tested with cxxtest 3.10.1.
If you want to unit-test code outside MCU (on desktop), check out libcheck:
https://libcheck.github.io/check/
I used it to test my own embedded code few times. It's pretty robust framework.
You can use emulare — you can drag and drop a microcontroller on a diagram and run your code in Eclipse. The documentation on the website tells you how to set it up.
Use Proteus VSM with an Arduino library to debug your code or to test it.
It is a best practice before getting your code onboard, but be sure with timings because the simulation does not run realtime as they run on the board.
Try Autodesk circuit simulator. It allows to test Arduino code and circuits with many other hardware components.
In basic Arduino is written with C and C++, even libraries of arduino are written in C and C++. So,in simple terms just handle the code as C and C++ and try doing the unit testing. Here, by the word "handle" I mean you to change all the basic syntax like serial.println to sysout, pinmode to varaibles, void loop to while() loop which breaks either in keystock or after some iteration.
I know this is little a long process and not so straight forward.On my personal experience, once you get to do with it, this turns to be more reliable.
-Nandha_Frost
In case you are interested in running an INO sketch and checkout the serial output, I have a working implementation of that in my Arduino NMEA checksum project.
The following script takes the file and uses Arduino CLI to compile it to a HEX file which is then loaded to SimAVR which evaluates it and prints the serial output. Since all Arduino programs run forever without really having an option of killing themselves (exit(0) doesn't work), I let the sketch run for a few seconds and then diff the captured output with expected output.
Download and extract Arduino CLI (in this case version 0.5.0 - latest at the time of writing):
curl -L https://github.com/arduino/arduino-cli/releases/download/0.5.0/arduino-cli_0.5.0_Linux_64bit.tar.gz -o arduino-cli.tar.gz
tar -xvzf arduino-cli.tar.gz
Now you can update the index and install the appropriate core:
./arduino-cli core update-index
./arduino-cli core install arduino:avr
Assuming your sketch is named nmea-checksum.ino, to get ELF and HEX, run:
./arduino-cli compile -b arduino:avr:uno nmea-checksum.ino
Next up, SimAVR to run the HEX (or ELF) - I build from source because the latest release didn't work for me:
sudo apt-get update
sudo apt-get install -y build-essential libelf-dev avr-libc gcc-avr freeglut3-dev libncurses5-dev pkg-config
git clone https://github.com/buserror/simavr.git
cd simavr
make
Successful compilation will give you simavr/run_avr which you can use to run the sketch. Like I said, timeout it otherwise it will never terminate:
cd simavr
timeout 10 ./run_avr -m atmega168 -f 16000000 ../../nmea-checksum.ino.arduino.avr.uno.elf &> nmea-checksum.ino.clog || true
The generated file will have ANSI color code control characters wrapping the serial output, to get rid of those:
cat nmea-checksum.ino.clog | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" > nmea-checksum.ino.log
cat nmea-checksum.ino.log
Now all you need to do is compared this file to a known good file:
diff nmea-checksum.ino.log ../../nmea-checksum.ino.test
If there are no differences, diff will exit with code 0, otherwise the script will fail.
I'd like to be able to unit test my Arduino code. Ideally, I would be able to run any tests without having to upload the code to the Arduino. What tools or libraries can help me with this?
There is an Arduino emulator in development which could be useful, but it doesn't yet seem to be ready for use.
AVR Studio from Atmel contains a chip simulator which could be useful, but I can't see how I would use it in conjunction with the Arduino IDE.
Don't Run Unit Tests on the Arduino Device or Emulator
The case against microcontroller Device/Emulator/Sim-based tests
There's a lot of discussion about what unit test means and I'm not
really trying to make an argument about that here. This post is not
telling you to avoid all practical testing on your ultimate target
hardware. I am trying to make a point about optimizing your
development feedback cycle by eliminating your target hardware from
your most mundane and frequent tests. The units under test are assumed
to be much smaller than the whole project.
The purpose of unit testing is to test the quality of your own code. Unit tests should generally never test the functionality of factors outside of your control.
Think about it this way: Even if you were to test functionality of the Arduino library, the microcontroller hardware, or an emulator, it is absolutely impossible for such test results to tell you anything about the quality of your own work. Hence, it is far more valuable and efficient to write unit tests that do not run on the target device (or emulator).
Frequent testing on your target hardware has a painfully slow cycle:
Tweak your code
Compile and upload to Arduino device
Observe behavior and guess whether your code is doing what you expect
Repeat
Step 3 is particularly nasty if you expect to get diagnostic messages via serial port but your project itself needs to use your Arduino's only hardware serial port. If you were thinking that the SoftwareSerial library might help, you should know that doing so is likely to disrupt any functionality that requires accurate timing like generating other signals at the same time. This problem has happened to me.
Again, if you were to test your sketch using an emulator and your time-critical routines ran perfectly until you uploaded to the actual Arduino, then the only lesson you're going to learn is that the emulator is flawed--and knowing this still reveals nothing about the quality of your own work.
If it's silly to test on the device or emulator, what should I do?
You're probably using a computer to work on your Arduino project. That computer is orders of magnitudes faster than the microcontroller. Write the tests to build and run on your computer.
Remember, the behavior of the Arduino library and microcontroller should be assumed to be either correct or at least consistently incorrect.
When your tests produce output contrary to your expectations, then you likely have a flaw in your code that was tested. If your test output matches your expectations, but the program does not behave correctly when you upload it to the Arduino, then you know that your tests were based on incorrect assumptions and you likely have a flawed test. In either case, you will have been given real insights on what your next code changes should be. The quality of your feedback is improved from "something is broken" to "this specific code is broken".
How to Build and Run Tests on Your PC
The first thing you need to do is identify your testing goals. Think about what parts of your own code you want to test and then make sure to construct your program in such a way that you can isolate discrete parts for testing.
If the parts that you want to test call any Arduino functions, you will need to provide mock-up replacements in your test program. This is much less work than it seems. Your mock-ups don't have to actually do anything but providing predictable input and output for your tests.
Any of your own code that you intend to test needs to exist in source files other than the .pde sketch. Don't worry, your sketch will still compile even with some source code outside of the sketch. When you really get down to it, little more than your program's normal entry point should be defined in the sketch file.
All that remains is to write the actual tests and then compile it using your favorite C++ compiler! This is probably best illustrated with a real world example.
An actual working example
One of my pet projects found here has some simple tests that run on the PC. For this answer submission, I'll just go over how I mocked-up some of Arduino library functions and the tests I wrote to test those mock-ups. This is not contrary to what I said before about not testing other people's code because I was the one who wrote the mock-ups. I wanted to be very certain that my mock-ups were correct.
Source of mock_arduino.cpp, which contains code that duplicates some support functionality provided by the Arduino library:
#include <sys/timeb.h>
#include "mock_arduino.h"
timeb t_start;
unsigned long millis() {
timeb t_now;
ftime(&t_now);
return (t_now.time - t_start.time) * 1000 + (t_now.millitm - t_start.millitm);
}
void delay( unsigned long ms ) {
unsigned long start = millis();
while(millis() - start < ms){}
}
void initialize_mock_arduino() {
ftime(&t_start);
}
I use the following mock-up to produce readable output when my code writes binary data to the hardware serial device.
fake_serial.h
#include <iostream>
class FakeSerial {
public:
void begin(unsigned long);
void end();
size_t write(const unsigned char*, size_t);
};
extern FakeSerial Serial;
fake_serial.cpp
#include <cstring>
#include <iostream>
#include <iomanip>
#include "fake_serial.h"
void FakeSerial::begin(unsigned long speed) {
return;
}
void FakeSerial::end() {
return;
}
size_t FakeSerial::write( const unsigned char buf[], size_t size ) {
using namespace std;
ios_base::fmtflags oldFlags = cout.flags();
streamsize oldPrec = cout.precision();
char oldFill = cout.fill();
cout << "Serial::write: ";
cout << internal << setfill('0');
for( unsigned int i = 0; i < size; i++ ){
cout << setw(2) << hex << (unsigned int)buf[i] << " ";
}
cout << endl;
cout.flags(oldFlags);
cout.precision(oldPrec);
cout.fill(oldFill);
return size;
}
FakeSerial Serial;
and finally, the actual test program:
#include "mock_arduino.h"
using namespace std;
void millis_test() {
unsigned long start = millis();
cout << "millis() test start: " << start << endl;
while( millis() - start < 10000 ) {
cout << millis() << endl;
sleep(1);
}
unsigned long end = millis();
cout << "End of test - duration: " << end - start << "ms" << endl;
}
void delay_test() {
unsigned long start = millis();
cout << "delay() test start: " << start << endl;
while( millis() - start < 10000 ) {
cout << millis() << endl;
delay(250);
}
unsigned long end = millis();
cout << "End of test - duration: " << end - start << "ms" << endl;
}
void run_tests() {
millis_test();
delay_test();
}
int main(int argc, char **argv){
initialize_mock_arduino();
run_tests();
}
This post is long enough, so please refer to my project on GitHub to see some more test cases in action. I keep my works-in-progress in branches other than master, so check those branches for extra tests, too.
I chose to write my own lightweight test routines, but more robust unit-test frameworks like CppUnit are also available.
In the absence of any pre-existing unit test frameworks for Arduino, I have created ArduinoUnit. Here's a simple Arduino sketch demonstrating its use:
#include <ArduinoUnit.h>
// Create test suite
TestSuite suite;
void setup() {
Serial.begin(9600);
}
// Create a test called 'addition' in the test suite
test(addition) {
assertEquals(3, 1 + 2);
}
void loop() {
// Run test suite, printing results to the serial port
suite.run();
}
I have considerable success unit testing my PIC code by abstracting out the hardware access and mocking it in my tests.
For example, I abstract PORTA with
#define SetPortA(v) {PORTA = v;}
Then SetPortA can easily be mocked, without adding overhead code in the PIC version.
Once the hardware abstraction has been tested a while I soon find that generally code goes from the test rig to the PIC and works first time.
Update:
I use a #include seam for the unit code, #including the unit code in a C++ file for the test rig, and a C file for the target code.
As an example I want to multiplex four 7 segment displays, one port driving the segments and a second selecting the display. The display code interfaces with the displays via SetSegmentData(char) and SetDisplay(char). I can mock these in my C++ test rig and check that I get the data I expect. For the target I use #define so that I get a direct assignment without the overhead of a function call
#define SetSegmentData(x) {PORTA = x;}
It seems that emulino would do the job perfectly.
Emulino is an emulator for the Arduino platform by Greg Hewgill. (Source)
GitHub repository
simavr is an AVR simulator using avr-gcc.
It already supports a few ATTiny and ATMega microcontrollers, and - according to the author - it's easy to add some more.
In the examples lies simduino, an Arduino emulator. It supports running the Arduino bootloader and can be programmed with avrdude through Socat (a modified Netcat).
You can unit test in Python with my project, PySimAVR. Arscons is used for building and simavr for simulation.
Example:
from pysimavr.sim import ArduinoSim
def test_atmega88():
mcu = 'atmega88'
snippet = 'Serial.print("hello");'
output = ArduinoSim(snippet=snippet, mcu=mcu, timespan=0.01).get_serial()
assert output == 'hello'
Start test:
$ nosetests pysimavr/examples/test_example.py
pysimavr.examples.test_example.test_atmega88 ... ok
I built arduino_ci for this purpose. Although it's limited to testing Arduino libraries (and not standalone sketches), it enables unit tests to be run either locally or on a CI system (like Travis CI or Appveyor).
Consider a very simple library in your Arduino Library directory, called DoSomething, with do-something.cpp:
#include <Arduino.h>
#include "do-something.h"
int doSomething(void) {
return 4;
};
You'd unit test it as follows (with a test file called test/is_four.cpp or some such):
#include <ArduinoUnitTests.h>
#include "../do-something.h"
unittest(library_does_something)
{
assertEqual(4, doSomething());
}
unittest_main() // this is a macro for main(). just go with it.
That's all. If that assertEqual syntax and test structure looks familiar, it's because I adopted some of Matthew Murdoch's ArduinoUnit library
that he referred to in his answer.
See Reference.md for more information about unit testing I/O pins, the clock, Serial ports, etc.
These unit tests are compiled and run using a script contained in a ruby gem. For examples of how to set that up, see the README.md or just copy from one of these examples:
A practical example, testing a Queue implementation
Another set of tests on another Queue project
A complex example, simulating a library that controls an interactive device over a SoftwareSerial connection as part of the Adafruit FONA library
The DoSomething example library shown above, used to test arduino_ci itself
I am not aware of any platform which can test Arduino code.
However, there is the Fritzing platform, which you can use to model the hardware and later on export PCB diagrams and stuff.
Worth checking.
We are using Arduino boards for data acquisition in a large scientific experiment. Subsequently, we have to support several Arduino boards with different implementations. I wrote Python utilities to dynamically load Arduino hex images during unit testing. The code found on the link below supports Windows and Mac OS X via a configuration file. To find out where your hex images are placed by the Arduino IDE, hit the shift key before you hit the build (play) button. Hit the shift key while hitting upload to find out where your avrdude (command line upload utility) is located on your system / version of Arduino. Alternatively, you can look at the included configuration files and use your install location (currently on Arduino 0020).
http://github.com/toddstavish/Python-Arduino-Unit-Testing
This program allows automated running of several Arduino unit tests. The testing process is started on the PC but the tests run on the actual Arduino hardware. One set of unit tests is typically used to test one Arduino library.
(this
Arduino Forum: http://arduino.cc/forum/index.php?topic=140027.0
GitHub project page: http://jeroendoggen.github.com/Arduino-TestSuite
Page in the Python Package Index: http://pypi.python.org/pypi/arduino_testsuite
The unit tests are written with the "Arduino Unit Testing Library": http://code.google.com/p/arduinounit
The following steps are performed for each set of unit tests:
Read the config file to find out which tests to run
The script compiles and uploads an Arduino sketch that contains the unit testing code.
The unit tests are run on the Arduino board.
The results of the test are printed over the serial port and analyzed by the Python script.
The script starts the next test, repeating the above steps for all test that are requested in the configuration file.
The script prints a summary showing an overview of all the failed/passed tests in the complete testsuite.
Keep hardware-specific code separate or abstracted away from the rest so you can test and debug that bigger "rest" on any platform for which you have good tools and with which you're familiar most.
Basically, try to build as much of the final code from as many known-to-work building blocks as possible. The remaining hardware-specific work will then be much easier and faster. You may finish it by using existing emulators and/or emulating devices on your own. And then, of course, you'll need to test the real thing somehow. Depending on circumstances, that may or may not be very well automatable (i.e. who or what will press buttons and provide other inputs? who or what will observe and interpret various indicators and outputs?).
James W. Grenning writes great books and this one is about unit testing embedded C code Test Driven Development for Embedded C.
I am using Searduino when writing Arduino code. Searduino is an Arduino simulator and a development environment (Makefiles, C code ...) that makes it easy to hack in C/C++ using your favorite editor. You can import Arduino sketches and run them in the simulator.
Screenshot of Searduino 0.8: http://searduino.files.wordpress.com/2014/01/jearduino-0-8.png
Searduino 0.9 will be released and a video will be recorded as soon as the lasts tests are done .... in a day or two.
Testing on the simulator is not to be considered as real tests, but it certainly have helped me a lot in finding stupid/logical mistakes (forgetting to do pinMode(xx, OUTPUT), etc.).
BTW: I am one of the people developing Searduino.
There is a project called ncore, which provides native core for Arduino. And allows you to write tests for Arduino code.
From the project description
The native core allows you to compile and run Arduino sketches on the
PC, generally with no modification. It provides native versions of
standard Arduino functions, and a command-line interepreter to give
inputs to your sketch that would normally come from the hardware
itself.
Also on the "what do I need to use it" section
If you want to build the tests, you'll need cxxtest from
http://cxxtest.tigris.org. NCORE has been tested with cxxtest 3.10.1.
If you want to unit-test code outside MCU (on desktop), check out libcheck:
https://libcheck.github.io/check/
I used it to test my own embedded code few times. It's pretty robust framework.
You can use emulare — you can drag and drop a microcontroller on a diagram and run your code in Eclipse. The documentation on the website tells you how to set it up.
Use Proteus VSM with an Arduino library to debug your code or to test it.
It is a best practice before getting your code onboard, but be sure with timings because the simulation does not run realtime as they run on the board.
Try Autodesk circuit simulator. It allows to test Arduino code and circuits with many other hardware components.
In basic Arduino is written with C and C++, even libraries of arduino are written in C and C++. So,in simple terms just handle the code as C and C++ and try doing the unit testing. Here, by the word "handle" I mean you to change all the basic syntax like serial.println to sysout, pinmode to varaibles, void loop to while() loop which breaks either in keystock or after some iteration.
I know this is little a long process and not so straight forward.On my personal experience, once you get to do with it, this turns to be more reliable.
-Nandha_Frost
In case you are interested in running an INO sketch and checkout the serial output, I have a working implementation of that in my Arduino NMEA checksum project.
The following script takes the file and uses Arduino CLI to compile it to a HEX file which is then loaded to SimAVR which evaluates it and prints the serial output. Since all Arduino programs run forever without really having an option of killing themselves (exit(0) doesn't work), I let the sketch run for a few seconds and then diff the captured output with expected output.
Download and extract Arduino CLI (in this case version 0.5.0 - latest at the time of writing):
curl -L https://github.com/arduino/arduino-cli/releases/download/0.5.0/arduino-cli_0.5.0_Linux_64bit.tar.gz -o arduino-cli.tar.gz
tar -xvzf arduino-cli.tar.gz
Now you can update the index and install the appropriate core:
./arduino-cli core update-index
./arduino-cli core install arduino:avr
Assuming your sketch is named nmea-checksum.ino, to get ELF and HEX, run:
./arduino-cli compile -b arduino:avr:uno nmea-checksum.ino
Next up, SimAVR to run the HEX (or ELF) - I build from source because the latest release didn't work for me:
sudo apt-get update
sudo apt-get install -y build-essential libelf-dev avr-libc gcc-avr freeglut3-dev libncurses5-dev pkg-config
git clone https://github.com/buserror/simavr.git
cd simavr
make
Successful compilation will give you simavr/run_avr which you can use to run the sketch. Like I said, timeout it otherwise it will never terminate:
cd simavr
timeout 10 ./run_avr -m atmega168 -f 16000000 ../../nmea-checksum.ino.arduino.avr.uno.elf &> nmea-checksum.ino.clog || true
The generated file will have ANSI color code control characters wrapping the serial output, to get rid of those:
cat nmea-checksum.ino.clog | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g" > nmea-checksum.ino.log
cat nmea-checksum.ino.log
Now all you need to do is compared this file to a known good file:
diff nmea-checksum.ino.log ../../nmea-checksum.ino.test
If there are no differences, diff will exit with code 0, otherwise the script will fail.