When to run unit-tests? - unit-testing

Question

We are currently setting up our build-process within an automated continious integration environment and facing the fundamental question, when to run unit-tests?
One way would be to run the unit test with every build task. So as soon as one unit-test fails, the whole build fails. This has the advantage, that the developer is always forced to keep the unit-tests green, as s/he is otherwise not able to run the application. On the other hand, you are always distracted by fixing the tests during a development process - which might force you to work in very small iterations. Besides that the time to run your application always increases, as you have to wait for the tests every time.
The other way would be, to let the CI-Server run the tests after each new commit and let the developer simply know, that something went wrong. In this way the developer is pretty free, at what time to care for the unit-tests, but also other developers on the same branch might suffer, because they cannot be sure, that all parts of the software work as expected and have to check theirselves, if the failing tests might also influence their work.
So do you have any best-practices or recommendations, which would be a good time to run the tests?
BTW: of course we also run bigger integration-tests, which are handled in a seperate CI-process.

Answer 1

Short answer: run all unit tests on the build server for every commit, on every branch. Assuming your unit tests don't take a really long time to run, there really is no downside to this. As for running all unit tests on every build task locally, that would be a overkill. Developers should have the discipline to decide when to run the tests and when not to.
You want to know as soon as possible when something is wrong so you can fix it promptly. You also want to know all of the tests that fail rather than just the first test that fails. When there are multiple issues it would be a pretty annoying workflow to only fix the one issue and then have to commit, push, and wait for the build to run again to see if there are more issues.

Answer 2

Your build process should have two targets: build and test. test should be the default target when not specifying anything else. The test can't run until the project was build, so the build target is a dependancy of test. So (suppose use use make): make or make test will build and test. make build will just build the project.
Now, if you're using some IDE, you could consider doing the test in some separate way "outside" of the IDE. So, maybe add a third target ide and let the ide build that one. It could then have the build target as normal dependency and as last step spawn a new job in background to do the testing in it's own terminal window, something like (under linux): ( xterm -e ./run-tests & ).
And if you're developing outside of an ide (like I do), then just have a separate terminal run the build & test. As soon as testing starts, you know the build process finished, so you can run you application already, even thou the tests are still running.
Just to demonstrate this (and as a proof of concept for having the test run in background) I just created some trivial test case:
bodo.c:
#include <stdio.h>
int main(int argc, char * argv[]) {
printf("Hallo %s", argc > 1 ? argv[1] : "Welt");
return 0;
}
Makefile:
test: build run-tests
ide: build run-tests-background
run-tests-background:
( xterm -e ./run-tests --wait & )
run-tests:
./run-tests
build: bodo
bodo: bodo.o
bodo.o: bodo.c
.PHONY: run-tests run-tests-background
run-tests:
#! /bin/sh
retval=true
if test "$(./bodo)" != "Hallo Welt"
then
echo "Test failed []"
retval=false
fi
if test "$(./bodo Bodo)" != "Hallo Bodo"
then
echo "Test failed [Bodo]"
retval=false
fi
if test "$(./bodo Fail)" != "Hallo Bodo"
then
echo "Test failed [Fail]"
retval=false
fi
sleep 5 # Simulate some more tests
if $retval
then
echo "All tests suceeded ;)"
else
echo "Some tests failed :("
fi
if test "$1" == "--wait"
then
read -p "Press ENTER to close" enter
fi
if $retval
then
exit 0
else
exit 2
fi
Usage:
Build the project but do not run the tests
make build
Build the project and do run the test in current terminal
make
Build the project and do run the test in separate terminal. Make will return once the build process completed and the test got started
make ide
And two helpers, which are not supposed to be run by hand:
Only run the tests in current terminal (this will fail, if the project wasn't built yet)
make run-tests
Only run the tests in separate terminal (this will fail, if the project wasn't built yet). Make will return immediatelly
make run-tests-background