In unit-testing and test driven development, why is it better to build test code as a library rather than as an executable for testing a c++ program? I have heard arguments for both.
You can build a separate executable for your test code and run it as a post-build event of your main application. This way, if the tests fail, the build fails. Most C++ IDEs (e.g. Visual Studio, Eclipse, QtCreator) support this.
The arguments for library vs executable depend on how you want your developers to use the tests.
If you want to have the tests integrated into the build process, you probably want a command-line executable. If you want to have the tests runnable from some kind of standalone GUI app, you might want a window based executable. If you want the tests to be run by a metrics gathering server, they may need to be hosted in a service.
If you want more than one of these methods, you could choose to compile the tests into a library, then link them into each of the executable frameworks. But if you only ever have need of command-line execution, then there'd be no need for the GUI or service options, and no advantage gained from building a separate static library.
Neither approach is "better". Choose the approach you need based on your team's particular situation, and your team's standards. It's also probably not that important right now. If you start with just an executable test harness, you can always split the tests into a static library later.
It's far more important that you start writing and running automated testing now than to pause and quibble about the test implementation details.
Related
I am using Google Test 1.7.0 in NetBeans IDE 7.3.1 on a Microsoft Windows 7 operating system together with Minimalist GNU for Windows. To make test cases run from inside the IDE, I followed this YouTube video.
Everything works fine, but running (more exactly building) the tests is horribly slow.
To conclude:
I've build Google Test and Google Mock as a static library. Both are not rebuilt on running the tests.
Both static libraries are linked for the test files only and together with POSIX Threads (-lpthreads).
The duration to compile and run eight test files is at least one minute for a project in which I am also linking Crypto++. This seems to be the major bottleneck, but the building of tests in a separated project (without a static library dependency) is also slow.
Therefore I have the following three questions:
General: Is the described build time "normal" for testing in C++, or is there something wrong with my environment?
I am coming from a dynamic programming language background, and imho the long test/implement cycles are not acceptable. TDD is more or less useless with these long build times.
NetBeans IDE related: Even if I select one test folder (with one test file) in NetBeans IDE and run the tests via the context menu for that folder, all existing tests for the project are rebuilt. In addition every test file is always rebuild, no matter if there have been any changes or not.
Is it possible to change that behaviour in NetBeans IDE? Or do I have to run the tests explicitly via the CLI then?
With respect to the other two questions: What is, in your opinion, the fastest and most efficient way to run automated tests for C++?
I know that C++ code must be compiled in contrast to other languages (e. g. Ruby or PHP - both use an interpreter) and therefore do understand the generally lower test/implement cycle times.
I have some projects developed with C++ builder XE.
I would like to add some unit test, but the DUnit framework installed is nice for Delphi, but the integration with C++ builder is not so good (and very limited).
What other xUnit framework can I easily work with ?
In your case I'd start by asking Embarcadero for assistance. They want to fully support the developers who use their stuff, and automated unit testing is really critical to keeping them happy.
Until then, CppUnit works on any C++ code, but does not really integrate all that well with IDEs. The approach we've used is to create a new project to contain the tests, and have its linker include the path to the existing production project's .OBJ files. We set up a a project dependency so the test project depends on the production project.
In the Test project, we'll use different main.cpp files, one each for Debug and Release, and use conditionals to include/exclude the appropriate one from the Debug and Release builds.
For some "fake" integration, at least as far as running the tests go, in the DebugMain.cpp we'll load up the MFC TestRunner GUI, so the developer can click to select the tests they want to execute. In the ReleaseMain.cpp, we'll use the command line test runner, using the CompilerOutputter object which will let the build process know of success or failure. The output is also compatible with IDEs that interpret stuff like that, so you can click on a failed test report in the Output window, and the IDE takes you to the failing test assertion.
I'm just getting started with TDD and am curious as to what approaches others take to run their tests. For reference, I am using the google testing framework, but I believe the question is applicable to most other testing frameworks and to languages other than C/C++.
My general approach so far has been to do one of three things:
Write the majority of the application in a static library, then create two executables. One executable is the application itself, while the other is the test runner with all of the tests. Both link to the static library.
Embed the testing code directly into the application itself, and enable or disable the testing code using compiler flags. This is probably the best approach I've used so far, but clutters up the code a bit.
Embed the testing code directly into the application itself, and, given certain command-line switches either run the application itself or run the tests embedded in the application.
None of these solutions are particularly elegant...
How do you do it?
Your approach no. 1 is the way I've always done it in C/C++ and Java. Most of the application code is in the static library and I try to keep the amount of extra code needed for the application to a minimum.
The way I approach TDD in Python and other dynamic languages is slightly different in that I leave the source code for the application and tests lying around and a test runner finds the tests and runs them.
I tend to favour static libs over dlls so most of my C++ code ends up in static libs anyway and, as you've found, they're as easy to test as dlls.
For code that builds into an exe I either have a separate test project which simply includes the source files that are under test and that are usually built into the exe OR I build a new static lib that contains most of the exe and test that in the same way that I test all of my other static libs. I find that I usually take the 'most code in a library' approach with new projects and the 'pull the source files from the exe project into the test project' approach when I'm retro fitting tests to existing applications.
I don't like your options 2 and 3 at all. Managing the build configurations for 2 is probably harder than having a separate test project that simply pulls in the sources it needs and including all of the tests into the exe as you suggest in 3 is just wrong ;)
I use two approaches, for dlls I just link my unit tests with the dll, easy. For executables I include the source files that are being tested in both the executable project and the unit test project. This adds slightly to the build time but means I don't need to separate the executable in to a static lib and a main function.
I use boost.test for unit testing and cmake to generate my project files and I find this the easiest approach. Also I am slowly introducing unit-testing to a large legacy code base so I am trying to introduce the least amount of changes, in case I inconvenience other developers and discourage them from unit testing. I would worry that using a static library just for unit testing might be seen as an excuse not adopt it.
Having said this, I think the static library approach is a nice one especially if you are starting from scratch.
For C/C++ apps I try to have as much code as possible in one or more dlls, with the main application being the bare minimum to start-up and hand-off to the dll. Dlls are much easier to test because they can export as many entry points as I like for a test application to use.
I use a seperate test application that links to the Dll(s). I'm strongly in favour of keeping test code and "product" code in seperate modules.
I go with #1, some reasons are
It allows to check that each lib links correctly
You don't want extra code in the product
It's easier to debug individual small test programs
You may need multiple executables for some tests (like communication tests)
For C++ build and test, I like to use CMake which can run a selection of the target executables as tests and print a summary of the results.
Personnally, I use another approach that relies a bit on yours:
I keep the project-to-test intact. If it's an executable, it should stay an executable. You simply create a post build action in order to aggregate all obj files into a static library.
Then, you can create you test project, linking the test framework and your previously generated static library.
Here are some topics corresponding to your question:
Visual Studio C++: Unit test exe project with google test?
Linker error - linking two "application" type projects in order to use Google Test
I'm using a third-party test-runners with their framework and including testing in build script. Tests are outside of production code (external dll).
What are the best policies for unit testing build files?
The reason I ask is my company produces highly reliable embedded devices. Software patches are just not an option, as they cost our customers thousands to distribute. Because of this we have very strict code quality procedures(unit tests, code reviews, tracability, etc). Those procedures are being applied to our build files (autotools if you must know, I expect pity), but if feels like a hack.
Uh... the project compiles... mark the build files as reviewed and unit tested.
There has got to be a better way. Ideas?
Here's the approach we've taken when building a large code base (many millions of lines of code) across more than a dozen platforms.
Makefile changes are reviewed by the build team. These people know the errors people tend to make in our build environment, and they are the ones who feel the brunt of it when a build breaks, so they're motivated to find issues.
Minimize what needs to go in a Makefile, so there are fewer opportunities for error. We have a layer on top of make, that generates the Makefile. A developer just has to indicate in the higher-level file, using tags, that for example a given target is a shared library or a unit test. Usually a target is defined on one line, which then results in multiple settings/targets in the generated Makefile. Similar things could be done with build tools like scons that allow one to abstract away things like platform-specific details, making targets very simple.
Unit tests of our build tool. The tool is written in Perl, so we use Perl's Test::More unit test framework there to verify that the tool generates the correct Makefile given our higher-level file. If we used something like scons instead, I'd use their testing framework.
Unit tests of our nightly build/test scripts. We have a set of scripts that start nightly builds on each platform, run static analysis tools, run unit tests, run functional tests, and report all results to a central database. We test the various scripts individually, mostly using the shunit2 unit-testing framework for sh/bash/ksh/etc.
End-to-end tests of our build/test process. I am working on an end-to-end test that operates on a tiny source tree rather than our production code, since the latter can take hours to build. These tests are mainly aimed at verifying that our build targets still work and report results into our central database even after, for example, upgrading our code coverage tool or making changes to our build scripts.
Have your build file to compile a known version of your software (or simpler piece of code that is similar from a build perspective) and compare the result obtained with your new build tools to a expected result (built with a validated version of the build tools).
In my projects build-files don't change very often. Even more, I can reuse build-files from earlier projects, only changing some variables (that I moved to an easy to recognize section). That's why for me it is unneeded to unit-test the build-files. That can be different in other projects.
My question is quite relevant to something asked before but I need some practical advice.
I have "Working effectively with legacy code" in my hands and I 'm using advice from the book as I read it in the project I 'm working on. The project is a C++ application that consists of a few libraries but the major portion of the code is compiled to a single executable. I 'm using googletest for adding unit tests to existing code when I have to touch something.
My problem is how can I setup my build process so I can build my unit tests since there are two different executables that need to share code while I am not able to extract the code from my "under test" application to a library. Right now I have made my build process for the application that holds the unit tests link against the object files generated from the build process of the main application but I really dislike it. Are there any suggestions?
Working Effectively With Legacy Code is the best resource for how to start testing old code. There are really no short term solutions that won't result in things getting worse.
I'll sketch out a makefile structure you can use:
all: tests executables
run-tests: tests
<commands to run the test suite>
executables: <file list>
<commands to build the files>
tests: unit-test1 unit-test2 etc
unit-test1: ,files that are required for your unit-test1>
<commands to build unit-test1>
That is roughly what I do, as a sole developer on my project
If your test app is only linking the object files it needs to test then you are effectively already treating them as a library, it should be possible to group those object files into a separate library for the main and the test app. If you can't then I don't see that what you are doing is too bad an alternative.
If you are having to link other object files not under test then that is a sign of dependencies that need to be broken, for which you have the perfect book.
We have similar problems and use a system like the one suggested by Vlion
I personally would continue doing as you are doing or consider having a build script that makes the target application and the unit tests at the same time (two resulting binaries off the same codebase). Yes it smells fishy but it is very practical.
Kudos to you and good luck with your testing.
I prefer one test executable per test. This enables link-time seams and also helps allow TDD as you can work on one unit and not worry about the rest of your code.
I make the libraries depend on all of the tests. Hopefully this means your tests are only run when the code actually changes.
If you do get a failure the tests will interrupt the build process at the right place.