I have some projects developed with C++ builder XE.
I would like to add some unit test, but the DUnit framework installed is nice for Delphi, but the integration with C++ builder is not so good (and very limited).
What other xUnit framework can I easily work with ?
In your case I'd start by asking Embarcadero for assistance. They want to fully support the developers who use their stuff, and automated unit testing is really critical to keeping them happy.
Until then, CppUnit works on any C++ code, but does not really integrate all that well with IDEs. The approach we've used is to create a new project to contain the tests, and have its linker include the path to the existing production project's .OBJ files. We set up a a project dependency so the test project depends on the production project.
In the Test project, we'll use different main.cpp files, one each for Debug and Release, and use conditionals to include/exclude the appropriate one from the Debug and Release builds.
For some "fake" integration, at least as far as running the tests go, in the DebugMain.cpp we'll load up the MFC TestRunner GUI, so the developer can click to select the tests they want to execute. In the ReleaseMain.cpp, we'll use the command line test runner, using the CompilerOutputter object which will let the build process know of success or failure. The output is also compatible with IDEs that interpret stuff like that, so you can click on a failed test report in the Output window, and the IDE takes you to the failing test assertion.
Related
In unit-testing and test driven development, why is it better to build test code as a library rather than as an executable for testing a c++ program? I have heard arguments for both.
You can build a separate executable for your test code and run it as a post-build event of your main application. This way, if the tests fail, the build fails. Most C++ IDEs (e.g. Visual Studio, Eclipse, QtCreator) support this.
The arguments for library vs executable depend on how you want your developers to use the tests.
If you want to have the tests integrated into the build process, you probably want a command-line executable. If you want to have the tests runnable from some kind of standalone GUI app, you might want a window based executable. If you want the tests to be run by a metrics gathering server, they may need to be hosted in a service.
If you want more than one of these methods, you could choose to compile the tests into a library, then link them into each of the executable frameworks. But if you only ever have need of command-line execution, then there'd be no need for the GUI or service options, and no advantage gained from building a separate static library.
Neither approach is "better". Choose the approach you need based on your team's particular situation, and your team's standards. It's also probably not that important right now. If you start with just an executable test harness, you can always split the tests into a static library later.
It's far more important that you start writing and running automated testing now than to pause and quibble about the test implementation details.
I have 63 DLL's with various C++ methods in each. I want to validate the output of some of the methods with fixed input values. I'm wondering if it is possible to do unit testing in the DLL itself during compilation build process.
So, the compilation build of DLL gives the results of the Unit Testing in the Output window of Visual Studio.
I know that I can validate this scenario by creating executable file and calling the methods. But, is it possible without executable file?
As others have said - testing "during compilation" does not make sense, so I'm assuming you mean testing during the build process, which is different and of course possible using post build steps etc.
You don't specify which version of Visual Studio you use, but if you have VS2012, there is an MSDN article that describes exactly how to do what you describe. See the link for the full instructions, I've attached a partial screenshot below
Taking your question verbatim, the answer is "no", because you can't test a DLL when you haven't even finished compiling it. Also, you need some kind of executable to load that DLL, so either you load it with a scripting language (Python with ctypes comes to mind) or you create an executable.
Calling that from a post-compile step in Visual Studio, as suggested by shivakumar is probably the only way to get the results into the output window. I personally prefer running this from an external build script, but I'm also cross-compiling a lot and I can't run things from a post-compile step there. This also makes it easier to debug the unit tests when something fails.
You have to wait compilation to complete so that there are no compilation error in the code.
In the post-build event you can add batch files which will run your unit test modules and validate the binaries generated after compilation.
You are asking for a thing that does not make sense. When you say "compiling" that means a very specific thing: invoking the compiler, before invoking the linker. But C++ code (and C++ unit tests) do not work like that. The compiler must finish compiling both your production code and your tests, and the object files must then be linked into libraries, executables, or both. A test framework must then execute the test code which calls your production code in order to get results. None of these steps are optional in C++.
Instead, you probably intended to ask if you could run the unit tests as part of the build (not compile). And the answer to that is an emphatic "yes!"
I'm guessing that your solution is likely structured into 63 or more individual DLL projects. For each production DLL you are going to test, such as Foo.DLL, I recommend you add a new FooTest project, with the unit test code added to the FooTest project. In FooTest, create a project dependency upon the Foo project, which will force FooTest to build after building Foo. In the FooTest project you would have two kinds of code modules: classes containing your unit tests, and a FooTest.cpp that would house the main() entrypoint of the FooTest.EXE program, invoking the testing framework, and outputting the results to the console.
Create your FooTest.cpp so that it's a console program. If you format your test executable's output so that it matches the output of the Visual Studio compiler, as in "filename.cpp(lineNo) : error: description of failure", Visual Studio will automatically navigate to the file and line if you click on it. Unit test frameworks such as CppUnit may already have a "CompilerOutputter" class that will properly format the output to match your compiler's errors.
In your FooTest project, you also need to set the input to the FooTest linker so that it can link in the production code you are trying to test. In the properties of the FooTest project, go to the Linker/Input tab and add the path to your Foo project's OBJ files to the Additional Dependencies. The line I use looks like this: $(SolutionDir)Foo\Debug\obj*.obj
In the Build Events properties of the FooTest project, invoke your new FooTest.EXE as a post-build step. Then, every time you click build, your code will be built and your unit tests will be executed. The project dependency will ensure that if you change your Foo code, you will compile, link, and execute the FooTest tests. And the console output ensures that your test results will appear as clickable output in your IDE.
You could create 63 separate unit test executables, or you could create one all-encompassing unit test executable. That's entirely your choice. If you are looking to make the builds and links happen quicker, you will probably want to have the separate executables; even though it's a bit more individual configuration work, you do it only once, and after that you retain the benefits of quick builds for small changes.
Now you're ready to do some serious coding.
I want to use Boost Test to unit test my code in Visual Studio 2010. I've downloaded and built the latest version of the library.
I've read a lot on the subject here and elsewhere on the internet and people seem to suggest having a second project within your solution exlusively for your tests.
Fine, sounds good. I'm having trouble actually setting this up however. I've yet to find a clear explanation of the best way to set this up.
Do I need to use a Project Reference to make my unit test project reference my main project?
If so, do I still need to add the Include & Source directories of my main project in the properties of my unit test project? If so, what's the advantage of using the Project Reference in the first place?
Do I have to have my main project output a library for my unit test project to link in? Again, I thought that Project References would make this unnecessary but it seems I don't really understand the Project References.
If at all possible could I get a very idiot proof, step by step procedure for setting up a Boost Test unit test project alongside a main project in VS2010?
Would I be better off going with the method laid out here (one project, different configurations to build tests or actual project exe):
http://blog.yastrebkov.com/2010/07/boost-test-setup-and-usage.html
Many thanks,
There is no magic behind setting up a Boost.Test project. Maybe because it's a regular C++ (executable) project in no way different from a "normal" application. This is what I do:
Create a new C++ project. I always choose Win32 Executable with precompiled headers. I have a naming convention, that all test projects using Boost.Test start with "tests.boost.testee_name..."
In "stdafx.h", add the include for <boost/unit_test.hpp> and define the BOOST_TEST_MODULE (I always choose the project name). Also, add all other includes for external components this project requires, e.g. other boost libraries, stl headers etc. This results in considerably faster compilation times.
The testee must be a library (dynamic or static). So "add reference" to all required dependencies. You can of course test header-only libraries, in that case do not add references.
Add source files to your test project, according to Boost.Test manual. The convention I enforce is one BOOST_FIXTURE_TEST_SUITE per file.
For convenience, I have a custom property sheet tailored for boost unit test, which I add to each boost test project. Among others it contains a post-build event, which runs the tests.
I have to add that, lately, I switched to MSTest with Visual Studio 2012 which allows a more comfortable way to manage the tests and test results. Nevertheless, for the most important parts of the software, I am still writing boost tests in order to ensure correctness with older toolsets and potentially other platforms.
Cheers,
Paul
As far as autotest is concerned, how do you do autotest for C++ programs? are there any autotest framework that can be utilized to do unit test and integration test?
Are you talking Autotest ala Ruby Autotest? If so, maybe Watchr would work for you. Yes, you would need to install the Ruby runtime on your development machine, but it looks like it can trigger pretty much anything that can be done on the command line when the file system changes. For example, if you wanted Watchr to build and run your C++ tests anytime a .c/.cpp/.h/.hpp file in your source tree changed you could do something like this:
watch('src/(.*)\.[h|cpp|hpp|c]') {system "build/buildAndRunTests.bat"}
This particular command obviously makes some assumptions about how your build process is set up (and obviously that you're on Windows), but that should be the gist of it. Our team configures our unit test projects with a post-build event that automatically runs the built unit test binary, so we can just trigger that part of our build process within the buildAndRunTests.bat script and have it print the results to the command-line. It might take some tweaking but it looks like Watchr may be a good choice. I'll update this response when I give it a shot (hopefully early next week).
UPDATE: I just tried this with one of my C# projects and got it working there. So I theoretically it should work with C++ projects as well.
autotest.watchr:
watch('./.*/.*\.cs$') {system "cd build && buildAndRunTests.bat && cd ..\\"}
Note the $ at the end of the regular expression. This is important because there are a lot of artifacts generated in the source tree at build time and if any of them match the string .cs it will trigger another run, effectively causing an infinite loop. Conceivably the same thing will happen if you generate/modify any source files at build time so you may have to find a way to compensate.
buildAndRunTests.bat:
pushd ..\
rem Build test project
"C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\devenv.com" Tests.Unit\Tests.Unit.csproj /rebuild Release
popd
rem Navigate to the directory containing the built files
pushd ..\Tests.Unit\bin\Release
rem Run the tests through nunit-console
..\..\..\Dependencies\NUnit-2.5.5-bin\net-2.0\nunit-console.exe Tests.Unit.dll /run=Tests.Unit
popd
Then, in a seperate console window just navigate to your project directory and run the following command (assumes autotest.watchr is at the top of your project tree, see below):
watchr autotest.watchr
Now, when any .cs files change in the source tree it will run the buildAndRunTests.bat script automatically. This is just an example from my local machine so it likely won't work verbatim on yours, but you should be able to tweak it to your needs.
This is the directory structure for reference:
/Project
/build
buildAndRunTests.bat
/Tests.Unit
/Dependencies
/NUnit-2.5.5-bin
/net-2.0
nunit-console.exe
autotest.watchr
I hope this helps.
You can use NUnit to achieve this, but there may be better ways. With NUnit you are writing test classes in managed C++/CLI which is calling your C++ code, which presumably runs as unmanaged. So for this option, some of your C++ code now runs as managed just for the sake of using NUnit. One may debate the "purity" of this approach. Another problem with this is attaching a debugger to NUnit (of course with both managed/native enabled) and trying to step through the managed C++/CLI bits in a sensible manner. Despite this, our office has been using NUnit for C++ unit and integration testing for a while now.
Just saw #Patrick's answer about CPPUnit, I will have to look at that.
The xUnit family can be used for unit tests. It exists for plain C++ code (CPPUNIT) and for .Net code (NUnit).
Boost have a test library you can have a look at among many others around.
Last time when I did some work in Qt, I've used Qt's QTestLib for unit tests. It did work well for my lo-fi needs. http://doc.qt.nokia.com/4.6/qtestlib-manual.html
I'm just getting started with TDD and am curious as to what approaches others take to run their tests. For reference, I am using the google testing framework, but I believe the question is applicable to most other testing frameworks and to languages other than C/C++.
My general approach so far has been to do one of three things:
Write the majority of the application in a static library, then create two executables. One executable is the application itself, while the other is the test runner with all of the tests. Both link to the static library.
Embed the testing code directly into the application itself, and enable or disable the testing code using compiler flags. This is probably the best approach I've used so far, but clutters up the code a bit.
Embed the testing code directly into the application itself, and, given certain command-line switches either run the application itself or run the tests embedded in the application.
None of these solutions are particularly elegant...
How do you do it?
Your approach no. 1 is the way I've always done it in C/C++ and Java. Most of the application code is in the static library and I try to keep the amount of extra code needed for the application to a minimum.
The way I approach TDD in Python and other dynamic languages is slightly different in that I leave the source code for the application and tests lying around and a test runner finds the tests and runs them.
I tend to favour static libs over dlls so most of my C++ code ends up in static libs anyway and, as you've found, they're as easy to test as dlls.
For code that builds into an exe I either have a separate test project which simply includes the source files that are under test and that are usually built into the exe OR I build a new static lib that contains most of the exe and test that in the same way that I test all of my other static libs. I find that I usually take the 'most code in a library' approach with new projects and the 'pull the source files from the exe project into the test project' approach when I'm retro fitting tests to existing applications.
I don't like your options 2 and 3 at all. Managing the build configurations for 2 is probably harder than having a separate test project that simply pulls in the sources it needs and including all of the tests into the exe as you suggest in 3 is just wrong ;)
I use two approaches, for dlls I just link my unit tests with the dll, easy. For executables I include the source files that are being tested in both the executable project and the unit test project. This adds slightly to the build time but means I don't need to separate the executable in to a static lib and a main function.
I use boost.test for unit testing and cmake to generate my project files and I find this the easiest approach. Also I am slowly introducing unit-testing to a large legacy code base so I am trying to introduce the least amount of changes, in case I inconvenience other developers and discourage them from unit testing. I would worry that using a static library just for unit testing might be seen as an excuse not adopt it.
Having said this, I think the static library approach is a nice one especially if you are starting from scratch.
For C/C++ apps I try to have as much code as possible in one or more dlls, with the main application being the bare minimum to start-up and hand-off to the dll. Dlls are much easier to test because they can export as many entry points as I like for a test application to use.
I use a seperate test application that links to the Dll(s). I'm strongly in favour of keeping test code and "product" code in seperate modules.
I go with #1, some reasons are
It allows to check that each lib links correctly
You don't want extra code in the product
It's easier to debug individual small test programs
You may need multiple executables for some tests (like communication tests)
For C++ build and test, I like to use CMake which can run a selection of the target executables as tests and print a summary of the results.
Personnally, I use another approach that relies a bit on yours:
I keep the project-to-test intact. If it's an executable, it should stay an executable. You simply create a post build action in order to aggregate all obj files into a static library.
Then, you can create you test project, linking the test framework and your previously generated static library.
Here are some topics corresponding to your question:
Visual Studio C++: Unit test exe project with google test?
Linker error - linking two "application" type projects in order to use Google Test
I'm using a third-party test-runners with their framework and including testing in build script. Tests are outside of production code (external dll).