I've recently started trying to put a venerable and large (>1 million lines) program under test. There are currently no unit tests. Also, the program is linked as each individual file being linked together - there are no component libraries. Furthermore, the objects are highly interdependent, and it is difficult (impossible?) to link to any object files without linking to at least half of them.
Yes, I know, my life sucks.
I'd like to do some refactoring (obviously), but I'd like to have some tests in place before I start moving things around. My current idea is to compile a single "test program" which runs all of the tests I create. This would drastically simplify the linking issues that I have and let me focus on the real problems. So I have two questions:
Is it possible to link multiple BOOST unit test files into one test executable?
Is there a better solution?
I guess, this is precisely how to use boost test.
I would keep one short main.cpp file consisting of literally 2 lines:
#define BOOST_TEST_MODULE "C++ Unit Tests for MyTangledLibrary"
#include <boost/test/included/unit_test.hpp>
And then I would keep adding test module *.cpp files compiled together into one executable
#include <boost/test/unit_test.hpp>
<< your include files >>
BOOST_AUTO_TEST_SUITE(FancyShmancyLogic)
BOOST_AUTO_TEST_CASE(TestingIf2x3equals6)
{
...
}
BOOST_AUTO_TEST_CASE(TestingIf2x2equals4)
{
...
}
BOOST_AUTO_TEST_SUITE_END()
Yes, you will be able to compile that main.cpp and all of your modules into one large executable.
Related
I have a C++ project in Visual Studio, and have added another project exclusively for testing. Both of these projects are EXEs (console apps). So how do I use the first project inside the second?
Just to clarify, the question here would be somewhat self-evident if the first project was a library that one could simply include in the second project but, being an EXE, this is where the problem lies.
Per your comments, you have a C++ console application (MyApp) for which you have developed some application-specific classes that you want to unit-test with googletest in
Visual Studio. How?
As you say, if you wanted to unit-test a library the way to do it would be
obvious. You would:
1) Create a project to create a unit-testing application (UnitTest).
2) Configure the include-search directories so that the compiler can find the library's headers.
3) Configure the library-search directories so that the linker can find the library itself.
4) Add the library itself to the linker inputs.
5) Make the UnitTest project dependent on the library project, so that building UnitTest ensures MyApp is up-to-date.
6) Code the UnitTest app per googletest docs.
But since the classes you want to unit-test are specific to MyApp, you don't have any
library.
A drill-sergeant answer to that is: You don't have a library containing the classes you want to unit-test? So make one!
That way you use 3 projects:-
MyAppLib, generating library that contains all the functionality you want to unit-test.
MyApp, generating the same executable as at present, but linking MyAppLib
UnitTest, generating an executable that unit-tests MyAppLib, also linking MyAppLib
However if you don't like the drill-sergeant answer you can work around it.
From the usual build-system point of view (the one designed into Visual Studio),
the important output of the MyApp project is the build-target - the .exe.
The .obj files generated are just intermediate by-products. VS offers you no support
for treating these by-products as automatic linker inputs of a dependent project, and if a dependent project was also an .exe of the same sort - as it is your case - then such automatic linkage would be impossible anyhow because the main entry point would be multiply defined.
But from the unit-testing point of view it's the other way round. The .exe is of no interest, whereas (some of) the .obj files wholly or partly contain the implementations of the classes you want to unit test. In the text-book case where class foo is defined in foo.h and implemented in foo.cpp, the object file foo.obj is needed in the linkage of UnitTest.
For simplicity, assume that MyApp employs just one application-specific class foo,
defined in foo.h and implemented in foo.cpp. Then you have two options for building UnitTest.
a) You can add foo.cpp to the source files of UnitTest. Don't copy it of course. Just Add an existing item from the source folder of MyApp. Then you're done, but this
course has the downside that foo.cpp is exposed to untoward editing within
the UnitTest project.
b) You can treat foo.obj just like a static library required for the linkage of UnitTest and follow steps 1) - 6) above. This means in particular at step 3) that the {Debug|Release} build of UnitTest is configured with library-search directories that include \path\to\MyApp\{Debug|Release} (either in relative or absolute form).
In reality, for option b), there's very likely more than one .obj file from MyApp that you will have to link in UnitTest, and quite likely that their number will grow with time. Maintaining the right linkage of UnitTest could become a chore, and you might come to the conclusion that the drill-sergeant was right after all.
Depends. Google Test is (primarily) a Unit Testing framework (oversimplifying, testing classes). You can absolutely use is for other types of tests, but it doesn't have "built in" functionality for other types of testing, you'll have to write it yourself.
If you are trying to system test your executable, than you can run the process. I suggest using Boost.Process if you are using a multi-platform system or already have a boost dependency. Else, look here: launch an exe/process with stdin stdout and stderr?
The "tests" you write will call the executable, and can input stdin or stdout accordingly.
For example:
std::string path_to_exectuable = "thepath";
TEST(FooTester,CheckHelpScriptReturns0)
{
using bp =::boost::process;
std::vector<std::string> args; args.push_back("--help");
bp::context ctx;
ctx.stdout_behavior = bp::capture_stream();
bp::child c = bp::launch(exec, args, ctx);
bp::status s = c.wait();
ASSERT_TRUE(s.exited())<<"process didn't exit!";
ASSERT_EQ(s.exit_status(),0)<<"Help didn't return 0";
}
I was in a similar situation and I set this up in a way that effectively accomplishes the same goal of Mike Kinghan's answer as far as the compiler is concerned, but goes about it a different way from the user's perspective.
What I did was create a custom Configuration that I called "Testing". You create a new configuration by opening the project settings, choosing "Configuration Manager..." and selecting "New..." in the configuration selection box.
When prompted, I chose to copy the settings from the default "Debug" configuration, so that I can use the debugger with my tests just the same as if I was in the "Debug" configuration.
Under the new Testing configuration, I set the options for the compiler and linker to use google test as you normally would.
The important change in the properties is that I define a preprocessor variable which I have called "TESTING".
I rewrote my "main.cpp" to look something like this:
...
// includes
// functions
// whatever
...
#ifdef TESTING
#include <gtest/gtest.h>
#endif
int main(int argc, char **argv) {
#ifdef TESTING
::testing::InitGoogleTest(&argc, argv);
int val = RUN_ALL_TESTS();
getchar(); // not necessary, but keeps the console open
return val;
#endif
// rest of main() as normal...
}
What I'm trying to indicate is that I only changed a few lines right around where main is defined, I don't have to make gross changes spread throughout the file.
Now that this is all set up I simply made a new source folder for my tests, and create ".cpp" files in there. To avoid bloating the normal executable, I wrap these files with a check for the TESTING variable, so I have something like this:
tests/Test.cpp:
#ifdef TESTING
#include <gtest/gtest.h>
#include "my_class_header.h"
TEST(TestMyClass, test_something) {
// perform some test on class
}
#endif
I think these files still get "hit" by the compiler under Debug and Release configurations, so having a ton of these might slow down the build, but the Debug and Release objects wont get bloated with testing code.
The two takeaways are:
Using this method, the testing code is still organized separately from the application code, but it still resides in the same Visual Studio project, which may or may not be beneficial. Personally I like not having to manage/worry about a second project.
Like Mike Kinghan said, managing and linking .obj files yourself can become a chore, but by using this method, the default Visual Studio settings manage this for you.
One downside is that effectively redundant copies of all object files will get created in the "Testing" output directory. With more configuration, surely there must be a way to "share" the Debug object files, but I didn't have a reason to go that far.
This is a very simple method which may be a lot easier than refactoring your application into separate libraries and a main. I don't love using preprocessor wankery, but in this case it's fairly straightforward, not too much code bloat, and accomplishes exactly what it needs to. You could always trigger the tests another way, without using the preprocessor.
If you are not very rigid about having the tests in a different project, you can write the tests in your application project. Then just make the application execute the tests when receiving certain command line arguments, and execute the normal application logic otherwise, i.e.
int main(int argc, char* argv[])
{
if (argc >= 2 && std::string(argv[1]) == "--tests")
{
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
else
{
// Application logic goes here
}
}
TEST(ExampleTests, TestSQRTCalculation) // assuming all the right headers are included
{
EXPECT_NEAR(2.0, std::sqrt(4.0), 0.000001);
}
This avoids creating an unnecessary library for the sole purpose of testing, although you can still do it if it is correct structure-wise. The downside is that the testing code goes into the executable you are going to release. If you don't want that I guess you need an additional configuration which specifies a pre-processor directive to disable the tests.
Debugging the tests or running them automatically at post build is easy, simply by specifying "--tests" as debug args or at post build command line respectively.
If you want to test a console app you can run a test that opens a console window and run the exe file of the first app.
Then in your googletest catch the standard output from the exe you just ran.
[For more control over the first app you might need to have the first app parse arguments sent to it, e.g. some flags like -x or what ever you need.]
I have prepared a github repo including Visual Studio 2015 solution in parralel of Mike's "drill-sergeant" suggestion. You can use it directly without any additional requirement or dependency.
https://github.com/fuatcoskun/GoogleTestVS2015
I hope it helps...
I have a basic question regarding Googletest in Eclipse.
I am using the test-runner plug in to run the Googletests.
But I need to specify a binary which runs my unit tests (of course that makes sense.)
The problem is that in my project I now have two main functions, one to run the actual program and one
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
to run the google tests.
Each time I want to run one I comment the other out, which of course is stupid.
But what practice do you use to handle this situation?
Googletest C++ is a unit-testing framework. That means it is intended for testing
implementations of C++ APIs. It isn't intended for testing programs.
For practical purposes a C++ API is what you get in a C++ header file. The
implementation of such an an API might be:
Just the header file itself. (The implementation is entirely inline)
The header file plus a single C++ source file
The header file plus a bunch of C++ source files
To generalize, the implementation of a C++ API is a header file plus
0 or more source files.
Say your program my_prog invokes an API that you or your team have developed
for managing gizmos. The implementation is something like:
gizmo.h
[gizmo_0.cpp,...gizmo_N.cpp]
where [...] means optionally ...
Maybe my_prog relies on other APIs for which you or your team are responsible,
but we'll stick with just one. my_prog uses the gizmo API by:-
Using #include "gizmo.h" in some source files.
Compiling the [gizmo_0.cpp,...gizmo_N.cpp] source files, if any.
Linking the [gizmo_0.o,...gizmo_N.o] object files, if any.
(gizmo_0.obj, etc. if you're on Windows)
Testing your implementation of the gizmo API with Googletest is supposed
to confirm that this implementation is correct, independently of my_prog
or any other program that relies on it to manage gizmos. So incorporating
the unit-testing of the implementation in the implementation of my_prog is misguided:-
Maybe your colleague writes another program that also needs to manage gizmos
with this implementation. Maybe you write another one. Is whoever writes this
other program supposed to repeat the process of incorporating gizmo unit-tests
into it - The same ones? Different ones? - and making the program conditionally
compile as either a gizmo test-harness or as whatever it's supposed to be in real life?
And how do you know that the gizmo implementation isn't somehow entangled with
functionality that's unique to my_prog, or with the implementation of some
other API that my_prog uses in the same way - so that when you or somebody
else tries to reuse it in another program, it breaks or behaves wrongly?
No program that relies on this gizmo implementation is the place to put
its unit-testing. Making my_prog conditionally compile different main functions so it can
double as a unit-test harness for your gizmo library is similar to cutting a hole in the
crotch of your jeans for your head to fit through.
The way you're supposed to unit-test the gizmo library is to write a program that is the
test-harness for this library, and nothing else. This program, say gizmo_test, will
use the gizmo API in just the same way as any other program would use it, but
for the sole purpose of testing the gizmo library. All that gizmo_test will do is execute tests
of the gizmo library, by invoking its API.
As a first approximation, the GoogleTest recipe for gizmo_test is:
Write a header file, gizmo_test.h
#include "gizmo.h" in it
#include <gtest/gtest.h> in it
Then write your Googletest test cases in it
Write the following source file gizmo_test.cpp
#include "gizmo_test.h"
int main(int argc, char** argv) {
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
Create a project gizmo_test - in Eclipse or whatever development environment or build system you use -
that builds the gizmo_test executable by:
Compiling the source files gizmo_test.cpp + [gizmo_0.cpp,...gizmo_N.cpp]
Linking the resulting object files gizmo_test.o + [gizmo_0.o,...gizmo_N.o], plus libgtest and any other libraries on which your gizmo library depends
You have two projects. The one that makes my_prog and the one that makes gizmo_test. In your development environment or
build system, make the build of my_prog depend on the build of gizmo_test, so that when you change anything that affects
the gizmo library and rebuild my_prog, gizmo_test gets rebuilt first.
That's a first approximation. Did you notice a while back that I started talking about your gizmo library? That's what
you've got (or should have). In C++ and programming generally, the implementation of an an API is called a library.
And maybe you also noticed some fragility, inconvenience and wastage in the recipe for gizmo_test. You have the same set of gizmo source files
[gizmo_0.cpp,...gizmo_N.cpp] in both projects. So you might edit, compile and link them differently in two projects. And they'll get compiled in both projects, either differently,
which is wrong, or identically, which is pointless.
Of course if this set of source files is empty - the gizmo library is nothing but gizmo.h - there's no such problem. But if it's not empty,
there is.
As you know, in C++ we don't use a library by building its source files in every program that uses it - not unless it's a header-only library.
A library is built by itself into an object library (either static or dynamic), and to use it a program just includes the library's
header file(s) and links the object library.
That's how a program should use your gizmo library too. So to the final approximation:-
Make a project libgizmo that builds a gizmo object library (static or dynamic, as you see fit).
Make a project gizmo_test as above, except that instead of compiling and linking [gizmo_0.cpp,...gizmo_N.cpp], it just links libgizmo, and make this project
depend on the libgizmo project.
Make a project my_prog as you have it now, but instead of compiling and linking [gizmo_0.cpp,...gizmo_N.cpp], just link libgizmo, and make this project
depend on the gizmo_test project.
So you have three projects by the time you build the first program that uses the gizmo library. Each subsequent program that uses the gizmo library needs one
more project, like the my_prog project.
Googletest is designed for testing C++ libraries, and this is how you're supposed to use it.
Now I know nothing about your program or how you are currently deploying Googletest test cases in your project. Maybe there aren't any well-defined API implementations in it
that those test cases are supposed to exercise, that you can factor out into free-standing libraries. Possibly that could be because your program is so very simple that
unit-testing its "components" is non-applicable, and you'd be wiser just to write blackbox tests of the program. More likely it would be because you've so far failed
to design a program architecture that is capable of being unit-tested. If that's what you find, you need to fix it, and then apply Googletest the right way. It will be worth the
effort.
And in case it needs pointed out, unit-tests are not program tests, so as well as unit-testing any libraries your program relies on, if they are your responsibility, you also need blackbox tests of your program.
Indeed a valid question. I believe that Mike Kinghan's detailed response explains how you should use google-test but nothing is set in stone and in the end, you can use google-test in any way you want.
So, since I actually encountered a similar situation, what I did was rather simple.
I modified the production file source code so that the main function will change from this:
void main(void)
to this:
#ifndef TDD_ENABLED // TDD_ENABLED is defined only in the unit test project
void main(void)
#else
void main_app(void)
#endif
this way when you are running unit tests, the original 'main' function is converted to 'main_app' and you avoid conflicts with the 'main' of google test. When you are running the original production code project, the 'main_app' function is named again 'main' as it should. So, technically, from your production code perspective, nothing has changed.
I have a Visual Studio solution organised like this:
ProjectA // A static library I'm working on
ProjectB // A static library containing UnitTest++ test suites for ProjectA
ProjectC // An executable test runner which links to ProjectA and ProjectB
ProjectB contains two files which look like this:
// RunTests.h
#ifndef RUNTESTS_H
#define RUNTESTS_H
#include "UnitTest++.h"
int runAllTests();
#endif
and this:
// RunTests.cpp
#include "RunTests.h"
int runAllTests()
{
return UnitTest::RunAllTests();
}
As well as several files containing test suites e.g.:
// FooTests.cpp
#include "RunTests.h" //
#include "Foo.h" // From ProjectA
TEST(SomeTest)
{
CHECK(true);
}
ProjectC consists of a single file:
// main.cpp
#include "RunTests.h" // from ProjectB
int main()
{
return runAllTests();
}
The reason I have the tests and the test runner separated, is that I have another project which uses the same tests to analyse code coverage, which I need to keep separate as it is not cross-platform, whereas the test runner is.
The issue is, that when I compile and run ProjectC, no tests are actually run (UnitTest++ runs, but with zero tests). This is because ProjectC does not reference any symbols relating to the tests from ProjectB, so the linker doesn't link the object files from ProjectB.lib.
It is my understanding that if ProjectB was an executable, I would not have this issue (presumably because the linker would link all the object files), as per the documentation:
The general idea is that you keep one Main.cpp file with the
entry-point which calls RunAllTests().
Then you can simply compile and link new .cpp files at will, typically
one per test suite.
Each of the Test*.cpp files will contain one or more TEST macro incantations with the associated
test code. There are no source-level dependencies between Main.cpp and
Test*.cpp, as the TEST macro handles the registration and setup
necessary for RunAllTests() to find all tests compiled into the same
final executable.
How can I resolve this problem without having to declare all the tests in header files that ProjectC can see (which would kill UnitTest++'s ease of use)? One possibility I've noticed in Visual Studio is:
Project Settings > Configuration Properties > Linker > Input > Force Symbol References
However it would be rather tedious to have to add every single symbol, every time I write a new unit test. Is there some way I can force it to include the entire contents of ProjectB.lib? Or perhaps some code-based solution?
EDIT: What I'm looking for is something like this but for Visual Studio.
I was trying to use UnitTest++ the same way you've described and ran into the same problem. I ran across a suggestion in another forum that seems to work for my unittest executable (i.e. ProjectC).
For ProjectC:
Project Settings > Common Properties > (select ProjectB) > Use Library Dependency Inputs: True
This worked for me. I think what this does is effectively link any object files from ProjectB into ProjectC (as opposed to linking the library).
I am using xcode (gcc) to compile my boost test suite and it takes too long.
The tests are minimal dummy tests, yet it takes several seconds (about 20) to compile them:
#include "boost/test/included/unit_test.hpp"
BOOST_AUTO_TEST_CASE(dummy)
{
BOOST_CHECK_EQUAL(2+2, 4);
}
BOOST_AUTO_TEST_CASE(dummyFail)
{
BOOST_CHECK_EQUAL(2+3, 4);
}
The manual suggests using the library version to speed up compilation. However, I am concerned this might not work - xcode already rebuilds my tests only. The whole framework isn't compiled again since the object files exist.
I guess it's the amount of header files and templates in Boost.Test that are responsible for most of the compilation time.
Do you have an idea of how to compile significantly faster? Would using it as library work? Would including only parts of boost.test work?
Any help is greatly appreciated!
The reason it's slow to compile is because boost/test/included/unit_test.hpp is huge. Using a library makes it faster because the huge header is compiled when the library is built and not thereafter. Your tests then include a smaller set of headers, leading to shorter build times.
Because I'm too lazy to build the library, an alternative I've used is to have one source file (which never changes, and so is rarely rebuilt) include the full boost test, and then have all the real test sources include just boost/test/unit_test.hpp. That gives most of the benefits of using the library.
Try using precompiled headers, this should reduce compilation time. Details can be found here:
http://www.boost.org/boost-build2/doc/html/bbv2/reference/precompiled_headers.html
I believe all the options are now described in the official documentation (see Usage variants).
The Static library usage variant is very convenient, and greatly reduces compilation times.
As described there, one can create a single source file including just two lines, compile that separately and link that in with the other tests.
A comment regarding the linked docs.
I believe that there is an error in that page, namely here:
One and only one translation unit should include following lines:
#define BOOST_TEST_MODULE test module name
#include <boost/test/unit_test.hpp>
This leads to "undefined reference" errors in the linking phase.
I believe it should be instead:
#define BOOST_TEST_MODULE test module name
#include <boost/test/included/unit_test.hpp>
I'm setting up a bunch of unit tests using CppUnit but am having the problem that none of the tests are being run. The project is divided up into several small libraries and I planned on dividing the unit test classes up the same way and then linking them all into a single test program. The problem is, then the test classes are in their own libraries, they don't get linked into the main test program unless I explicitly call them, i.e. I have to put in
runner.addTest( TestClass::suite() );
individually for each test class and can't use the TestFactoryRegistry's makeTests() method to get the list of tests. If I just compile them all together in the top directory the makeTests() method works fine but I don't want to have all the test classes in one location if I can help it.
The CppUnit documentation gives the following little hint
Linking problem when using Helper
macros ?
When you create a project and write
its unit test suites, the work is made
easier through the use of the
so-called helper macros :
CPPUNIT_TEST_SUITE_NAMED_REGISTRATION,
CPPUNIT_REGISTRY_ADD and
CPPUNIT_REGISTRY_ADD_TO_DEFAULT. The
problem is that if you use those
macros in the source code file of a
TestFixture class (say MyTest as an
example), and if you use a line like
this one
runner.addTest( CppUnit::TestFactoryRegistry::getRegistry().makeTest()
);
in your main() function in file
main.cpp, there will have no test run
at all !
The reason is simply that the link
stage, one of the step of the build
process, do not insert the object
files (.obj or .o files) in the final
executable if there is no undefined
symbol in your main.cpp.
That way, the object code which
contains the AutoRegister static
variables instantiation is not part of
the final executable and is not able
to insert oneself into the runner in
the main() function.
You have to create an undefined symbol
in main.cpp so that the mytest.o file
is integrated with main.o into the
final executable.
Trick committed by Michel Nolard
but doesn't say how to make this work and I'm just dense enough not to be able to figure it out myself or find an example on-line.
Now I could just make a separate executable test for each library, and in the end I may go that way, but I wanted to try to get this working first so I just had one single test program to run to test the whole thing. Any ideas/examples of how to get this to work?
By adding an undefined symbol to main, he just means create any random external symbol to force the linker into searching your external libraries that contain the test code.
For example, assuming two test libraries fred and barney, in fredTestLib.cpp you'd just add this line:
int fredDummyInt = 0; // declare a unique symbol for the linker to resolve
and in barneyTestLib.cpp, you'd add a similar line:
int barneyDummyInt = 0; // a different unique symbol for the linker to resolve
You could compile each library separately in different steps. In the main test program, you then force the linker to resolve them. So add these lines to main.cpp:
extern int fredDummyInt;
extern int barneyDummyInt;
...
main () {
...
fredDummyInt++; // give the linker some symbols to resolve
barneyDummyInt++;
...
The idea (according to what the author of the trick above is saying) is that because the linker is already searching fredTest.lib for fredDummyInt, it will also find and resolve your automatically registered tests.
Note: I have not tried this to see if it works! I'm just answering your question about externals.
Another approach to consider would be to create your tests in DLLs, and use LoadLibrary() to explicitly bring them in to run. For overkill, if you use the MfcUi::TestRunner, you could probably build a little drop-down GUI thing that lets you pick the library to load, loads it, then displays the tests to run in that library, then runs them.
The solution for this problem is rather simple as statet before (but may not be very elegant).
For each TestFixture that is located in an external library you have to add the following two lines of code in the main module
#include <CppUnitTestFixtureExample.h>
CppUnitTestFixtureExample Test1;
It creates an unused dummy variable that is not used, it just forces the linker to link the test fixture. Now the test runner that is located in the main module is able to run the test.
I realize this post is now quite old, but for anyone else who comes across it:
One way to address this w/out having references in your code is to instruct (force) the linker to include the entire static library in the binary. Details area available from the gcc & ld man pages, and this post covers it too:
How to force gcc to link unreferenced, static C++ objects from a library
Per ld's man page, it's important to consider explicitly turning the option off (also shown in one of the examples above).