I am starting to use GoogleTest. It seems that it needs a main file for running the tests:
Separate test cases across multiple files in google test
But currently in my demo application I already have a main file:
src/
-> MyType.h
-> main.cpp
-> Makefile
Which will eventually be my "production" application. I don't want to clutter that with gtest includes, macros etc.
Should I just create another main.cpp file in another folder e.g.: test/ that will contain all the specific gtest configuration so I would end up with:
src/
-> MyType.h
-> main.cpp
-> Makefile // Makefile for producing production code/binaries
Test/
-> MyTypeTest.h // Unittest for MyType
-> main.cpp // The "Test runner"
-> Makefile // Makefile for producing test executable
EDIT:
Found this based on cmake:
http://www.kaizou.org/2014/11/gtest-cmake/
which seems to be exactly what I am looking for.
The most sensible approach to this is to have a library for your production code and then two executables, one for production and another one for tests:
|-lib/
| |-Makefile
| |-mytype.h
| `-mytype.cpp
|-app/
| |-Makefile
| `-main.cpp
`-test/
|-Makefile
`-mytypetest.cpp
Notice that gtest distribution provides the gtest library and a gtest_main library with the standard main function for your test executable. So unless you need a custom main (rare case) you don't need to provide a main.cpp for your tests and can simply link against gtest_main, e.g. $(CC) mytypetest.cpp -o apptests -lapplib -lgtest_main -lgtest.
The library approach involves slightly more complex Makefiles, but it pays off in compilation time, since not having it implies you need to compile mytype.cpp once for production application and once for test executable.
There are probably a lot of ways to do this, but generally speaking, yes, you should add a test-specific main function to your project. This makes compilation a little bit more complex since you'll have to produce two separate binaries (one for your application and another for your tests) but this is a fairly typical setup.
I'd simply add a test.cpp file with a main and create a test target in my makefile so that I could either make - to build my production code - or make test - to build the tests. In actual projects I use cmake in very similar fashion (I sometimes bundle all common dependencies in a core.a library and then link both main and test against it).
Related
This is the simplified folder structure I have:
+ production
+-- ClassToTest.cpp (contains '#includes "DragonHeader.h"')
+-- ClassToTest.h
+-- DragonHeader.h (has a lot of dependencies, ClassToTest only need one simple static function with no dependencies at all.)
+ tests
+-- tst_ClassToTest.cpp
+-- DragonHeader.h (which defines and implements the needed simple function)
So for the test case I'd like that tests/DragonHeader.h shadows production/DragonHeader.h. => ClassToTest.cpp includes tests/DragonHeader.h. How can that be achieved? (I am working with MSVC 2015 if it is relevant...)
The CMakeLists.txt in the tests folder looks like
project( tst_ClassToTest )
set( ${PROJECT_NAME}_SOURCES tst_ClassToTest.cpp DragonHeader.h ../production/ClassToTest.cpp )
# How to force ClassToTest.cpp to use DragonHeader.h, not production/DragonHeader.h?
add_executable( ${PROJECT_NAME} WIN32 ${${PROJECT_NAME}_SOURCES} )
Thanks!
Disclaimer: I seek for a solution that does not require me to touch ClassToTest.cpp or to touch the code in production including the related CMakeLists.txt files at all. One solution, however, is to use CMake to copy ClassToTest.cpp/h into the tests folder, but I consider it sub-optimal.
My first thought was to use different include_directory statements for both production and unit test. But in this case the other header file ClassToTest.h must be in a different folder. Or you have to moveDragonHeader.h into another folder.
# CMake for production
include_directory(production)
include_directory(production_dragon)
#CMake for test
include_directory(production)
include_directory(test_dragon)
Another way could be an
#ifndef USE_TEST_HEADERS
#include "DragonHeaderTest.h"
#else
#include "DragonHeader.h"
#endif
in ClassToTest.cpp. To set the define add into your test/CMakeLists.txt file the string -DUSE_TEST_HEADERS to the compiler flags.
Third solution is to replace production/DragonHeader.h by a copy of test/DragonHeader.h in advanced of make. This works only in case production/DragonHeader.h can be restored. Either from a file system copy or from your Configuration Mgmt. System. This can by achieved by wrap the bild into a script. Don't know whether MSVC 2015 can trigger such a script instead of build.
rem Build for tests
rem Run cmake
cmake PARAMETERS
rem Prepare unit test
cp WHATEVER
rem nmake
name PARAMETERS
I am using QT Creator to work on a medium-sized project in C++.
The project structure basically looks like this
Project
Group A
Library A1
Group B
Library B1
Library B2
...etc
Test
LibA1_Test
LibB1_Test
LibB2_Test
...etc
The libraries are tested by the executables in the test project. I've managed to compile the tests themselves with gcov enabled, and produce code coverage reports with lcov, but all that they were showing the coverage for were the test cases, not the actual code that I'm testing. I've also tried compiling the static libraries with gcov as well, but when I run the tests against those libraries it does not generate any of the gcov output files.
How could I generate the gcov output files by linking my project libraries against the tests? I want to see if there are any gaps in my unit tests.
From the ld manual
--whole-archive
For each archive mentioned on the command line after the --whole-archive option, include every object file in the archive
in the link, rather than searching the archive for the required object
files.
So link your static-library into your test using --whole-archive, which will result in your test binary having the entire static-library, and give gcov visibility of the entire code
To be honest, you don't really need --whole-archive, it's enough to just add the following to all the staticlibs and Unit-Tests, that you want gcov information from:
# only apply codecoverage when CodeCoverage is set and debug-mode is active
CodeCoverage
{
debug:QMAKE_CXXFLAGS_DEBUG += --coverage # -fprofile-arcs -ftest-coverage
debug:QMAKE_LFLAGS_DEBUG += --coverage # -lgcov
}
--coverage is context-sensitive and the meaning of it depends on whether you pass it to CXXFLAGS or LFLAGS:
https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html#Instrumentation-Options
Also mentioned in https://stackoverflow.com/a/5352713/7820869.
So no need to add -fprofile-arcs or any other mixture of flags.
In your case, add those Flags to A1.pro, B1.pro and B2.pro.
The added scope around the flags just prevent that the codecoverage-flags pollute your build in Release- or Debug-mode. Just pass "CONFIG+=CodeCoverage" to qmake.exe, so it will build with code-coverage:
https://doc.qt.io/archives/qt-4.8/qmake-advanced-usage.html#configuration-and-scopes
After that just check whether after compilation the *.gcno files and after running the Unit-Test's the *.gcda files have been generated in the build-directory and call lcov on them:
https://gcc.gnu.org/onlinedocs/gcc/Gcov-Data-Files.html#Gcov-Data-Files
Additional Info
The stackoverflow, that helped me solving my problems I run into:
Where are the gcov symbols?
in case you are running into linker errors like (undefined reference to __gcov_init and __gcov_merge_add), watch out that you are really adding those flags to every unit:
https://www.spinics.net/lists/gcchelp/msg29518.html
We have C++ project that has a relatively big number of test suites implemented in Boost/Test. All tests are kept out of main project's tree, every test suite is located in separate .cpp file. So, our current CMakeLists.txt for tests looks like this:
cmake_minimum_required(VERSION 2.6)
project(TEST_PROJECT)
find_package(Boost COMPONENTS unit_test_framework REQUIRED)
set(SPEC_SOURCES
main.cpp
spec_foo.cpp
spec_bar.cpp
...
)
set(MAIN_PATH some/path/to/our/main/tree)
set(MAIN_SOURCES
${MAIN_PATH}/foo.cpp
${MAIN_PATH}/bar.cpp
...
)
add_executable (test_project
${SPEC_SOURCES}
${MAIN_SOURCES}
)
target_link_libraries(test_project
${Boost_UNIT_TEST_FRAMEWORK_LIBRARY}
)
add_test(test_project test_project)
enable_testing()
It works ok, but the problem is SPEC_SOURCES and MAIN_SOURCES are fairly long lists and someone occasionally breaks something in either one of the files in main tree or spec sources. This, in turn, makes it impossible to build target executable and test the rest. One has to manually figure out what was broken, go into CMakeLists.txt and comment out parts that fail to compile.
So, the question: is there a way to ignore tests that fail to build automatically in CMake, compile, link and run the rest (ideally, marking up ones that failed as "failed to build")?
Remotely related question
Best practice using boost test and tests that should not compile suggests to try_compile command in CMake. However, in its bare form it justs executes new ad hoc generated CMakeList (which will fail just as the original one) and doesn't have any hooks to remove uncompilable units.
I think you have some issues in your testing approach.
One has to manually figure out what was broken, go into CMakeLists.txt and comment out parts that fail to compile.
If you have good coverage by unit-tests you should be able to identify and locate problems really quickly. Continuous integration (e.g. Jenkins, Buildbot, Travis (GitHub)) can be very helpful. They will run your tests even if some developers have not done so before committing.
Also you assume that a non-compiling class (and its test) would just have to be removed from the build. But what about transitive dependencies, where a non-compiling class breaks compilation of other classes or leads to linking errors. What about tests that break the build? All these things happen during development.
I suggest you separate your build into many libraries each having its own test runner. Put together what belongs together (cohesion). Try to minimize dependencies in your compilation also (dependency injection, interfaces, ...). This will allow to keep development going by having compiling libraries and test runners even if some libs do not compile (for some time).
I guess you could create one test executable per spec source, (using a foreach() loop) and then do something like:
make spec_foo && ./spec_foo
This will only try to build the binary matching the test you want to run
But if your build often fails it may be a sign of some bad design in your production code ...
It seems like there are a few answers that kind-of, sort-of make sense, but that I don't know how to carry out. And I haven't found a comprehensive answer.
The First Problem
Google Test should not be an installed library, it should be built with the project. (See the FAQ.) As far as I can tell, this means the Google Test libraries are a dependency of my unit tests, and should be built when I run "make check" within my project for the first time. This should build Google Test libraries in some directory. I don't know how to do this. It mentions some autotools script that's deprecated, and I'm not sure what they're talking about or how to point my build at it properly.
The Second Problem
Assuming the build is successful, how do I write a test that uses my locally-compiled version of Google Test to run tests? I assume that there are a bunch of Makefile.am commands I put in my tests directory. But what are they? And what's an example of a unit test that uses Google Test?
I have solved the problem to my satisfaction! I will move on entirely now. This is basically asking for a tutorial. There are a lot of decisions that must be made, hopefully logically, so that Google Test dovetails nicely into autotools. So I apologize in advance for the long answer, but all the details should be there.
The First Problem
In order to understand the answer, the question needs to be rephrased a little. We are compiling Google Test as a library which our test code will link to. The library will not be installed. The question we want to ask is
"How do we configure autotools to compile Google Test as a library
which our test code can link against?"
In order to do that, we need to download Google Test and place it into our project. I use Github, so I do that by adding a submodule in the root path of my project:
$ git submodule add git#github.com:google/googletest.git
$ git submodule init
$ git submodule update
This downloads googletest into my root of my project:
/:
Makefile.am
configure.ac
src/:
(files for my project)
tests/:
(test files)
googletest/:
googletest/:
include/:
(headers, etc., to be included)
gtest/:
gtest.h
m4/:
(directory for m4 scripts and things)
src/:
(source files for Google Test)
I need to compile per the instructions. I only want the Google Test library to be built upon running make check, so I will use check_LTLIBRARIES. I add the following to my tests Makefile.am in /tests:
check_LTLIBRARIES = libgtest.la
libgtest_la_SOURCES = ../googletest/googletest/src/gtest-all.cc
libgtest_la_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/googletest/googletest
libgtest_la_LDFLAGS = -pthread
This requires subdir-objects to be enabled in configure.ac. That is accomplished by adding it to the AM_INIT_AUTOMAKE line. I also need to include the makefile in AC_CONFIG_FILES. We also want to use libtool, because we are compiling library files (I'll explain why and how that works in a moment). To use libtool, we add AM_PROG_AR, LT_INIT. We want autoreconf to install m4 macros to /m4, and then we want automake to find them, so we need AC_CONFIG_MACRO_DIRS. My configure.ac has lines updated:
AM_INIT_AUTOMAKE([-Wall -Werror subdir-objects])
...
AM_PROG_AR
LT_INIT
AC_CONFIG_MACRO_DIRS([m4])
...
AC_CONFIG_FILES([Makefile
src/Makefile
tests/Makefile
])
I also need to include the subdirectory and a line pointing to the macros in the /m4 macros directory in my /Makefile.am:
ACLOCAL_AMFLAGS = -I m4
SUBDIRS = src tests
What has this done? Libtool has been enabled with AM_PROG_AR and LT_INIT. The check_LTLIBRARIES means we will use libtool to create what's called a convenience library called libgtest.la. With subdir-objects enabled, it will be built into the /tests directory, but not installed. This means that, whenever we want to update our tests, we don't have to recompile the Google Test library libgtest.la. This will save time when testing and help us iterate faster. Then, we will want to compile our unit tests against it later as we update them. The library will only be compiled upon running make check, saving time by not compiling it if all we want to do is make or make install.
The Second Problem
Now, the second problem needs to be refined: How do you (a) create a test (b) that is linked to the Google Test libraries and thus uses them? The questions are kind of intertwined, so we answer them at once.
Creating a test is just a matter of putting the following code into a gtest.cpp file located at /tests/gtest.cpp:
#include "gtest/gtest.h" // we will add the path to C preprocessor later
TEST(CategoryTest, SpecificTest)
{
ASSERT_EQ(0, 0);
}
int main(int argc, char **argv)
{
::testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
This runs only the simple test 0=0. To create a test for your library, you need to read the primer. You'll notice we don't need a header for this (yet). We are linking to the file "gtest/gtest.h", so we'll need to make sure that we tell automake to include a directory that has gtest/gtest.h.
Next, we need to tell automake that we want to build a test and run it. The test is going to build into an executable that we don't want to install. Then automake is going to run that executable. It will report whether that executable says the tests passed or failed.
Automake does that by looking in the makefile for the variable check_PROGRAMS. These are the programs it will compile, but it won't necessarily run them. So we add to /tests/Makefile.am:
check_PROGRAMS = gtest
gtest_SOURCES = gtest.cpp
gtest_LDADD = libgtest.la
gtest_LDFLAGS = -pthread
gtest_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/googletest/googletest -pthread
The gtest_SOURCES finds the /tests/gtest.cpp file and compiles it. gtest_LDADD links against libgtest.la which will be compiled into the /tests directory. Google wants us to use the gtest_LDFLAGS line to enable pthreads. Finally, we need to include the location where the header "gtest/gtest.h" will be found, and that is the gtest_CPPFLAGS line. Google also wants us to include the /googletest/googletest location, and include the
The state of things: The Google Test library libgtest.la will compile with make into the directory /tests, but not be installed. The binary gtest will only be compiled with make check, but will not be installed.
Next we want to tell automake to actually run the compiled binary gtest and report errors. This is accomplished by adding a line to /tests/Makefile.am:
TESTS = gtest
The final /tests/Makefile.am looks like this:
check_LTLIBRARIES = libgtest.la
libgtest_la_SOURCES = ../googletest/googletest/src/gtest-all.cc
libgtest_la_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/googletest/googletest -pthread
check_PROGRAMS = gtest demo
gtest_SOURCES = gtest.cpp ../src/fields.cpp
gtest_LDADD = libgtest.la
gtest_LDFLAGS = -pthread
gtest_CPPFLAGS = -I$(top_srcdir)/googletest/googletest/include -I$(top_srcdir)/src
demo_SOURCES = demo.cpp ../src/fields.cpp
demo_CPPFLAGS = -I$(top_srcdir)/src
TESTS = gtest
Now, autoreconf -fiv (note any errors and hopefully fix them) from /, and make check and you should get a test that runs:
build(dev)$ make check
Making check in tests
/Applications/Xcode.app/Contents/Developer/usr/bin/make gtest
make[2]: `gtest' is up to date.
/Applications/Xcode.app/Contents/Developer/usr/bin/make check-TESTS
PASS: gtest
============================================================================
Testsuite summary for IonMotion 0.0.1
============================================================================
# TOTAL: 1
# PASS: 1
# SKIP: 0
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
Here is a sample Makefile.am for the unit test project (project name: TestProject). It depends on GTEST and GMOCK:
Makefile.am
#######################################
# The list of executables we are building seperated by spaces
# the 'bin_' indicates that these build products will be installed
# in the $(bindir) directory. For example /usr/bin
#bin_PROGRAMS=exampleProgram
# Because a.out is only a sample program we don't want it to be installed.
# The 'noinst_' prefix indicates that the following targets are not to be
# installed.
noinst_PROGRAMS=utTestProject
#######################################
# Build information for each executable. The variable name is derived
# by use the name of the executable with each non alpha-numeric character is
# replaced by '_'. So a.out becomes a_out and the appropriate suffex added.
# '_SOURCES' for example.
# Sources for the a.out
utTestProject_SOURCES= \
utTestProject.cpp
# Library dependencies
utTestProject_LDADD = \
$(top_srcdir)/../TestProject/build/${host}/libTestProject/.libs/libTestProject.a \
../$(PATH_TO_GTEST)/lib/libgtest.a \
../$(PATH_TO_GMOCK)/lib/libgmock.a
# Compiler options for a.out
utTestProject_CPPFLAGS = \
-std=c++11 \
-I../$(PATH_TO_GTEST)/include \
-I../$(PATH_TO_GMOCK)/include \
-I$(top_srcdir)/include \
-I$(top_srcdir)/..
TESTS = utTestProject
TESTS_ENVIRONMENT = export UT_FOLDER_PATH=$(top_srcdir)/utTestProject; \
export GTEST_OUTPUT="xml";
Compiling gtest:
# Useful vars
SourceVersionedArchiveFolderName="gtest-1.7.0"
#
# Make it
#
pushd .
cd ./${SourceVersionedArchiveFolderName}/make
make gtest.a
if [ $? != 0 ]; then
echo "$0: Make failed"
exit 1
fi
popd
It's worth noting that Googletest doesn't officially maintain its Autotools integration anymore:
Before settling on CMake, we have been providing hand-maintained build
projects/scripts for Visual Studio, Xcode, and Autotools. While we
continue to provide them for convenience, they are not actively
maintained any more. We highly recommend that you follow the
instructions in the above sections to integrate Google Test with your
existing build system.
https://github.com/google/googletest/tree/master/googletest#legacy-build-scripts
It's now recommended to build Googletest with CMake.
Making GoogleTest's source code available to the main build can be
done a few different ways:
Download the GoogleTest source code manually and place it at a known
location. This is the least flexible approach and can make it more
difficult to use with continuous integration systems, etc.
Embed the
GoogleTest source code as a direct copy in the main project's source
tree. This is often the simplest approach, but is also the hardest to
keep up to date. Some organizations may not permit this method.
Add
GoogleTest as a git submodule or equivalent. This may not always be
possible or appropriate. Git submodules, for example, have their own
set of advantages and drawbacks.
Use CMake to download GoogleTest as
part of the build's configure step. This is just a little more
complex, but doesn't have the limitations of the other methods.
https://github.com/google/googletest/tree/master/googletest#incorporating-into-an-existing-cmake-project
I have e.g. 10 C++ projects in a solution SolA and want to UnitTest them with GoogleTest:
so I created a new solution SolATest and for every project of SolA an unit test project in SolATest!
Is it a good approach to load the SolA libraries implicit in SolATest/Projects and run every test project as an executable:
#include <iostream>
#include "gmock/gmock.h"
#include "gtest/gtest.h"
int main(int argc, char **argv)
{
::testing::InitGoogleMock(&argc, argv);
int value = RUN_ALL_TESTS();
std::getchar(); // to hold the terminal open
return value;
}
or is there a more convenience way -> e.g. only have one executable in SolATest and load the other test projects as libraries (IMHO to have all cpp files in one test project is confusing)?!
Thx for any help
Either approach should work; it just depends on your preference. I tend to follow a project structure like the following:
Solution
|-- ProjectA
|-- ProjectATests
|-- ProjectB
|-- ProjectBTests
`-- TestLib
Where the projects (ProjectA, ProjectB) are libraries, and each test project (ProjectATests, ProjectBTests) is an executable. Note that we do not separate unit tests into a separate solution; they are always built and run alongside the production code. I like this structure for a few reasons:
It's easier to run just the tests that are related to your changes.
The development workflow is a bit more efficient, since when making changes to one library you only have to rebuild and link the corresponding test.
Whether you create a single test project or multiple, I would definitely recommend putting the project in the same solution as the code under test. Further, I would recommend setting up a post-build step for the test project(s) which runs the tests and fails the build if they don't pass.
Lastly, you might be wondering about that 'TestLib' project. I use that for the gtest/gmock fused sources, the definition of main(), and any other utilities that are shared between tests. That eliminates (or at least, reduces) code duplication between the various test projects.