In setting up my most recent project I am attempting to use GoogleTest. Currently I have this:
macro(add_test_l TEST_NAME)
# Create the executable and use the last arguments as source files.
add_executable(${TEST_NAME} ${ARGN})
# Link in the AMR and gtest libraries.
target_link_libraries(${TEST_NAME}
${PROJECT_NAME}
gtest
gmock
gtest_main
)
# Add the test to gtest.
add_test(
NAME ${TEST_NAME}
COMMAND ${TEST_NAME}
)
list(APPEND TEST_SOURCE_FILES ${ARGN})
endmacro()
Then I can add my tests like add_test_l(Element Sets/Element.test.cpp) which is working and convenient. But this of course creates an executable for each test, which is not a bad thing as it allows for quick testing of a single file.
Though I do want the ability to run all the tests with a single exe (that way CI will be easier) so I have that list at the end of my macro and after adding all my individual tests I have:
add_executable(all_tests ${TEST_SOURCE_FILES})
target_link_libraries(all_tests
${PROJECT_NAME}
gtest
gmock
gtest_main
)
Which creates an EXE to run all my test cases.
This does not seem efficient as I compile all my files twice. Is there a better way for me to achieve the desired outcome? Perhaps I can just add an option to enable / disable individual vs all tests exes.
It is unnecessary to have an executable per each file. Build one executable for all tests and learn the gtest option --gtest_filter. You can run each test individually:
all_tests --gtest_filter=Element.Test
Or you can run all Element tests like the macro add_test_l does it:
all_tests --gtest_filter=Element.*
More info about command line options is available:
all_tests --help
One of the useful commands:
all_tests --gtest_list_tests
Related
I'm learning CMake through a C++ hobby project using Visual Studio 2017, and the way I have it set up, I have one folder for source code and one folder for test code. I build the source code as a static library, and build the test code as an executable (using Catch2). The problem I have now is that these are two separate targets, and whenever one or both of these targets are rebuilt I want to run the testing executable. Now I can find out how to run a post-build event using ADD_CUSTOM_COMMAND, but that only works for a single target. Putting multiple targets after "TARGET" leads to only the last target being used (I tested this), and duplicating the custom command can lead to the tests being run twice, and also it seems like poor code style. Is there any way to do this elegantly? My CMake file looks like this:
# CMakeList.txt : Top-level CMake project file, do global configuration
# and include sub-projects here.
#
cmake_minimum_required (VERSION 3.8)
project ("SheepyEngine")
set (CMAKE_CXX_STANDARD 17)
set (HEADER_FILES 3rdParty/CImg/CImg.h)
set (SOURCE_DIRECTORY Source)
set (TEST_DIRECTORY Test)
# Include sub-projects.
add_subdirectory ("Source")
add_subdirectory ("Test")
# Include libraries
include_directories (
"${CMAKE_CURRENT_LIST_DIR}/3rdParty/CImg"
"${CMAKE_CURRENT_LIST_DIR}/3rdParty/Catch2/single_include"
)
add_library (SheepyEngine STATIC
"${SOURCE_DIRECTORY}/Game.cpp"
"${SOURCE_DIRECTORY}/Game.h"
"${SOURCE_DIRECTORY}/GameObject.h"
${HEADER_FILES})
target_include_directories(SheepyEngine PRIVATE ${CMAKE_CURRENT_LIST_DIR}/3rdParty/CImg/)
add_executable(SheepyEngineTest "${TEST_DIRECTORY}/test.cpp" "3rdParty/Catch2/single_include/catch.hpp")
target_include_directories(SheepyEngineTest PRIVATE ${CMAKE_CURRENT_LIST_DIR}/3rdParty/Catch2/)
# TODO: Add tests and install targets if needed.
if(${RUN_TESTS})
ADD_CUSTOM_COMMAND(
TARGET SheepyEngineTest SheepyEngine
POST_BUILD
COMMAND ${CMAKE_CURRENT_LIST_DIR}/Build/Debug/SheepyEngineTest.exe
)
endif()
The SheepyTestProgram target needs to be dependent on SheepyEngine:
target_link_libraries(SheepyEngineTest SheepyEngine)
Then the target of add_custom_command will be just SheepyEngineTest (add_custom_command accepts only a single target).
I wanted to integrate ctest to a c++/c project. I use google tests to write unit tests.
Relevant part of my CMakeLists.txt looks like this:
...
####### CREATING EXE #######
add_executable(test_exe main.cpp test.cpp)
target_link_libraries(test_exe GTest::GTest GTest::Main)
set_target_properties (test_exe PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${UNIT_TEST_BIN_OUTPUT_DIR})
add_test(test_exe test_exe)
As you can see i specified the output directory of my executable (UNIT_TEST_BIN_OUTPUT_DIR).
The executable works fine on its own when I use the terminal:
cd <UNIT_TEST_BIN_OUTPUT_DIR>
./test_exe
I want to use ctest to execute my tests. So I go to the "ctest folder" generated by cmake. Here I want to use ctest to execute all test added by "add_test" in cmake.
user#user:~/<dir to cmake>/cmake/unit_tests$ ctest
Test project /<dir to cmake>/cmake/unit_tests
Start 1: test_exe
Could not find executable test_exe
Looked in the following places:
test_exe
test_exe
Release/test_exe
Release/test_exe
Debug/test_exe
Debug/test_exe
MinSizeRel/test_exe
MinSizeRel/test_exe
RelWithDebInfo/test_exe
RelWithDebInfo/test_exe
Deployment/test_exe
Deployment/test_exe
Development/test_exe
Development/test_exe
Unable to find executable: test_exe
1/1 Test #1: test_exe ......***Not Run 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.00 sec
The following tests FAILED:
1 - test_exe (Not Run)
Errors while running CTest
If I put the "test_exe" in one of the shown paths it works fine. But I don't want them to be there.
My Question:
Is there a way to tell ctest it should look in UNIT_TEST_BIN_OUTPUT_DIR in order to find the executable?
Documentation for add_test specifies WORKING_DIRECTORY option for long form of the command. Value of this option is used as a directory in which test operates:
add_test(NAME test_exe COMMAND test_exe WORKING_DIRECTORY ${UNIT_TEST_BIN_OUTPUT_DIR})
If you just want the test to find the executable, it is sufficient to use
add_test(NAME test_exe COMMAND test_exe)
This is a long form of add_test command. In this form, CMake checks whether COMMAND is a target name, and, if it is so, replaces it with an absolute path to the executable corresponded to that target. Such way the test can be run from any directory.
Note, that automatic replacement of the target doesn't work for a short form of add_test which you use.
Using CMake 3.20 and greater, you can tell CTest which directory contains your tests by using a CLI option:
ctest --test-dir /path/to/your/tests
This is a less-invasive solution for existing tests, for which you don't want to modify the CMake files.
In our projects we always specify the path when we call add_test(), e.g.:
add_test( ${filename} ${CMAKE_CURRENT_BINARY_DIR}/${filename} )
My configuration has CMake 3.6, Visual Studio 2015 and latest Google test from GitHub. I add my unit tests through one of my cmake functions addGtest and do the build. After this I can run the test from my RUN_TESTS target or using ctrl + F5 in VS and works as expected.
The final goal is to run the unit tests at build time using the CMake dependency management. For now, as a first step, I have enhanced my function to create a custom_target (included the entire function, in case there are unforeseen issues in the working part), but not build it:
function (addGtest)
# vvvv this part works as explained vvvv #
set (optBOOLS)
set (optSINGLES EXE)
set (optLISTS DLL_LIST)
cmake_parse_arguments (myARGS
"${optBOOLS}" "${optSINGLES}" "${optLISTS}" ${ARGN})
# addExecutable is a function that adds target executables
set(myARGS_DLL_LIST gtest_main gtest "${myARGS_DLL_LIST}")
addExecutable (EXE ${myARGS_EXE} DLL_LIST ${myARGS_DLL_LIST} ${myARGS_UNPARSED_ARGUMENTS})
add_test (NAME ${myARGS_EXE} COMMAND ${myARGS_EXE} WORKING_DIRECTORY
${CMAKE_INSTALL_PREFIX}/$<$<CONFIG:Release>:Release>$<$<CONFIG:Debug>:Debug>/bin
) # so it can be run using ctest
# ^^^^ this part works as explained ^^^^ #
add_custom_target (${myARGS_EXE}.tgt DEPENDS ${myARGS_EXE}
COMMAND ${myARGS_EXE} --gtest_output="xml:${myARGS_EXE}.xml"
WORKING_DIRECTORY ${CMAKE_INSTALL_PREFIX}/$<$<CONFIG:Release>:Release>$<$<CONFIG:Debug>:Debug>/bin
)
endfunction (addGtest)
As expected when I perform the build, a new target, say, utMyTest.tgt is added to VS, but it is not built. Now when I build this new target by hand in VS, I expect that the test will be run. But it doesn't and gives the following error:
1> The filename, directory name, or volume label syntax is incorrect.
I tried providing full path to the COMMAND option, removing double quotes around --gtest_output value, but to no avail. On the other hand when I cd to the working directory in a command line window and invoke the exe, it works fine!!
The first question is how do I fix it to run the test by building this new target? After that, I plan to add_custom_target (${myARGS_EXE}.run) and add_dependencies (${myARGS_EXE}.run ${myARGS_EXE}.tgt). Would this then run the test whenever the exe changes? Or should I do something else? Thank you for your help.
Could not add so much details in the comment, hence this answer.
1. Answer to the original problem
Since I needed the configuration dependent path in the WORKING_DIRECTORY option of the add_custom_target command, but cannot pass generator expressions to it, the idea is to use the CMAKE_CFG_INTDIR variable so:
add_custom_target (${myARGS_EXE}.tgt
DEPENDS ${myARGS_EXE}
COMMAND ${myARGS_EXE} --gtest_output=xml:${myARGS_EXE}.xml
WORKING_DIRECTORY ${CMAKE_INSTALL_PREFIX}/${CMAKE_CFG_INTDIR}/bin
)
Now, when you build the above target, the unit test is run in the WORKING_DIRECTORY which is not entirely desirable, since that is the install directory for libs and exes. It would be really nice to ...
2. Run the unit test from its build directory
While, at the same time, picking up the DLL paths from within Visual Studio, and storing the Gtest generated .xml file in the build directory. This is the solution:
In CMake version 3.10 CMAKE_MSVCIDE_RUN_PATH property was added. In the project wide CMakeLists.txt, set(CMAKE_MSVCIDE_RUN_PATH ${CMAKE_INSTALL_PREFIX}/${CMAKE_CFG_INTDIR}/bin) - thanks to this solution #3, we can appendPATH to point to our install directory. And then replace the above add_custom_target command with this:
add_custom_command (
TARGET ${myARGS_EXE} POST_BUILD
COMMAND ${myARGS_EXE} --gtest_output=xml:${myARGS_EXE}.xml
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}
)
This solution avoids the mess of creating additional targets. Clearly only when myARGS_EXE is built, the unit test is run. Obviously myARGS_EXE's transitive dependency on other DLLs is covered also.
If you have other elegant solutions, please post.
So firstly I'm new to testing frameworks and relatively new to C++ but am trying to wrap my head around GoogleTest. I'm working on a Windows machine, running "Git for Windows" (MSYS) and MinGW whilst using Sublime Text as my code editor. I am using make as my build tool, although the more I learn about cmake and its cross-platform focus makes me wonder if I should switch to let cmake create makefiles for me. (that's probably a whole other question)
What I'm struggling to understand is what precisly to do with the GoogleTest source package. I realise that I need to build the source into the library and then include that when compiling my tests, but how should I go about doing this? Google includes a cmake build script that generates env/compilier specific makefiles for building. Should I be using this? I feel like if i do so and it blindly works a lot of what is happening under the hood will go over my head. The readme file isn't eliviating my issues, as it implies that i should be building the library and my tests each time i wish the run them. Shouldn't a library be a standalone archive that needs compiling only once? I'm confused and I'm sure its my fault but i'd appreciate it if someone shed some light on this process for me.
You should keep in mind that make will not rebuild gtest if you don't change anything in gtest source code.
Below is my approach to the usage of cmake and gtest for unit testing.
You can add gtest source code by adding it as subdirectory in the root CMakeLists.txt file.
add_subdirectory(${CMAKE_SOURCE_DIR}/thirdparty/gtest ${CMAKE_CURRENT_BINARY_DIR}/gtest)
include_directories(${CMAKE_SOURCE_DIR}/thirdparty/gtest/include ${CMAKE_SOURCE_DIR}/thirdparty/gtest)
My application consist of individual modules containing test folder for unit testing. I have the following boilerplate loop to add each test to the global scope.
file(GLOB TEST_SRC_FILES *.cpp)
foreach(TEST_SRC_PATH ${TEST_SRC_FILES})
#get filename of your test without extension
get_filename_component(TEST_NAME ${TEST_SRC_PATH} NAME_WE)
add_executable(${TEST_NAME} ${TEST_NAME})
#here you link the test executable with gtest
target_link_libraries(${TEST_NAME} gtest gtest_main)
#-----------------------------
# you can link here your test to external libraries
#-----------------------------
add_test(${TEST_NAME} ${TEST_NAME})
#this is a list of all tests
set(PROJECT_TEST_NAMES ${PROJECT_TEST_NAMES} ${TEST_NAME})
endforeach()
#This assigns the list of tests to a property. This make the list available from the root scope.
get_property(UNIT_TESTS GLOBAL PROPERTY UNIT_TESTS)
set(UNIT_TESTS ${UNIT_TESTS} ${PROJECT_TEST_NAMES})
set_property(GLOBAL PROPERTY UNIT_TESTS ${UNIT_TESTS} )
Finally, in the root scope, I add a custom target named check which runs ctest on my unit tests.
#-----------------------------
# Running unit tests
#-----------------------------
get_property(UNIT_TESTS GLOBAL PROPERTY UNIT_TESTS)
if(DEFINED UNIT_TESTS)
add_custom_target(check COMMAND ctest -VV
DEPENDS ${UNIT_TESTS})
endif()
When I run make check, it runs unit tests from all modules, whereas make compiles without tests.
Sometimes it's good to check that certain things fail to build, e.g.:
// Next line should fail to compile: can't convert const iterator to iterator.
my_new_container_type::iterator it = my_new_container_type::const_iterator();
Is it possible to incorporate these types of things into CMake/CTest? I'm looking for something like this in CMakeLists.txt:
add_build_failure_executable(
test_iterator_conversion_build_failure
iterator_conversion_build_failure.cpp)
add_build_failure_test(
test_iterator_conversion_build_failure
test_iterator_conversion_build_failure)
(Of course, these specific CMake directives don't exist, to the best of my knowledge.)
You can do this more or less as you described. You can add a target which will fail to compile, then add a test which invokes cmake --build to try to build the target. All that remains is to set the test property WILL_FAIL to true.
So, say you have your tests in a file named "will_fail.cpp" which contains:
#if defined TEST1
non-compiling code for test 1
#elif defined TEST2
non-compiling code for test 2
#endif
Then you can have something like the following in your CMakeLists.txt:
cmake_minimum_required(VERSION 3.0)
project(Example)
include(CTest)
# Add a couple of failing-to-compile targets
add_executable(will_fail will_fail.cpp)
add_executable(will_fail_again will_fail.cpp)
# Avoid building these targets normally
set_target_properties(will_fail will_fail_again PROPERTIES
EXCLUDE_FROM_ALL TRUE
EXCLUDE_FROM_DEFAULT_BUILD TRUE)
# Provide a PP definition to target the appropriate part of
# "will_fail.cpp", or provide separate files per test.
target_compile_definitions(will_fail PRIVATE TEST1)
target_compile_definitions(will_fail_again PRIVATE TEST2)
# Add the tests. These invoke "cmake --build ..." which is a
# cross-platform way of building the given target.
add_test(NAME Test1
COMMAND ${CMAKE_COMMAND} --build . --target will_fail --config $<CONFIGURATION>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
add_test(NAME Test2
COMMAND ${CMAKE_COMMAND} --build . --target will_fail_again --config $<CONFIGURATION>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
# Expect these tests to fail (i.e. cmake --build should return
# a non-zero value)
set_tests_properties(Test1 Test2 PROPERTIES WILL_FAIL TRUE)
You can obviously wrap all of this into a function or macro if you have a lot of these to write.
#Fraser's answer is a good approach, in particular the WILL_FAIL property is good advice. There is an alternative to making the failing target part of the main project though. The use case in the question is pretty much what the ctest --build-and-test mode is meant for. Rather than making the expected-to-fail target part of the main build, you can put it in its own separate mini project which is then built as part of a test. An example of how this might look in the main project goes something like this:
add_test(NAME iter_conversion
COMMAND ${CMAKE_CTEST_COMMAND}
--build-and-test
${CMAKE_CURRENT_LIST_DIR}/test_iter
${CMAKE_CURRENT_BINARY_DIR}/test_iter
--build-generator ${CMAKE_GENERATOR}
--test-command ${CMAKE_CTEST_COMMAND}
)
set_tests_properties(iter_conversion PROPERTIES WILL_FAIL TRUE)
This has the advantage that it will be part of the project's test results and will therefore be more likely to get executed regularly as part of normal testing processes. In the above example, the test_iter directory is essentially it's own separate project. If you need to pass information to it from the main build, you can do that by adding --build-options to define cache variables to pass to it's CMake run. Check the latest docs for recently corrected/clarified help on this area.
For the particular example in the question I agree with the comment. This is should be tested with a static_assert rather than a build failure test.
For a way of adding build failure test with CMake (in the cases where this is a good idea) I recently created a library which more or less allows adding such tests with a single CMake function call. It expands on the accepted answer and allows build-failure test and checks that the failure happened because of a specific error (the expected error can be provided in CMake or in a source file): https://github.com/iboB/icm/blob/master/icm_build_failure_testing.cmake