I'm trying CTest in CMake in order to automatically run some of my tests using make test target. The problem is CMake does not "understand" that the test I'm willing to run has to be built since it is part of the project.
So I'm looking for a way to explicitly specify this dependency.
It is arguably a bug in CMake (previously tracked here) that this doesn't work out of the box. A workaround is to do the following:
add_test(TestName ExeName)
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND}
DEPENDS ExeName)
Then you can run make check and it will compile and run the test. If you have several tests, then you would have to use DEPENDS exe1 exe2 exe3 ... in the above line.
There is actually a way to use make test. You need to define the build of the test executable as one of the tests and then add dependencies between the tests. That is:
ADD_TEST(ctest_build_test_code
"${CMAKE_COMMAND}" --build ${CMAKE_BINARY_DIR} --target test_code)
ADD_TEST(ctest_run_test_code test_code)
SET_TESTS_PROPERTIES(ctest_run_test_code
PROPERTIES DEPENDS ctest_build_test_code)
I use a variant of richq's answer. In the top-level CMakeLists.txt, I add a custom target, build_and_test, for building and running all tests:
find_package(GTest)
if (GTEST_FOUND)
enable_testing()
add_custom_target(build_and_test ${CMAKE_CTEST_COMMAND} -V)
add_subdirectory(test)
endif()
In the various sub-project CMakeLists.txt files under test/, I add each test executable as a dependency of build_and_test:
include_directories(${CMAKE_SOURCE_DIR}/src/proj1)
include_directories(${GTEST_INCLUDE_DIRS})
add_executable(proj1_test proj1_test.cpp)
target_link_libraries(proj1_test ${GTEST_BOTH_LIBRARIES} pthread)
add_test(proj1_test proj1_test)
add_dependencies(build_and_test proj1_test)
With this approach, I just need to make build_and_test instead of make test (or make all test), and it has the benefit of only building test code (and its dependencies). It's a shame I can't use the target name test. In my case, it's not so bad because I have a top-level script that does out-of-tree debug and release (and cross-compiled) builds by calling cmake and then make, and it translates test into build_and_test.
Obviously, the GTest stuff isn't required. I just happen to use/like Google Test, and wanted to share a complete example of using it with CMake/CTest. IMHO, this approach also has the benefit of allowing me to use ctest -V, which shows the Google Test output while the tests run:
1: Running main() from gtest_main.cc
1: [==========] Running 1 test from 1 test case.
1: [----------] Global test environment set-up.
1: [----------] 1 test from proj1
1: [ RUN ] proj1.dummy
1: [ OK ] proj1.dummy (0 ms)
1: [----------] 1 test from proj1 (1 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test case ran. (1 ms total)
1: [ PASSED ] 1 test.
1/2 Test #1: proj1_test ....................... Passed 0.03 sec
If you are using CMake >= 3.7, then the recommended approach is to use fixtures:
add_executable(test test.cpp)
add_test(test_build
"${CMAKE_COMMAND}"
--build "${CMAKE_BINARY_DIR}"
--config "$<CONFIG>"
--target test
)
set_tests_properties(test_build PROPERTIES FIXTURES_SETUP test_fixture)
add_test(test test)
set_tests_properties(test PROPERTIES FIXTURES_REQUIRED test_fixture)
This does the following:
Adds a test executable target built from test.cpp
Adds a test_build "test" that runs Cmake to build target test
Marks the test_build test to be a setup task of fixture test_fixture
Add a test test that just runs the test executable
Marks the test test to need fixture test_fixture.
So, every time test test is to be run, it first runs test test_build, which builds the necessary executable.
If you are trying to emulate make check, you may find this wiki entry usefull :
http://www.cmake.org/Wiki/CMakeEmulateMakeCheck
I have just checked that is does what it says with success (CMake 2.8.10).
Save yourself the headache:
make all test
Works out of the box for me and will build dependencies before running the test. Given how simple this is, it almost makes the native make test functionality convenient because it gives you the option of running the last compiling tests even if your code is broken.
For CMake 3.10 or later, another option is to use the TEST_INCLUDE_FILES directory property to set up a script that triggers a build before a test is run. In your outermost CMakeLists.txt add the following code:
set_property(DIRECTORY APPEND
PROPERTY TEST_INCLUDE_FILES "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake")
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/BuildTestTarget.cmake"
"execute_process(COMMAND \"${CMAKE_COMMAND}\""
" --build \"${CMAKE_BINARY_DIR}\""
" --config \"\$ENV{CMAKE_CONFIG_TYPE}\")")
The actual test configuration is passed through to the build via the environment variable CMAKE_CONFIG_TYPE. Optionally you can add a --target option to only build targets required by the test.
This is what I hammered out and have been using:
set(${PROJECT_NAME}_TESTS a b c)
enable_testing()
add_custom_target(all_tests)
foreach(test ${${PROJECT_NAME}_TESTS})
add_executable(${test} EXCLUDE_FROM_ALL ${test}.cc)
add_test(NAME ${test} COMMAND $<TARGET_FILE:${test}>)
add_dependencies(all_tests ${test})
endforeach(test)
build_command(CTEST_CUSTOM_PRE_TEST TARGET all_tests)
string(CONFIGURE \"#CTEST_CUSTOM_PRE_TEST#\" CTEST_CUSTOM_PRE_TEST_QUOTED ESCAPE_QUOTES)
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake" "set(CTEST_CUSTOM_PRE_TEST ${CTEST_CUSTOM_PRE_TEST_QUOTED})" "\n")
YMMV
Derrick's answer, simplified and commented:
# It is impossible to make target "test" depend on "all":
# https://gitlab.kitware.com/cmake/cmake/-/issues/8774
# Set a magic variable in a magic file that tells ctest
# to invoke the generator once before running the tests:
file(WRITE "${CMAKE_BINARY_DIR}/CTestCustom.cmake"
"set(CTEST_CUSTOM_PRE_TEST ${CMAKE_MAKE_PROGRAM})\n"
)
It is not perfectly correct, as it does not solve the concurrency problem of running ninja all test, in case anyone does that. On the contrary, because now, you have two ninja processes.
(Ftr, I also shared this solution here.)
All above answers are perfect. But actually CMake use CTest as its testing tools, so the standard method (I think it is) to do the mission is:
enable_testing ()
add_test (TestName TestCommand)
add_test (TestName2 AnotherTestCommand)
Then run cmake and make to build the targets. After that, you can either run make test, or just run
ctest
you will get the result. This is tested under CMake 2.8 .
Check details at: http://cmake.org/Wiki/CMake/Testing_With_CTest#Simple_Testing
All the answers are good, but they imply a breach of tradition to run a test by command make test. I've done this trick:
add_test(NAME <mytest>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
COMMAND sh -c "make <mytarget>; $<TARGET_FILE:<mytarget>>")
This means that the test consists of building (optionally) and running of executable target.
Related
In setting up my most recent project I am attempting to use GoogleTest. Currently I have this:
macro(add_test_l TEST_NAME)
# Create the executable and use the last arguments as source files.
add_executable(${TEST_NAME} ${ARGN})
# Link in the AMR and gtest libraries.
target_link_libraries(${TEST_NAME}
${PROJECT_NAME}
gtest
gmock
gtest_main
)
# Add the test to gtest.
add_test(
NAME ${TEST_NAME}
COMMAND ${TEST_NAME}
)
list(APPEND TEST_SOURCE_FILES ${ARGN})
endmacro()
Then I can add my tests like add_test_l(Element Sets/Element.test.cpp) which is working and convenient. But this of course creates an executable for each test, which is not a bad thing as it allows for quick testing of a single file.
Though I do want the ability to run all the tests with a single exe (that way CI will be easier) so I have that list at the end of my macro and after adding all my individual tests I have:
add_executable(all_tests ${TEST_SOURCE_FILES})
target_link_libraries(all_tests
${PROJECT_NAME}
gtest
gmock
gtest_main
)
Which creates an EXE to run all my test cases.
This does not seem efficient as I compile all my files twice. Is there a better way for me to achieve the desired outcome? Perhaps I can just add an option to enable / disable individual vs all tests exes.
It is unnecessary to have an executable per each file. Build one executable for all tests and learn the gtest option --gtest_filter. You can run each test individually:
all_tests --gtest_filter=Element.Test
Or you can run all Element tests like the macro add_test_l does it:
all_tests --gtest_filter=Element.*
More info about command line options is available:
all_tests --help
One of the useful commands:
all_tests --gtest_list_tests
I wanted to integrate ctest to a c++/c project. I use google tests to write unit tests.
Relevant part of my CMakeLists.txt looks like this:
...
####### CREATING EXE #######
add_executable(test_exe main.cpp test.cpp)
target_link_libraries(test_exe GTest::GTest GTest::Main)
set_target_properties (test_exe PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${UNIT_TEST_BIN_OUTPUT_DIR})
add_test(test_exe test_exe)
As you can see i specified the output directory of my executable (UNIT_TEST_BIN_OUTPUT_DIR).
The executable works fine on its own when I use the terminal:
cd <UNIT_TEST_BIN_OUTPUT_DIR>
./test_exe
I want to use ctest to execute my tests. So I go to the "ctest folder" generated by cmake. Here I want to use ctest to execute all test added by "add_test" in cmake.
user#user:~/<dir to cmake>/cmake/unit_tests$ ctest
Test project /<dir to cmake>/cmake/unit_tests
Start 1: test_exe
Could not find executable test_exe
Looked in the following places:
test_exe
test_exe
Release/test_exe
Release/test_exe
Debug/test_exe
Debug/test_exe
MinSizeRel/test_exe
MinSizeRel/test_exe
RelWithDebInfo/test_exe
RelWithDebInfo/test_exe
Deployment/test_exe
Deployment/test_exe
Development/test_exe
Development/test_exe
Unable to find executable: test_exe
1/1 Test #1: test_exe ......***Not Run 0.00 sec
0% tests passed, 1 tests failed out of 1
Total Test time (real) = 0.00 sec
The following tests FAILED:
1 - test_exe (Not Run)
Errors while running CTest
If I put the "test_exe" in one of the shown paths it works fine. But I don't want them to be there.
My Question:
Is there a way to tell ctest it should look in UNIT_TEST_BIN_OUTPUT_DIR in order to find the executable?
Documentation for add_test specifies WORKING_DIRECTORY option for long form of the command. Value of this option is used as a directory in which test operates:
add_test(NAME test_exe COMMAND test_exe WORKING_DIRECTORY ${UNIT_TEST_BIN_OUTPUT_DIR})
If you just want the test to find the executable, it is sufficient to use
add_test(NAME test_exe COMMAND test_exe)
This is a long form of add_test command. In this form, CMake checks whether COMMAND is a target name, and, if it is so, replaces it with an absolute path to the executable corresponded to that target. Such way the test can be run from any directory.
Note, that automatic replacement of the target doesn't work for a short form of add_test which you use.
Using CMake 3.20 and greater, you can tell CTest which directory contains your tests by using a CLI option:
ctest --test-dir /path/to/your/tests
This is a less-invasive solution for existing tests, for which you don't want to modify the CMake files.
In our projects we always specify the path when we call add_test(), e.g.:
add_test( ${filename} ${CMAKE_CURRENT_BINARY_DIR}/${filename} )
I am building a test executable using CMake. During the build process, I would like to run the executable, which returns whether the tests pass or not. If not, I would like the build to fail. However, when I use add_custom_command(... POST_BUILD ... ), and use a Makefile generator, the test executable will be deleted (explain in this question: Why does GNU make delete a file).
Is there a way to have CMake treat the executable as a .PRECIOUS, or otherwise change the CMakeLists.txt such that the executable doesn't get deleted if the tests fail?
For reference, my CMakeList.txt looks like the following (simplified from actual):
add_executable(UnitTest unittest.cpp)
add_custom_command(TARGET UnitTest POST_BUILD COMMAND $<TARGET_FILE:UnitTest>)
The solution that I was alluding to was to use add_custom_target instead of add_custom_command. While it will not delete the executable if the test fails and the build process as a whole fails if runUnitTest fails, this target does not get built as a result of building the UnitTest target specifically.
add_executable(UnitTest unittest.cpp)
add_custom_target(runUnitTest UnitTest COMMAND $<TARGET_FILE:UnitTest> DEPENDS UnitTest)
I struggled with this same problem: I have a unit test I only want to run in the following conditions:
The test has been modified.
The code under test has been modified.
The test failed its previous execution.
In the event the test is run and fails I want the test binary to remain for debugging.
The solution I ended up with uses a flag file to indicate the test has run and no longer needs to execute again. Note the flag must live in the binary directory so that other builds are not impacted. However the WORKING_DIRECTORY is set so the test can access read-only data files relative to its source location. Here's what it all looks like -- I stuck this in a macro so all my various unit tests can invoke this, the only input to the macro is the test executable:
set (TEST_FLAG_FILE ${CMAKE_CURRENT_BINARY_DIR}/${TEST_EXECUTABLE}.PASSED)
# This isn't normally needed since a rebuild will have a newer time stamp.
# But it adds robustness to handle changes to the system clock.
add_custom_command(TARGET ${TEST_EXECUTABLE}
COMMENT "Unit test ${TEST_EXECUTABLE} rebuilt; clearing flag ${TEST_FLAG_FILE}."
COMMAND ${CMAKE_COMMAND} -E remove -f ${TEST_FLAG_FILE}
POST_BUILD
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
# This command only runs if the flag isn't present or is older than the executable.
add_custom_command(OUTPUT ${TEST_FLAG_FILE}
COMMENT "Unit Test Execution: ${TEST_EXECUTABLE}"
COMMAND ${TEST_EXECUTABLE}
COMMAND ${CMAKE_COMMAND} -E touch ${TEST_FLAG_FILE}
MAIN_DEPENDENCY ${TEST_EXECUTABLE}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR})
# Any "make all" will evaluate the preceding custom command.
add_custom_target(run_${TEST_EXECUTABLE} ALL DEPENDS ${TEST_FLAG_FILE})
My configuration has CMake 3.6, Visual Studio 2015 and latest Google test from GitHub. I add my unit tests through one of my cmake functions addGtest and do the build. After this I can run the test from my RUN_TESTS target or using ctrl + F5 in VS and works as expected.
The final goal is to run the unit tests at build time using the CMake dependency management. For now, as a first step, I have enhanced my function to create a custom_target (included the entire function, in case there are unforeseen issues in the working part), but not build it:
function (addGtest)
# vvvv this part works as explained vvvv #
set (optBOOLS)
set (optSINGLES EXE)
set (optLISTS DLL_LIST)
cmake_parse_arguments (myARGS
"${optBOOLS}" "${optSINGLES}" "${optLISTS}" ${ARGN})
# addExecutable is a function that adds target executables
set(myARGS_DLL_LIST gtest_main gtest "${myARGS_DLL_LIST}")
addExecutable (EXE ${myARGS_EXE} DLL_LIST ${myARGS_DLL_LIST} ${myARGS_UNPARSED_ARGUMENTS})
add_test (NAME ${myARGS_EXE} COMMAND ${myARGS_EXE} WORKING_DIRECTORY
${CMAKE_INSTALL_PREFIX}/$<$<CONFIG:Release>:Release>$<$<CONFIG:Debug>:Debug>/bin
) # so it can be run using ctest
# ^^^^ this part works as explained ^^^^ #
add_custom_target (${myARGS_EXE}.tgt DEPENDS ${myARGS_EXE}
COMMAND ${myARGS_EXE} --gtest_output="xml:${myARGS_EXE}.xml"
WORKING_DIRECTORY ${CMAKE_INSTALL_PREFIX}/$<$<CONFIG:Release>:Release>$<$<CONFIG:Debug>:Debug>/bin
)
endfunction (addGtest)
As expected when I perform the build, a new target, say, utMyTest.tgt is added to VS, but it is not built. Now when I build this new target by hand in VS, I expect that the test will be run. But it doesn't and gives the following error:
1> The filename, directory name, or volume label syntax is incorrect.
I tried providing full path to the COMMAND option, removing double quotes around --gtest_output value, but to no avail. On the other hand when I cd to the working directory in a command line window and invoke the exe, it works fine!!
The first question is how do I fix it to run the test by building this new target? After that, I plan to add_custom_target (${myARGS_EXE}.run) and add_dependencies (${myARGS_EXE}.run ${myARGS_EXE}.tgt). Would this then run the test whenever the exe changes? Or should I do something else? Thank you for your help.
Could not add so much details in the comment, hence this answer.
1. Answer to the original problem
Since I needed the configuration dependent path in the WORKING_DIRECTORY option of the add_custom_target command, but cannot pass generator expressions to it, the idea is to use the CMAKE_CFG_INTDIR variable so:
add_custom_target (${myARGS_EXE}.tgt
DEPENDS ${myARGS_EXE}
COMMAND ${myARGS_EXE} --gtest_output=xml:${myARGS_EXE}.xml
WORKING_DIRECTORY ${CMAKE_INSTALL_PREFIX}/${CMAKE_CFG_INTDIR}/bin
)
Now, when you build the above target, the unit test is run in the WORKING_DIRECTORY which is not entirely desirable, since that is the install directory for libs and exes. It would be really nice to ...
2. Run the unit test from its build directory
While, at the same time, picking up the DLL paths from within Visual Studio, and storing the Gtest generated .xml file in the build directory. This is the solution:
In CMake version 3.10 CMAKE_MSVCIDE_RUN_PATH property was added. In the project wide CMakeLists.txt, set(CMAKE_MSVCIDE_RUN_PATH ${CMAKE_INSTALL_PREFIX}/${CMAKE_CFG_INTDIR}/bin) - thanks to this solution #3, we can appendPATH to point to our install directory. And then replace the above add_custom_target command with this:
add_custom_command (
TARGET ${myARGS_EXE} POST_BUILD
COMMAND ${myARGS_EXE} --gtest_output=xml:${myARGS_EXE}.xml
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}
)
This solution avoids the mess of creating additional targets. Clearly only when myARGS_EXE is built, the unit test is run. Obviously myARGS_EXE's transitive dependency on other DLLs is covered also.
If you have other elegant solutions, please post.
Sometimes it's good to check that certain things fail to build, e.g.:
// Next line should fail to compile: can't convert const iterator to iterator.
my_new_container_type::iterator it = my_new_container_type::const_iterator();
Is it possible to incorporate these types of things into CMake/CTest? I'm looking for something like this in CMakeLists.txt:
add_build_failure_executable(
test_iterator_conversion_build_failure
iterator_conversion_build_failure.cpp)
add_build_failure_test(
test_iterator_conversion_build_failure
test_iterator_conversion_build_failure)
(Of course, these specific CMake directives don't exist, to the best of my knowledge.)
You can do this more or less as you described. You can add a target which will fail to compile, then add a test which invokes cmake --build to try to build the target. All that remains is to set the test property WILL_FAIL to true.
So, say you have your tests in a file named "will_fail.cpp" which contains:
#if defined TEST1
non-compiling code for test 1
#elif defined TEST2
non-compiling code for test 2
#endif
Then you can have something like the following in your CMakeLists.txt:
cmake_minimum_required(VERSION 3.0)
project(Example)
include(CTest)
# Add a couple of failing-to-compile targets
add_executable(will_fail will_fail.cpp)
add_executable(will_fail_again will_fail.cpp)
# Avoid building these targets normally
set_target_properties(will_fail will_fail_again PROPERTIES
EXCLUDE_FROM_ALL TRUE
EXCLUDE_FROM_DEFAULT_BUILD TRUE)
# Provide a PP definition to target the appropriate part of
# "will_fail.cpp", or provide separate files per test.
target_compile_definitions(will_fail PRIVATE TEST1)
target_compile_definitions(will_fail_again PRIVATE TEST2)
# Add the tests. These invoke "cmake --build ..." which is a
# cross-platform way of building the given target.
add_test(NAME Test1
COMMAND ${CMAKE_COMMAND} --build . --target will_fail --config $<CONFIGURATION>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
add_test(NAME Test2
COMMAND ${CMAKE_COMMAND} --build . --target will_fail_again --config $<CONFIGURATION>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
# Expect these tests to fail (i.e. cmake --build should return
# a non-zero value)
set_tests_properties(Test1 Test2 PROPERTIES WILL_FAIL TRUE)
You can obviously wrap all of this into a function or macro if you have a lot of these to write.
#Fraser's answer is a good approach, in particular the WILL_FAIL property is good advice. There is an alternative to making the failing target part of the main project though. The use case in the question is pretty much what the ctest --build-and-test mode is meant for. Rather than making the expected-to-fail target part of the main build, you can put it in its own separate mini project which is then built as part of a test. An example of how this might look in the main project goes something like this:
add_test(NAME iter_conversion
COMMAND ${CMAKE_CTEST_COMMAND}
--build-and-test
${CMAKE_CURRENT_LIST_DIR}/test_iter
${CMAKE_CURRENT_BINARY_DIR}/test_iter
--build-generator ${CMAKE_GENERATOR}
--test-command ${CMAKE_CTEST_COMMAND}
)
set_tests_properties(iter_conversion PROPERTIES WILL_FAIL TRUE)
This has the advantage that it will be part of the project's test results and will therefore be more likely to get executed regularly as part of normal testing processes. In the above example, the test_iter directory is essentially it's own separate project. If you need to pass information to it from the main build, you can do that by adding --build-options to define cache variables to pass to it's CMake run. Check the latest docs for recently corrected/clarified help on this area.
For the particular example in the question I agree with the comment. This is should be tested with a static_assert rather than a build failure test.
For a way of adding build failure test with CMake (in the cases where this is a good idea) I recently created a library which more or less allows adding such tests with a single CMake function call. It expands on the accepted answer and allows build-failure test and checks that the failure happened because of a specific error (the expected error can be provided in CMake or in a source file): https://github.com/iboB/icm/blob/master/icm_build_failure_testing.cmake