Expected build-failure tests in CMake - c++

Question

Sometimes it's good to check that certain things fail to build, e.g.:
// Next line should fail to compile: can't convert const iterator to iterator.
my_new_container_type::iterator it = my_new_container_type::const_iterator();
Is it possible to incorporate these types of things into CMake/CTest? I'm looking for something like this in CMakeLists.txt:
add_build_failure_executable(
test_iterator_conversion_build_failure
iterator_conversion_build_failure.cpp)
add_build_failure_test(
test_iterator_conversion_build_failure
test_iterator_conversion_build_failure)
(Of course, these specific CMake directives don't exist, to the best of my knowledge.)

Answer 1

You can do this more or less as you described. You can add a target which will fail to compile, then add a test which invokes cmake --build to try to build the target. All that remains is to set the test property WILL_FAIL to true.
So, say you have your tests in a file named "will_fail.cpp" which contains:
#if defined TEST1
non-compiling code for test 1
#elif defined TEST2
non-compiling code for test 2
#endif
Then you can have something like the following in your CMakeLists.txt:
cmake_minimum_required(VERSION 3.0)
project(Example)
include(CTest)
# Add a couple of failing-to-compile targets
add_executable(will_fail will_fail.cpp)
add_executable(will_fail_again will_fail.cpp)
# Avoid building these targets normally
set_target_properties(will_fail will_fail_again PROPERTIES
EXCLUDE_FROM_ALL TRUE
EXCLUDE_FROM_DEFAULT_BUILD TRUE)
# Provide a PP definition to target the appropriate part of
# "will_fail.cpp", or provide separate files per test.
target_compile_definitions(will_fail PRIVATE TEST1)
target_compile_definitions(will_fail_again PRIVATE TEST2)
# Add the tests. These invoke "cmake --build ..." which is a
# cross-platform way of building the given target.
add_test(NAME Test1
COMMAND ${CMAKE_COMMAND} --build . --target will_fail --config $<CONFIGURATION>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
add_test(NAME Test2
COMMAND ${CMAKE_COMMAND} --build . --target will_fail_again --config $<CONFIGURATION>
WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
# Expect these tests to fail (i.e. cmake --build should return
# a non-zero value)
set_tests_properties(Test1 Test2 PROPERTIES WILL_FAIL TRUE)
You can obviously wrap all of this into a function or macro if you have a lot of these to write.

Answer 2

#Fraser's answer is a good approach, in particular the WILL_FAIL property is good advice. There is an alternative to making the failing target part of the main project though. The use case in the question is pretty much what the ctest --build-and-test mode is meant for. Rather than making the expected-to-fail target part of the main build, you can put it in its own separate mini project which is then built as part of a test. An example of how this might look in the main project goes something like this:
add_test(NAME iter_conversion
COMMAND ${CMAKE_CTEST_COMMAND}
--build-and-test
${CMAKE_CURRENT_LIST_DIR}/test_iter
${CMAKE_CURRENT_BINARY_DIR}/test_iter
--build-generator ${CMAKE_GENERATOR}
--test-command ${CMAKE_CTEST_COMMAND}
)
set_tests_properties(iter_conversion PROPERTIES WILL_FAIL TRUE)
This has the advantage that it will be part of the project's test results and will therefore be more likely to get executed regularly as part of normal testing processes. In the above example, the test_iter directory is essentially it's own separate project. If you need to pass information to it from the main build, you can do that by adding --build-options to define cache variables to pass to it's CMake run. Check the latest docs for recently corrected/clarified help on this area.

Answer 3

For the particular example in the question I agree with the comment. This is should be tested with a static_assert rather than a build failure test.
For a way of adding build failure test with CMake (in the cases where this is a good idea) I recently created a library which more or less allows adding such tests with a single CMake function call. It expands on the accepted answer and allows build-failure test and checks that the failure happened because of a specific error (the expected error can be provided in CMake or in a source file): https://github.com/iboB/icm/blob/master/icm_build_failure_testing.cmake