How to unit test deliberate compilation errors of template code - c++

Question

Please note that this is NOT a duplicate of How write a unit test for verifying compiling error? as I'm not concerned about testing the correctness of external libraries or the compiler itself.
It is typical in C++, particularly when dealing with templates, to employ techniques that prevent some particular piece of code from being compiled. As these can get convoluted, what is the best way to ensure that particular pieces of code do indeed generate compiler errors?
As the test shouldn't even get compiled, you can't rely on things such as boost-test, so I guess it should be integrated in the build system? How are these issues usually approached?

Answer 1

Do it in the similar way compiler tests are written. You will have a bit of testing code in some scripting language (shell, perl, tcl etc.) that will run compiler on given snippets of code and check whether the right ones compiled and the right ones did not.
gcc uses DejaGnu, which is built on top of expect, which is itself built on top of Tcl.
If you use shell script (probably easier, DejaGnu is probably overkill), you might want to look at shUnit2.
Perl's Test::Harness system should be mostly easy to use as is.
After all, it's not that much more work to run process from C++, so writing a function to try to call compiler on a given string and check whether it outputs error for line where you expect it would not be that hard and than you can integrate it into the other boost.test-based tests.

Answer 2

Testing for a negative feature, hence provide a guarantee that certain construct will fail to compile is possible using c++20 requires expressions as follows:
Simple example
Below, I check if overloads to the function func exist in static assertions, when used with a test framework, the boolean should be used on one of the run time tests in order to not block the other tests from compiling:
#include <concepts>
/// Arbitrary restrictions in order to test:
/// if T != double -> zero args
template <typename T> void func(){};
/// if T == double -> arbitrary args.
template<std::same_as<double> ...T> void func(T... as){};
template <typename T, typename... a> constexpr bool applies_to_func = requires(a... as) {
func<T>(as...);
};
/// compiles:
static_assert(applies_to_func<int>);
static_assert(applies_to_func<double, double>);
static_assert(applies_to_func<double, double, double>);
/// function fails to compile:
static_assert(!applies_to_func<int, int>);
The code is available on Compiler Explorer: https://godbolt.org/z/direWo
C++17
I recently tried tried to do a similar thing for a project in which I can only use c++17. In my code I also check if the function's return type matches the caller's expectations. Aside from some limitations regarding the non-type template parameters, a similiar thing can be achieved as demonstrated below. In this case I could not enfroce double as input to the overload, due to the implicit conversion, applies_to_func(void, int, int) will evaluate to true in the code snipplet below.
#include <utility>
#include <string>
/// Create compile-time tests that allow checking a specific function's type
#define COMPILE_TIME_TEST(func) COMPILE_TIME_TEST_FUNCTION(func, func)
#define COMPILE_TIME_TEST_FUNCTION(name, func) \
namespace detail { \
template<typename R, auto... args> struct name ## FromArgs:std::false_type{}; \
template<auto... args> struct name ## FromArgs<decltype(func(args...)), args...> : std::true_type{}; \
template<typename R, typename... Args> struct name ## FromType:std::false_type{}; \
template<typename... Args> struct name ## FromType<decltype(func(std::declval<Args>()...)), Args...> : std::true_type{};\
} \
template<typename R, auto ...Args> \
static constexpr auto name ## _compiles = detail::name ## FromArgs<R, Args...>::value; \
template<typename ...Args> \
static constexpr auto name ## _compiles_from_type = detail::name ## FromType<Args...>::value;\
int func();
template <typename T> void func(T);
void func(double);
void func(double, double );
void func(double, double, double);
// create the structs from above macros for the function `func`
COMPILE_TIME_TEST(func);
static_assert(!func_compiles<void>);
static_assert(func_compiles<int>);
static_assert(func_compiles_from_type<void, double, double>);
static_assert(!func_compiles_from_type<void, double, double, double, double>);
static_assert(func_compiles<void, 1>);
static_assert(!func_compiles<void, 1, nullptr>);

Answer 3

You would have to rely on an external framework to run a set of compilation tests, e.g. makefiles, or hudson jobs and check for either compiler output or compiler artifacts. If the compilation is supposed to fail then there should not be an object file for the file under compilation. I am guessing you could write a plugin for hudson to do that or a simple batch script that runs a makefile that compiles all the testfiles that should fail or succeed and flag successes or failures accordingly.
In the simplest case you would just check for the existance of the '.o' file to see whether your test succeeded, in more complex cases you might want to look at the compiler output and verify that the error that is produce concurs with the error that you are expecting. That would depend on the compiler that you are using.
Going one level deeper would probably mean writing a compiler extension to do that (LLVM might be able to handle what you are asking for)

Answer 4

You might want to check out metatest - Unit testing framework for C++ template metaprograms (author's original post to the Boost mailing list). Get it here.
Publications related to the libraries here.

Answer 5

[This is a variant of #mutableVoid's answer above, but a little more explicit on two things that I had to think about a little. It concerns only concepts, which is what I wanted to get under test, so it only applies to C++20 onwards].
First, assume we have a concept that we wish to test. This can be simple and just defer to existing type traits:
template <class T>
concept integral = std::is_integral_v<T>;
Or it could be more complex, in this case, a concept that checks that type T has a size function that returns the type's size_type (and you can add more requirements to this "compound requirement", say, that it has a [] operator):
template<typename T>
concept sizeable = requires(T x) {
{x.size()} -> std::same_as<typename T::size_type>;
};
So, now we have some concepts. We want to test that these concepts work like we expect.
What we will need to do is to get a bool to test. Well, that's easy, because that's what concepts gives us naturally:
std::cout << std::string(typeid(integral<int>).name()) << std::endl;
std::cout << std::string(typeid(sizeable<int>).name()) << std::endl;
produces (with GCC):
b
b
So, we can check these, either with static_assert (i.e. mutableVoid's answer), which will fail your compilation if your concept isn't working:
static_assert(integral<int>);
static_assert(!integral<float>);
static_assert(sizeable<std::vector<int>>);
static_assert(!sizeable<int>);
You can prove to yourself that this works by removing a ! and observing that the compilation fails.
However, you may not want compilation to fail with a compiler error. If you'd rather feed this to your unit test framework, it doesn't have to be static_asserted:
void say(bool b, const std::string& s) {
std::cout << s << " = " << b << std::endl;
}
int main() {
say(integral<int>, "integral, int");
say(integral<float>, "integral, float");
say(sizeable<std::vector<int>>, "sizeable, vector of int");
say(sizeable<int>, "sizeable, int");
return 0;
}
This produces something like this:
integral, int = 1
integral, float = 0
sizeable, vector of int = 1
sizeable, int = 0
Now, you can plug this into whatever unit testing library you want, and you can check that your concepts aren't accidentally permitting types that you expect to fail.
However, do note there are some limitations:
You can't check that there are no unexpected type acceptances, because your tests would be infinite in size (the same as any negative test, really).
Although these tests will compile and run even if the concepts are broken, it's possible that other code will choke if the concepts are "allowing" types though that they shouldn't, and your overall build may well still fail. For example, if functions or classes are static_asserting their types for their own purposes.
Indeed, you may still want static_asserts in the main program compilation to prevent non-compliant code even being written in the first place: after all, compile-time correctness is a good thing. However, this gives you a chance to make sure that your concepts and static assertions are working together as expected. If the static_asserts are for some reason not compiled or are changed to be too permissive, you may not notice the concept is now defective.