I want to unit test a C++ function which throws aborting assertion error on invalid input.
The function is as follows:
uint64_t FooBar::ReadTimeStamp(std::string& name) {
auto iter = hash_table_.find(name);
assert(iter != hash_table_.end());
....
}
In unit test, I use CPPUNIT_ASSERT_ASSERTION_FAIL to assert on assertion failure:
void FooBarTest::
TestReadNonexistentTimestamp() {
CPPUNIT_ASSERT_ASSERTION_FAIL(ReadTimestamp("NON_EXISTENT"));
}
But I got abort message and unit test failed.
I read this page. It's not clear to me if I need to throw exception here and what the correct way to unit test this scenario would be. Thanks!
Firstly, your misunderstanding is caused by the different ways to use the term "assertion". The test framework itself talks about assertions, but it does not mean the assert() macro provided by the standardlibrary. Since the standard assertion failure causes program termination, you get those results.
Now, how to fix that:
Don't use assert(). Instead, you could throw an exception.
Don't test this code path. Since this is a programming error that's not recoverable anyway, it can only be caused by misuse of your code (i.e. violating preconditions). Since it's not your code that's at fault, not testing it doesn't have any negative impact on its quality.
Hijack assert() to throw a failure that CppUnit understands. This could be tricky because for one, assert() is part of C++ and shouldn't be redefined carelessly (perhaps substituting it with a different macro would be better). Further, you now have three different behaviours: Throw in tests, abort() in regular use, UB with NDEBUG defined.
Which of these works best is up to you to decide, based on your use case and, admittedly, personal preference.
Related
I'm learning to write unit tests, and have started with an easy "Calculator"-class that I wanted to test.
I figured out how to use the EXPECT/ASSERT functions, and what test cases etc. are, but I got a problem when I wanted to test the division by zero. Is there any possibility to test it? I mean, what should I write as test result? Is there anything like "ERROR"? Or do I have to use exceptions?
These are my tests so far:
TEST(TestCalc, TestPos)
{
Calc calculate;
EXPECT_EQ(10.0, calculate.add(5.0, 5.0));
EXPECT_EQ(9, calculate.mul(3, 3));
EXPECT_EQ(9, calculate.div(27, 3));
EXPECT_EQ(9, calculate.sub(12, 3));
}
TEST(TestCalc, TestNeg)
{
Calc calculate;
EXPECT_EQ(-1.0, calculate.add(5.0, -6.0));
EXPECT_EQ(-9, calculate.mul(3, -3));
EXPECT_EQ(-9, calculate.div(27, -3));
EXPECT_EQ(15, calculate.sub(12, -3));
}
TEST(TestCalc, TestZero)
{
Calc calculate;
EXPECT_EQ(10.0, calculate.add(5.0, 0));
EXPECT_EQ(9, calculate.mul(3, 0));
EXPECT_EQ(, calculate.div(27,0));
EXPECT_EQ(12, calculate.sub(12,0));
}
I don't agree with #Ketzu. You have an expectation how calculator should behave, when dividing by zero.
EXPECT_EQ(, calculate.div(27,0));
This expectation is perhaps not well formulated in this test.
If calculate.div(27,0) throws an exception, than you can catch this exception and your test fails, if it is not thrown. You can write something like this
TEST(ExceptionTest, ExpectThrowsSpecificException) {
try {
calculate.div(27,0);
FAIL() << "calculate.div(27,0) should throw an error, since a division by zero is not valid\n";
} catch (TestException& exception) {
EXPECT_THAT(std::string(exception.what()), Eq("VALID_SETTING"));
EXPECT_THAT(exception.errorCode, Eq(20));
}
}
See here for a detailed discussion.
If no exception is thrown, how do you detect the abnormal usage of calculate.div?
Is there any possibility to test it? I mean, what should I write as test result? Is there anything like "ERROR"? Or do I have to use exceptions?
This is a core question you need to ask (and answer) yourself - how should your calculator (in this case the function Calc::div()) behave if an invalid input is given. There are several ways it can behave:
Crash. This (usually) is the behavior you get if you divide by zero in C++ (technically the behaviour is undefined, so the compiler may do ANYTHING. Fortunately (for us) most compilers agree that terminating the entire process is the "correct" ANYTHING to do here)
Return some value. You could return (in the case of division by zero) infinity. Or possibly NaN. Or any other value that makes sense in your context. This approach however mixes the result with error handling, which is discouraged nowadays (as it would force you to check each and every invokation of the function for "error results" - and if you forget a single check you get nasty bugs as you continue operations with invalid/bogus values.)
Throw an exception. You can throw an exception, signalling that something went wrong. This is the usually used method nowadays (except for extremly performance sensitve stuff, where every microsecond counts), as it neatly separates the normal path (return result value) from the error path (exception) (and if you forget to handle the exception you will notice at once instead of using bad value like in option 2)
Once you have decided on what your behaviour should be you can test it.
For Option 1 gtest provides death tests.
For Option 2 you can simply validate that you get your expected result.
For Option 3 you can catch and evalute the exception, either via exception assertions or a homemade try { } catch { } with a FAIL() at the end of the try block (so you notice if the function fails to throw an exception when you expect it to)
Your comment
I thought I could test it
suggest a misunderstanding about tests.
Your tests only test if the behaviour of your classes is the way you want it to be. Tests are in no way error handling mechanisms!
If one of your functions is creating an application crash with certain parameters, that is not something your tests should mark as correct.
For how to solve it (exception and suitable macro) see john's comment.
Here it's discussed how to catch failing assert, e.g. you setup your fixture so that assert() fails and you see nice output. But what I need is the opposite. I want to test that assert() succeeds. But in case it fails I want to have nice output. At that point it just terminates when it snags on assert().
#define LIMIT 5
struct Obj {
int getIndex(int index) {
assert(index < LIMIT);
// do stuff;
}
}
Obj obj;
TEST(Fails_whenOutOfRange) {
ASSERT_DEATH(obj->getIndex(6), "");
}
TEST(Succeeds_whenInRange) {
obj->getIndex(4);
}
Above is contrived example. I want second test not to terminate in case it fails, for example if I set LIMIT to 3. After all, ASSERT_DEATH suppresses somehow termination when assert() fails.
You should try using the command line option --gtest_break_on_failure
It is meant to run tests within a debugger, so you get a breakpoint upon test failure. If you don't use a debugger you'll just get a SEGFAULT and execution will stop.
The following is just my opinion, but it seems for me that you are either testing a wrong thing, or using a wrong tool.
Assert (C assert()) is not for verifying input, it is for catching impossible situations. It will disappear from release code, for example, so you can't rely on it.
What you should test is your function specification rather than implementation. And you should decide, what is your specification for invalid input values:
Undefined behavior, so assert is fine, but you can't test it with unit-test, because undefined behavior is, well, undefined.
Defined behavior. Then you should be consistent regardless of NDEBUG presence. And throwing exception, in my opinion, is the right thing to do here, instead of calling std::abort, which is almost useless for user (can't be intercepted and processed properly).
If assert triggers (fails) you get "nice output" (or a crash or whatever assert does in your environment). If assert does not trigger then nothing happens and execution continues.
What more do you need to know?
This (hack) adds a EXPECT_NODEATH macro to Google Test. It is the "opposite" of EXPECT_DEATH in that it will pass if the statement does not assert, abort, or otherwise fail.
The general idea was simple, but I did not take the time to make the error messages any nicer. I tried to leave Google Test as untouched as possible and just piggy-back on what is already there. You should be able to include this without any side effects to the rest of Google Test
For your case:
TEST(Succeeds_whenInRange) {
EXPECT_NODEATH(obj->getIndex(4), "");
}
GTestNoDeath.h
CPPUNIT_ASSERT_THROW(Expression, ExceptionType) does not seem to allow checking for exceptions of multiple types i.e. for a statement that can throw more than one kind of exceptions.
For e.x. an expression may throw Excp1 on one platform, or Excp2 on another platform. Is there a workaround to test such statements using CPPUNIT_ASSERT_THROW?
First test, you make your test conditions such that it throws exeption 1.
If it fails to throw, that is a test failure.
if it does throw, you catch it as an exception 1, and accept it as passing.
If it throws something else, the framework catches it.
Second test, you make using conditional compilation to enable code for platform 2 only. You make your test conditions such that it throws exception 2.
IF it fails to throw, that is a test failure.
If it does throw, you catch it as exception 2, and accept it as passing.
If it throws something else, the framework catches it.
On the first platform the test simply passes, as there is nothing for it to do.
On the second platform you catch exception 2 as expected.
There is no direct support for this feature in cppunit but you have basically two solutions how you can implement it easily in your code.
So the basic idea behind this assert is the following code:
bool expected_exception_thrown = false;
try
{
yourExpression();
}
catch(const ExpectedException&)
{
expected_exception_thrown = true;
}
catch(...)
{
}
if (!expected_exception_thrown)
CPPUNIT_FAIL();
Of course the actual implementation is a bit fancier and involves some additional features (like better messages for unexpected std::exception and the missing support for an error message) but the general idea is the same.
So now you can easily extend that pattern to support as many exceptions as you need. You can have a look at the existing implementation in include/cppunit/TestAssert.h and either use that implementation and extend it or use the simplified one that I posted above.
example of using assertions during debugging:
char* append(char* pStr, const char* pAddStr)
{
// Verify non-null pointers
assert(pStr != nullptr);
assert(pAddStr != nullptr);
// Code to append pAddStr to pStr...
}
Calling the append() function with a null pointer argument in a simple program produced the following diagnostic message on my machine:
Assertion failed: pStr != nullptr, file c:\beginning visual c++ 2010\examples visual studio project files\tryassertion\tryassertion\tryassertion.cpp, line 10
I would like to know if assertions are neccesary. What is the point of using them if i can use if-else expressions to output my own error messages?
An assertion is a conditional check; it's just a macro that's something like this (simplified):
#define assert(b) if (!(b)) { std::cerr << ...; exit(1); }
I can think of two advantages:
When you compile in release mode (i.e. not in debug mode), then all the asserts will compile away to nothing, so you will incur no runtime overhead.
assert is an idiom; other programmers know to look for them, and they're also distinct from "normal" control flow.
assertions are used to ensure that certain basic assumptions are met. Basically you put in an assertation in every case which "couldnt possibly happen", most commonly assertions how the API has to be used e.g. preconditions and postconditions (like your example, which contains asserts which check the correct usage of your append function). For other errors, which are known to occur during runtime and which cannot be prevented beforehand (e.g. file-not-found or insufficient-permissions kind of errors), you will have to write error handling code.
Assertions are not included in release compilations, so they can only be used to catch critical programming errors, which will already occur in debug-mode test runs.
You should use assertions when the only case in which they are violated is an error in the program logic. You use normal if-then-else conditionals for things that may indeed happen because of input or external possible conditions (i.e. a file is missing).
When you know for sure that the program logic is faulty there is not really much sense in trying to continue execution... your program is working differently from what you think (because otherwise the assertion wouldn't have been triggered) and therefore the best option is just to yell what the problem is and die immediately.
Often assertion are removed when the code is compiled in "release" mode even if however it may make sense to keep them in place if the program logic is very complex and if continuing execution generating bad output is going to create bigger problems than stopping execution.
Note that a "trap" that sometimes novice programmers fall into with assertion is that when the assertion code is removed for release mode the expression inside the assert is not evaluated any more, and therefore if you program depends on side effects of that expression then you're going to be in trouble... for example:
...
assert(insert_record(db, data) == DB_OK); // <== this is bad
...
when assertion are defined away the insertion will not happen at all, leaving you with a program that doesn't work in release mode and that works instead when you try to debug the problem.
I use the Boost Test framework to unit test my C++ code and wondered if it is possible to test if a function will assert? Yes, sounds a bit strange but bear with me! Many of my functions check the input parameters upon entry, asserting if they are invalid, and it would be useful to test for this. For example:
void MyFunction(int param)
{
assert(param > 0); // param cannot be less than 1
...
}
I would like to be able to do something like this:
BOOST_CHECK_ASSERT(MyFunction(0), true);
BOOST_CHECK_ASSERT(MyFunction(-1), true);
BOOST_CHECK_ASSERT(MyFunction(1), false);
...
You can check for exceptions being thrown using Boost Test so I wondered if there was some assert magic too...
Having the same problem, I digged through the documentation (and code) and
found a "solution".
The Boost UTF uses boost::execution_monitor (in
<boost/test/execution_monitor.hpp>). This is designed with the aim to catch
everything that could happen during test execution. When an assert is found
execution_monitor intercepts it and throws boost::execution_exception. Thus,
by using BOOST_REQUIRE_THROW you may assert the failure of an assert.
so:
#include <boost/test/unit_test.hpp>
#include <boost/test/execution_monitor.hpp> // for execution_exception
BOOST_AUTO_TEST_CASE(case_1)
{
BOOST_REQUIRE_THROW(function_w_failing_assert(),
boost::execution_exception);
}
Should do the trick. (It works for me.)
However (or disclaimers):
It works for me. That is, on Windows XP, MSVC 7.1, boost 1.41.0. It might
be unsuitable or broken on your setup.
It might not be the intention of the author of Boost Test.
(although it seem to be the purpose of execution_monitor).
It will treat every form of fatal error the same way. I e it could be
that something other than your assert is failing. In this case you
could miss e g a memory corruption bug, and/or miss a failed failed assert.
It might break on future boost versions.
I expect it would fail if run in Release config, since the assert will be
disabled and the code that the assert was set to prevent will
run. Resulting in very undefined behavior.
If, in Release config for msvc, some assert-like or other fatal error
would occur anyway it would not be caught. (see execution_monitor docs).
If you use assert or not is up to you. I like them.
See:
http://www.boost.org/doc/libs/1_41_0/libs/test/doc/html/execution-monitor/reference.html#boost.execution_exception
the execution-monitor user-guide.
Also, thanks to Gennadiy Rozental (Author of Boost Test), if you happen to
read this, Great Work!!
There are two kinds of errors I like to check for: invariants and run-time errors.
Invariants are things that should always be true, no matter what. For those, I use asserts. Things like you shouldn't be passing me a zero pointer for the output buffer you're giving me. That's a bug in the code, plain and simple. In a debug build, it will assert and give me a chance to correct it. In a retail build, it will cause an access violation and generate a minidump (Windows, at least in my code) or a coredump (Mac/unix). There's no catch that I can do that makes sense to deal with dereferencing a zero pointer. On Windows catch (...) can suppress access violations and give the user a false sense of confidence that things are OK when they've already gone horribly, horribly wrong.
This is one reason why I've come to believe that catch (...) is generally a code smell in C++ and the only reasonable place where I can think of that being present is in main (or WinMain) right before you generate a core dump and politely exit the app.
Run-time errors are things like "I can't write this file because of permissions" or "I can't write this file because the disk is full". For these sorts of errors throwing an exception makes sense because the user can do something about it like change the permission on a directory, delete some files or choose an alternate location to save the file. These run-time errors are correctable by the user. A violation of an invariant can't be corrected by the user, only by a programmer. (Sometimes the two are the same, but typically they aren't.)
Your unit tests should force code to throw the run-time error exceptions that your code could generate. You might also want to force exceptions from your collaborators to ensure that your system under test is exception safe.
However, I don't believe there is value in trying to force your code to assert against invariants with unit tests.
I don't think so. You could always write your own assert which throws an exception and then use BOOST_CHECK_NOTHROW() for that exception.
I think this question, and some of replies, confuse run-time errors detection with bug detection. They also confuse intent and mechanism.
Run-time error is something that can happen in a 100% correct program. It need detection, and it needs proper reporting and handling, and it should be tested. Bugs also happen, and for programmer's convenience it's better to catch them early using precondition checks or invariant checks or random assert. But this is programmer's tool. The error message will make no sense for ordinary user, and it does not seem reasonable to test function behaviour on the data that properly written program will never pass to it.
As for intent and mechanism, it should be noted that exception is nothing magic. Some time ago, Peter Dimov said on Boost mailing list (approximately) that "exceptions are just non-local jump mechanism". And this is very true. If you have application where it's possible to continue after some internal error, without the risk that something will be corrupted before repair, you can implement custom assert that throws C++ exception. But it would not change the intent, and won't make testing for asserts much more reasonable.
At work I ran into the same problem. My solution is to use a compile flag. When my flag GROKUS_TESTABLE is on my GROKUS_ASSERT is turned into an exception and with Boost you can test code paths that throw exceptions. When GROKUS_TESTABLE is off, GROKUS_ASSERT is translated to c++ assert().
#if GROKUS_TESTABLE
#define GROKUS_ASSERT ... // exception
#define GROKUS_CHECK_THROW BOOST_CHECK_THROW
#else
#define GROKUS_ASSERT ... // assert
#define GROKUS_CHECK_THROW(statement, exception) {} // no-op
#endif
My original motivation was to aid debugging, i.e. assert() can be debugged quickly and exceptions often are harder to debug in gdb. My compile flag seems to balance debuggability and testability pretty well.
Hope this helps
Sorry, but you're attacking your problem the wrong way.
"assert" is the spawn of the devil (a.k.a. "C") and is useless with any language that has proper exceptions. It's waaaaaay better to reimplement an assert-like functionality with exceptions. This way you actually get a chance of handling errors the right way (incl proper cleanup procedures) or triggering them at will (for unit testing).
Besides, if your code ever runs in Windows, when you fail an assertion you get a useless popup offering you to debug/abort/retry. Nice for automated unit tests.
So do yourself a favor and re-code an assert function that throws exceptions. There's one here:
How can I assert() without using abort()?
Wrap it in a macro so you get _ _FILE _ _ and _ _ LINE _ _ (useful for debug) and you're done.