On Windows, my assert macro essentially looks like this:
#define MYASSERT(condition) (if (!(condition)) { ReportFailture( #condition, __FILE__, __LINE__, __FUNCTION__); __debugbreak(); }
And in Google Test I am trying to check the output of a bad condition to test out of bound assertions, etc:
ASSERT_DEATH( { MYASSERT(false); }, "");
However all this does it report the following message:
Running main() from gtest_main.cc
..\Test\FormatUnitTest\Test_Format.cpp(59): error: Death test: { if (!(false)) { ReportFailture( "false", ..\\Test\\UnitTest\\Test.cpp", 59, __FSTREXP __FUNCTION_
_ ); __debugbreak(); }; }
Result: illegal return in test statement.
Error msg:
[ DEATH ]
It seems that GoogleTest is handling the Debug Exception in the structured exception handler (SEH) as a special case. However, I want to catch the assert and validate it's contents.
What's the right move here? Do I need to define a special assert macro for google test? If so what should it do?
I note that replacing my assert with assert(false) (included via assert.h) doesn't call this problem - what is it doing differently?
The problem is that you are calling __debugbreak(), which causes a breakpoint exception to be thrown by your code. As you can see in documentation for death tests (https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests), if code throws an exception it is not considered "death" by death tests in googletest.
As far as you other question goes, assert from assert.h calles std::abort (which causes program to terminate). This is "death" by definition of a death test.
Related
I am using Google Test on code I expect to fail. As part of this failure the code calls a custom assert macro, which contains std::abort().
Unfortunately Google Test's EXPECT_EXIT() is not "catching" the std::abort().
This is a self-contained example to emulate what I'm trying to achieve:
// A placeholder for my assert macro
void MyFunction()
{
std::abort();
}
TEST(TestGroup1, TestName)
{
EXPECT_EXIT(MyFunction(), ::testing::ExitedWithCode(SIGABRT), ".*");
}
I get this failure output:
Death test: MyFunction()
Result: died but not with expected exit code:
Terminated by signal 6 (core dumped)
Actual msg:
[ DEATH ]
Is this possible to achieve?
I needed to use ::testing::KilledBySignal(SIGABRT)
Our project uses Boost.Test for our units tests. We would like to get minidumps when unexpected exceptions occur during our test cases as well, so we've started integrating Google Breakpad to write the minidumps.
It looks like Boost.Test is intercepting all thrown exceptions from user tests - I'm assuming because Boost test cases wrap each function with a try / catch and the unit test simply fails if an unexpected exception is thrown. This prevents the Breakpad exception handler from firing and writing minidumps.
Is it possible to have Boost.Test not just catch and fail on unexpected exceptions in unit tests? And to instead let the exceptions go unhandled (or rethrow) so Breakpad or another exception handler can be triggered to write a minidump?
Boost.Test is designed to catch all exceptions from a test case in order to continue executing other test cases. I don't think you can turn this behavior off, but you can always consult the source code.
To achieve what you're after, I'd wrap the body of the test cases myself and catch exceptions and have breakpad minidumps written out when unexpected exceptions are thrown. You can genericize this by writing the catch/dump handler as a function that accepts a functor that is your test case body and invoking your handler with the test case body in a lambda:
void handler(std::function<void()> test_case)
{
try {
test_case();
} catch (...) {
write_minidump();
}
}
BOOST_AUTO_TEST_CASE(doit)
{
handler([] {
// do testing here
});
}
It will be more involved if you're using fixtures, but the same idea would apply.
I tried a few different approaches but the following solution provides the best result. Defining a macro to wrap the BOOST_AUTO_TEST_CASE macro, and surrounding the calling code with SEH __try/__except and piping the exception data into Breakpad.
#define CUSTOM_AUTO_TEST_CASE( test_name ) \
void test_name##_custom_wrapper(); \
\
BOOST_AUTO_TEST_CASE( test_name ) \
{ \
__try \
{ \
test_name##_custom_wrapper(); \
} \
__except(pHandler->WriteMinidumpForException(GetExceptionInformation()), EXCEPTION_EXECUTE_HANDLER) {} \
} \
\
void test_name##_custom_wrapper() \
Where pHandler is a Breakpad ExceptionHandler pointer.
The downside is you have to replace every occurrence of BOOST_AUTO_TEST_CAST with the wrapper macro. But it does the trick.
The following assertion will produce a . or an F when PHPUnit is run:
$this->assertEquals('foo', $bar);
But what if I wanted to know the result of the assertion within the test itself? Something similar to:
if ($this->assertEquals('foo', $bar)) {
// log or do something
} else {
// do something else
}
There's probably no good reason to conditionally run assertions (or maybe there is), but this is a question purely from the debugging point of view.
PHPUnit uses exceptions to signify that a test has failed.
If we follow the assertEquals() code through, we can see that on line 137 of isEqual.php, a new PHPUnit_Framework_ExpectationFailedException is thrown if the operands are not equal. The test runner then converts this to a "F" for the console output and also uses if for reports etc.
You can catch the Exception and do what you like with it in your testcase. You may want to rethrow it if you still want the test to fail:
try {
$this->assertEquals('foo', $bar);
// do something else
} catch (PHPUnit_Framework_ExpectationFailedException $e) {
// log or do something
throw $e; // rethrow to make sure that the test still fails
}
In a test case I would like to test a function which in debug mode generates an assertion for invalid input. This unfortunately stops Catch test runner. Is there any way to bypass this assertion so that the test runner keeps going ?
Here is my test case:
SCENARIO("Simple test case", "[tag]") {
GIVEN("some object") {
MyObject myobject;
WHEN("object is initialized with invalid data") {
// method init generates an assertion when parameters are invalid
bool result = myObject.init(nullptr, nullptr, nullptr, nullptr);
REQUIRE(false == result);
THEN("data processing can't be started") {
}
}
}
}
Usually assert is a macro doing something like
#define assert(e) ((void) ((e) \
? 0
: (void)printf ("%s:%u: failed assertion `%s'\n", __FILE__, __LINE__, #e),
abort(), // <-- this aborts you program after printf's above
0
)
and this macro is enabled in debug builds. For more specifics, look into your standard assert.h
So, if you have a binary library against which you link your test cases, you will need to tell the dev team that, unless they gave you a release build with no asserts enabled, you won't be able to unit-test their API for negative test cases.
If you need to test a header-only library or you compile against the source code to be tested, you'll need to
compile your test cases/suite with -DNDEBUG; and/or
define your own assert macro (e,g. to throw an error which you'll catch) and hope that your specific assert.h will test if already defined and won't try to define it again (again, look for specifics in your compiler/std libs assert.h header)
I'm trying to test a c++ code on an Nunit framework but I keep getting the following Exception
System.Runtime.InteropServices.SEHException : External Component has thrown an exception.
which is supposedly perfectly normal (I assume) anyway I wanna ignore it. (i.e. Use ExpectedException) This is my .h file
[Test, Description("Tests if an Entity has been successfully Locked")]
void test_LockOperation();
and the .cpp file
void TestDmObstacles::test_LockOperation()
{
lockVal = DbtoDmObstaclesAdapter::lock( CmnGuid::parseString( L"3B6DB8F8-4BA7-DD11-B6A7-001E8CDE165C" ) );
//When lock is successful the lockVal is 0
Assert::AreEqual(0, lockVal);
}
I wanna use ExpectedException but I don't know how to do it in c++. I tried the try/catch method as well but it didn't work (I just put the Assertion in the catch block)
PS: I can't use another framework it has to be Nunit
EDIT
Here is the try/catch approach I used
void TestDmObstacles::test_LockOperation()
{
try
{
lockVal = DbtoDmObstaclesAdapter::lock( CmnGuid::parseString( L"3B6DB8F8-4BA7-DD11-B6A7-001E8CDE165C" ) );
}
catch (...)
{
//Assert::Fail();
Assert::AreEqual(0, lockVal);
}
}
Is the exception expected, or is the exception acceptable?
If it is expected, then your unit test framework should have some kind of API that allows you to state the expected exception, and to fail the test if it does not occur. A quick trawl through the documentation yields the incantation:
[ExpectedException( "System.ArgumentException" )]
(replace System.ArgumentException with the exception you're expecting.)
If the exception is merely acceptable, then I would say that either your code or your test is broken. A unit test is to test that expected things happen. If there is a result in your test that only may yield a particular result, then you are not testing a consistent view of the unit from test to test. Hence, you're not really testing it.
It might indicate, for example, that your code is leaking an unexpected exception that it should be handling instead.
Your code sample doesn't match what you are trying to achieve : if the exception is expected, than catching it is not supposed to fail the test.
Note that I wouldn't recommend (at all) for the test to catch (...) : any thrown exception will induce the same test result, which I doubt is what you want.