In a test case I would like to test a function which in debug mode generates an assertion for invalid input. This unfortunately stops Catch test runner. Is there any way to bypass this assertion so that the test runner keeps going ?
Here is my test case:
SCENARIO("Simple test case", "[tag]") {
GIVEN("some object") {
MyObject myobject;
WHEN("object is initialized with invalid data") {
// method init generates an assertion when parameters are invalid
bool result = myObject.init(nullptr, nullptr, nullptr, nullptr);
REQUIRE(false == result);
THEN("data processing can't be started") {
}
}
}
}
Usually assert is a macro doing something like
#define assert(e) ((void) ((e) \
? 0
: (void)printf ("%s:%u: failed assertion `%s'\n", __FILE__, __LINE__, #e),
abort(), // <-- this aborts you program after printf's above
0
)
and this macro is enabled in debug builds. For more specifics, look into your standard assert.h
So, if you have a binary library against which you link your test cases, you will need to tell the dev team that, unless they gave you a release build with no asserts enabled, you won't be able to unit-test their API for negative test cases.
If you need to test a header-only library or you compile against the source code to be tested, you'll need to
compile your test cases/suite with -DNDEBUG; and/or
define your own assert macro (e,g. to throw an error which you'll catch) and hope that your specific assert.h will test if already defined and won't try to define it again (again, look for specifics in your compiler/std libs assert.h header)
Related
The following assertion will produce a . or an F when PHPUnit is run:
$this->assertEquals('foo', $bar);
But what if I wanted to know the result of the assertion within the test itself? Something similar to:
if ($this->assertEquals('foo', $bar)) {
// log or do something
} else {
// do something else
}
There's probably no good reason to conditionally run assertions (or maybe there is), but this is a question purely from the debugging point of view.
PHPUnit uses exceptions to signify that a test has failed.
If we follow the assertEquals() code through, we can see that on line 137 of isEqual.php, a new PHPUnit_Framework_ExpectationFailedException is thrown if the operands are not equal. The test runner then converts this to a "F" for the console output and also uses if for reports etc.
You can catch the Exception and do what you like with it in your testcase. You may want to rethrow it if you still want the test to fail:
try {
$this->assertEquals('foo', $bar);
// do something else
} catch (PHPUnit_Framework_ExpectationFailedException $e) {
// log or do something
throw $e; // rethrow to make sure that the test still fails
}
On Windows, my assert macro essentially looks like this:
#define MYASSERT(condition) (if (!(condition)) { ReportFailture( #condition, __FILE__, __LINE__, __FUNCTION__); __debugbreak(); }
And in Google Test I am trying to check the output of a bad condition to test out of bound assertions, etc:
ASSERT_DEATH( { MYASSERT(false); }, "");
However all this does it report the following message:
Running main() from gtest_main.cc
..\Test\FormatUnitTest\Test_Format.cpp(59): error: Death test: { if (!(false)) { ReportFailture( "false", ..\\Test\\UnitTest\\Test.cpp", 59, __FSTREXP __FUNCTION_
_ ); __debugbreak(); }; }
Result: illegal return in test statement.
Error msg:
[ DEATH ]
It seems that GoogleTest is handling the Debug Exception in the structured exception handler (SEH) as a special case. However, I want to catch the assert and validate it's contents.
What's the right move here? Do I need to define a special assert macro for google test? If so what should it do?
I note that replacing my assert with assert(false) (included via assert.h) doesn't call this problem - what is it doing differently?
The problem is that you are calling __debugbreak(), which causes a breakpoint exception to be thrown by your code. As you can see in documentation for death tests (https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests), if code throws an exception it is not considered "death" by death tests in googletest.
As far as you other question goes, assert from assert.h calles std::abort (which causes program to terminate). This is "death" by definition of a death test.
I'm testing some code that uses CHECK from glog and I'd like to test that this check fails in certain scenarios. My code looks like:
void MyClass::foo() {
// stuff...
// It's actually important that the binary gets aborted if this flag is false
CHECK(some_flag) << "flag must be true";
// more stuff...
}
I've done some research into gtest and how I might be able to test for this. I found EXPECT_FATAL_FALIURE, EXPECT_NONFATAL_FAILURE, and HAS_FATAL_FAILURE but I haven't managed to figure out how to use them. I'm fairly confident that if I change CHECK(some_flag) to EXPECT_TRUE(some_flag) then EXPECT_FATAL_FAILURE will work correctly but then I'm introducing test dependencies in non-test files which is...icky.
Is there a way for gtest to catch the abort signal (or whatever CHECK raises) and expect it?
aaaand I found an answer 5 minutes after posting this question. Typical.
This can be done using Death tests from gtest. Here's how my test looks:
TEST(MyClassTest, foo_death_test) {
MyClass clazz(false); // make some_flag false so the CHECK fails
ASSERT_DEATH( { clazz.foo(); }, "must be true");
}
This passes. Woohoo!
I'm using Google Test and Google Mock frameworks for a project's unit tests. I have various unit tests projects and want to automate my build so to run all of them.
I was expecting the unit tests executable to return 0 on success and 1 (or any other value) on any test failure, but I'm getting 1 when all tests passed. I'm getting some GMOCK warnings but couldn't find any documentation about warnings affecting return value.
I tried running the tests filtering to run just one test case where no GMOCK warnings are triggered and still get 1 as return value.
I had a couple of DISABLED test cases, so I commented them out. Still getting 1 as return value.
According to documentation and code comments for RUN_ALL_TESTS macro, the return value should be 0.
I can't think of anything else causing return value 1. Am I missing anything?
If you look at the definition of the RUN_ALL_TESTS() macro from gtest.h there's clearly stated that 0 is returned on no failures:
// Use this macro in main() to run all tests. It returns 0 if all
// tests are successful, or 1 otherwise.
//
// RUN_ALL_TESTS() should be invoked after the command line has been
// parsed by InitGoogleTest().
#define RUN_ALL_TESTS()\
(::testing::UnitTest::GetInstance()->Run())
Appearantly even warnings (from gmock) may result in a return value of 1. Try what happens if you get rid of the gmock warnings (e.g. using s.th. like NiceMock<> to wrap your mock class instance).
I'm trying to test a c++ code on an Nunit framework but I keep getting the following Exception
System.Runtime.InteropServices.SEHException : External Component has thrown an exception.
which is supposedly perfectly normal (I assume) anyway I wanna ignore it. (i.e. Use ExpectedException) This is my .h file
[Test, Description("Tests if an Entity has been successfully Locked")]
void test_LockOperation();
and the .cpp file
void TestDmObstacles::test_LockOperation()
{
lockVal = DbtoDmObstaclesAdapter::lock( CmnGuid::parseString( L"3B6DB8F8-4BA7-DD11-B6A7-001E8CDE165C" ) );
//When lock is successful the lockVal is 0
Assert::AreEqual(0, lockVal);
}
I wanna use ExpectedException but I don't know how to do it in c++. I tried the try/catch method as well but it didn't work (I just put the Assertion in the catch block)
PS: I can't use another framework it has to be Nunit
EDIT
Here is the try/catch approach I used
void TestDmObstacles::test_LockOperation()
{
try
{
lockVal = DbtoDmObstaclesAdapter::lock( CmnGuid::parseString( L"3B6DB8F8-4BA7-DD11-B6A7-001E8CDE165C" ) );
}
catch (...)
{
//Assert::Fail();
Assert::AreEqual(0, lockVal);
}
}
Is the exception expected, or is the exception acceptable?
If it is expected, then your unit test framework should have some kind of API that allows you to state the expected exception, and to fail the test if it does not occur. A quick trawl through the documentation yields the incantation:
[ExpectedException( "System.ArgumentException" )]
(replace System.ArgumentException with the exception you're expecting.)
If the exception is merely acceptable, then I would say that either your code or your test is broken. A unit test is to test that expected things happen. If there is a result in your test that only may yield a particular result, then you are not testing a consistent view of the unit from test to test. Hence, you're not really testing it.
It might indicate, for example, that your code is leaking an unexpected exception that it should be handling instead.
Your code sample doesn't match what you are trying to achieve : if the exception is expected, than catching it is not supposed to fail the test.
Note that I wouldn't recommend (at all) for the test to catch (...) : any thrown exception will induce the same test result, which I doubt is what you want.