How to run Boost Test and produce Minidumps? - c++

Our project uses Boost.Test for our units tests. We would like to get minidumps when unexpected exceptions occur during our test cases as well, so we've started integrating Google Breakpad to write the minidumps.
It looks like Boost.Test is intercepting all thrown exceptions from user tests - I'm assuming because Boost test cases wrap each function with a try / catch and the unit test simply fails if an unexpected exception is thrown. This prevents the Breakpad exception handler from firing and writing minidumps.
Is it possible to have Boost.Test not just catch and fail on unexpected exceptions in unit tests? And to instead let the exceptions go unhandled (or rethrow) so Breakpad or another exception handler can be triggered to write a minidump?

Boost.Test is designed to catch all exceptions from a test case in order to continue executing other test cases. I don't think you can turn this behavior off, but you can always consult the source code.
To achieve what you're after, I'd wrap the body of the test cases myself and catch exceptions and have breakpad minidumps written out when unexpected exceptions are thrown. You can genericize this by writing the catch/dump handler as a function that accepts a functor that is your test case body and invoking your handler with the test case body in a lambda:
void handler(std::function<void()> test_case)
{
try {
test_case();
} catch (...) {
write_minidump();
}
}
BOOST_AUTO_TEST_CASE(doit)
{
handler([] {
// do testing here
});
}
It will be more involved if you're using fixtures, but the same idea would apply.

I tried a few different approaches but the following solution provides the best result. Defining a macro to wrap the BOOST_AUTO_TEST_CASE macro, and surrounding the calling code with SEH __try/__except and piping the exception data into Breakpad.
#define CUSTOM_AUTO_TEST_CASE( test_name ) \
void test_name##_custom_wrapper(); \
\
BOOST_AUTO_TEST_CASE( test_name ) \
{ \
__try \
{ \
test_name##_custom_wrapper(); \
} \
__except(pHandler->WriteMinidumpForException(GetExceptionInformation()), EXCEPTION_EXECUTE_HANDLER) {} \
} \
\
void test_name##_custom_wrapper() \
Where pHandler is a Breakpad ExceptionHandler pointer.
The downside is you have to replace every occurrence of BOOST_AUTO_TEST_CAST with the wrapper macro. But it does the trick.

Related

QT test fail on exception and continue next test rather than abort

I've had a look at some of the qtest source code and documentation and can't figure it out.
Would like to run tests and have test cases fail if an exception is thrown, and then continue on to the next test.
Is there a way short of recompiling qt with various macros defined/undefined? or adding try/catch blocks to every test?
or some way to get something like this to work?
QT_BEGIN_NAMESPACE
#define QCOMPARE_NOEXCEP( actual, expected )\
try{\
QCOMPARE( actual, expected );\
} catch( ... ) {\
QFAIL( "Exception thrown" );\
return;\
}\
QT_END_NAMESPACE

How to bypass assert in unit test in Debug mode?

In a test case I would like to test a function which in debug mode generates an assertion for invalid input. This unfortunately stops Catch test runner. Is there any way to bypass this assertion so that the test runner keeps going ?
Here is my test case:
SCENARIO("Simple test case", "[tag]") {
GIVEN("some object") {
MyObject myobject;
WHEN("object is initialized with invalid data") {
// method init generates an assertion when parameters are invalid
bool result = myObject.init(nullptr, nullptr, nullptr, nullptr);
REQUIRE(false == result);
THEN("data processing can't be started") {
}
}
}
}
Usually assert is a macro doing something like
#define assert(e) ((void) ((e) \
? 0
: (void)printf ("%s:%u: failed assertion `%s'\n", __FILE__, __LINE__, #e),
abort(), // <-- this aborts you program after printf's above
0
)
and this macro is enabled in debug builds. For more specifics, look into your standard assert.h
So, if you have a binary library against which you link your test cases, you will need to tell the dev team that, unless they gave you a release build with no asserts enabled, you won't be able to unit-test their API for negative test cases.
If you need to test a header-only library or you compile against the source code to be tested, you'll need to
compile your test cases/suite with -DNDEBUG; and/or
define your own assert macro (e,g. to throw an error which you'll catch) and hope that your specific assert.h will test if already defined and won't try to define it again (again, look for specifics in your compiler/std libs assert.h header)

Catching custom assert in GoogleTest using __debugbreak

On Windows, my assert macro essentially looks like this:
#define MYASSERT(condition) (if (!(condition)) { ReportFailture( #condition, __FILE__, __LINE__, __FUNCTION__); __debugbreak(); }
And in Google Test I am trying to check the output of a bad condition to test out of bound assertions, etc:
ASSERT_DEATH( { MYASSERT(false); }, "");
However all this does it report the following message:
Running main() from gtest_main.cc
..\Test\FormatUnitTest\Test_Format.cpp(59): error: Death test: { if (!(false)) { ReportFailture( "false", ..\\Test\\UnitTest\\Test.cpp", 59, __FSTREXP __FUNCTION_
_ ); __debugbreak(); }; }
Result: illegal return in test statement.
Error msg:
[ DEATH ]
It seems that GoogleTest is handling the Debug Exception in the structured exception handler (SEH) as a special case. However, I want to catch the assert and validate it's contents.
What's the right move here? Do I need to define a special assert macro for google test? If so what should it do?
I note that replacing my assert with assert(false) (included via assert.h) doesn't call this problem - what is it doing differently?
The problem is that you are calling __debugbreak(), which causes a breakpoint exception to be thrown by your code. As you can see in documentation for death tests (https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests), if code throws an exception it is not considered "death" by death tests in googletest.
As far as you other question goes, assert from assert.h calles std::abort (which causes program to terminate). This is "death" by definition of a death test.

Exception handling for Unit Tests in c++

I'm trying to test a c++ code on an Nunit framework but I keep getting the following Exception
System.Runtime.InteropServices.SEHException : External Component has thrown an exception.
which is supposedly perfectly normal (I assume) anyway I wanna ignore it. (i.e. Use ExpectedException) This is my .h file
[Test, Description("Tests if an Entity has been successfully Locked")]
void test_LockOperation();
and the .cpp file
void TestDmObstacles::test_LockOperation()
{
lockVal = DbtoDmObstaclesAdapter::lock( CmnGuid::parseString( L"3B6DB8F8-4BA7-DD11-B6A7-001E8CDE165C" ) );
//When lock is successful the lockVal is 0
Assert::AreEqual(0, lockVal);
}
I wanna use ExpectedException but I don't know how to do it in c++. I tried the try/catch method as well but it didn't work (I just put the Assertion in the catch block)
PS: I can't use another framework it has to be Nunit
EDIT
Here is the try/catch approach I used
void TestDmObstacles::test_LockOperation()
{
try
{
lockVal = DbtoDmObstaclesAdapter::lock( CmnGuid::parseString( L"3B6DB8F8-4BA7-DD11-B6A7-001E8CDE165C" ) );
}
catch (...)
{
//Assert::Fail();
Assert::AreEqual(0, lockVal);
}
}
Is the exception expected, or is the exception acceptable?
If it is expected, then your unit test framework should have some kind of API that allows you to state the expected exception, and to fail the test if it does not occur. A quick trawl through the documentation yields the incantation:
[ExpectedException( "System.ArgumentException" )]
(replace System.ArgumentException with the exception you're expecting.)
If the exception is merely acceptable, then I would say that either your code or your test is broken. A unit test is to test that expected things happen. If there is a result in your test that only may yield a particular result, then you are not testing a consistent view of the unit from test to test. Hence, you're not really testing it.
It might indicate, for example, that your code is leaking an unexpected exception that it should be handling instead.
Your code sample doesn't match what you are trying to achieve : if the exception is expected, than catching it is not supposed to fail the test.
Note that I wouldn't recommend (at all) for the test to catch (...) : any thrown exception will induce the same test result, which I doubt is what you want.

How do I write NUnit unit tests without having to surround them with try catch statements?

At my company we are writing a bunch of unit tests. What we'd like to have done is for the unit tests to execute and whenever one succeeds or fails at the end of the test we can write that somewhere but we don't want to put that logic in every test.
Any idea how we could just write tests without having to surround the content of the test with the try catch logic that we've been using?
I'm guessing you do something like this:
[Test]
public void FailBecauseOfException()
{
try
{
throw new Exception();
}
catch (Exception e)
{
Assert.Fail(e.Message);
}
}
There is no need for this. The tests will fail automatically if they throw an exception. For example, the following test will show up as a failure:
[Test]
public void FailBecauseOfException()
{
throw new Exception();
}
I'm not entirely sure what you are trying to do here. Are you saying you are wrapping it in a try/catch so that you can catch when an exception occurs and log this?
If so, then a better way, probably, is just to get NUnit to write an output file and use this. I haven't used NUnit for about a year, but IIRC you can redirect its output to any file you like using the /out directive.
If there is a reason why you have to log it the way you say, then you'll either have to add your custom code to each test, or have a common "runner" that takes your code (for each test) as an anonymous method and runs it inside a single try..catch. That would prevent you having to repeat the try..catch for every test.
Apologies if I've misunderstood the question.
MSTest has TestCleanup, which runs after every test. In NUnit, the attribute to be used is TearDown (after every test) or TestFixtureTearDown (after all the test are completely). This executes after the end of each test.
If you want something to run just in case a test passes, you could have a member variable shouldRunExtraMethod, which is initialized to false before each test, and is changed to true at the end of the test. And on the TearDown, you only execute it depending on this variable value
If your unit test method covers the scenario in which you expect exceptions to be thrown, use the ExpectedException attribute. There's a post here on SO about using that attribute.
Expect exceptions in nUnit...
NUnit assert statements all have an option to print a message for each test for when it fails.
Although if you'd like to have it write out something somewhere at the end of each test, you can set it up in the teardown of each method. Just set the string to what you want written inside the test itself, and during teardown (which happens after each test) It can do whatever you want with it.
I'm fairly certain teardown occurs even if an exception is thrown. That should do what you're wanting.
The problem you have is that the NUnit Assert.* methods will throw an AssertionException whenever an assert fails - but it does nothing else. So it doesn't look like you can check anything outside of the unit test to verify whether the test failed or not.
The only alternative I can think of is to use AOP (Aspect Oriented Programming) with a tool such as PostSharp. This tool allows you to create aspects that can act on certain events. For example:
public class ExceptionDialogAttribute : OnExceptionAspect
{
public override void OnException(MethodExecutionEventArgs eventArgs)
{
string message = eventArgs.Exception.Message;
Window window = Window.GetWindow((DependencyObject) eventArgs.Instance);
MessageBox.Show(window, message, "Exception");
eventArgs.FlowBehavior = FlowBehavior.Continue;
}
}
This aspect is code which runs whenever an exception is raised:
[ExceptionDialog]
[Test]
public void Test()
{
assert.AreEqual(2, 4);
}
Since the above test will raise an exception, the code in ExceptionDialogAttribute will run. You can get information about the method, such as it's name, so that you can log it into a file.
It's been a long time since I used PostSharp, so it's worth checking out the examples and experimenting with it.