I am using Google Test on code I expect to fail. As part of this failure the code calls a custom assert macro, which contains std::abort().
Unfortunately Google Test's EXPECT_EXIT() is not "catching" the std::abort().
This is a self-contained example to emulate what I'm trying to achieve:
// A placeholder for my assert macro
void MyFunction()
{
std::abort();
}
TEST(TestGroup1, TestName)
{
EXPECT_EXIT(MyFunction(), ::testing::ExitedWithCode(SIGABRT), ".*");
}
I get this failure output:
Death test: MyFunction()
Result: died but not with expected exit code:
Terminated by signal 6 (core dumped)
Actual msg:
[ DEATH ]
Is this possible to achieve?
I needed to use ::testing::KilledBySignal(SIGABRT)
Related
When I run the following program with cargo test:
use std::panic;
fn assert_panic_func(f: fn() -> (), msg: String) {
let result = panic::catch_unwind(|| {
f();
});
assert!(result.is_err(), msg);
}
macro_rules! assert_panic {
($test:expr , $msg:tt) => {{
fn wrapper() {
$test;
}
assert_panic_func(wrapper, $msg.to_string())
}};
}
fn main() {
println!("running main()");
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn t_1() {
assert_panic!(
panic!("We are forcing a panic"),
"This is asserted within function (raw panic)."
);
// assert_panic!(true, "This is asserted within function (raw true).");
}
}
I get the expected output:
running 1 test
test tests::t_1 ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
If I uncomment the second assert_panic!(...) line, and rerun cargo test, I get the following output:
running 1 test
test tests::t_1 ... FAILED
failures:
---- tests::t_1 stdout ----
thread 'tests::t_1' panicked at 'We are forcing a panic', src/lib.rs:29:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tests::t_1' panicked at 'This is asserted within function (raw true).', src/lib.rs:7:5
failures:
tests::t_1
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
The second panic is legitimate, and that is what I am looking for, but the first panic seems to be being triggered by the line that was not triggering a panic in the first invocation.
What is going on, and how do I fix it?
The stderr output
thread 'tests::t_1' panicked at 'We are forcing a panic', src/main.rs:30:23
is logged independently of whether a panic is caught, the test running just does not show any logged output unless a test fails. To suppress that text entirely, you would need to separately swap out the panic notification hook using std::panic::set_hook.
fn assert_panic_func(f:fn()->(), msg: String) {
let previous_hook = panic::take_hook();
// Override the default hook to avoid logging panic location info.
panic::set_hook(Box::new(|_| {}));
let result = panic::catch_unwind(|| {
f();
});
panic::set_hook(previous_hook);
assert!(result.is_err(), msg );
}
All that said, I second #SCappella's answer about using #[should_panic].
Even if std::panic::catch_unwind catches the panic, any output from that panic will be printed. The reason you don't see anything with the first test (with the commented out second panic) is that cargo test doesn't print output from successful tests.
To see this behavior more clearly, you can use main instead of a test. (playground)
fn main() {
let _ = std::panic::catch_unwind(|| {
panic!("I don't feel so good Mr. catch_unwind");
});
println!("But the execution of main still continues.");
}
Running this gives the output
thread 'main' panicked at 'I don't feel so good Mr. catch_unwind', src/main.rs:3:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
But the execution of main still continues.
Note that panics usually output to stderr, rather than stdout, so it's possible to filter these out.
See also Suppress panic output in Rust when using panic::catch_unwind.
I'm not sure if this is what you're trying to do, but if you want to ensure that a test panics, use the should_panic attribute. For example,
#[test]
#[should_panic]
fn panics() {
panic!("Successfully panicked");
}
At the time I was not aware that unit tests would suppress output messages. I later became aware of the suppression of output messages when researching why println!(...) would not work within unit tests. That this might also be an answer to why panics sometimes display and sometimes do not does make sense.
Nonetheless, it does seem to me to be perverse that panics produce output even when I explicitly tell Rust that I wish to prevent the panic from having any effect, but if that is what Rust does, however perverse that might seem, then one has to live with it.
I was aware of the #[should_panic] attribute, but was not happy with this solution for two reasons:
Firstly, it requires that each test becomes a separate function, whereas I tend to put a number of closely related tests (many of the tests being no more than a single assert!(...) statement) into one function.
Secondly, it would be nice to have a single model to express each test. To my mind, testing whether an expression raises a panic (or fails to raise a panic) is no different from testing whether the result is equal to, or unequal to, some particular value. It makes far more sense to me to create a single model to express both tests, hence my desire to have an assert_panic!(...) macro that behaved analogous to the assert!(...) or assert_eq!(...) macro. It seems that this is simply not an achievable objective within Rust.
Thank you for clearing that up.
The following assertion will produce a . or an F when PHPUnit is run:
$this->assertEquals('foo', $bar);
But what if I wanted to know the result of the assertion within the test itself? Something similar to:
if ($this->assertEquals('foo', $bar)) {
// log or do something
} else {
// do something else
}
There's probably no good reason to conditionally run assertions (or maybe there is), but this is a question purely from the debugging point of view.
PHPUnit uses exceptions to signify that a test has failed.
If we follow the assertEquals() code through, we can see that on line 137 of isEqual.php, a new PHPUnit_Framework_ExpectationFailedException is thrown if the operands are not equal. The test runner then converts this to a "F" for the console output and also uses if for reports etc.
You can catch the Exception and do what you like with it in your testcase. You may want to rethrow it if you still want the test to fail:
try {
$this->assertEquals('foo', $bar);
// do something else
} catch (PHPUnit_Framework_ExpectationFailedException $e) {
// log or do something
throw $e; // rethrow to make sure that the test still fails
}
On Windows, my assert macro essentially looks like this:
#define MYASSERT(condition) (if (!(condition)) { ReportFailture( #condition, __FILE__, __LINE__, __FUNCTION__); __debugbreak(); }
And in Google Test I am trying to check the output of a bad condition to test out of bound assertions, etc:
ASSERT_DEATH( { MYASSERT(false); }, "");
However all this does it report the following message:
Running main() from gtest_main.cc
..\Test\FormatUnitTest\Test_Format.cpp(59): error: Death test: { if (!(false)) { ReportFailture( "false", ..\\Test\\UnitTest\\Test.cpp", 59, __FSTREXP __FUNCTION_
_ ); __debugbreak(); }; }
Result: illegal return in test statement.
Error msg:
[ DEATH ]
It seems that GoogleTest is handling the Debug Exception in the structured exception handler (SEH) as a special case. However, I want to catch the assert and validate it's contents.
What's the right move here? Do I need to define a special assert macro for google test? If so what should it do?
I note that replacing my assert with assert(false) (included via assert.h) doesn't call this problem - what is it doing differently?
The problem is that you are calling __debugbreak(), which causes a breakpoint exception to be thrown by your code. As you can see in documentation for death tests (https://github.com/google/googletest/blob/master/googletest/docs/advanced.md#death-tests), if code throws an exception it is not considered "death" by death tests in googletest.
As far as you other question goes, assert from assert.h calles std::abort (which causes program to terminate). This is "death" by definition of a death test.
I'm testing some code that uses CHECK from glog and I'd like to test that this check fails in certain scenarios. My code looks like:
void MyClass::foo() {
// stuff...
// It's actually important that the binary gets aborted if this flag is false
CHECK(some_flag) << "flag must be true";
// more stuff...
}
I've done some research into gtest and how I might be able to test for this. I found EXPECT_FATAL_FALIURE, EXPECT_NONFATAL_FAILURE, and HAS_FATAL_FAILURE but I haven't managed to figure out how to use them. I'm fairly confident that if I change CHECK(some_flag) to EXPECT_TRUE(some_flag) then EXPECT_FATAL_FAILURE will work correctly but then I'm introducing test dependencies in non-test files which is...icky.
Is there a way for gtest to catch the abort signal (or whatever CHECK raises) and expect it?
aaaand I found an answer 5 minutes after posting this question. Typical.
This can be done using Death tests from gtest. Here's how my test looks:
TEST(MyClassTest, foo_death_test) {
MyClass clazz(false); // make some_flag false so the CHECK fails
ASSERT_DEATH( { clazz.foo(); }, "must be true");
}
This passes. Woohoo!
I am using WPToolkitTestFx Unit test framework for the Windows Phone 8 application. I am getting below error when I am executing Assert.Fail() or Assert.IsFalse(true).
A first chance exception of type 'Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException' occurred in Microsoft.VisualStudio.QualityTools.UnitTesting.Phone.DLL
Any solution to the above error.
Here the source code
[TestMethod]
[Asynchronous]
[Description("Test2: Sample asynchronous test")]
public void Test2()
{
// this test executes asynchronously
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
// ... and then fails
Assert.IsFalse(true);
EnqueueTestComplete();
});
}
Thanks,
Ragu
I can see something similar when following the sample code from http://blogs.windows.com/windows_phone/b/wpdev/archive/2012/11/20/windows-phone-toolkit-overview.aspx.
It is normal for the exception to occur, and see:
A first chance exception of type 'Microsoft.VisualStudio.TestTools.UnitTesting.AssertFailedException' occurred in Microsoft.VisualStudio.QualityTools.UnitTesting.Phone.DLL
Also if the debugger is attached to your device/emulator, the debugger will break so you can find out where the test failure was.
private void Application_UnhandledException(object sender, ApplicationUnhandledExceptionEventArgs e)
{
if (Debugger.IsAttached)
{
// An unhandled exception has occurred; break into the debugger
Debugger.Break(); // <- !!! application will stop in debugger here !!!
}
}
However if you continue the project (press F5), or are not running under the debugger, you will see that the application continues to run and doesn't get exited.
That allows you to see your test results.
Question, is it a normal part of your test to fail the assertion? if so, you need to instruct the framework to expect such exception to happen
[TestMethod, Asynchronous]
[Description("Test2: Sample asynchronous test")]
[ExpectedException(typeof(AssertFailedException))]
public void Test2()
{
// this test executes asynchronously
Deployment.Current.Dispatcher.BeginInvoke(() =>
{
// ... and then fails
Assert.IsFalse(true);
//TestComplete();
});
}
Also I notice that you are marking the method as Asynchronous, but you are not using the asynchronous invocation methods of EnqueueDelay(TimeSpan), EnqueueCallback(), EnqueueTestComplete() are the ones that make the method function asynchronous.