I'm testing some code that uses CHECK from glog and I'd like to test that this check fails in certain scenarios. My code looks like:
void MyClass::foo() {
// stuff...
// It's actually important that the binary gets aborted if this flag is false
CHECK(some_flag) << "flag must be true";
// more stuff...
}
I've done some research into gtest and how I might be able to test for this. I found EXPECT_FATAL_FALIURE, EXPECT_NONFATAL_FAILURE, and HAS_FATAL_FAILURE but I haven't managed to figure out how to use them. I'm fairly confident that if I change CHECK(some_flag) to EXPECT_TRUE(some_flag) then EXPECT_FATAL_FAILURE will work correctly but then I'm introducing test dependencies in non-test files which is...icky.
Is there a way for gtest to catch the abort signal (or whatever CHECK raises) and expect it?
aaaand I found an answer 5 minutes after posting this question. Typical.
This can be done using Death tests from gtest. Here's how my test looks:
TEST(MyClassTest, foo_death_test) {
MyClass clazz(false); // make some_flag false so the CHECK fails
ASSERT_DEATH( { clazz.foo(); }, "must be true");
}
This passes. Woohoo!
Related
I have a unit test, written with JUnit 5 (Jupiter), that is failing. I do not currently have time to fix the problem, so I would like to mark the test as an expected failure. Is there a way to do that?
I see #Disable which causes the test to not be run. I would like the test to still run (and ideally fail the build if it starts to work), so that I remember that the test is there.
Is there such an annotation in Junit 5? I could use assertThrows to catch the error, but I would like the build output to indicate that this is not a totally normal test.
You can disable the failing test with the #Disabled annotation. You can then add another test that asserts the first one does indeed fail:
#Test
#Disabled
void fixMe() {
Assertions.fail();
}
#Test
void fixMeShouldFail() {
assertThrows(AssertionError.class, this::fixMe);
}
When I create a TEST or TEST_F test, how can I know that my assertion is actually executing?
The problem I have is, when I have an empty TEST_F, for example,
TEST_F(myFixture, test1) {}
When it runs, gtest says this test passes. I would have expected the test to fail, until I write test code. Anyway.
So, my problem is that when gtest says that when test is "OK" or that it passed, I can't trust it, because a test could "pass" if there is no test code.
It would be nice to print what my EXPECT_ or ASSERT calls are doing and then see that they pass. Problem is, if I do any std::cout calls, that seems to be out of sync with the test results at the end. The output messages are not in sync with any of my own std::cout calls.
Is there a verbose option to google test? How can I be sure the EXPECT that I coded is actually running?
You might consider looking at TDD, Test Driven Development, https://en.wikipedia.org/wiki/Test-driven_development
write one test => it will fail
write code to make the test pass => test passes
Rinse and repeat: express each requirement as a test, that initially fails. Write code to make that test pass.
I'm writing tests using gtest and gmock. Most of my test cases are supposed to crash with an custom assert (that I mock).
Here come troubles : whereas the assert is well triggered, I have plenty of problems with the expected calls.
The following code is the three steps I have been through in order to make it works ('cause yes, this part works) :
class MyTestedObject : public testing::Test {
public:
static MockObject * myMockedObject;
};
void assertFailure() {
exit(1);
}
TEST_F(MyTestedObjectDeathTest, nullInputConstructors) {
MockAssertHandler assertHandler;
EXPECT_CALL(assertHandler, failure(_,_,_,_))
.Times(1)
.WillRepeatedly(InvokeWithoutArgs(assertFailure));
setHandler(assertHandler);
testing::Mock::AllowLeak(myMockedObject);
testing::Mock::AllowLeak(&assertHandler);
EXPECT_DEATH(new MyTestedObject(NULL, NULL,0), ".*");
}
MyTestedObject's constructor begins to check whether arguments are NULL. It's supposed to trigger an assert if at least one of them is. But the test fails because failure is 'never called'. Debug reveals it is called.
Then I tried to comment the Times part, just to be sure it came from here and it was the only issue. It works, but is not adequate : I want to be sure the program die from my assert. As the evaluation of EXPECT_CALL is done when the mock object is destroyed, I guessed the exit calls was messing up the whole thing, so I tried this, which works :
void testHelper() {
MockAssertHandler assertHandler;
EXPECT_CALL(assertHandler, failure(_,_,_,_))
.Times(1)
.WillRepeatedly(InvokeWithoutArgs(assertFailure));
setHandler(assertHandler);
testing::Mock::AllowLeak(MyTestedObjectTest::myMockObject);
testing::Mock::AllowLeak(&assertHandler);
new MyTestedObject(NULL, NULL,0);
}
TEST_F(MyTestedObjectDeathTest, nullInputConstructors) {
EXPECT_DEATH(testHelper(), ".*");
}
Now, I'd like to be sure no calls are made on others functions.
I tried to put myMockedObject in a StrictMock structure and to put EXPECT_CALL(...).Times(0), but I got the same pattern as at first : the 'exit' call seems to block all EXPECT_CALL evaluation.
Any hint /workaround ? :)
EDIT : Forgot to tell : Execution env. is Windows 7 with Visual Studio 2008.
Google Test's wiki explains this:
Since statement runs in the child process, any in-memory side effect
(e.g. modifying a variable, releasing memory, etc) it causes will not
be observable in the parent process.
That includes Google Mock's tracking of calls made in the death test statement. In short, Google Mock and death tests don't mix.
My advice here is to separate those tests. Use Google Mock to verify that a failure handler is invoked and use death tests to verify that your failure handler does indeed do what it's supposed to (terminate the program, print specified output, etc.)
I want to add a testcase for functionality not yet implemented and mark this test case as "it's ok that I fail".
Is there a way to do this?
EDIT:
I want the test to be executed and the framework should verify it is failing as long as the testcase is in the "expected fail" state.
EDIT2:
It seems that the feature I am interested in does not exist in google-test, but it does exist in the Boost Unit Test Framework, and in LIT.
EXPECT_NONFATAL_FAILURE is what you want to wrap around the code that you expect to fail. Note you will hav to include the gtest-spi.h header file:
#include "gtest-spi.h"
// ...
TEST_F( testclass, testname )
{
EXPECT_NONFATAL_FAILURE(
// your code here, or just call:
FAIL()
,"Some optional text that would be associated with"
" the particular failure you were expecting, if you"
" wanted to be sure to catch the correct failure mode" );
}
Link to docs: https://github.com/google/googletest/blob/955c7f837efad184ec63e771c42542d37545eaef/docs/advanced.md#catching-failures
You can prefix the test name with DISABLED_.
I'm not aware of a direct way to do this, but you can fake it with something like this:
try {
// do something that should fail and throw and exception
...
EXPECT_TRUE(false); // this should not be reached!
} catch (...) {
// return or print a message, etc.
}
Basically, the test will fail if it reaches the contradictory expectation.
It would be unusual to have a unit test in an expected-to-fail state. Unit tests can test for positive conditions ("expect x to equal 2") or negative conditions ("expect save to throw an exception if name is null"), and can be flagged not to run at all (if the feature is pending and you don't want the noise in your test output). But what you seem to be asking for is a way to negate a feature's test while you're working on it. This is against the tenants of Test Driven Development.
In TDD, what you should do is write tests that accurately describe what a feature should do. If that feature isn't written yet then, by definition, those tests will and should fail. Then you implement the feature until, one by one, all those tests pass. You want all the tests to start as failing and then move to passing. That's how you know when your feature is complete.
Think of how it would look if you were able to mark failing tests as passing as you suggest: all tests would pass and everything would look complete when the feature didn't work. Then, once you were done and the feature worked as expected, suddenly your tests would start to fail until you went in and unflagged them. Beyond being a strange way to work, this workflow would be very prone to error and false-positives.
At my company we are writing a bunch of unit tests. What we'd like to have done is for the unit tests to execute and whenever one succeeds or fails at the end of the test we can write that somewhere but we don't want to put that logic in every test.
Any idea how we could just write tests without having to surround the content of the test with the try catch logic that we've been using?
I'm guessing you do something like this:
[Test]
public void FailBecauseOfException()
{
try
{
throw new Exception();
}
catch (Exception e)
{
Assert.Fail(e.Message);
}
}
There is no need for this. The tests will fail automatically if they throw an exception. For example, the following test will show up as a failure:
[Test]
public void FailBecauseOfException()
{
throw new Exception();
}
I'm not entirely sure what you are trying to do here. Are you saying you are wrapping it in a try/catch so that you can catch when an exception occurs and log this?
If so, then a better way, probably, is just to get NUnit to write an output file and use this. I haven't used NUnit for about a year, but IIRC you can redirect its output to any file you like using the /out directive.
If there is a reason why you have to log it the way you say, then you'll either have to add your custom code to each test, or have a common "runner" that takes your code (for each test) as an anonymous method and runs it inside a single try..catch. That would prevent you having to repeat the try..catch for every test.
Apologies if I've misunderstood the question.
MSTest has TestCleanup, which runs after every test. In NUnit, the attribute to be used is TearDown (after every test) or TestFixtureTearDown (after all the test are completely). This executes after the end of each test.
If you want something to run just in case a test passes, you could have a member variable shouldRunExtraMethod, which is initialized to false before each test, and is changed to true at the end of the test. And on the TearDown, you only execute it depending on this variable value
If your unit test method covers the scenario in which you expect exceptions to be thrown, use the ExpectedException attribute. There's a post here on SO about using that attribute.
Expect exceptions in nUnit...
NUnit assert statements all have an option to print a message for each test for when it fails.
Although if you'd like to have it write out something somewhere at the end of each test, you can set it up in the teardown of each method. Just set the string to what you want written inside the test itself, and during teardown (which happens after each test) It can do whatever you want with it.
I'm fairly certain teardown occurs even if an exception is thrown. That should do what you're wanting.
The problem you have is that the NUnit Assert.* methods will throw an AssertionException whenever an assert fails - but it does nothing else. So it doesn't look like you can check anything outside of the unit test to verify whether the test failed or not.
The only alternative I can think of is to use AOP (Aspect Oriented Programming) with a tool such as PostSharp. This tool allows you to create aspects that can act on certain events. For example:
public class ExceptionDialogAttribute : OnExceptionAspect
{
public override void OnException(MethodExecutionEventArgs eventArgs)
{
string message = eventArgs.Exception.Message;
Window window = Window.GetWindow((DependencyObject) eventArgs.Instance);
MessageBox.Show(window, message, "Exception");
eventArgs.FlowBehavior = FlowBehavior.Continue;
}
}
This aspect is code which runs whenever an exception is raised:
[ExceptionDialog]
[Test]
public void Test()
{
assert.AreEqual(2, 4);
}
Since the above test will raise an exception, the code in ExceptionDialogAttribute will run. You can get information about the method, such as it's name, so that you can log it into a file.
It's been a long time since I used PostSharp, so it's worth checking out the examples and experimenting with it.