We use CPPUnit to test our Test framework
The tests are organized in Test fixtures (inherited from CPPUNIT_NS::TestFixture)
There is a new requirement - To flush out the application buffer at the end of a test ONLY if it has failed.
I can do this in the overloaded teardown() function in the Test Fixture.
But how to know if a test has failed.
The result of a test is checked using CPPUNIT_ASSERT.
There are around 12 test fixtures with each fixture having around 10 tests.
How to achieve this with minimal code change?
I think it depends a bit on how you call your tests but my first idea would be to use a TestListener and react to the TestListener::addFailure call.
Note however that the tearDown can in theory also throw an exception (possibly through a CPPUNIT_ASSERT) which would also call the TestListener::addFailure.
If that does not work an obvious but really ugly solution is to set a flag at the end of each test method that signals that test finished successfully and call your code when the flag is not set.
Related
When I create a TEST or TEST_F test, how can I know that my assertion is actually executing?
The problem I have is, when I have an empty TEST_F, for example,
TEST_F(myFixture, test1) {}
When it runs, gtest says this test passes. I would have expected the test to fail, until I write test code. Anyway.
So, my problem is that when gtest says that when test is "OK" or that it passed, I can't trust it, because a test could "pass" if there is no test code.
It would be nice to print what my EXPECT_ or ASSERT calls are doing and then see that they pass. Problem is, if I do any std::cout calls, that seems to be out of sync with the test results at the end. The output messages are not in sync with any of my own std::cout calls.
Is there a verbose option to google test? How can I be sure the EXPECT that I coded is actually running?
You might consider looking at TDD, Test Driven Development, https://en.wikipedia.org/wiki/Test-driven_development
write one test => it will fail
write code to make the test pass => test passes
Rinse and repeat: express each requirement as a test, that initially fails. Write code to make that test pass.
I'm reading "Dependency Injection for .NET" and following along with some of the samples in the book and it lead me to a question about unit testing.
How does a unit test know when to pass or fail?
Not sure why, but I've always assumed you needed to Assert something and if the Assert is true, the unit test passes otherwise the test fails.
However, that seems to be not the case. Look at this sample below (using NUnit and Moq).
[Test]
public void Test_UserProvidedMessage()
{
Mock<IMessageWriter> m = new Mock<IMessageWriter>();
Salutation s = new Salutation(m.Object);
s.Exclaim("use this message silly");
m.Verify(w => w.Write("use this message silly"));
}
The unit test output is controlled through the Mock.Verify method. I understand that for this example but now I'm questioning what I know about unit tests passing or failing.
How does a unit test know when to pass or fail?
What criteria does the unit test framework use to determine if the output is pass or fail?
A unit test fails if it throws an exception.
All assertion methods and mock verifiers throw special exceptions that provide more detail about the failure.
I'm guessing maybe the Verify method throws an exception if it can't verify, and this causes the unit test to fail. And if there's no exception it means it passed.
I use the framework cppunit to test my classes,I want to know if the methods TestFixture::setUp() and TestFixture::tearDown() are called one time for TEST_SUITE or they are called for each method added to this suite
The methods wrap each individual test case. From the docs:
Each test runs in its own fixture so there can be no side effects
among test runs.
I'm writing unit tests (using NUnit & Moq) for my MVC 2 controllers, and am following examples in the Pro ASP.net MVC 2 Framework book by Steven Sanderson (great book, btw). However, I've run into problems, which I think are just due to my lack of understanding of NUnit.
Here's an excerpt, with the irrelevant parts removed:
[Test]
public void Cannot_Save_Invalid_Event()
{
...
repository.Setup(x => x.SaveEvent(evt)).Callback(Assert.Fail);
...
repository.Verify(x => x.SaveEvent(evt));
}
This test is passing for me, although from what I understand, those two statements should directly conflict with each other. The second one wasn't there originally, but I put it in to verify that it was passing for the right reasons.
From what I understand, my repository is set up to fail if "repository.SaveEvent(evt)" is called. However, later in the test, I try to verify that "repository.SaveEvent(evt)" was called. Since it passes, doesn't this mean that it was both called, and not called? Perhaps those statements don't act as I suspect they do.
Can someone explain how these two statements are not opposites, and how they can both exist and the test still pass?
Maybe your tests doesn-t fail beacuse it has a catch-everything block that also hides the assert/verify-exception that is necessary for the test to fail.
Note: the following unittest will allways pass
[Test]
public void HidingAssertionFailure()
{
try {
Assert.AreEqual(0,1); // this should fail
} catch (Exception ex) {
// this will hide the assertion failure
}
}
The reason for this behavior was that it was running "SaveEvent()", however, since the mocked repository didn't define that action, it was throwing an exception in my controller, which my controller was catching.
So, it seems that the callback will only execute if control returns successfully.
I'm writing some NUnit tests for database operations. Obviously, if Add() fails, then Get() will fail as well. However, it looks deceiving when both Add() and Get() fail because it looks like there's two problems instead of just one.
Is there a way to specify an 'order' for tests to run in, in that if the first test fails, the following tests are ignored?
In the same line, is there a way to order the unit test classes themselves? For example, I would like to run my tests for basic database operations first before the tests for round-tripping data from the UI.
Note: This is a little different than having tests depend on each other, it's more like ensuring that something works first before running a bunch of tests. It's a waste of time to, for example, run a bunch of database operations if you can't get a connection to the database in the first place.
Edit: It seems that some people are missing the point. I'm not doing this:
[Test]
public void AddTest()
{
db.Add(someData);
}
[Test]
public void GetTest()
{
db.Get(someData);
Assert.That(data was retrieved successfully);
}
Rather, I'm doing this:
[Test]
public void AddTest()
{
db.Add(someData);
}
[Test]
public void GetTest()
{
// need some way here to ensure that db.Add() can actually be performed successfully
db.Add(someData);
db.Get(somedata);
Assert.That(data was retrieved successfully);
}
In other words, I want to ensure that the data can be added in the first place before I can test whether it can be retrieved. People are assuming I'm using data from the first test to pass the second test when this is not the case. I'm trying to ensure that one operation is possible before attempting another that depends on it.
As I said already, you need to ensure you can get a connection to the database before running database operations. Or that you can open a file before performing file operations. Or connect to a server before testing API calls. Or...you get the point.
NUnit supports an "Assume.That" syntax for validating setup. This is documented as part of the Theory (thanks clairestreb). In the NUnit.Framework namespace is a class Assume. To quote the documentation:
/// Provides static methods to express the assumptions
/// that must be met for a test to give a meaningful
/// result. If an assumption is not met, the test
/// should produce an inconclusive result.
So in context:
public void TestGet() {
MyList sut = new MyList()
Object expecting = new Object();
sut.Put(expecting);
Assume.That(sut.size(), Is(1));
Assert.That(sut.Get(), Is(expecting));
}
Tests should never depend on each other. You just found out why. Tests that depend on each other are fragile by definition. If you need the data in the DB for the test for Get(), put it there in the setup step.
I think the problem is that you're using NUnit to run something other than the sort of Unit Tests that NUnit was made to run.
Essentially, you want AddTest to run before GetTest, and you want NUnit to stop executing tests if AddTest fails.
The problem is that that's antithetical to unit testing - tests are supposed to be completely independent and run in any order.
The standard concept of Unit Testing is that if you have a test around the 'Add' functionality, then you can use the 'Add' functionality in the 'Get' test and not worry about if 'Add' works within the 'Get' test. You know 'Add' works - you have a test for it.
The 'FIRST' principle (http://agileinaflash.blogspot.com/2009/02/first.html) describes how Unit tests should behave. The test you want to write violates both 'I' (Isolated) and 'R' (Repeatable).
If you're concerned about the database connection dropping between your two tests, I would recommend that rather than connect to a real database during the test, your code should use some sort of a data interface, and for the test, you should be using a mock interface. If the point of the test is to exercise the database connection, then you may simply be using the wrong tool for the job - that's not really a Unit test.
I don't think that's possible out-of-box.
Anyway, your test class design as you described will make the test code very fragile.
MbUnit seems to have a DependsOnAttribute that would allow you to do what you want.
If the other test fixture or test
method fails then this test will not
run. Moreover, the dependency forces
this test to run after those it
depends upon.
Don't know anything about NUnit though.
You can't assume any order of test fixture execution, so any prerequisites have to be checked for within your test classes.
Segregate your Add test into one test-class e.g. AddTests, and put the Get test(s) into another test-class, e.g. class GetTests.
In the [TestFixtureSetUp] method of the GetTests class, check that you have working database access (e.g. that Add's work), and if not, Assert.Ignore or Inconclusive, as you deem appropriate.
This will abort the GetTests test fixture when its prerequisites aren't met, and skip trying to run any of the unit tests it contains.
(I think! I'm an nUnit newbie.)
Create a global variable and return in the test for Get unless Add set it to true (do this in the last line of Add):
public boolean addFailed = false;
public void testAdd () {
try {
... old test code ...
} catch (Throwable t) { // Catch all errors
addFailed = true;
throw t; // Don't forget to rethrow
}
}
public void testGet () {
if (addFailed) return;
... old test code ...
}