I'm reading "Dependency Injection for .NET" and following along with some of the samples in the book and it lead me to a question about unit testing.
How does a unit test know when to pass or fail?
Not sure why, but I've always assumed you needed to Assert something and if the Assert is true, the unit test passes otherwise the test fails.
However, that seems to be not the case. Look at this sample below (using NUnit and Moq).
[Test]
public void Test_UserProvidedMessage()
{
Mock<IMessageWriter> m = new Mock<IMessageWriter>();
Salutation s = new Salutation(m.Object);
s.Exclaim("use this message silly");
m.Verify(w => w.Write("use this message silly"));
}
The unit test output is controlled through the Mock.Verify method. I understand that for this example but now I'm questioning what I know about unit tests passing or failing.
How does a unit test know when to pass or fail?
What criteria does the unit test framework use to determine if the output is pass or fail?
A unit test fails if it throws an exception.
All assertion methods and mock verifiers throw special exceptions that provide more detail about the failure.
I'm guessing maybe the Verify method throws an exception if it can't verify, and this causes the unit test to fail. And if there's no exception it means it passed.
Related
We use CPPUnit to test our Test framework
The tests are organized in Test fixtures (inherited from CPPUNIT_NS::TestFixture)
There is a new requirement - To flush out the application buffer at the end of a test ONLY if it has failed.
I can do this in the overloaded teardown() function in the Test Fixture.
But how to know if a test has failed.
The result of a test is checked using CPPUNIT_ASSERT.
There are around 12 test fixtures with each fixture having around 10 tests.
How to achieve this with minimal code change?
I think it depends a bit on how you call your tests but my first idea would be to use a TestListener and react to the TestListener::addFailure call.
Note however that the tearDown can in theory also throw an exception (possibly through a CPPUNIT_ASSERT) which would also call the TestListener::addFailure.
If that does not work an obvious but really ugly solution is to set a flag at the end of each test method that signals that test finished successfully and call your code when the flag is not set.
I have the following Groovy unit test code:
class MyTest extends GroovyTestCase {
#Test(expected = IllegalArgumentException.class)
public void testReadFileMissing() {
// does something which causes an IllegalArgumentException
}
// more tests
}
This works perfectly with my Java unit tests, but with groovy tests, mvn clean test runs the test, but the test fails because an IllegalArgumentException was thrown. In other words, my "expected" annotation attribute seems to be completely ignored.
I could of course simply use a try/catch block to check the behaviour I'm interested in, but I'd like to use the JUnit API if possible because that's what it's for and I find the resulting code simpler to read and understand.
So, can anyone tell me what I'm doing wrong?
either don't use GroovyTestCase and the according JUnit class instead as base or go full groovy and use shouldFail. examples are here http://mrhaki.blogspot.de/2009/11/groovy-goodness-testing-for-expected.html
You can use shouldFail or shouldFailWithCause showcasing exactly which kind of exception is expected from the code under test.
this is one of my questions about unit testing.
I'm reading The Art Of Unit Testing and at chapter 3 the author shows how to remove dependency between one or more classes. That seems clear to me. What's not absolutely clear is the following point.
When I configure a test method with a stub, I configure it to return a specific value. Then I call the tested method exposed by the tested class. This method executes some logic and uses the return value of the stub. The problem is: if the stub is configured to return the wrong value my test will probably fail.
So the question is: when I use stubs, should I ALWAYS configure them to return the expected value? In my opininon this should be the correct way to test as if the stub return always the expected value I'm sure to test only the logic inside the tested method.
What do you think about this? Is there some case in which has some kind of sense to oblige the stub to return uncorrect values?
Thanks a lot,
Marco
You are testing how the sut (system under test) works under several conditions:
the good path = configuring stubs to return good values and test that the sut behaves accordingly
the sad path(s) = configure stubs with wrong values and verify that the sut can handle such cases (e.g. you can test that it throws an Exception using the ExpectedException attribute if you're using nUnit)
You could configure the stub method to return value depending on the test setup in some scenarios. In others to return default value, which should be valid.
I'm writing unit tests (using NUnit & Moq) for my MVC 2 controllers, and am following examples in the Pro ASP.net MVC 2 Framework book by Steven Sanderson (great book, btw). However, I've run into problems, which I think are just due to my lack of understanding of NUnit.
Here's an excerpt, with the irrelevant parts removed:
[Test]
public void Cannot_Save_Invalid_Event()
{
...
repository.Setup(x => x.SaveEvent(evt)).Callback(Assert.Fail);
...
repository.Verify(x => x.SaveEvent(evt));
}
This test is passing for me, although from what I understand, those two statements should directly conflict with each other. The second one wasn't there originally, but I put it in to verify that it was passing for the right reasons.
From what I understand, my repository is set up to fail if "repository.SaveEvent(evt)" is called. However, later in the test, I try to verify that "repository.SaveEvent(evt)" was called. Since it passes, doesn't this mean that it was both called, and not called? Perhaps those statements don't act as I suspect they do.
Can someone explain how these two statements are not opposites, and how they can both exist and the test still pass?
Maybe your tests doesn-t fail beacuse it has a catch-everything block that also hides the assert/verify-exception that is necessary for the test to fail.
Note: the following unittest will allways pass
[Test]
public void HidingAssertionFailure()
{
try {
Assert.AreEqual(0,1); // this should fail
} catch (Exception ex) {
// this will hide the assertion failure
}
}
The reason for this behavior was that it was running "SaveEvent()", however, since the mocked repository didn't define that action, it was throwing an exception in my controller, which my controller was catching.
So, it seems that the callback will only execute if control returns successfully.
At my company we are writing a bunch of unit tests. What we'd like to have done is for the unit tests to execute and whenever one succeeds or fails at the end of the test we can write that somewhere but we don't want to put that logic in every test.
Any idea how we could just write tests without having to surround the content of the test with the try catch logic that we've been using?
I'm guessing you do something like this:
[Test]
public void FailBecauseOfException()
{
try
{
throw new Exception();
}
catch (Exception e)
{
Assert.Fail(e.Message);
}
}
There is no need for this. The tests will fail automatically if they throw an exception. For example, the following test will show up as a failure:
[Test]
public void FailBecauseOfException()
{
throw new Exception();
}
I'm not entirely sure what you are trying to do here. Are you saying you are wrapping it in a try/catch so that you can catch when an exception occurs and log this?
If so, then a better way, probably, is just to get NUnit to write an output file and use this. I haven't used NUnit for about a year, but IIRC you can redirect its output to any file you like using the /out directive.
If there is a reason why you have to log it the way you say, then you'll either have to add your custom code to each test, or have a common "runner" that takes your code (for each test) as an anonymous method and runs it inside a single try..catch. That would prevent you having to repeat the try..catch for every test.
Apologies if I've misunderstood the question.
MSTest has TestCleanup, which runs after every test. In NUnit, the attribute to be used is TearDown (after every test) or TestFixtureTearDown (after all the test are completely). This executes after the end of each test.
If you want something to run just in case a test passes, you could have a member variable shouldRunExtraMethod, which is initialized to false before each test, and is changed to true at the end of the test. And on the TearDown, you only execute it depending on this variable value
If your unit test method covers the scenario in which you expect exceptions to be thrown, use the ExpectedException attribute. There's a post here on SO about using that attribute.
Expect exceptions in nUnit...
NUnit assert statements all have an option to print a message for each test for when it fails.
Although if you'd like to have it write out something somewhere at the end of each test, you can set it up in the teardown of each method. Just set the string to what you want written inside the test itself, and during teardown (which happens after each test) It can do whatever you want with it.
I'm fairly certain teardown occurs even if an exception is thrown. That should do what you're wanting.
The problem you have is that the NUnit Assert.* methods will throw an AssertionException whenever an assert fails - but it does nothing else. So it doesn't look like you can check anything outside of the unit test to verify whether the test failed or not.
The only alternative I can think of is to use AOP (Aspect Oriented Programming) with a tool such as PostSharp. This tool allows you to create aspects that can act on certain events. For example:
public class ExceptionDialogAttribute : OnExceptionAspect
{
public override void OnException(MethodExecutionEventArgs eventArgs)
{
string message = eventArgs.Exception.Message;
Window window = Window.GetWindow((DependencyObject) eventArgs.Instance);
MessageBox.Show(window, message, "Exception");
eventArgs.FlowBehavior = FlowBehavior.Continue;
}
}
This aspect is code which runs whenever an exception is raised:
[ExceptionDialog]
[Test]
public void Test()
{
assert.AreEqual(2, 4);
}
Since the above test will raise an exception, the code in ExceptionDialogAttribute will run. You can get information about the method, such as it's name, so that you can log it into a file.
It's been a long time since I used PostSharp, so it's worth checking out the examples and experimenting with it.