I'm learning GwtMockito but having trouble getting consistent verify() method results in one of my tests.
I'm trying to test the correct GwtEvents are being fired by my application. So I've mocked the Event Bus like this in my #Before method:
eventBus = mock(HandlerManager.class);
This test passes as expected:
// Passes as expected
verify(eventBus).fireEvent(any(ErrorOccurredEvent.class));
I wanted to force the test to fail just to know it was running correctly. So I changed it to this and it still passes:
// Expected this to fail, but it passes
verify(eventBus).fireEvent(any(ErrorOccurredEvent.class));
verifyZeroInteractions(eventBus).fireEvent(any(ErrorOccurredEvent.class));
This seems contradictory to me. So I removed the first test:
// Fails as expected
verifyZeroInteractions(eventBus).fireEvent(any(ErrorOccurredEvent.class));
Finally I added an unrelated event that should cause it to fail
// Expected to fail, but passes
verify(eventBus).fireEvent(any(ErrorOccurredEvent.class));
verify(eventBus).fireEvent(any(ModelCreatedEvent.class)); // This event is not used at all by the class that I'm testing. It's not possible for it to be fired.
I'm not finding any documentation that explains what's going on. Both ErrorOccurredEvent and ModelCreatedEvent extend GwtEvent, and have been verified in manual testing. Am I testing my EventBus incorrectly? If so, what is a better way to go about it?
Update
I've done some additional experimenting. It appears to be an issue I'm having with the Mockito matcher. When I get the test to fail the exception reports the method signature as eventBus.fireEvent(<any>) so it doesn't appear to be taking into account the different classes I'm passing into the any method. Not sure what to do about this yet, but including it here for anyone else researching this problem.
The method you're looking for is isA, instead of any.
This doesn't explain my first attempt to force the test to fail, but it does explain the other confusion. From the Mockito documentation:
public static T any(java.lang.Class clazz)
Matches any object, including nulls
This method doesn't do type checks with the given parameter, it is
only there to avoid casting in your code. This might however change
(type checks could be added) in a future major release.
So by design it doesn't do the type checks I was hoping for. I'll have to work out another way to design these tests. But this explains why they weren't behaving as I expected.
Related
Consider unit testing a dictionary object. The first unit tests you might write are a few that simply adds items to the dictionary and check exceptions. The next test may be something like testing that the count is accurate, or that the dictionary returns a correct list of keys or values.
However, each of these later cases requires that the dictionary can first reliably add items. If the tests which add items fail, we have no idea whether our later tests fail because of what they're testing is implemented incorrectly, or because the assumption that we can reliably add items is incorrect.
Can I declare a set of unit tests which cause a given unit test to be inconclusive if any of them fail? If not, how should I best work around this? Have I set up my unit tests wrong, that I'm running into this predicament?
It's not as hard as it might seem. Let's rephrase the question a bit:
If I test my piece of code which requires System.Collections.Generic.List<T>.Add to work, what should I do when one day Microsoft decides to break .Add on List<T>? Do I make my tests depending on this to work inconclusive?
Answer to the above is obvious; you don't. You let them fail for one simple reason - your assumptions have failed, and test should fail. It's the same here. Once you get your add tests to work, from that point on you assume add works. You shouldn't treat your tested code any differently than 3rd party tested code. Once it's proven to work, you assume it indeed does.
On a different note, you can utilize concept called guard assertions. In your remove test, after the arrange phase you introduce additional assert phase, which verifies your initial assumptions (in this case - that the add is working). More information about this technique can be found here.
To add an example, NUnit uses the concept above disguised under the name Theory. This does exactly what you proposed (yet it seems to be more related to data driven testing rather than general utility):
The theory itself is responsible for ensuring that all data supplied meets its assumptions. It does this by use of the Assume.That(...) construct, which works just like Assert.That(...) but does not cause a failure. If the assumption is not satisfied for a particular test case, that case returns an Inconclusive result, rather than a Success or Failure.
However, I think what Mark Seemann states in an answer to the question I linked makes the most sense:
There may be many preconditions that need to be satisfied for a given test case, so you may need more than one Guard Assertion. Instead of repeating those in all tests, having one (and one only) test for each precondition keeps your test code more mantainable, since you will have less repetition that way.
Nice question, I often ponder this and had this problem the other day. What I did was get the basics of our collection working using a dictionary behind the scenes. For example:
public class MyCollection
{
private IDictionary<string, int> backingStore;
public MyCollection(IDictionary<string, int> backingStore)
{
_backingStore = backingStore;
}
}
Then we test drove the addition implementation. As we had the dictionary by reference we could assert that after adding items our business logic was correct.
For example the pseudo code for the additon was something like:
public void Add(Item item)
{
// Check we have not added before
// More business logic...
// Add
}
Then the test could be written such as:
var subject = new MyCollection(backingStore);
subject.Add(new Item())
Assert.That(backingStore.Contains(itemThatWeAdded)
We then went on to drive out the other methods such as retrieval, and deletion.
Your question is what should you do with regards the addition breaking, in turn breaking the retrieval. This is a catch 22 scenario. Personally I'd rather ditch the backing store and use this as an implementation detail. So this is what we did. We refactored the tests to use the system under test, rather than the backing store for the asserts. The great thing about the backing store being public initially is it allows you test drive small parts of the codebase, rather than having to implement both addition and retrieval in one go.
The test for addition then looked like the following after we refactored the collection to not expose the backing store.
var subject = new MyCollection();
var item = new Item()
subject.Add(item)
Assert.That(subject.Has(item), Is.True);
In this case I think this is fine. If you can not add items successfully then you sure as hell cannot retrieve anything because you've not added them. As long as your tests are named well any developer seeing some test such as "CanOnlyAddUniqueItemsToCollection" will point future developers in the right direction, in other words, the addition is broken. Just make sure your tests are named well and you should be giving as much help as possible.
I don't see this as too much of a problem. If your Dictionary class is not too big, and the unit test for that class is the only unit test testing that code, then when your add method is broken and multiple tests fail, you still know the problem is in the Dictionary class and can identify it, debug and fix it easily.
Where it becomes a problem is when you have other code smells or design problems such as:
unit tests tests are testing many application classes, using mocks instead can help here.
unit tests are actually system tests creating and testing many application classes at once.
the Dictionary class is too big and complex so when it breaks and tests fail it's difficult to figure out what part is broken.
This is very interesting. We use NUnit and the best I can tell it runs test-methods alphabetically. That might be an overly-artificial way to order your tests, but if you built up your test classes such that alphabetically/numerically-named pre-req methods came first you might accomplish what you want.
I find myself writing a test method, firing just it to watch it fail, and then writing the code to make it pass. When I'm all done I can run the whole class and everything passes - it doesn't matter what order the tests ran in becuase everything 'works' becuase of the incremental dev I did.
Now later on if I break something in the thing i'm testing who knows what all will fail in the harness. I guess it doesn't really matter to me - I've got a long list of failures and I can tease out what went wrong.
Suppose I have several unit tests in a test class ([TestClass] in VSUnit in my case). I'm trying to test just one thing in each test (doesn't mean just one Assert though). Imagine there's one test (e.g. Test_MethodA() ) that tests a method used in other tests as well. I do not want to put an assert on this method in other tests that use it to avoid duplicity/maintainability issues so I have the assert in only this one test. Now when this test fails, all tests that depend on correct execution of that tested method fail as well. I want to be able to locate the problem faster so I want to be somehow pointed to Test_MethodA. It would e.g. help if I could make some of the tests in the test class execute in a particular order and when they fail I'd start looking for cause of the failure in the first failing test. Do you have any idea how to do this?
Edit: By suggesting that a solution would be to execute the tests in a particular order I have probably went too far and in the wrong direction. I don't care about the order of the tests. It's just that some of the tests will always fail if a prequisite isn't valid. E.g. I have a test class that tests a DAO class (ok, probably not a UNIT test, but there's logic in the database stored procedures that needs to be tested but that's not the point here I think). I need to insert some records into a table in order to test that a method responsible for retrieving the records (let's call it GetAll()) gets them all in the correct order e.g. I do the insert by using a method on the DAO class. Let's call it Insert(). I have tests in place that verify that the Insert() method works as expected. Now I want to test the GetAll() method. In order to get the database in a desired state I use the Insert() method. If Insert() doesn't work, most tests for GetAll() will fail. I'd prefer to mark the tests that can't pass because Insert() doesn't work as inconclusive rather than failed. It would ease finding the cause of the problem if I know which method/test to look into first.
You can't (and shouldn't) execute unit tests in a specific order. The underlying reason for this is to prevent Interacting Tests - I realize that your motivation for requesting such a feature is different, but that's the reason why unit test frameworks don't allow you to order tests. In fact, last time I checked, xUnit.net even randomizes the order.
One could argue that the fact that some of your tests depend on a different method call on the same class is a symptom of tight coupling, but that's not always the case (state machines come to mind).
However, if possible, consider using a Back Door instead of the other method in question.
If you can't do either that or decouple the interdependency (e.g. by making the first method virtual and using the Extract and Override technique), you will have to live with it.
Here's an example:
public class MyClass
{
public virtual void FirstMethod() { // do something... }
public void SecondMethod() {}
}
Since FirstMethod is virtual, you can derive from MyClass and override its behavior. You can also use a dynamic mock to do that for you. With Moq, it would look like this:
var sutStub = new Mock<MyClass>();
// by default, Moq overrides all virtual methods without calling base
// Now invoke both methods in sequence:
sutStub.Object.FirstMethod(); // overriden by Moq, so it does nothing
sutSutb.Object.SecondMethod();
I think I would indeed have the assertion on the method_A() result in every tests relying on its result, even if this introduces some duplication. Then I would use the assertion message to point to the method_A() failure.
assert("method_A() returned true", true, rc);
Perhaps will I end extracting the method_A() call and the assertion into an helper function to remove the duplication.
Now let's imagine method_A() queries an object and returns it, or NULL when no object is found. Then this assertion is a guard ; and it is necessary with languages suchas C, C++ that do not have NullPointerException.
I'm afraid you can't do this. The only solution is to redesign your code and break it up into smaller methods so that unit tests can call these one by one. Of course this isn't always desirable.
With Visual Studio you can order your tests: see here. But I'd like to advise you to stay away from this technique as much as possible: unit tests are meant to be run anywhere, anytime and in every order.
EDIT: why is this a problem for you? All failing tests point to the same method anyway...
I think understand the definition of State / Interaction based testing (read the Fowler thing, etc). I found that I started state based but have been doing more interaction based and I'm getting a bit confused on how to test certain things.
I have a controller in MVC and an action calls a service to deny a package:
public ActionResult Deny(int id)
{
service.DenyPackage(id);
return RedirectToAction("List");
}
This seems clear to me. Provide a mock service, verify it was called correctly, done.
Now, I have an action for a view that lets the user associate a certificate with a package:
public ActionResult Upload(int id)
{
var package = packageRepository.GetPackage(id);
var certificates = certificateRepository.GetAllCertificates();
var view = new PackageUploadViewModel(package, certificates);
return View(view);
}
This one I'm a bit stumped on. I'm doing Spec style tests (possibly incorrectly) so to test this method I have a class and then two tests: verify the package repository was called, verify the certificate repository was called. I actually want a third to test to verify that the constructor was called but have no idea how to do that! I'm get the impression this is completely wrong.
So for state based testing I would pass in the id and then test the ActionResult's view. Okay, that makes sense. But wouldn't I have a test on the PackageUploadViewModel constructor? So if I have a test on the constructor, then part of me would just want to verify that I call the constructor and that the action return matches what the constructor returns.
Now, another option I can think of is I have a PackageUploadViewModelBuilder (or something equally dumbly named) that has dependency on the two repositories and then I just pass the id to a CreateViewModel method or something. I could then mock this object, verify everything, and be happy. But ... well ... it seems extravagant. I'm making something simple ... not simple. Plus, controller.action(id) returning builder.create(id) seems like adding a layer for no reason (the controller is responsible for building view models.. right?)
I dunno... I'm thinking more state based testing is necessary, but I'm afraid if I start testing return values then if Method A can get called in 8 different contexts I'm going to have a test explosion with a lot of repetition. I had been using interaction based testing to pass some of those contexts to Method B so that all I have to do is verify Method A called Method B and I have Method B tested so Method A can just trust that those contexts are handled. So interaction based testing is building this hierarchy of tests but state based testing is going to flatten it out some.
I have no idea if that made any sense.
Wow, this is long ...
I think Roy Osherove recently twitted that as a rule of thumb, your tests should be 95 percent state-based and 5 percent interaction-based. I agree.
What matters most is that your API does what you want it to, and that is what you need to test. If you test the mechanics of how it achieves what it needs to do, you are very likely to end up with Overspecified Tests, which will bite you when it comes to maintainability.
In most cases, you can design your API so that state-based testing is the natural choice, because that is just so much easier.
To examine your Upload example: Does it matter that GetPackage and GetAllCertificates was called? Is that really the expected outcome of the Upload method?
I would guess not. My guess is that the purpose of the Upload method - it's very reason for existing - is to populate and serve the correct View.
So state-based testing would examine the returned ViewResult and its ViewModel and verify that it has all the correct values.
Sure, as the code stands right now, you will need to provide Test Doubles for packageRepository and certificateRepository, because otherwise exceptions will be thrown, but it doesn't look like it is important in itself that the repository methods are being called.
If you use Stubs instead of Mocks for your repositories, your tests are no longer tied to internal implementation details. If you later on decide to change the implementation of the Upload method to use cached instances of packages (or whatever), the Stub will not be called, but that's okay because it's not important anyway - what is important is that the returned View contains the expected data.
This is much more preferrable than having the test break even if all the returned data is as it should be.
Interestingly, your Deny example looks like a prime example where interaction-based testing is still warranted, because it is only by examining Indirect Outputs that you can verify that the method performed the correct action (the DenyPackage method returns void).
All this, and more, is explained very well in the excellent book xUnit Test Patterns.
The question to ask is "if this code worked, how could I tell?" That might mean testing some interactions or some state, it depends on what's important.
In your first test, the Deny changes the world outside the target class. It requires a collaboration from a service, so testing an interaction makes sense. In your second test, you're making queries on the neighbours (not changing anything outside the target class), so stubbing them makes more sense.
That's why we have a heuristic of "Stub Queries, Mock Actions" in http://www.mockobjects.com/book
I'm trying to get my head around TDD methodology and have run into - what I think is - a chicken-and-egg problem: what to do if a bug fix involves the changing of a method's signature.
Consider the following method signature:
string RemoveTokenFromString (string delimited, string token)
As the name suggests, this method removes all instances of a token from delimited and returns the resultant string.
I find later that this method has a bug (e.g. the wrong bits are being removed from the string). So I write a test case describing the scenario where the bug occurs and make sure that the test fails.
When fixing the bug, I find that the method needs more information to be able to do its job properly - and this bit of information can only be sent in as a parameter (the method under test is part of a static class).
What do I do then? If I fix the bug, this compels me to change the unit test - would that be 'correct' TDD methodology?
You have fallen into the most dangerous trap in TDD: you think TDD is about testing, but it isn't. However, it is easy to fall into that trap, since all the terminology in TDD is about testing. This is why BDD was invented: it is essentially TDD, but without the confusing terminology.
In TDD, tests aren't really tests, they are examples. And assertions aren't really assertions, they are expectations. And you aren't dealing with units, you are dealing with behaviors. BDD just calls them that. (Note: BDD has evolved since it was first invented, and it now incorporates things that aren't part of TDD, but the original intention was just "many people do TDD wrong, so use different words to help them do it right".)
Anyway, if you think of a test not as a test, but a behavioral example of how the method should work, it should become obvious that as you develop a better understanding of the expected behavior, deleting or changing the test is not only allowed by TDD, it is the only correct choice! Always keep that in mind!
There is absolutely nothing wrong with bombing your tests, when you discover that the intended behaviour of the unit changes.
//Up front
[Test]
public void should_remove_correct_token_from_string()
{
var text = "do.it.correctly..";
var expected = "doitcorrectly";
Assert.AreEqual(StaticClass.RemoveTokenFromString(text, "."), expected);
}
//After finding that it doesn't do the right thing
//Delete the old test and *design* a new function that
//Does what you want through a new test
//Remember TDD is about design, not testing!
[Test]
public void should_remove_correct_token_from_string()
{
var text = "do.it.correctly..";
var expected = "doitcorrectly";
Assert.AreEqual(
StaticClass.RemoveTokenFromString(
text,
".",
System.Text.Encoding.UTF8), expected);
}
//This will force you to add a new parameter to your function
//Obviously now, there are edge cases to deal with your new parameter etc.
//So more test are required to further design your new function
Keep it simple.
If your unit test is wrong, or obsolete, you have to rewrite it. If your specs change, or certain specs are no longer relevant, your unit tests have to reflect that.
Red, green, refactor also applies to your unit tests, not just the code you are testing.
There is a refactoring called Add Parameter that could help here.
If your language supports method overloading, you could create the new function with the new parameter first, copying the body of the existing function and fixing your problem.
Then when the problem is fixed, you can modify all the tests, one by one, to call the new method. Last you can delete the old method.
With a language that does not support method overloading, create a new function with a different name, copy the body on the existing function in that new function, have the existing function calling the new function, possibly with a dummy value for the new parameter. Then you could have all your tests passing. Make your old tests call the new function, one by one. When the old method is not used anymore, it can be deleted and the new function renamed.
This is a bit process-extensive, but I think this is the TDD way to follow red-green-refactor.
Default value for parameter could also help, if there are available.
Red, Green, Refactor.
Whatever you do, you first want to get to a state where you have a compiling but failing test case that reproduces the bug. You then can proceed on adding just the parameter to the test and the implementation, but do nothing with it so you still have Red.
I'd say don't fret about the 'right'/'correct' way... whatever helps you get you closer to the solution quicker.
If you find that you need to take in an extra parameter,
update the call in the test case
add the new parameter to the actual method
verify that your code builds and the test fails again.
proceed with making it green.
Only in cases where adding a new parameter would result in zillions of compile errors, would I recommend - taking it in baby steps... you dont want to update the whole source base before finding out you really didn't need the third param or you need a fourth one.. time lost. So get the new version of the method 'working' before updating all references. (As philippe says here)
write a new overload with the added parameter
Move the code of the old method into the new overload
Make the old overload relay or delegate to the new overload with some default value for the new param
Now you can get back to the task at hand and get the new test to go green.
If you dont need the old overload anymore, delete it and fix the resulting compile errors.
If a method is not doing the job correctly then it needs to be fixed and if the fix requires change in signature then noting wrong in that. As per TDD you write the test case first which will certainly fail and then you write the method to satisfy the test. As per this approach if the method call in test requires a parameter for it to function then you need to do it.
Description for Assert.Equals() from the MSDN Documentation:
Do not use this method.
That's it, the full explanation. Uh.. ok, but then ... why is it there?
Is it a deprecated method from an earlier version of the framework? Something that's supposed to be used only by other Microsoft Assemblies?
It just makes me want to use it all the more knowing I'm not supposed to. ;-)
Does anyone know?
.Equals is inherited from object. It's listed as "Do not use this method" so users won't confuse it with the AreEqual method.
All objects in .NET derive from Object.
Object has a .Equals() method.
Apparently the .Equals() method for this particular object doesn't do anything useful, so the docs are warning you that it doesn't do anything useful.
It was changed in 2008 (Maybe SP1) to fail a test when called, so that people who were using it by accident were told they really shouldn't be using it.
Assert.Equals, just like its based class method Object.Equals, is perfectly useful for comparing objects. However, neither method is useful for stand-alone detection and reporting or errors in unit testing, since Object.Equals returns a boolean rather than throws if the values are not equal. This is a problem if used like this in a unit test:
Assert.Equals(42, ComputeMeaningOfLife());
Aside from the problem of this unit test possibly running too long :-), this test would silently succeed even if the Compute method provides the wrong result. The right method to use is Assert.AreEqual, which doesn't return anything, but throws an exception if the parameters aren't equal.
Assert.Equals was added so code like in the sample above doesn't fall back to Object.Equals and silently neuter the unit test. Instead, when called from a unit test, Assert.Equals always throws an exception reminding you not to use it.