When using Rhino Mocks, when is it appropriate to use "VerifyAll" and when should I do "Asserts"?
VerifyAll and Verify are used to check that some methods have been called (and possibly verify the parameters with which they were called). This is considered "interaction-based testing", and is used when you want to verify that the system-under-test calls a method on one of its dependencies.
Asserts normally means that you want to make sure the value returned has the correct value. Asserts are used for what is called "state-based testing", which is essentially verifying the state of the system-under-test after it has been acted upon.
verifyall, check out this.
Also, differentiate Mock and Stub .
I believe VerifyAll belongs to the older style of using RhinoMocks, where you would have a record step and a playback step, after which you would verify all Expectations. In this model you would during the record step set up an expectation (eg, Expect that this method will be called with parameters x, y, and z, etc).
The newer versions of RhinoMocks introduce Arrange-Act-Assert (AAA) syntax as the preferred pattern; Using this pattern, it makes more sense to use Assertions at the end of your test method. It is still possible to use VerifyAllExpectations(), but personally I think it reads easier if all of your Assertions happen in a block at the end of the test.
So I guess the answer (to me anyway) is that it is personal preference; See the link above where he has several examples of the same test and choose the one that reads best to you.
Related
I'm quite new to unit testing. I've read around on here as well as done some Googling, but am still a bit confused as to the meaning of each of the four meanings. I came across the following definition which help....
Stub - stubs are minimal implementations of interfaces or base classes
Spy - a spy will record which members were invoked
Fake - more complex, a fake may resemble a production implementation
Mock - A mock is usually dynamically created by a mock library and depending on its configuration, a mock can behave like a dummy, a stub, or a spy
However, I'd like to simplify the meaning (if possible) and ask a few questions.
Do all of the above only relate to functions, or can they be objects or any other type?
Stub - Is Stubbing bascially a way to provide dummy info instead of making the calling to the actual database? So for example, if I had an API call, instead of actually making a call to the API, I just instead make a GET request to a JSON file which is in my tests folder which has some dummy data in, and use that instead of making the API call?
Spy - so is this a way of tracking what happens to a function for example. Meaning you follow when it's called, where it gets passed around to?
Fake - Is this for example a function which you create inside the test file to mimic the real function or be a simpified version of the actual function?
Thanks in advance.
There are multiple attempts at definitions. To my knowledge there is no fully consistent definition, probably due to the fact that mocking frameworks defines things slightly differently. Martin Fowler lists the following (https://martinfowler.com/bliki/TestDouble.html):
Dummy objects are passed around but never actually used. Usually they are just used to fill parameter lists.
Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an InMemoryTestDatabase is a good example).
Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what's programmed in for the test.
Spies are stubs that also record some information based on how they were called. One form of this might be an email service that records how many messages it was sent.
Mocks are pre-programmed with expectations which form a specification of the calls they are expected to receive. They can throw an exception if they receive a call they don't expect and are checked during verification to ensure they got all the calls they were expecting.
Fowler has them from this page: http://xunitpatterns.com/Test%20Double.html where you can read about them in a bit more detail.
No, they apply to more then just functions
Yes
Kind of, in most cases you will spy on an object
Yes
The implementation and nomenclatur will depend of what framework you use.
I know that many mocking libraries let the programmer check that a method was called only once. But why is that useful?
Also, why is it useful to verify the parameters to a mock's method?
When doing unit tests you are testing a isolated method: all other methods are supposed to work correctly, and you test only that your method behaves in the expected (specified...) way.
But in many occasions the expected way implies calling methods of classes you depend on (via dependency injection, if you want to do unit testing).
For these reason you need to assure that this calls are really done... and of course they are called with the expected parameters.
Example:
In your real application you have a repository class that stores all your changes in the database (and does only this!). But for unit test your "business" class (where all your business rules are defined), you should mock that "repository" class: then you must check that this mocked class receives the correct updated data.
In general, the answers to both of those questions are the same.
It's useful if the requirements of the unit/method you're testing specify that that behavior is required. If that behavior is required, then that's what you need to verify is actually happening.
If it's important to ensure that a particular method is only called once, then you can do that. If it doesn't matter that a method is called more than once, then don't test for it explicitly. Note that the default for the Mockito "verify" method is "times(1)", which means that it confirms that the method was called once and only once.
Concerning the second question, you might want to verify the parameters if it's entirely possible the method could be called with different parameters, and you don't want to count those occurrences, you only care about a specific set of parameter values.
I have started using moq for mocking. Can someone explain me the concept of strict and non-strict mocks? How can they can be used in moq?
edit:
in which scenario do we use which type of mock?
I'm not sure about moq specifically, but here's how strict mocks work in Rhino. I declare that I expect a call to foo.Bar on my object foo:
foo.Expect(f => f.Bar()).Returns(5);
If the calling code does
foo.Bar();
then I'm fine because the expectations are exactly met.
However, if the calling code is:
foo.Quux(12);
foo.Bar();
then my expectation failed because I did not explicitly expect a call to foo.Quux.
To summarize, a strict mock will fail immediately if anything differs from the expectations. On the other hand, a non-strict mock (or a stub) will gladly "ignore" the call to foo.Quux and it should return a default(T) for the return type T of foo.Quux.
The creator of Rhino recommends that you avoid strict mocks (and prefer stubs) because you generally don't want your test to fail when receiving an unexpected call as above. It makes refactoring your code much more difficult when you have to fix dozens of test that relied on the exact original behavior.
Ever come across Given / When / Then?
Given a context
When I perform some events
Then an outcome should occur
This pattern appears in BDD's scenarios, and is also relevant for unit tests.
If you're setting up context, you're going to use the information which that context provides. For instance, if you're looking up something by Id, that's context. If it doesn't exist, the test won't run. In this case, you want to use a NiceMock or a Stub or whatever - Moq's default way of running.
If you want to verify an outcome, you can use Moq's verify. In this case, you want to record the relevant interactions. Fortunately, this is also Moq's default way of running. It won't complain if something happens that you weren't interested in for that test.
StrictMock is there for when you want no unexpected interactions to occur. It's how old-style mocking frameworks used to run. If you're doing BDD-style examples, you probably won't want this. It has a tendency to make tests a bit brittle and harder to read than if you separate the aspects of behaviour you're interested in. You have to set up expectations for both the context and the outcome, for all outcomes which will occur, regardless of whether they're of interest or not.
For instance, if you're testing a controller and mocking out both your validator and your repository, and you want to verify that you've saved your object, with a strict mock you also have to verify that you've validated the object first. I prefer to see those two aspects of behaviour in separate examples, because it makes it easier for me to understand the value and behaviour of the controller.
In the last four years I haven't found a single example which required the use of a strict mock - either it was an outcome I wanted to verify (even if I verify the number of times it's called) or a context for which I can tell if I respond correctly to the information provided. So in answer to your question:
non-strict mock: usually
strict mock: preferably never
NB: I am strongly biased towards BDD, so hard-core TDDers may disagree with me, and it will be right for the way that they are working.
Here's a good article.
I usually end up having something like this
public class TestThis {
private final Collaborator1 collaborator1;
private final Collaborator2 collaborator2;
private final Collaborator2 collaborator3;
TestThis(Collaborator1 collaborator1, Collaborator2 collaborator2, Collaborator3 collaborator3) {
this.collaborator1 = collaborator1;
this.collaborator2 = collaborator2;
this.collaborator3 = collaborator3;
}
public Login login(String username) {
User user = collaborator1.getUser(username);
collaborator2.notify(user);
return collaborator3.login(user);
}
}
...and I use Strict mocks for the 3 collaborators to test login(username). I don't see how Strict Mocks should never be used.
I have a simple convention:
Use strict mocks when the system under test (SUT) is delegating the call to the underlying mocked layer without really modifying or applying any business logic to the arguments passed to itself.
Use loose mocks when the SUT applies business logic to the arguments passed to itself and passes on some derived/modified values to the mocked layer.
For eg:
Lets say we have database provider StudentDAL which has two methods:
Data access interface looks something like below:
public Student GetStudentById(int id);
public IList<Student> GetStudents(int ageFilter, int classId);
The implementation which consumes this DAL looks like below:
public Student FindStudent(int id)
{
//StudentDAL dependency injected
return StudentDAL.GetStudentById(id);
//Use strict mock to test this
}
public IList<Student> GetStudentsForClass(StudentListRequest studentListRequest)
{
//StudentDAL dependency injected
//age filter is derived from the request and then passed on to the underlying layer
int ageFilter = DateTime.Now.Year - studentListRequest.DateOfBirthFilter.Year;
return StudentDAL.GetStudents(ageFilter , studentListRequest.ClassId)
//Use loose mock and use verify api of MOQ to make sure that the age filter is correctly passed on.
}
If a function just calls another function or performs actions. How do I test it? Currently, I enforce all the functions should return a value so that I could assert the function return values. However, I think this approach mass up the API because in the production code. I don't need those functions to return value. Any good solutions?
I think mock object might be a possible solution. I want to know when should I use assert and when should I use mock objects? Is there any general guide line?
Thank you
Let's use BufferedStream.Flush() as an example method that doesn't return anything; how would we test this method if we had written it ourselves?
There is always some observable effect, otherwise the method would not exist. So the answer can be to test for the effect:
[Test]
public void FlushWritesToUnderlyingStream()
{
var memory = new byte[10];
var memoryStream = new MemoryStream(memory);
var buffered = new BufferedStream(memoryStream);
buffered.Write(0xFF);
Assert.AreEqual(0x00, memory[0]); // not yet flushed, memory unchanged
buffered.Flush();
Assert.AreEqual(0xFF, memory[0]); // now it has changed
}
The trick is to structure your code so that these effects aren't too hard to observe in a test:
explicitly pass collaborator objects,
just like how the memoryStream is passed
to the BufferedStream in the constructor.
This is called dependency
injection.
program against an interface, just
like how BufferedStream is programmed
against the Stream interface. This enables
you to pass simpler, test-friendly implementations (like MemoryStream in this case) or use a mocking framework (like MoQ or RhinoMocks), which is all great for unit testing.
Sorry for not answering straight but ... are you sure you have the exact balance in your testing?
I wonder if you are not testing too much ?
Do you really need to test a function that merely delegates to another?
Returns only for the tests
I agree with you when you write you don't want to add return values that are useful only for the tests, not for production. This clutters your API, making it less clear, which is a huge cost in the end.
Also, your return value could seem correct to the test, but nothing says that the implementation is returning the return value that corresponds to the implementation, so the test is probably not proving anything anyway...
Costs
Note that testing has an initial cost, the cost of writing the test.
If the implementation is very easy, the risk of failure is ridiculously low, but the time spend testing still accumulates (over hundred or thousands cases, it ends up being pretty serious).
But more than that, each time you refactor your production code, you will probably have to refactor your tests also. So the maintenance cost of your tests will be high.
Testing the implementation
Testing what a method does (what other methods it calls, etc) is critized, just like testing a private method... There are several points made:
this is fragile and costly : any code refactoring will break the tests, so this increases the maintenance cost
Testing a private method does not bring much safety to your production code, because your production code is not making that call. It's like verifying something you won't actually need.
When a code delegates effectively to another, the implementation is so simple that the risk of mistakes is very low, and the code almost never changes, so what works once (when you write it) will never break...
Yes, mock is generally the way to go, if you want to test that a certain function is called and that certain parameters are passed in.
Here's how to do it in Typemock (C#):
Isolate.Verify.WasCalledWithAnyArguments(()=> myInstance.WeatherService("","", null,0));
Isolate.Verify.WasCalledWithExactArguments(()=> myInstance. StockQuote("","", null,0));
In general, you should use Assert as much as possible, until when you can't have it ( For example, when you have to test whether you call an external Web service API properly, in this case you can't/ don't want to communicate with the web service directly). In this case you use mock to verify that a certain web service method is correctly called with correct parameters.
"I want to know when should I use assert and when should I use mock objects? Is there any general guide line?"
There's an absolute, fixed and important rule.
Your tests must contain assert. The presence of assert is what you use to see if the test passed or failed. A test is a method that calls the "component under test" (a function, an object, whatever) in a specific fixture, and makes specific assertions about the component's behavior.
A test asserts something about the component being tested. Every test must have an assert, or it isn't a test. If it doesn't have assert, it's not clear what you're doing.
A mock is a replacement for a component to simplify the test configuration. It is a "mock" or "imitation" or "false" component that replaces a real component. You use mocks to replace something and simplify your testing.
Let's say you're going to test function a. And function a calls function b.
The tests for function a must have an assert (or it's not a test).
The tests for a may need a mock for function b. To isolate the two functions, you test a with a mock for function b.
The tests for function b must have an assert (or it's not a test).
The tests for b may not need anything mocked. Or, perhaps b makes an OS API call. This may need to be mocked. Or perhaps b writes to a file. This may need to be mocked.
I think understand the definition of State / Interaction based testing (read the Fowler thing, etc). I found that I started state based but have been doing more interaction based and I'm getting a bit confused on how to test certain things.
I have a controller in MVC and an action calls a service to deny a package:
public ActionResult Deny(int id)
{
service.DenyPackage(id);
return RedirectToAction("List");
}
This seems clear to me. Provide a mock service, verify it was called correctly, done.
Now, I have an action for a view that lets the user associate a certificate with a package:
public ActionResult Upload(int id)
{
var package = packageRepository.GetPackage(id);
var certificates = certificateRepository.GetAllCertificates();
var view = new PackageUploadViewModel(package, certificates);
return View(view);
}
This one I'm a bit stumped on. I'm doing Spec style tests (possibly incorrectly) so to test this method I have a class and then two tests: verify the package repository was called, verify the certificate repository was called. I actually want a third to test to verify that the constructor was called but have no idea how to do that! I'm get the impression this is completely wrong.
So for state based testing I would pass in the id and then test the ActionResult's view. Okay, that makes sense. But wouldn't I have a test on the PackageUploadViewModel constructor? So if I have a test on the constructor, then part of me would just want to verify that I call the constructor and that the action return matches what the constructor returns.
Now, another option I can think of is I have a PackageUploadViewModelBuilder (or something equally dumbly named) that has dependency on the two repositories and then I just pass the id to a CreateViewModel method or something. I could then mock this object, verify everything, and be happy. But ... well ... it seems extravagant. I'm making something simple ... not simple. Plus, controller.action(id) returning builder.create(id) seems like adding a layer for no reason (the controller is responsible for building view models.. right?)
I dunno... I'm thinking more state based testing is necessary, but I'm afraid if I start testing return values then if Method A can get called in 8 different contexts I'm going to have a test explosion with a lot of repetition. I had been using interaction based testing to pass some of those contexts to Method B so that all I have to do is verify Method A called Method B and I have Method B tested so Method A can just trust that those contexts are handled. So interaction based testing is building this hierarchy of tests but state based testing is going to flatten it out some.
I have no idea if that made any sense.
Wow, this is long ...
I think Roy Osherove recently twitted that as a rule of thumb, your tests should be 95 percent state-based and 5 percent interaction-based. I agree.
What matters most is that your API does what you want it to, and that is what you need to test. If you test the mechanics of how it achieves what it needs to do, you are very likely to end up with Overspecified Tests, which will bite you when it comes to maintainability.
In most cases, you can design your API so that state-based testing is the natural choice, because that is just so much easier.
To examine your Upload example: Does it matter that GetPackage and GetAllCertificates was called? Is that really the expected outcome of the Upload method?
I would guess not. My guess is that the purpose of the Upload method - it's very reason for existing - is to populate and serve the correct View.
So state-based testing would examine the returned ViewResult and its ViewModel and verify that it has all the correct values.
Sure, as the code stands right now, you will need to provide Test Doubles for packageRepository and certificateRepository, because otherwise exceptions will be thrown, but it doesn't look like it is important in itself that the repository methods are being called.
If you use Stubs instead of Mocks for your repositories, your tests are no longer tied to internal implementation details. If you later on decide to change the implementation of the Upload method to use cached instances of packages (or whatever), the Stub will not be called, but that's okay because it's not important anyway - what is important is that the returned View contains the expected data.
This is much more preferrable than having the test break even if all the returned data is as it should be.
Interestingly, your Deny example looks like a prime example where interaction-based testing is still warranted, because it is only by examining Indirect Outputs that you can verify that the method performed the correct action (the DenyPackage method returns void).
All this, and more, is explained very well in the excellent book xUnit Test Patterns.
The question to ask is "if this code worked, how could I tell?" That might mean testing some interactions or some state, it depends on what's important.
In your first test, the Deny changes the world outside the target class. It requires a collaboration from a service, so testing an interaction makes sense. In your second test, you're making queries on the neighbours (not changing anything outside the target class), so stubbing them makes more sense.
That's why we have a heuristic of "Stub Queries, Mock Actions" in http://www.mockobjects.com/book