After reading an interesting article about unit testing behavior instead of state, I came to realize that my unit tests often are tightly coupled to my code because I am using mocks.
I cannot image writing unit tests without mocks but the fact is that these mocks are coupling my unit test very much to my code because of the expect andReturn calls.
For example when I create a test that uses a mock, I record all calls to the specific mock and assign return values.
Now when I change the implementation of the actual code for whatever reason, a lot of tests break because that call was not expected by the mock, forcing me to update the unit test also, and effectively forcing me to implement every change twice...
This happens a lot.
Is this issue intrinsic to using mocks, and should I learn to live with it, or am I doing something fundamentally wrong?
Please enlighten me :)
Clear examples coming with the explanation are most welcome of course.
when I create a test that uses a mock,
I record all calls to the specific
mock and assign return values
It sounds like you may be over-specifying expectations.
Try to build as little setup code as possible into your tests: stub (rather than expect) all behavior that doesn't pertain to the current test and only specify return values that are absolutely needed to make your test work.
This answer includes a concise example (as well as an alternative, more detailed explanation).
My experience is to use mocks only at the bounderies of (sub)systems. If I have two classes that are strongly related I do not mock them appart but test them together. An example might be an composite and a visitor. If I test a concrete visitor I do not use a mock for the composite but create real composites. One might argue that this is not a unit test (depends on the definition of what is a unit). But that doesn't matter that much. What I try to achieve is:
Write readable tests (tests without mocks are most of the time more easy to read).
Test only a focused area of code (in the example the concreate visitor and the relevant part of the composite).
Write fast tests (as long as I instantiate only a few classes, in the example the concrete composites, this is not a concern ... watch for transitive creations).
Only if I encounter the boundary of a subsystem I use mocks. Example: I have a composite that can render itself to a renderer I would mock out the renderer if I test the render logic of the composite.
Testing behavior instead of state looks promosing at first but in general I would test state as the resulting tests are easiear to maintain. Mocks are a cannon. Don't crack a nut with a sledgehammer.
If you are fixing the tests because they break, you are not using them as intended.
If the behaviour of a method changes, in test driven development you would first change the test to expect the new behaviour, then implement the new behaviour.
Several good answers here already, but for me a good rule of thumb is to test the requirements of the method, not the implementation. Sometimes that may mean using a mock object because the interaction is the requirement, but you're usually better off testing the return value of the method or the change in state of the object.
Related
I was assigned a task regarding writing unit tests for some already-implemented classes.
Currently I am not really understand the purposes of each class. My concern is that whether I should (1) deeply dive into implementation, or (2) just simply see input - expected output of each class (as well as its methods)?
IMO:
(1) has an advantage that I can see some potential bugs so that I can design unit tests that can cover those cases.
In the other hand, (1) has disadvantages that are (a) I can get biased by the implementation and (b): takes time.
Actually, you do both.
First, and foremost, you focus on the public contract of the classes under test. So yes, looking at input/output should be your "first stop".
You see - ideally, your unit test do not need any kind of mocking. You create some instance underTest; you call a method; you observe/verify some behavior (for example asserting an expected against the actual return value).
But of course, sometimes it is worth looking into the details of the implementation.
So, a reasonable procedure is:
write "black box" tests that don't know about implementation details (where possible)
use coverage to understand how "good" your tests are covering the class under test
From the coverage numbers you can draw further conclusions, like:
your black box tests aren't sufficient, and you need to add certain "white box" tests in order to get into specific methods. Meaning: you look into the class under test to understand what it is doing; and how to get into important corners.
your tests are actually great - but there is unused code in those classes under test. Maybe that can then be deleted (deleting source code is the second best thing you can do as software engineer!)
Finally: it is more of an anti pattern to create unit tests that (more or less) re-program the implementation. You really want to avoid that your unit tests do nothing else but configure mocks for those calls that your production code is making. Thing is: when you test implementation details, then any implementation change (like a simple refactoring/reordering) can break your unit tests.
I'm still getting to grips with the whole TDD concept. Should I be writing tests for properties? Should one only write tests on properties containing sufficient logic? Any thoughts or examples on this would be great.
Writing a test for public getter of a private field will not give you much if this getter does nothing except returning your private field. But if it does contain some logic (or just something that can fail, like converting your private Int32 field to Byte), testing such property starts to make sense.
Test things that have a reasonable chance of failure - you gain no extra confidence by testing properties with no logic beyond get/set.
Simple rule of thumb I use when doing TDD: always write tests that fail.
If a test fails at first it's a good test for TDD. It means something is not yet implemented, or not implemented as it should be. Then you can change code to make it pass. A test that succeeds at first is a bad test. You don't even know if it succeeded because you made some mistake writing the test, or because what you are testing is already working.
If you are able to produce a test failure using properties, then write tests for those properties. Typically you should begin by writing a test testing a setter or a getter before implementing it. A not implemented setter or getter can look trivial but makes test fail. And why would you write any line of code even a setter or getter if it's not driven by a test failure ?
Other kinds of tests, like tests as documentation showing how to use an API are also very useful, and good agile practice, but that is not TDD. TDD is about trying actively to break the code until you can't any more. Then you run functional tests and if you pushed unit tests hard enough, and if there is not some integration or system problem going into the way, all should be fine.
I usually write junit tests for accessors. It doesn't add much when they're written, except to keep coverage statistics pretty. But if someone adds "sufficient logic" later to the production code, the tests will already be in place to catch any mistakes.
Also it is the work of moments to write a test to check the value returned by a getter.
Instead of thinking of them as tests, think of each test as an example of how someone could use your code.
Instead of just testing properties, think about the behaviour which changes when the properties have different values, and give an example of the behaviour of the class in each meaningful context.
If it's really just a data property you can test by inspection, or with automated acceptance tests, or manually, maybe with a tester's help. Otherwise, don't worry about testing each method, or each property - just show how you can use the code and how you expect it to behave.
Think of TDD and Unit testes as a way to "see ahead", you write the test in a way that you feel would be best for your public signature, of your classes, to be.
In the article Test for Required Behavior, not Incidental Behavior, Kevlin Henney advises us that:
"[...] a common pitfall in testing is to hardwire tests to the specifics of an implementation, where those specifics are incidental and have no bearing on the desired functionality."
However, when using TDD, I often end up writing tests for incidental behaviour. What do I do with these tests? Throwing them away seems wrong, but the advice in the article is that these tests can reduce agility.
What about separating them into a separate test suite? That sounds like a start, but seems impractical intuitively. Does anyone do this?
In my experience implementation-dependent tests are brittle and will fail massively at the very first refactoring. What I try to do is focus on deriving a proper interface for a class while writing the tests, effectively avoiding such implementation details in the interface. This not only solves the brittle tests, but it also promotes cleaner design.
This still allows for extra tests that check for the risky parts of my selected implementation, but only as extra protection to a good coverage of the "normal" interface of my class.
For me the big paradigma shift came when I started writing tests before even thinking about the implementation. My initial surprise was that it became much easier to generate "extreme" test cases. Then I recognized the improved interface in turn helped shape the implementation behind it. The result is that my code nowadays doesn't do much more than the interface exposes, effectively reducing the need for most "implementation" tests.
During refactoring of the internals of a class, all tests will hold. Only in cases where the exposed interface changes, the test set may need to be extended or modified.
The problem you describe is very real and very easy to encounter when TDD'ing. In general you can say that it isn't testing incidental behavior itself which is a problem, but rather if tons of tests depend on that incidental behavior.
The DRY principle applies to test code as well as to production code. That can often be a good guideline when writing test code. The goal should be that all the 'incidental' behavior you specify along the way is isolated so that only a few tests out of the entire test suite use them. In that way, if you need to refactor that behavior, you only need to modify a few tests instead of a large fraction of the entire test suite.
This is best achieved by copious use of interfaces or abstract classes as collaborators, because this means that you get low class coupling.
Here's an example of what I mean. Assume that you have some kind of MVC implementation where a Controller should return a View. Assume that we have a method like this on a BookController:
public View DisplayBookDetails(int bookId)
The implementation should use an injected IBookRepository to get the book from the database and then convert that to a View of that book. You could write a lot of tests to cover all aspects of the DisplayBookDetails method, but you could also do something else:
Define an additional IBookMapper interface and inject that into the BookController in addition to the IBookRepository. The implementation of the method could then be something like this:
public View DisplayBookDetails(int bookId)
{
return this.mapper.Map(this.repository.GetBook(bookId);
}
Obviously this is a too simplistic example, but the point is that now you can write one set of tests for your real IBookMapper implementation, which means that when you test the DisplayBookDetails method, you can just use a Stub (best generated by a dynamic mock framework) to implement the mapping, instead of trying to define a brittle and complex relationship between a Book Domain object and how it is mapped.
The use of an IBookMaper is definitely an incidental implementation detail, but if you use a SUT Factory or better yet an auto-mocking container, the definition of that incidental behavior is isolated which means that you if later on you decide to refactor the implementation, you can do that by only changing the test code in a few places.
"What about separating them into a separate test suite?"
What would you do with that separate suite?
Here's the typical use case.
You wrote some tests which test implementation details they shouldn't have tested.
You factor those tests out of the main suite into a separate suite.
Someone changes the implementation.
Your implementation suite now fails (as it should).
What now?
Fix the implementation tests? I think not. The point was to not test an implementation because it leads to way to much maintenance work.
Have tests that can fail, but the overall unittest run is still considered good? If the tests fail, but the failure doesn't matter, what does that even mean? [Read this question for an example: Non-critical unittest failures An ignored or irrelevant test is just costly.
You have to discard them.
Save yourself some time and aggravation by discarding them now, not when they fail.
I you really do TDD the problem is not so big as it may seem at once because you are writing tests before code. You should not even think about any possible implementation before writing test.
Such problems of testing incidental behavior is much more common when you write tests after implementation code. Then the easy way is just checking that the function output is OK and does what you want, then writing test using that output. Really that's cheating, not TDD, and the cost of cheating is tests that will break if implementation change.
The good thing is that such tests will break yet more easily than good tests (good test meaning here tests depending only of the wanted feature, not implementation dependent). Having tests so generic they never break is quite worse.
Where I work what we do is simply fix such tests when we stumble upon them. How we fix them depends on the kind of incidental test performed.
the most common such test is probably the case where testing results occurs in some definite order overlooking this order is really not guaranteed. The easy fix is simple enough: sort both result and expected result. For more complex structures use some comparator that ignore that kind of differences.
every so often we test innermost function, while it's some outer most function that perform the feature. That's bad because refactoring away the innermost function becomes difficult. The solution is to write another test covering the same feature range at outermost function level, then remove the old test, and only then we can refactor the code.
when such test break and we see an easy way to make them implementation independant we do it. Yet, if it's not easy we may choose to fix them to still be implementation dependant but depending on the new implementation. Tests will break again at the next implementation change, but it's not necessarily a big deal. If it's a big deal then definitely throw away that test and find another one to cover that feature, or change the code to make it easier to test.
another bad case is when we have written tests using some Mocked object (used as stub) and then the mocked object behavior change (API Change). This one is bad because it does not break code when it should because changing the mocked object behavior won't change the Mock mimicking it. The fix here is to use the real object instead of the mock if possible, or fix the Mock for new behavior. In that case both the Mock behavior and the real object behavior are both incidental, but we believe tests that does not fail when they should are a bigger problem than tests breaking when they shouldn't. (Admitedly such cases can also be taken care of at integration tests level).
This question already has answers here:
When should I mock?
(4 answers)
Closed 9 years ago.
Classes that use other classes (as members, or as arguments to methods) need instances that behave properly for unit test. If you have these classes available and they introduce no additional dependencies, isn't it better to use the real thing instead of a mock?
I say use real classes whenever you can.
I'm a big believer in expanding the boundaries of "unit" tests as much as possible. At this point they aren't really unit tests in the traditional sense, but rather just an automated regression suite for your application. I still practice TDD and write all my tests first, but my tests are a little bigger than most people's and my green-red-green cycles take a little longer. But now that I've been doing this for a little while I'm completely convinced that unit tests in the traditional sense aren't all they're cracked up to be.
In my experience writing a bunch of tiny unit tests ends up being an impediment to refactoring in the future. If I have a class A that uses B and I unit test it by mocking out B, when I decide to move some functionality from A to B or vice versa all of my tests and mocks have to change. Now if I have tests that verify that the end to end flow through the system works as expected then my tests actually help me to identify places where my refactorings might have caused a change in the external behavior of the system.
The bottom line is that mocks codify the contract of a particular class and often end up actually specifying some of the implementation details too. If you use mocks extensively throughout your test suite your code base ends up with a lot of extra inertia that will resist any future refactoring efforts.
It is fine to use the "real thing" as long as you have absolute control over the object. For example if you have an object that just has properties and accessors you're probably fine. If there is logic in the object you want to use, you could run into problems.
If a unit test for class a uses an instance of class b and an change introduced to b breaks b, then the tests for class a are also broken. This is where you can run into problems where as with a mock object you could always return the correct value. Using "the real thing" Can kind of convolute tests and hide the real problem.
Mocks can have downsides too, I think there is a balance with some mocks and some real objects you will have to find for yourself.
There is one really good reason why you want to use stubs/mocks instead of real classes. I.e. to make your unit test's (pure unit test) class under test isolated from everything else. This property is extremely useful and the benefits for keeping tests isolated are plentiful:
Tests run faster because they don't need to call the real class implementation. If the implementation is to run against file system or relational database then the tests will become sluggish. Slow tests make developers not run unit tests as often. If you're doing Test Driven Development then time hogging tests are together a devastating waste of developers time.
It will be easier to track down problems if the test is isolated to the class under test. In contrast to a system test it will be much more difficult to track down nasty bugs that are not apparently visible in stack traces or what not.
Tests are less fragile on changes done on external classes/interfaces because you're purely testing the class that is under test. Low fragility is also an indication of low coupling, which is a good software engineering.
You're testing against external behaviour of a class rather than the internal implementation which is more useful when deciding code design.
Now if you want to use real class in your test, that's fine but then it is NOT a unit test. You're doing a integration test instead, which is useful for the purpose of validating requirements and overall sanity check. Integration tests are not run as often as unit tests, in practice it is mostly done before committing to favorite code repository, but is equally important.
The only thing you need to have in mind is the following:
Mocks and stubs are for unit tests.
Real classes are for integration/system tests.
Extracted and extended from an answer of mine How do I unit-test inheriting objects?">here:
You should always use real objects where possible.
You should only use mock objects if the real objects do something you dont want to set up (like use sockets, serial ports, get user input, retrieve bulky data etc). Essentially, mock objects are for when the estimated effort to implement and maintain a test using a real object is greater than that to implement and maintain a test using a mock object.
I dont buy into the "dependant test failure" argument. If a test fails because a depended-on class broke, the test did exactly what it should have done. This is not a smell! If a depended-on interface changes, I want to know!
Highly mocked testing environments are very high-maintenance, particularly early in a project when interfaces are in flux. Ive always found it better to start integration testing ASAP.
I always use a mock version of a dependency if the dependency accesses an external system like a database or web service.
If that isn't the case, then it depends on the complexity of the two objects. Testing the object under test with the real dependency is essentially multiplying the two sets of complexities. Mocking out the dependency lets me isolate the object under test. If either object is reasonably simple, then the combined complexity is still workable and I don't need a mock version.
As others have said, defining an interface on the dependency and injecting it into the object under test makes it much easier to mock out.
Personally, I'm undecided about whether it's worth it to use strict mocks and validate every call to the dependency. I usually do, but it's mostly habit.
You may also find these related questions helpful:
What is object mocking and when do I need it?
When should I mock?
How are mocks meant to be used?
And perhaps even, Is it just me, or are interfaces overused?
Use the real thing only if it has been unit tested itself first. If it introduces dependencies that prevent that (circular dependencies or if it requires certain other measures to be in place first) then use a 'mock' class (typically referred to as a "stub" object).
If your 'real things' are simply value objects like JavaBeans then thats fine.
For anything more complex I would worry as mocks generated from mocking frameworks can be given precise expectations about how they will be used e.g. the number of methods called, the precise sequence and the parameters expected each time. Your real objects cannot do this for you so you risk losing depth in your tests.
I've been very leery of mocked objects since I've been bitten by them a number of times. They're great when you want isolated unit tests, but they have a couple of issues. The major issue is that if the Order class needs a a collection of OrderItem objects and you mock them, it's almost impossible to verify that the behavior of of the mocked OrderItem class matches the real-world example (duplicating the methods with appropriate signatures is generally not enough). More than once I've seen systems fail because the mocked classes don't match the real ones and there weren't enough integration tests in place to catch the edge cases.
I generally program in dynamic languages and I prefer merely overriding the specific methods which are problematic. Unfortunately, this is sometimes hard to do in static languages. The downside of this approach is that you're using integration tests rather than unit tests and bugs are sometimes harder to track down. The upside is that you're using the actual code that is written, rather than a mocked version of that code.
If you don't care for verifying expectations on how your UnitUnderTest should interact with the Thing, and interactions with the RealThing have no other side-effects (or you can mock these away) then it is in my opinion perfectly fine to just let your UnitUnderTest use the RealThing.
That the test then covers more of your code base is a bonus.
I generally find it is easy to tell when I should use a ThingMock instead of a RealThing:
When I want to verify expectations in the interaction with the Thing.
When using the RealThing would bring unwanted side-effects.
Or when the RealThing is simply too hard/troublesome to use in a test setting.
If you write your code in terms of interfaces, then unit testing becomes a joy because you can simply inject a fake version of any class into the class you are testing.
For example, if your database server is down for whatever reason, you can still conduct unit testing by writing a fake data access class that contains some cooked data stored in memory in a hash map or something.
It depends on your coding style, what you are doing, your experience and other things.
Given all that, there's nothing stopping you from using both.
I know I use the term unit test way too often. Much of what I do might be better called integration test, but better still is to just think of it as testing.
So I suggest using all the testing techniques where they fit. The overall aim being to test well, take little time doing it and personally have a solid feeling that it's right.
Having said that, depending on how you program, you might want to consider using techniques (like interfaces) that make mocking less intrusive a bit more often. But don't use Interfaces and injection where it's wrong. Also if the mock needs to be fairly complex there is probably less reason to use it. (You can see a lot of good guidance, in the answers here, to what fits when.)
Put another way: No answer works always. Keep your wits about you, observe what works what doesn't and why.
A while ago I read the Mocks Aren't Stubs article by Martin Fowler and I must admit I'm a bit scared of external dependencies with regards to added complexity so I would like to ask:
What is the best method to use when unit testing?
Is it better to always use a mock framework to automatically mock the dependencies of the method being tested, or would you prefer to use simpler mechanisms like for instance test stubs?
As the mantra goes 'Go with the simplest thing that can possibly work.'
If fake classes can get the job done, go with them.
If you need an interface with multiple methods to be mocked, go with a mock framework.
Avoid using mocks always because they make tests brittle. Your tests now have intricate knowledge of the methods called by the implementation, if the mocked interface or your implementation changes... your tests break. This is bad coz you'll spend additional time getting your tests to run instead of just getting your SUT to run. Tests should not be inappropriately intimate with the implementation.
So use your best judgment.. I prefer mocks when it'll help save me writing-updating a fake class with n>>3 methods.
Update Epilogue/Deliberation:
(Thanks to Toran Billups for example of a mockist test. See below)
Hi Doug, Well I think we've transcended into another holy war - Classic TDDers vs Mockist TDDers. I think I'm belong to the former.
If I am on test#101 Test_ExportProductList and I find I need to add a new param to IProductService.GetProducts(). I do that get this test green. I use a refactoring tool to update all other references. Now I find all the mockist tests calling this member now blow up. Then I have to go back and update all these tests - a waste of time. Why did ShouldPopulateProductsListOnViewLoadWhenPostBackIsFalse fail? Was it because the code is broken? Rather the tests are broken. I favor the one test failure = 1 place to fix. Mocking freq goes against that. Would stubs be better? If it I had a fake_class.GetProducts().. sure One place to change instead of shotgun surgery over multiple Expect calls. In the end it's a matter of style.. if you had a common utility method MockHelper.SetupExpectForGetProducts() - that'd also suffice.. but you'll see that this is uncommon.
If you place a white strip on the test name, the test is hard to read. Lot of plumbing code for the mock framework hides the actual test being performed.
requires you to learn this particular flavor of a mocking framework
I generally prefer to use mocks because of Expectations. When you call a method on a stub that returns a value, it typically just gives you back a value. But when you call a method on a mock, not only does it return a value, it also enforces the expectation that you set up that the method was even called in the first place. In other words, if you set up an expectation and then don't call that method, an exception gets thrown. When you set an expectation, you are essentially saying "If this method doesn't get called, something went wrong." And the opposite is true, if you call a method on a mock and did not set an expectation, it will throw an exception, essentially saying "Hey, what are you doing calling this method when you didn't expect it."
Sometimes you don't want expectations on every method you're calling, so some mocking frameworks will allow "partial" mocks that are like mock/stub hybrids, in that only the expectations you set are enforced, and every other method call is treated more like a stub in that it just returns a value.
One valid place to use stubs I can think of off the top, though, is when you are introducing testing into legacy code. Sometimes it's just easier to make a stub by subclassing the class you are testing than refactoring everything to make mocking easy or even possible.
And to this...
Avoid using mocks always because they make tests brittle. Your tests now have intricate knowledge of the methods called by the implementation, if the mocked interface changes... your tests break. So use your best judgment..<
...I say if my interface changes, my tests had better break. Because the whole point of unit tests is that they accurately test my code as it exists right now.
It's best to use a combination, and you'll have to use your own judgement. Here's the guidelines I use:
If making a call to external code is part of your code's expected (outward-facing) behavior, this should be tested. Use a mock.
If the call is really an implementation detail which the outside world doesn't care about, prefer stubs. However:
If you're worried that later implementations of the tested code might accidentally go around your stubs, and you want to notice if that happens, use a mock. You're coupling your test to your code, but it's for the sake of noticing that your stub is no longer sufficient and your test needs re-working.
The second kind of mock is a sort of necessary evil. Really what's going on here is that whether you use a stub or a mock, in some cases you have to couple to your code more than you'd like. When that happens, it's better to use a mock than a stub only because you'll know when that coupling breaks and your code is no longer written the way your test thought it would be. It's probably best to leave a comment in your test when you do this so that whoever breaks it knows that their code isn't wrong, the test is.
And again, this is a code smell and a last resort. If you find you need to do this often, try rethinking the way you write your tests.
It just depends on what type of testing you are doing. If you are doing behavior based testing you might want a dynamic mock so you can verify that some interaction with your dependancy occurs. But if you are doing state based testing you might want a stub so you verify values/etc
For example, in the below test you notice that I stub out the view so I can verify a property value is set (state based testing). I then create a dynamic mock of the service class so I can make sure a specific method gets called during the test (interaction / behavior based testing).
<TestMethod()> _
Public Sub Should_Populate_Products_List_OnViewLoad_When_PostBack_Is_False()
mMockery = New MockRepository()
mView = DirectCast(mMockery.Stub(Of IProductView)(), IProductView)
mProductService = DirectCast(mMockery.DynamicMock(Of IProductService)(), IProductService)
mPresenter = New ProductPresenter(mView, mProductService)
Dim ProductList As New List(Of Product)()
ProductList.Add(New Product())
Using mMockery.Record()
SetupResult.For(mView.PageIsPostBack).Return(False)
Expect.Call(mProductService.GetProducts()).Return(ProductList).Repeat.Once()
End Using
Using mMockery.Playback()
mPresenter.OnViewLoad()
End Using
'Verify that we hit the service dependency during the method when postback is false
Assert.AreEqual(1, mView.Products.Count)
mMockery.VerifyAll()
End Sub
Never mind Statist vs. Interaction. Think about the Roles and Relationships. If an object collaborates with a neighbour to get its job done, then that relationship (as expressed in an interface) is a candidate for testing using mocks. If an object is a simple value object with a bit of behaviour, then test it directly. I can't see the point of writing mocks (or even stubs) by hand. That's how we all started and refactored away from that.
For a longer discussion, consider taking a look at http://www.mockobjects.com/book
Read Luke Kanies' discussion of exactly this question in this blog post. He references a post from Jay Fields which even suggests that using [a equivalent to ruby's/mocha's] stub_everything is preferrable to make the tests more robust. To quote Fields' final words: "Mocha makes it as easy to define a mock as it is to define a stub, but that doesn't mean you should always prefer mocks. In fact, I generally prefer stubs and use mocks when necessary."