I have finally got in my mind what worried me about Dependency Injection and similar techniques that should make unit tests easier. Let's take this example:
public interface IRepository { void Item Find(); a lot of other methods here; }
[Test]
public void Test()
{
var repository = Mock<IRepository>();
repository.Expect(x => x.Find());
var service = new Service(repository);
service.ProcessWithItem();
}
Now, what's wrong with the code above? It's that our test roughly peeks into ProcessWithItem() implementation. What if it wants to do "from x in GetAll() where x..." - but no, our test knows what is going to happen there. And that's just a simple example. Imaging few calls that our test now is tied with, and when we want to change from GetAll() to a better GetAllFastWithoutStuff() inside the method... our test(s) are broken. Please change them. A lot of crappy work that happens so often without any real need.
And that's what often makes me to stop write tests. I just don't see how I can test without knowing implementation details. And knowing them, tests are now very fragile and pain to do.
Sure, it's not about interface (or DI) only. POCOs (and POJOs, why not) also suffer from the same thing, but they're now tied with the data, not with the interface. But the principle is the same - our final assertion is tightly coupled with our knowledge of what our SUT is going to do. "Yes you HAVE to provide this field, sir, and this better be of this value".
As a consequence, tests ARE going to fail - soon and often. This is pain. And the problem.
Are there any techniques to deal with this? AutoMockingContainer (which basically takes care all ALL methods and nested DI hierarchies) looks promising, but with its own drawback. Anything else?
Dependency Injection, per se, would let you inject an implementation of IRepository that accepts whatever calls are made on it, checks that the invariants and preconditions are satisfied, and returns results satisfying the postconditions. When you choose to inject a mock object that has very specific expectations for what methods will be called, then yes, you're doing highly implementation-specific testing -- but Dependency Injection is totally innocent in the matter, since it never dictates WHAT you should inject; rather, your beef appears to be with Mocking -- in fact, specifically the somewhat-automated mocking approach that you have chosen to use, which is one based on very specific expectations.
Mocking with very specific expectations IS indeed useful for white-box testing only. Depending on the tools / frameworks / libraries you're using (and you're not even specifying the exact programming language in a tag, so I assume your question is totally open ended) you may be able to specify the degrees of freedom allowed (these calls are allowed to come in any orders, these arguments must only satisfy the following preconditions, etc, etc). However, I don't know of an automated tool to perform exactly what you need for opaque-box testing, which is the "generic, tolerant implementation of yonder interface with all the ''programming by contract'' checks that are needed and no other".
What I tend to do over the life of a project is to build up a library of "not quite mocks" for the major interfaces needed. In some cases those may be somewhat obvious from the start, but in other cases they emerge incrementally as I'm considering some major refactoring, as follows (typical scenario)...:
The early stages of the refactoring break some aspect of the fragile strong-expectations mocking that I have cheaply put in place initially, I ponder whether to just tweak the expectations or go whole hog, if I decide it's not a one-off (i.e. the return in future refactorings and tests will justify the investment) then I hand-code a good "not quite mock" and stash it away in the project's specific bag of tricks -- actually often reusable across projects; such classes/packages as MockFilesystem, MockBigtable, MockDom, MockHttpClient, MockHttpServer, etc etc, go into a project-agnostic repository and get reused for testing all kinds of future projects (and in fact may be shared with other teams across the company, if several teams are using filesystem interfaces, bigtable interfaces, DOMs, http client/server interfaces, etc etc, that are uniform across the teams).
I acknowledge that the use of the word "mock" may be slightly inappropriate here if you take "mock" to refer specifically to the precise-expectation style of "fake implementation for testing purposes" of interfaces. Maybe Stub, Shim, Fake, Test, or some other prefix yet might be preferable (I do tend to use Mock for historical reasons, except when I remember to specifically call it Fake or the like;-).
If I was using languages with clear and precise way to express in the language itself the various design-by-contract specs in an interface, I imagine I'd get automatic tool support for most of this faking/shimming/etc; however I mostly code in other languages so I have to do a bit more manual work here. But I think that's a separate issue.
I read the excellent book http://www.manning.com/rainsberger/.
I would like to provide some insight I gained from it.
I believe several advice could help you to reduce the coupling between your tests and your implementation.
Edited: included in this coupling is the test asserting that the code under test calls some methods. Calling some method is never a functional need, it is an implementation concern. It relates to an interface other than the one being tested.
In many cases, the testing should be about the external behavior of an interface, and be completely black-box testing them.
The author gives the example that the test classes should be in a different package than the class to test. At first, I was sure this was wrong, because it makes it more difficult to test protected and package methods. But he argues that you should only test the external behavior of a system, that is the public methods. The non-public methods are implementation-details, and testing it results in coupling the test with the implementation. This was very insightful to me.
By the way, this book has so many excellent practical advice on how to design tests (say JUnit tests), that I would buy it on my own money if it wasn't provided by the company! ;-)
An excellent other advice from the book was to test at the functionality level, not the method level. For example, testing the add() method for a list requires trusted size() and get() methods, but they in turn require add() so we have a loop, we can't test safely. But testing the list's behavior globally (accross all methods) when adding involves testing the three methods at the same time, not proving that each is correct in isolation, but checking that together they provide the expected behavior. Often, when you try to test one of your methods in isolation, you cannot write a sensible test without using other methods, so you end up testing the implementation instead ; the consequence are coupling between test and implementation.
Only test functionalities, not methods.
Also, note that testing using external ressources (the database being the more common, but many others exist) is much slower, requires some access (IP, licence etc) from the executing machine, require a started container, may be sensitive to simultaneous access (a database can't run reliably multiple JUnit campaign at the same time), and has many other drawbacks. If all your tests use external resources, then you are in trouble, you can't run all your tests all the time, from any machine, from many machines at once, etc. So I understood (still from the book):
Test only once each external resource (database for example), in a dedicated test that is not a unit-test, but an integration test (although it can still use the same JUnit technology if appropriate).
Test enough dedicated tests to trust the resource is working. Then, other tests should never test it again, this is a waste, they should trust it.
Note that the current Maven best-practices give similar advice (see free book "Better builds with Maven"). I believe this is not a coincidence:
The JUnits in the test directory of a project are real unit tests. They run every time you do something with your project (except just compile).
The integration and functional tests should be provided in a different project, an integration-test project. They only run in a much later (optional) phase, after you have deployed your whole application in the container.
As a consequence, tests ARE going to
fail - soon and often. This is pain.
And the problem.
Well yes, unit tests can depend on internal implementation details. And sure, such "white box" tests are more brittle than "black box" tests which only rely on the externally published contract.
But I don't agree that this has to cause regular test failures. Think about how you arrived at testing with mocks in the first place: you've used dependency injection to limit the responsibilities of the class, to decrease coupling to other code, and to enable testing the class in isolation.
Are there any techniques to deal with
this?
A good unit test can only fail if you change the class under test, even if it depends on internal implementation details. And you can limit the responsibilities and coupling (to other classes) of your class, so that you will rarely have to change it.
In practice you'll have to be pragmatic; every now and then you'll write "unit tests" that are actually integration tests involving multiple classes or over-sized classes. Brittle tests depending on internal implementation details are more dangerous in that case. But for truly TDD-style classes, not so much.
Remember when you're writing a test you're not testing your repository, you're testing your Service class. In this specific example ProcessWithItem method. You create your expectations for repository object. By the way, you forgot to specify expected return for your x.Find method. That's the beauty of DI that you isolate everything from the code you about to write (I assume you do TDD).
To be honest I cannot relate to the problem you describe.
Yeah, that's one of the big problems with unit testing. That, and refactoring. And design changes that are a regular occurrence with Agile. And the inexperience of those creating the tests. And etc etc...
I think the only thing the average non-critical-systems developer can do is pick and choose your battles wisely. Early in development identify the truly critical paths and test those. Weigh the likelihood of that code changing before spending lots of time testing the rest of it.
If anybody figures it all out please let us know.
Related
The tests at my new job are nothing like the tests I have encountered before.
When they're writing their unit tests (presumably before the code), they create a class starting with "When". The name describes the scenario under which the tests will run (the fixture). They'll created subclasses for each branch through the code. All of the tests within the class start with "should" and they test different aspects of the code after running. So, they will have a method for verifying that each mock (DOC) is called correctly and for checking the return value, if applicable. I am a little confused by this method because it means the exact same execution code is being run for each test and this seems wasteful. I was wondering if there is a technique similar to this that they may have adapted. A link explaining the style and how it is supposed to be implemented would be great. I sounds similar to some approaches of BDD I've seen.
I also noticed that they've moved the repeated calls to "execute" the SUT into the setup methods. This causes issues when they are expecting exceptions, because they can't use built-in tools for performing the check (Python unittest's assertRaises). This also means storing the return value as a backing field of the test class. They also have to store many of the mocks as backing fields. Across class hierarchies it becomes difficult to tell the configuration of each mock.
They also test code a little differently. It really comes down to what they consider an integration test. They mock out anything that steals the context away from the function being tested. This can mean private methods within the same class. I have always limited mocking to resources that can affect the results of the test, such as databases, the file system or dates. I can see some value in this approach. However, the way it is being used now, I can see it leading to fragile tests (tests that break with every code change). I get concerned because without an integration test, in this case, you could be using a 3rd party API incorrectly but your unit tests would still pass. I'd like to learn more about this approach as well.
So, any resources about where to learn more about some of these approaches would be nice. I'd hate to pass up a great learning opportunity just because I don't understand they way they are doing things. I would also like to stop focusing on the negatives of these approaches and see where the benefits come in.
If I understood you explanation in the first paragraph correctly, that's quite similar to what I often do. (Depending on whether the testing framework makes it easy or not. Also many mocking frameworks don't support it, but spy frameworks like Mockito do better.)
For example see the stack example here which has a common setup (adding things to the stack) and then a bunch of independent tests which each check one thing. Here's still another example, this time one where none of the tests (#Test) modify the common fixture (#Before), but each of them focuses on checking just one independent thing that should happen. If the tests are very well focused, then it should be possible to change the production code to make any single test fail while all other tests pass (I wrote about that recently in Unit Test Focus Isolation).
The main idea is to have each test check a single feature/behavior, so that when tests fail it's easier to find out why it failed. See this TDD tutorial for more examples and to learn that style.
I'm not worried about the same code paths executed multiple times, when it takes a millisecond to run one test (if it takes more than a couple of seconds to run all unit tests, the tests are probably too big). From your explanation I'm more worried that the tests might be too tightly coupled to the implementation, instead of the feature, if it's systematic that there is one test for each mock. The name of the test would be a good indicator of how well structured or how fragile the tests are - does it describe a feature or how that feature is implemented.
About mocking, a good book to read is Growing Object-Oriented Software Guided by Tests. One should not mock 3rd party APIs (APIs which you don't own and can't modify), for the reason you already mentioned, but one should create an abstraction over it which better fits the needs of the system using it and works the way you want it. That abstraction needs to be integration tested with the 3rd party API, but in all tests using the abstraction you can mock it.
First, the pattern that you are using is based on Cucumber - here's a link. The style is from the BDD (Behavior-driven development) approach. It has two advantages over traditional TDD:
Language - one of the tenants of BDD is that the language you use influences the thoughts you have by forcing you to speak in the language of the end user, you will end up writing different tests than when you write tests from the focus of a programmer
Tests lock code - BDD locks the code at the appropriate level. One problem common in testing is that you write a large number of tests, which makes your codebase more brittle as when you change the code you must also change a large number of tests too. BDD forces you to lock the behavior of your code, rather than the implementation of your code. This way, when a test breaks, it is more likely to be meaningful.
It is worth noting that you do not have to use the Cucumber style of testing to achieve these affects and using it does add an extra layer of overhead. But very few programmers have been successful in keeping the BDD mindset while using traditional xUnit tools (TDD).
It also sounds like you have some scenarios where you would like to say 'When I do , then verify '. Because the current BDD xUnit frameworks only allow you to verify primitives (strings, ints, doubles, booleans....), this usually results in a large number of individual tests (one for each Assert). It is possible to do more complicated verifications using a Golden Master paradigm test tool, such as ApprovalTests. Here's a video example of this.
Finally, here's a link to Dan North's blog - he started it all.
In Osherove's great book "The Art of Unit Testing" one of the test anti-patterns is over-specification which is basically the same as testing the internal state of the object instead of some expected output. To my experience, using Isolation frameworks can cause the same unwanted side effects as testing internal behavior because one tends to only implement the behavior necessary to make your stub interact with the object under test. Now if your implementation changes later on (but the contract remains the same), your test will suddenly break because you are expecting some data from the stub which was not implemented.
So what do you think is the best approach to counter this?
1) Implement your stubs/mocks fully, this has the negative side-effect of potentially making your test less readable and also specifying more than necessary to make your test pass.
2) Favor manual, fully implemented fakes.
3) Implement your stubs/fakes so that they make your test just pass, and then deal with the brittleness that this might introduce.
I do not think you should favor manual testing - unless you prefer to test instead of code.
Instead you have another option - if you test the functionality and not the implementation, try to avoid testing private methods (that can be refactored) and in general write less-fragile tests you'll see that using a mocking/isolation framework does not require you to over specify the system nor does it cause your tests to become more fragile.
In a nutshell - writing fragile tests can be done with or without fakes/mocks and vise-versa.
I tend to use mocks instead of stubbed/fake objects. I find them a lot less trouble and they are way better at keeping test code under control because it's not cluttered with all sorts of half baked implementations. They also help to clarify what is being tested.
Another advantage is that I only have to address where the class under test needs something specific from the mock. So I don't have to code where it's not important. As for verification, again I only have to very the calls from the class under test to the mock that I care about and consider important aspects of the test.
I think, the problem is always the same, although it comes in different flavours: If you have tests that somehow cover the internals of a class, then you will break the tests that cover this internal code.
IMHO there are two ways to deal with that:
Your tests only cover the public contract of a class - a test strategy which is widely adopted for that exact reason: You don't have to change your tests as long as the public contract remains constant. Unfortunately, this is not, what you will have when doing Test-driven development.
If your tests come from a TDD process, then they will regularly cover non-public code. This means that they will break if you change the code. The only way to keep things in sync here is to 'fix' the tests together with the code. This means more maintenance during development. There's no recipe to easily deal with that (other than throw away the test, of course...).
My personal 'way out' is think in terms of 'code elements' rather than just code. A code element consists of three parts: Documentation, test, code. So if you change one part of the element, you have to also adjust the other two - otherwise you leave a broken code element behind.
I recently discussed with a colleague about mocking. He said that mocking classes is very bad and should not be done, only in few cases.
He says that only interfaces should be mocked, otherwise it's an architecture fault.
I wonder why this statement (I fully trust him) is so correct? I don't know it and would like to be convinced.
Did I miss the point of mocking (yes, I read Martin Fowler's article)
Mocking is used for protocol testing - it tests how you'll use an API, and how you'll react when the API reacts accordingly.
Ideally (in many cases at least), that API should be specified as an interface rather than a class - an interface defines a protocol, a class defines at least part of an implementation.
On a practical note, mocking frameworks tend to have limitations around mocking classes.
In my experience, mocking is somewhat overused - often you're not really interested in the exact interaction, you really want a stub... but mocking framework can be used to create stubs, and you fall into the trap of creating brittle tests by mocking instead of stubbing. It's a hard balance to get right though.
IMHO, what your colleague means is that you should program to an interface, not an implementation. If you find yourself mocking classes too often, it's a sign you broke the previous principle when designing your architecture.
Mocking classes (in contrast to mocking interfaces) is bad because the mock still has a real class in the background, it is inherited from, and it is possible that real implementation is executed during the test.
When you mock (or stub or whatever) an interface, there is no risk of having code executed you actually wanted to mock.
Mocking classes also forces you to make everything, that could possibly be mocked, to be virtual, which is very intrusive and could lead to bad class design.
If you want to decouple classes, they should not know each other, this is the reason why it makes sense to mock (or stub or whatever) one of them. So implementing against interfaces is recommended anyway, but this is mentioned here by others enough.
I would suggest to stay away from mocking frameworks as far as possible. At the same time, I would recommend to use mock/fake objects for testing, as much as possible. The trick here is that you should create built-in fake objects together with real objects. I explain it more in detail in a blog post I wrote about it: http://www.yegor256.com/2014/09/23/built-in-fake-objects.html
Generally you'd want to mock an interface.
While it is possible to mock a regular class, it tends to influence your class design too much for testability. Concerns like accessibility, whether or not a method is virtual, etc. will all be determined by the ability to mock the class, rather than true OO concerns.
There is one faking library called TypeMock Isolator that allows you to get around these limitations (have cake, eat cake) but it's pretty expensive. Better to design for testability.
The answer, like most questions about practices, is "it depends".
Overuse of mocks can lead to tests that don't really test anything. It can also lead to tests which are virtual re-implementations of the code under test, tightly bound to a specific implementation.
On the other hand, judicious use of mocks and stubs can lead to unit tests which are neatly isolated and test one thing and one thing alone - which is a good thing.
It's all about moderation.
It makes sense to mock classes so tests can be written early in the development lifecycle.
There is a tendency to continue to use mock classes even when concrete implementations become available. There is also the tendency to develop against mock classes (and stubs) necessary early in a project when some parts of the system have not been built.
Once a piece of the system has been built it is necessary to test against it and continue to test against it (for regression). In this case starting with mocks is good but they should be discarded in favour of the implementation as soon as possible. I have seen projects struggle because different teams continue to develop against the behaviour of the mock rather than the implementation (once it is available).
By testing against mocks you are assuming that the mock is characteristic of the system. Often this involves guessing what the mocked component will do. If you have a specification of the system you are mocking then you don't have to guess, but often the 'as-built' system doesn't match the original specification due to practical considerations discovered during construction. Agile development projects assume this will always happen.
You then develop code that works with the mock. When it turns out that the mock does not truly represent the behaviour of the real as-built system (eg. latency issues not seen in the mock, resource and efficiency issues not seen in the mock, concurrency issues, performance issues etc) you then have a bunch of worthless mocking tests you must now maintain.
I consider the use of mocks to be valuable at the start of development but these mocks should not contribute to project coverage. It is best later if the mocks are removed and proper integration tests are created to replace them otherwise your system will not be getting tested for the variety of conditions which your mock did not simulate (or simulates incorrectly relative to the real system).
So, the question is whether or not to use mocks, it is a matter of when to use them and when to remove them.
It depends how often you use (or are forced by bad design) mocks.
If instantiating the object becomes too hard (and it happens more than often), then it is a sign the code may need some serious refactoring or change in design (builder? factory?).
When you mock everything you end up with tests that know everything about your implementation (white box testing). Your tests no longer document how to use the system - they are basically a mirror of its implementation.
And then comes potential code refactoring..
From my experience it's one of the biggest issues related to overmocking. It becomes painful and takes time, lots of it.
Some developers become fearful of refactoring their code knowing how long will it take.
There is also question of purpose - if everything is mocked, are we really testing the production code?
Mocks of course tend to violate DRY principle by duplicating code in two places: once in the production code and once in the tests.
Therefore, as I mentioned before, any change to code has to be made in two places (if tests aren't written well, it can be in more than that..).
Edit: Since you have clarified that your colleague meant mock class is bad but mock interface is not, the answer below is outdated. You should refer to this answer.
I am talking about mock and stub as defined by Martin Fowler, and I assume that's what your colleague meant, too.
Mocking is bad because it can lead to overspecification of tests. Use stub if possible and avoid mock.
Here's the diff between mock and stub (from the above article):
We can then use state verification on
the stub like this.
class OrderStateTester...
public void testOrderSendsMailIfUnfilled() {
Order order = new Order(TALISKER, 51);
MailServiceStub mailer = new MailServiceStub();
order.setMailer(mailer);
order.fill(warehouse);
assertEquals(1, mailer.numberSent());
}
Of course this is a very simple test -
only that a message has been sent.
We've not tested it was send to the
right person, or with the right
contents, but it will do to illustrate
the point.
Using mocks this test would look quite
different.
class OrderInteractionTester...
public void testOrderSendsMailIfUnfilled() {
Order order = new Order(TALISKER, 51);
Mock warehouse = mock(Warehouse.class);
Mock mailer = mock(MailService.class);
order.setMailer((MailService) mailer.proxy());
mailer.expects(once()).method("send");
warehouse.expects(once()).method("hasInventory")
.withAnyArguments()
.will(returnValue(false));
order.fill((Warehouse) warehouse.proxy());
}
}
In order to use state verification on the stub, I need to make some extra methods on the >stub to help with verification. As a result the stub implements MailService but adds extra >test methods.
Wherever possible I use TDD:
I mock out my interfaces
I use IOC so my mocked ojbects can be injected
I ensure my tests run and that the coverage increases and I am happy.
then...
I create derived classes that actually do stuff, like going to a database, or writing to a message queue etc.
This is where code coverage decreases - and I feel sad.
But then, I liberally spread [CoverageExclude] over these concrete classes and coverage goes up again.
But then instead of feeling sad, I feel dirty. I somehow feel like I'm cheating even though it's not possible to unit-test the concrete classes.
I'm interested in hearing how your projects are organised, i.e. how do you physically arrange code that can be tested against code that can't be tested.
I'm thinking that perhaps a nice solution would be to separate out untestable concrete types into their own assembly and then ban the use of [CoverageExclude] in the assemblies that do contain testable code. This'd also make it easier to create an NDepend rule to fail the build when this attribute is incorrectly found in the testable assemblies.
Edit: the essense of this question touches on the fact that you can test the things that USE your mocked interfaces but you can't (or shouldn't!) UNIT-test the objects that ARE the real implementations of those interfaces. Here's an example:
public void ApplyPatchAndReboot( )
{
_patcher.ApplyPatch( ) ;
_rebooter.Reboot( ) ;
}
patcher and rebooter are injected in the constructor:
public SystemUpdater(IApplyPatches patcher, IRebootTheSystem rebooter)...
The unit test looks like:
public void should_reboot_the_system( )
{
... new SystemUpdater(mockedPatcher, mockedRebooter);
update.ApplyPatchAndReboot( );
}
This works fine - my UNIT-TEST coverage is 100%. I now write:
public class ReallyRebootTheSystemForReal : IRebootTheSystem
{
... call some API to really (REALLY!) reboot
}
My UNIT-TEST coverage goes down and there's no way to UNIT-TEST the new class. Sure, I'll add a functional test and run it when I've got 20 minutes to spare(!).
So, I suppose my question boils down to the fact that it's nice to have near 100% UNIT-TEST coverage. Said another way, it's nice to be able to unit-test near 100% of the behaviour of the system. In the above example, the BEHAVIOUR of the patcher should reboot the machine. This we can verify for sure. The ReallyRebootTheSytemForReal type isn't strictly just behaviour - it has side effects which means it can't be unit-tested. Since it can't be unit-test it affects the test-coverage percentage. So,
Does it matter that these things reduce unit-test coverage percantage?
Should they be segregated into their own assemblies where people expect 0% UNIT-TEST coverage?
Should concrete types like this be so small (in Cyclomatic Complexity) that a unit-test (or otherwise) is superfluous
You are on the right track. Some of the concrete implementations you probably can test, such as Data Access Components. Automated testing against a relational database is most certainly possible, but should also be factored out into its own library (with a corresponding unit test library).
Since you are already using Dependency Injection, it should be a piece of cake for you compose such a dependency back into your real application.
On the other hand, there will also be concrete dependencies that are essentially un-testable (or de-testable, as Fowler once joked). Such implementations should be kept as thin as possible. Often, it is possible to design the API that such a Dependency exposes in such a way that all the logic happens in the consumer, and the complexity of the real implementation is very low.
Implementing such concrete Dependencies is an explicit design decision, and when you make that decision, you simultaneously decide that such a library should not be unit tested, and thus code coverage should not be measured.
Such a library is called a Humble Object. It (and many other patterns) are described in the excellent xUnit Test Patterns.
As a rule of thumb I accept that code is untested if it has a Cyclomatic Complexity of 1. In that case, it's more or less purely declarative. Pragmatically, untestable components are in order as long as they have low Cyclomatic Complexity. How low 'low' is you must decide for yourself.
In any case, [CoverageExclude] seems like a smell to me (I didn't even know it existed before I read your question).
I don't understand how your concrete classes are untestable. This smells horrible to me.
If you have a concrete class that is writing to a message queue, you should be able to pass it a mock queue, and test all it's methods just fine. If your class is going to a database, then you should be able to hand it a mock database to go to.
There can be situations that might lead to untestable code, I won't deny that - but that should be the exception, not the rule. All your concrete class worker objects? Something isn't right.
To expand on womps answer: I suspect you are considering more to be "untestable" than really is. Untestable in the strict "one unit at a time" unit testing without testing any of the dependencies simultanously? Sure. But it should be easily achievable with slower and more infrequently running integration style tests.
You mention accessing database and writing messages to queues. As womp mentions, you can feed them mock databases and mock queues during unit testing and the testing the actual concrete beahviour in integration tests. Personally I don't see anything wrong with testing concrete implementations directly as unit tests either, at least when they are not remote (or legacy). Sure they run a bit slower, but hey, at least they get covered by automated tests.
Would you put a system into production where messages are written to queues and not having actually tested that the messages are written to the actual physical/logical queue? I wouldn't.
This question already has answers here:
When should I mock?
(4 answers)
Closed 9 years ago.
Classes that use other classes (as members, or as arguments to methods) need instances that behave properly for unit test. If you have these classes available and they introduce no additional dependencies, isn't it better to use the real thing instead of a mock?
I say use real classes whenever you can.
I'm a big believer in expanding the boundaries of "unit" tests as much as possible. At this point they aren't really unit tests in the traditional sense, but rather just an automated regression suite for your application. I still practice TDD and write all my tests first, but my tests are a little bigger than most people's and my green-red-green cycles take a little longer. But now that I've been doing this for a little while I'm completely convinced that unit tests in the traditional sense aren't all they're cracked up to be.
In my experience writing a bunch of tiny unit tests ends up being an impediment to refactoring in the future. If I have a class A that uses B and I unit test it by mocking out B, when I decide to move some functionality from A to B or vice versa all of my tests and mocks have to change. Now if I have tests that verify that the end to end flow through the system works as expected then my tests actually help me to identify places where my refactorings might have caused a change in the external behavior of the system.
The bottom line is that mocks codify the contract of a particular class and often end up actually specifying some of the implementation details too. If you use mocks extensively throughout your test suite your code base ends up with a lot of extra inertia that will resist any future refactoring efforts.
It is fine to use the "real thing" as long as you have absolute control over the object. For example if you have an object that just has properties and accessors you're probably fine. If there is logic in the object you want to use, you could run into problems.
If a unit test for class a uses an instance of class b and an change introduced to b breaks b, then the tests for class a are also broken. This is where you can run into problems where as with a mock object you could always return the correct value. Using "the real thing" Can kind of convolute tests and hide the real problem.
Mocks can have downsides too, I think there is a balance with some mocks and some real objects you will have to find for yourself.
There is one really good reason why you want to use stubs/mocks instead of real classes. I.e. to make your unit test's (pure unit test) class under test isolated from everything else. This property is extremely useful and the benefits for keeping tests isolated are plentiful:
Tests run faster because they don't need to call the real class implementation. If the implementation is to run against file system or relational database then the tests will become sluggish. Slow tests make developers not run unit tests as often. If you're doing Test Driven Development then time hogging tests are together a devastating waste of developers time.
It will be easier to track down problems if the test is isolated to the class under test. In contrast to a system test it will be much more difficult to track down nasty bugs that are not apparently visible in stack traces or what not.
Tests are less fragile on changes done on external classes/interfaces because you're purely testing the class that is under test. Low fragility is also an indication of low coupling, which is a good software engineering.
You're testing against external behaviour of a class rather than the internal implementation which is more useful when deciding code design.
Now if you want to use real class in your test, that's fine but then it is NOT a unit test. You're doing a integration test instead, which is useful for the purpose of validating requirements and overall sanity check. Integration tests are not run as often as unit tests, in practice it is mostly done before committing to favorite code repository, but is equally important.
The only thing you need to have in mind is the following:
Mocks and stubs are for unit tests.
Real classes are for integration/system tests.
Extracted and extended from an answer of mine How do I unit-test inheriting objects?">here:
You should always use real objects where possible.
You should only use mock objects if the real objects do something you dont want to set up (like use sockets, serial ports, get user input, retrieve bulky data etc). Essentially, mock objects are for when the estimated effort to implement and maintain a test using a real object is greater than that to implement and maintain a test using a mock object.
I dont buy into the "dependant test failure" argument. If a test fails because a depended-on class broke, the test did exactly what it should have done. This is not a smell! If a depended-on interface changes, I want to know!
Highly mocked testing environments are very high-maintenance, particularly early in a project when interfaces are in flux. Ive always found it better to start integration testing ASAP.
I always use a mock version of a dependency if the dependency accesses an external system like a database or web service.
If that isn't the case, then it depends on the complexity of the two objects. Testing the object under test with the real dependency is essentially multiplying the two sets of complexities. Mocking out the dependency lets me isolate the object under test. If either object is reasonably simple, then the combined complexity is still workable and I don't need a mock version.
As others have said, defining an interface on the dependency and injecting it into the object under test makes it much easier to mock out.
Personally, I'm undecided about whether it's worth it to use strict mocks and validate every call to the dependency. I usually do, but it's mostly habit.
You may also find these related questions helpful:
What is object mocking and when do I need it?
When should I mock?
How are mocks meant to be used?
And perhaps even, Is it just me, or are interfaces overused?
Use the real thing only if it has been unit tested itself first. If it introduces dependencies that prevent that (circular dependencies or if it requires certain other measures to be in place first) then use a 'mock' class (typically referred to as a "stub" object).
If your 'real things' are simply value objects like JavaBeans then thats fine.
For anything more complex I would worry as mocks generated from mocking frameworks can be given precise expectations about how they will be used e.g. the number of methods called, the precise sequence and the parameters expected each time. Your real objects cannot do this for you so you risk losing depth in your tests.
I've been very leery of mocked objects since I've been bitten by them a number of times. They're great when you want isolated unit tests, but they have a couple of issues. The major issue is that if the Order class needs a a collection of OrderItem objects and you mock them, it's almost impossible to verify that the behavior of of the mocked OrderItem class matches the real-world example (duplicating the methods with appropriate signatures is generally not enough). More than once I've seen systems fail because the mocked classes don't match the real ones and there weren't enough integration tests in place to catch the edge cases.
I generally program in dynamic languages and I prefer merely overriding the specific methods which are problematic. Unfortunately, this is sometimes hard to do in static languages. The downside of this approach is that you're using integration tests rather than unit tests and bugs are sometimes harder to track down. The upside is that you're using the actual code that is written, rather than a mocked version of that code.
If you don't care for verifying expectations on how your UnitUnderTest should interact with the Thing, and interactions with the RealThing have no other side-effects (or you can mock these away) then it is in my opinion perfectly fine to just let your UnitUnderTest use the RealThing.
That the test then covers more of your code base is a bonus.
I generally find it is easy to tell when I should use a ThingMock instead of a RealThing:
When I want to verify expectations in the interaction with the Thing.
When using the RealThing would bring unwanted side-effects.
Or when the RealThing is simply too hard/troublesome to use in a test setting.
If you write your code in terms of interfaces, then unit testing becomes a joy because you can simply inject a fake version of any class into the class you are testing.
For example, if your database server is down for whatever reason, you can still conduct unit testing by writing a fake data access class that contains some cooked data stored in memory in a hash map or something.
It depends on your coding style, what you are doing, your experience and other things.
Given all that, there's nothing stopping you from using both.
I know I use the term unit test way too often. Much of what I do might be better called integration test, but better still is to just think of it as testing.
So I suggest using all the testing techniques where they fit. The overall aim being to test well, take little time doing it and personally have a solid feeling that it's right.
Having said that, depending on how you program, you might want to consider using techniques (like interfaces) that make mocking less intrusive a bit more often. But don't use Interfaces and injection where it's wrong. Also if the mock needs to be fairly complex there is probably less reason to use it. (You can see a lot of good guidance, in the answers here, to what fits when.)
Put another way: No answer works always. Keep your wits about you, observe what works what doesn't and why.