As far as I can see there are two ways, both with their drawbacks.
Get the object you are unit testing from the dependency injection system. This is low maintenance as you don’t have to manage anything when you change the framework around. But you are essentially testing the whole system from the point of view of your object, if a component fails it can blow up a lot of unit tests and it may not be apparent which one is failing.
is to manage the dependencies manually in the unit tests, and in some cases create test objects so that you can test each object in isolation. This keeps the unit tests discreet but dramatically increases the maintenance of the unit tests themselves. It also means that you don’t pick up on bugs cause by the way the objects interact on your live system.
Is either approach right or wrong? Should a compromise be used? Has anyone had any success stories either way.
If you're writing a unit test you should be using mocks for your dependencies and an IoC container shouldn't come into the picture. You should instantiate your class-under-test with mocks for the dependencies injected by hand.
If you're getting your object from the IoC container already wired up then what you're writing is integration tests which are very different.
Your goal for writing a unit test should be to write your test in isolation from the rest of the system.
You want to use an auto mocking container to manage the dependencies for you. There is one built into StructureMap and it's not that hard to roll your own.
The basic premise of an auto mocking container is that you set expectations on the dependencies you need to/care about and it creates mocks for any other dependencies for you. This means that if you add a dependency to a class you don't necessarily have to go back and alter every test for it; just those that will use the new dependency.
In my opinion the first approach is wrong as - as you you mentioned - it's not necessarily the object that you are testing that will cause a test to fail.
Related
Sometime back I used mocks in TDD (Test Driven Development) when the implementation of dependent interfaces was not available. So I mocked those dependencies and tested my work.
Now implementations of the dependencies are available. So should I remove the mock or should I convert them to spy or similar? I think integration tests can only be done after removing those mocks, or I may have to duplicate them. I am little confused, any suggestion on this?
As far as unit tests are concerned - never. Usually it is because you want to keep your unit tests isolated:
Your unit tests (just as any other classes) should usually have single reason to change/break. When a bug happens you want to reduce possible number of offenders to absolute minimum. Having unit test failing because some dependency might have failed, or some other dependency might have failed, or maybe finally actual tested class might have failed is undersirable.
In reality, this is quite a strech because even changing an interface of a dependency will result in changes in unit tests of class using this dependency. What you want to avoid tho, is some guy from some other team changing implementation of some dependency you are using and as a result your tests break, apparently for no reason. This is again, highly undesirable.
Mocks should be used to isolate your units. Of course, when writing different types of tests (say integration) you don't want isolation. You want real components collaborating together. This is one of the reasons to use actual implementations rather than mocks - but! - this does not invalidate your unit tests. They should remain isolated.
As always: It depends.
If you are using a mock, you are testing against a defined interface, so your unit test is more focused and there could be less breaking changes in your tests (such as when something internal needs to be refactored).
If you are using the actual dependencies, then your unit tests are not as focused (they might be integration tests). So these tests are often slower, more difficult to set up and more prone to breaking when refactoring. On the other hand, since you are testing against the actual implementation, you are more likely to find bugs due to the actual implementation behaving differently than you expected.
At my office we have a dispute regarding the necessity of unit tests in addition to integration tests for the classes that have the main responsibility of interacting with a filesystem (DB, etc).
The integration tests we have, are almost unit tests, as the tested object doesn't interact with other objects at all. The only reason, why we call the tests integration, is that the real filesystem is used in tests. And it is proposed to make the tested class use filesystem layer component, then mock it in tests (so we will call them unit tests), and check interaction with this component, rather than real filesystem results. Necessity of this change is what we discuss.
One point of view we have, is that unit tests are always required, because:
Writing unit tests makes code much better
Having unit tests, you don't need to care about real filesystem and side-effects of files, appearing at wrong locations
A developer can fully test results by making the tested class use filesystem mock and setting proper expectations for that mock
It is ok to tie the mock expectations to the specific internal algorithm of the tested class, because we do white-box testing with unit tests
Thus, the unit tests must always be written for such a tested class. And a filesystem layer component must always be used by that class for the purpose of testing.
Another point of view, is that unit tests are not needed for specific edge cases of classes that are devoted to filesystem interactions, because:
It is not possible to properly verify, that a tested class works, just by having simple mock instead of real filesystem (or its full emulation). Filesystem is such a complex component, that:
A tested class can work in many different ways in order to achieve successful result. The mock expectations cover just one-two possible scenarios, so a unit test erroneously shows failures for a class that properly implements good algorithm, which is different from the one expected.
A tested class can work in a way, detected by mock as a successful scenario, while the class still does not produce right result. This can be because of quite complex reasons in real filesystem. All these reasons are impossible to be covered just by a mock.
A unit test with mock and expectations is very fragile, because it is very tied to the tested class's internal algorithm. And the test erroneously fails upon even right changes of the algorithm.
The integration testing is a proper and full replacement for unit testing for a case, when the class has just 1-2 public methods, and only dependency is a filesystem. Integration testing gives same benefits as unit testing for this case - clear dependencies, more readable code, etc.
Thus the unit testing with filesystem mocking is not needed in our case. It is fragile and is not accurate for this particular case of classes.
So, to sum it up, the question is:
Is integration testing fully enough for an edge case of having not a complex class, which has main responsibility to work with a filesystem (DB, etc.)?
The only difference between integration and unit tests for this class, is that with unit tests the filesystem mock would be used (class would be fully isolated), while with integration tests the real filesystem is used.
I would appreciate, if you can add the references to classic books, or maybe articles / presentations of well-known industry people, so we can have a really strong ground to support the resulting conclusion.
The short answer here is yes, you could fully test a class with 'integration' tests. The better question, though, is should you do so?
I think you're getting too hung up on the difference in definitions between a 'unit test' (no outside dependencies) and an 'integration test' (has such dependencies). The goal with testing is to give you confidence that your code is working at all times, while keeping the associated costs of having that confidence down. So your question
Is integration testing fully enough for an edge case of having not a
complex class, which has main responsibility to work with a filesystem
(DB, etc.)?
is somewhat incomplete.
The most useful part of that distinction between 'unit' and 'integration' for our discussion is this: unit tests are easier and cheaper to write, maintain, and run.
To write a unit test, you just need to know the code. If a unit test fails, you know it's because of changes to the code. Writing an integration test requires setting up dependencies, eg creating files with specific contents, inserting rows in to a database, etc. If an integration test fails, it could be your code, or it could be your dependencies. For these reasons and others, integration tests are more complex, and therefore expensive, to create, maintain and run.
That increased expense should push the developer to separate classes encapsulating business logic from classes that handle interaction with outside systems, in an effort to minimize the number of integration tests required. The business logic can be tested with unit tests, which are cheaper.
Edit
It is possible that your class has complicated logic that itself is complicated because it has to handle complicated behavior in the underlying external dependency (ie, the file system in question). In that case, mocking the file system may be quite difficult in itself, and it may be cheaper/easier to just use a properly set up file system and write 'integration' tests.
The important point to keep in mind is what you're trying to achieve: confidence at a acceptable cost. If 'integration' tests are cheap enough, great. If you can get the same confidence cheaper using 'unit' tests, even better. The exact mix depends on the problem at hand.
It would be preferable to have a known state of the filesystem or DB for the tests. As an example, you do not want to have a test fail because it is trying to insert a record that already exists. This failure is not due to the code but a problem with the DB. Same thing can happen in the filesystem. However, you should write the best test that you are able to. If you can't easily mock the filesystem or whatever, then interact with it. Just realize that if the test fails it may not be a problem with the code.
An ugle test is better than no test. --The Way of Testivus
http://www.artima.com/weblogs/viewpost.jsp?thread=203994
Now even if you do have tests with mocks, that does not mean that you should not have QA or some sort of integration test to make sure that everything connects correctly. I view that unit tests are for verifying that the internals of the code works correctly and integration tests tell me that all the pieces work together.
I don't know what language you are using but the documentation for PHPUnit gives some ideas about testing the DB and filesystem.
http://www.phpunit.de/manual/current/en/database.html
http://www.phpunit.de/manual/current/en/test-doubles.html#test-doubles.stubbing-and-mocking-web-services
http://www.phpunit.de/manual/current/en/test-doubles.html#test-doubles.mocking-the-filesystem
A unit test with mock and expectations is very fragile, because it is
very tied to the tested class's internal algorithm. And the test
erroneously fails upon even right changes of the algorithm.
For testing with mocks, you should not be tying to the algorithm. All that you are testing for is the expected behavior of the class. Not how it goes about doing it.
I'm currently broadening my Unit Testing by utilising Mock objects (nSubsitute in this particular case). However I'm wondering what the current wisdom when creating a Mock objects. For instance, I'm working with an object that contains various routines to grab and process data - no biggie here but it will be utilised in a fair number of tests.
Should I create a shared function that returns the Mock Object with all the appropriate methods and behaviours mocked for pretty much most of the Testing project and call that object into my Unit Tests? Or shall I Mock the object into every Unit Test, only mocking the behaviour I need for that test (although there will be times I'll be mocking the same behaviour more than one occasion).
Thoughts or advice is gratefully received...
I'm not sure if there is an agreed "current wisdom" on this, but here's my 2 cents.
First, as #codebox pointed out, re-creating your mocks for each unit test is a good idea, as you want your unit tests to run independently of each other. Doing otherwise can result in tests that pass when run together but fail when run in isolation (or vis versa). Creating mocks required for tests is commonly done in test setup ([SetUp] in NUnit, constructor in XUnit), so each test will get a newly created mock.
In terms of configuring these mocks, it depends on the situation and how you test. My preference is to configure them in each test with the minimum amount of configuration necessary. This is a good way of communicating exactly what that test requires of its dependencies. There is nothing wrong with some duplication in these cases.
If a number of tests require the same configuration, I would consider using a scenario-based test fixture (link disclaimer: shameless self-promotion). A scenario could be something like When_the_service_is_unavailable, and the setup for that scenario could configure the mocked service to throw an exception or return an error code. Each test then makes assertions based on that common configuration/scenario (e.g. should display error message, should send email to admin etc).
Another option if you have lots of duplicated bits of configuration is to use a Test Data Builder. This gives you reusable ways of configuring a number of different aspects of your mock or other any other test data.
Finally, if you're finding a large amount of configuration is required it might be worth considering changing the interface of the test dependency to be less "chatty". By looking for a valid abstraction that reduces the number of calls required by the class under test you'll have less to configure in your tests, and have a nice encapsulation of the responsibilities on which that class depends.
It is worth experimenting with a few different approaches and seeing what works for you. Any removal of duplication needs to be balanced with keeping each test case independent, simple, maintainable and reliable. If you find you have a large number of tests fail for small changes, or that you can't figure out the configuration an individual tests needs, or if tests fail depending on the order in which they are run, then you'll want to refine your approach.
I would create new mocks for each test - if you re-use them you may get unexpected behaviour where the state of the mock from earlier tests affects the outcome of later tests.
It's hard to provide a general answer without looking at a specific case.
I'd stick with the same approach as I do everywhere else: first look at the tests as independent beings, then look for similarities and extract the common part out.
Your goal here is to follow DRY, so that your tests are maintainable in case the requirements change.
So...
If it's obvious that every test in a group is going to use the same mock behaviour, provide it in your common set-up
If each of them is significantly different, as in: the content of the mock constitutes a significant part of what you're testing and the test/mock relationship looks like 1:1, then it's reasonable to keep them close to the tests
If the mocks differ between them, but only to some degree, you still want to avoid redundancy. A common SetUp won't help you, but you may want to introduce an utility like PrepareMock(args...) that will cover different cases. This will make your actual test methods free of repetitive set-up, but still let you introduce any degree of difference between them.
The tests look nice when you extract all similarities upwards (to a SetUp or helper methods) so that the only thing that remains in test methods is what's different between them.
I am new to unit testing, but tend to think that I believe in beautifully written code, and properly designed architectures.
My question is. Aren't unit tests focusing too much on dependencies between objects? What do you do when your unit test fails because a dependency your method used to call befor is no longer called (a design decision) or your method calls another method or a dependency (again a design decision) Do you redesign your tests? If that's the case, then unit testing helps very little to reduce couple and improve cohesion between components.
Maybe my opinion is too broad, but in general how do people treat dependencies in properly mannered unit tests. I guess that the best way would be to have no dependencies at all, and every method relied on the parameters that were given to it, but this is hardly the case in reality. In addition, faking every dependency method for every possible call is also a bit subjective and time wasting, because at a future point in time, the class under test may simply no longer need the dependency.
I would suggest that you look at Test Driven Development (TDD) as I believe this technique will help you with your design issues. By writing unit tests before writing the production code, you will need to think about how to make your production code testable. This is better then the test later approach, where you write the production code first and then try to shoe-horn tests around them.
To deal with dependencies, think about what dependencies are causing you problems.
External Dependencies
If your tests use an external resource, such as a file, then you are writing an integration test, not a unit test. I've written many tests that use an external file, and I simply created a copy of the file in my test project. This file copy will contain dummy data required for my tests.
If your test requires a database, then again your writing an integration test. Personally I create a local copy of the database on my PC and run my tests against it.
Object Dependencies
If you are worried about code dependencies (e.g. your test will fail if a private method's signature is changed) then you are testing at the wrong level of abstraction. By that I mean make sure that your tests are calling public API's and not private ones. To cement this point, use interfaces for your objects to ensure an expected contract for an object that implements it.
I would also recommend that you try using a mocking framework such as RhinoMocks, Moq or TypeMock
A mocking framework will help you remove the dependency on, for example, having a database available for your tests. I personally use TypeMock, it's not cheap but it's by far the most powerful tool out there.
If you are talking about Unit testing you have no dependencies, cause a unit test tests only a single class (Java, C++, Ruby, Python). What you are talking about sounds more like integration testing which is different. Furthermore if you have to much dependencies your coupling is to high which is not very good, but of course not always avoidable.
Unit tests shall test the behavior, not the implementation. That way, one can rely on the unit tests when changing the implementation, or when refactoring the code. Removing a dependency (via inlining the class for instance), does not break the test.
Testing the implementation leads to brittle tests, that gets in the way when refactoring.
How do you decide what to choose:
use mock objects for a test OR
create a test object/ object graph using an IoC framework and run test on that data
It depends what you are trying to test. Unit tests with collaborators mocked out are great because
They are really, really fast
They are small and easy to understand
They don't have dependencies on the wider world which makes them easy to run
They provide excellent defect localisation
However, pure unit tests cannot tell you if you have configured your objects correctly in your IoC container, if the database connection string works etc. You need a test which runs up your IoC container and really reaches out to the Db to prove these things.
If you write as many of your tests as pure, standalone unit tests as possible then your build will stay fast. This is crucial, as a slow bulid gets run less. Even so, don't forget to add a sprinkling of wired tests to prove that your application'hangs together'.
For example, we have a (single) test for every service in our container that proves that we can request it from the IoC container. This proves we're wired up, from then on it is unit tests all the way. We have lots of pure unit tests.
The whole lot is then wrapped in some application level functional tests to prove that the app itself does what the user wants.
The thing to bear in mind is the time cost of each test type. Moving from pure unit -> wired -> functional tests costs an order of magnitude of execution time and complexity when they break.
I am very happy with using IoC for much of my app, and especially I appreciate that test-datasources can be injected for testing.
For more problematic backend connections (currently a single ESB call) or functions that need complicated state I mock.
For unit tests: If an object is not the object tested, mock or stub it.
In that way, you can directly control it so it returns the data that you want.
If you create a test object/object-graph, you have to set it up so that it provides the data that you want. That is probably a lot more work than you want.
For integration tests, of course you'd test a whole object graph at a time.
If you need to write a lot of initialization code - a mocking framework would probably help you write better, easy to understand Unit Tests.
There is no need to re write code that a mocking framework can save you.