I am new to unit testing, but tend to think that I believe in beautifully written code, and properly designed architectures.
My question is. Aren't unit tests focusing too much on dependencies between objects? What do you do when your unit test fails because a dependency your method used to call befor is no longer called (a design decision) or your method calls another method or a dependency (again a design decision) Do you redesign your tests? If that's the case, then unit testing helps very little to reduce couple and improve cohesion between components.
Maybe my opinion is too broad, but in general how do people treat dependencies in properly mannered unit tests. I guess that the best way would be to have no dependencies at all, and every method relied on the parameters that were given to it, but this is hardly the case in reality. In addition, faking every dependency method for every possible call is also a bit subjective and time wasting, because at a future point in time, the class under test may simply no longer need the dependency.
I would suggest that you look at Test Driven Development (TDD) as I believe this technique will help you with your design issues. By writing unit tests before writing the production code, you will need to think about how to make your production code testable. This is better then the test later approach, where you write the production code first and then try to shoe-horn tests around them.
To deal with dependencies, think about what dependencies are causing you problems.
External Dependencies
If your tests use an external resource, such as a file, then you are writing an integration test, not a unit test. I've written many tests that use an external file, and I simply created a copy of the file in my test project. This file copy will contain dummy data required for my tests.
If your test requires a database, then again your writing an integration test. Personally I create a local copy of the database on my PC and run my tests against it.
Object Dependencies
If you are worried about code dependencies (e.g. your test will fail if a private method's signature is changed) then you are testing at the wrong level of abstraction. By that I mean make sure that your tests are calling public API's and not private ones. To cement this point, use interfaces for your objects to ensure an expected contract for an object that implements it.
I would also recommend that you try using a mocking framework such as RhinoMocks, Moq or TypeMock
A mocking framework will help you remove the dependency on, for example, having a database available for your tests. I personally use TypeMock, it's not cheap but it's by far the most powerful tool out there.
If you are talking about Unit testing you have no dependencies, cause a unit test tests only a single class (Java, C++, Ruby, Python). What you are talking about sounds more like integration testing which is different. Furthermore if you have to much dependencies your coupling is to high which is not very good, but of course not always avoidable.
Unit tests shall test the behavior, not the implementation. That way, one can rely on the unit tests when changing the implementation, or when refactoring the code. Removing a dependency (via inlining the class for instance), does not break the test.
Testing the implementation leads to brittle tests, that gets in the way when refactoring.
Related
Sometime back I used mocks in TDD (Test Driven Development) when the implementation of dependent interfaces was not available. So I mocked those dependencies and tested my work.
Now implementations of the dependencies are available. So should I remove the mock or should I convert them to spy or similar? I think integration tests can only be done after removing those mocks, or I may have to duplicate them. I am little confused, any suggestion on this?
As far as unit tests are concerned - never. Usually it is because you want to keep your unit tests isolated:
Your unit tests (just as any other classes) should usually have single reason to change/break. When a bug happens you want to reduce possible number of offenders to absolute minimum. Having unit test failing because some dependency might have failed, or some other dependency might have failed, or maybe finally actual tested class might have failed is undersirable.
In reality, this is quite a strech because even changing an interface of a dependency will result in changes in unit tests of class using this dependency. What you want to avoid tho, is some guy from some other team changing implementation of some dependency you are using and as a result your tests break, apparently for no reason. This is again, highly undesirable.
Mocks should be used to isolate your units. Of course, when writing different types of tests (say integration) you don't want isolation. You want real components collaborating together. This is one of the reasons to use actual implementations rather than mocks - but! - this does not invalidate your unit tests. They should remain isolated.
As always: It depends.
If you are using a mock, you are testing against a defined interface, so your unit test is more focused and there could be less breaking changes in your tests (such as when something internal needs to be refactored).
If you are using the actual dependencies, then your unit tests are not as focused (they might be integration tests). So these tests are often slower, more difficult to set up and more prone to breaking when refactoring. On the other hand, since you are testing against the actual implementation, you are more likely to find bugs due to the actual implementation behaving differently than you expected.
At my office we have a dispute regarding the necessity of unit tests in addition to integration tests for the classes that have the main responsibility of interacting with a filesystem (DB, etc).
The integration tests we have, are almost unit tests, as the tested object doesn't interact with other objects at all. The only reason, why we call the tests integration, is that the real filesystem is used in tests. And it is proposed to make the tested class use filesystem layer component, then mock it in tests (so we will call them unit tests), and check interaction with this component, rather than real filesystem results. Necessity of this change is what we discuss.
One point of view we have, is that unit tests are always required, because:
Writing unit tests makes code much better
Having unit tests, you don't need to care about real filesystem and side-effects of files, appearing at wrong locations
A developer can fully test results by making the tested class use filesystem mock and setting proper expectations for that mock
It is ok to tie the mock expectations to the specific internal algorithm of the tested class, because we do white-box testing with unit tests
Thus, the unit tests must always be written for such a tested class. And a filesystem layer component must always be used by that class for the purpose of testing.
Another point of view, is that unit tests are not needed for specific edge cases of classes that are devoted to filesystem interactions, because:
It is not possible to properly verify, that a tested class works, just by having simple mock instead of real filesystem (or its full emulation). Filesystem is such a complex component, that:
A tested class can work in many different ways in order to achieve successful result. The mock expectations cover just one-two possible scenarios, so a unit test erroneously shows failures for a class that properly implements good algorithm, which is different from the one expected.
A tested class can work in a way, detected by mock as a successful scenario, while the class still does not produce right result. This can be because of quite complex reasons in real filesystem. All these reasons are impossible to be covered just by a mock.
A unit test with mock and expectations is very fragile, because it is very tied to the tested class's internal algorithm. And the test erroneously fails upon even right changes of the algorithm.
The integration testing is a proper and full replacement for unit testing for a case, when the class has just 1-2 public methods, and only dependency is a filesystem. Integration testing gives same benefits as unit testing for this case - clear dependencies, more readable code, etc.
Thus the unit testing with filesystem mocking is not needed in our case. It is fragile and is not accurate for this particular case of classes.
So, to sum it up, the question is:
Is integration testing fully enough for an edge case of having not a complex class, which has main responsibility to work with a filesystem (DB, etc.)?
The only difference between integration and unit tests for this class, is that with unit tests the filesystem mock would be used (class would be fully isolated), while with integration tests the real filesystem is used.
I would appreciate, if you can add the references to classic books, or maybe articles / presentations of well-known industry people, so we can have a really strong ground to support the resulting conclusion.
The short answer here is yes, you could fully test a class with 'integration' tests. The better question, though, is should you do so?
I think you're getting too hung up on the difference in definitions between a 'unit test' (no outside dependencies) and an 'integration test' (has such dependencies). The goal with testing is to give you confidence that your code is working at all times, while keeping the associated costs of having that confidence down. So your question
Is integration testing fully enough for an edge case of having not a
complex class, which has main responsibility to work with a filesystem
(DB, etc.)?
is somewhat incomplete.
The most useful part of that distinction between 'unit' and 'integration' for our discussion is this: unit tests are easier and cheaper to write, maintain, and run.
To write a unit test, you just need to know the code. If a unit test fails, you know it's because of changes to the code. Writing an integration test requires setting up dependencies, eg creating files with specific contents, inserting rows in to a database, etc. If an integration test fails, it could be your code, or it could be your dependencies. For these reasons and others, integration tests are more complex, and therefore expensive, to create, maintain and run.
That increased expense should push the developer to separate classes encapsulating business logic from classes that handle interaction with outside systems, in an effort to minimize the number of integration tests required. The business logic can be tested with unit tests, which are cheaper.
Edit
It is possible that your class has complicated logic that itself is complicated because it has to handle complicated behavior in the underlying external dependency (ie, the file system in question). In that case, mocking the file system may be quite difficult in itself, and it may be cheaper/easier to just use a properly set up file system and write 'integration' tests.
The important point to keep in mind is what you're trying to achieve: confidence at a acceptable cost. If 'integration' tests are cheap enough, great. If you can get the same confidence cheaper using 'unit' tests, even better. The exact mix depends on the problem at hand.
It would be preferable to have a known state of the filesystem or DB for the tests. As an example, you do not want to have a test fail because it is trying to insert a record that already exists. This failure is not due to the code but a problem with the DB. Same thing can happen in the filesystem. However, you should write the best test that you are able to. If you can't easily mock the filesystem or whatever, then interact with it. Just realize that if the test fails it may not be a problem with the code.
An ugle test is better than no test. --The Way of Testivus
http://www.artima.com/weblogs/viewpost.jsp?thread=203994
Now even if you do have tests with mocks, that does not mean that you should not have QA or some sort of integration test to make sure that everything connects correctly. I view that unit tests are for verifying that the internals of the code works correctly and integration tests tell me that all the pieces work together.
I don't know what language you are using but the documentation for PHPUnit gives some ideas about testing the DB and filesystem.
http://www.phpunit.de/manual/current/en/database.html
http://www.phpunit.de/manual/current/en/test-doubles.html#test-doubles.stubbing-and-mocking-web-services
http://www.phpunit.de/manual/current/en/test-doubles.html#test-doubles.mocking-the-filesystem
A unit test with mock and expectations is very fragile, because it is
very tied to the tested class's internal algorithm. And the test
erroneously fails upon even right changes of the algorithm.
For testing with mocks, you should not be tying to the algorithm. All that you are testing for is the expected behavior of the class. Not how it goes about doing it.
I've used unit tests successfully for a while, but I'm beginning to think they're only useful for classes/methods that actually perform a fair amount of logic - parsers, doing math, complex business logic - all good candidates for testing, no question. I'm really struggling to figure out how to use testing for another class of objects: those which operate mostly via delegation.
Case in point: my current project coordinates a lot of databases and services. Most classes are just collections of service methods, and most methods perform some basic conditional logic, maybe a for-each loop, and then invoke other services.
With objects like this, mocks are really the only viable strategy for testing, so I've dutifully designed mocks for several of them. And I really, really don't like it, for the following reasons:
Using mocks to specify expectations for behavior makes things break whenever I change the class implementation, even if it's not the sort of change that ought to make a difference to a unit test. To my mind, unit tests ought to test functionality, not specify "the methods needs to do A, then B, then C, and nothing else, in that order." I like tests because I am free to change things with the confidence that I'll know if something breaks - but mocks just make it a pain in the ass to change anything.
Writing the mocks is often more work than writing the classes themselves, if the intended behavior is simple.
Because I'm using a completely different implementation of all the services and component objects in my test, in the end, all my tests really verify is the most basic skeleton of the behavior: that "if" and "for" statements still work. Boring. I'm not worried about those.
The core of my application is really how all the pieces work together, so I'm considering
ditching unit tests altogether (except for places where they're clearly appropriate) and moving to external integration tests instead - harder to set up, coverage of less possible cases, but actually exercise the system as it is mean to be run.
I'm not seeing any cases where using mocks is actually useful.
Thoughts?
If you can write integration tests that are fast and reliable, then I would say go for it.
Use mocks and/or stubs only where necessary to keep your tests that way.
Notice, though, that using mocks is not necessarily as painful as you described:
Mocking APIs let you use loose/non-strict mocks, which will allow all invocations from the unit under test to its collaborators. Therefore, you don't need to record all invocations, but only those which need to produce some required result for the test, such as a specific return value from a method call.
With a good mocking API, you will have to write little test code to specify mocking. In some cases you may get away with a single field declaration, or a single annotation applied to the test class.
You can use partial mocking so that only the necessary methods of a service/component class are actually mocked for a given test. And this can be done without specifying said methods in strings.
To my mind, unit tests ought to test
functionality, not specify "the
methods needs to do A, then B, then C,
and nothing else, in that order."
I agree. Behavior testing with mocks can lead to brittle tests, as you've found. State-based testing with stubs reduces that issue. Fowler weighs in on this in Mocks Aren't Stubs.
Writing the mocks is often more work
than writing the classes themselves
For mocks or stubs, consider using an isolation (mocking) framework.
in the end, all my tests really verify
is the most basic skeleton of the
behavior: that "if" and "for"
statements still work
Branches and loops are logic; I would recommend testing them. There's no need to test getters and setters, one-line pure delegation methods, and so forth, in my opinion.
Integration tests can be extremely valuable for a composite system such as yours. I would recommend them in addition to unit tests, rather than instead of them.
You'll definitely want to test the classes underlying your low-level or composing services; that's where you'll see the biggest bang for the buck.
EDIT: Fowler doesn't use the "classical" term the way I think of it (which likely means I'm wrong). When I talk about state-based testing, I mean injecting stubs into the class under test for any dependencies, acting on the class under test, then asserting against the class under test. In the pure case I would not verify anything on the stubs.
Writing Integration Tests is a viable option here, but should not replace Unit Tests. But since you stated your writing mocks yourself, I suggest using an Isolation Framework (aka Mocking Framework), which I am pretty sure of will be available for your environment too.
Being that you've posted several questions in one I'll answer them one by one.
How do I write useful unit tests for a mostly service-oriented app?
Do not rely on unit tests for a "mostly service-oriented app"! Yes I said that in a sentence. These types of apps are meant to do one thing: integrate services. It's therefore more pressing that you write integration tests instead of unit tests to very that the integration is working correctly.
I'm not seeing any cases where using mocks is actually useful.
Mocks can be extremely useful, but I wouldn't use them on controllers. Controllers should be covered by integration tests. Services can be covered by unit tests but it may be wise to have them as separate modules if the amount of testing slows down your project.
Thoughts?
For me, I tend to think about a few things:
What is my application doing?
How expensive would it be to perform system level / integration tests?
Can I split my application up into modules that can be tested separately?
In the scenario you've provided, I'd say your application is an integration of many services. Therefore, I'd lean heavily on integration tests over unit tests. I'd bet most of the Mocks you've written have been for http related classes etc.
I'm a bigger fan of integration / system level tests wherever possible for the following reasons:
In this day and age of "moving fast", re-factoring the designs of yesterday happens at an ever increasing rate. Integration tests aren't concerned about implementation details at all so this facilitates rapid change. Dynamic languages are in full swing making mocks even more dangerous / brittle. With a static lang, mocks are much safer because your tests won't compile if they're trying to stub out a non existent or misspelled method name.
The amount of code written in an integration test is usually 60% less than the amount of code written in a unit test to achieve the same level of coverage so development time is less. "Yes but it takes longer to run integration tests..." that's where you need to be pragmatic until it actually slows you down to run integration tests.
Integration tests catch more bugs. Mocking is often contrived and removes the developer from the realities of what their changes will do to the application as a whole. I've allowed way more bugs into production under the "safety net" of 100% unit test coverage than I would have with integration tests.
If integration testing is slow for my application then I haven't split it up into separate modules. This is often an indicator early on that I need to do some extracting into separation.
Integration tests do way more for you than reach code coverage, they're also an indicator of performance issues or network problems etc.
As far as I can see there are two ways, both with their drawbacks.
Get the object you are unit testing from the dependency injection system. This is low maintenance as you don’t have to manage anything when you change the framework around. But you are essentially testing the whole system from the point of view of your object, if a component fails it can blow up a lot of unit tests and it may not be apparent which one is failing.
is to manage the dependencies manually in the unit tests, and in some cases create test objects so that you can test each object in isolation. This keeps the unit tests discreet but dramatically increases the maintenance of the unit tests themselves. It also means that you don’t pick up on bugs cause by the way the objects interact on your live system.
Is either approach right or wrong? Should a compromise be used? Has anyone had any success stories either way.
If you're writing a unit test you should be using mocks for your dependencies and an IoC container shouldn't come into the picture. You should instantiate your class-under-test with mocks for the dependencies injected by hand.
If you're getting your object from the IoC container already wired up then what you're writing is integration tests which are very different.
Your goal for writing a unit test should be to write your test in isolation from the rest of the system.
You want to use an auto mocking container to manage the dependencies for you. There is one built into StructureMap and it's not that hard to roll your own.
The basic premise of an auto mocking container is that you set expectations on the dependencies you need to/care about and it creates mocks for any other dependencies for you. This means that if you add a dependency to a class you don't necessarily have to go back and alter every test for it; just those that will use the new dependency.
In my opinion the first approach is wrong as - as you you mentioned - it's not necessarily the object that you are testing that will cause a test to fail.
I do TDD, and I've been fairly loose in organizing my unit tests. I tend to start with a file representing the next story or chunk of functionality and write all the unit-tests to make that work.
Of course, if I'm introducing a new class, I usually make a separate unit-test module or file for that class, but I don't organize the tests themselves into any higher level structure. The result is I write code fast and I believe my actual program is reasonably well structured, but the unit tests themselves are "messy". Especially, their structure tends to recapitulate the phylogeny of the development process. Sometimes I see myself as trading laziness in the code for laziness in the tests.
How big a problem is this? Who here continually refactors and reorganizes their unit tests to try to improve their overall structure? Any tips for this? What should the overall structure of tests look like.
(Note, that I'm not so much asking the "how many assertions per function" question asked here : How many unit tests should I write per function/method? I'm talking about the bigger picture.)
Divide your tests in 2 sets:
functional tests
units tests
Functional tests are per-user story. Unit tests are per-class. The former check that you actually support the story, the latter exercise and document your functionality.
There is one directory (package) for functional tests. Unit tests should be closely bound with functionality they exercise (so they're scattered). You move them around and refactor them as you move & refactor your code around.
The less important part is organizing the tests.
I start by putting the tests into a class that relates to the class under test, so com.jeffreyfredrick.Foo has a test com.jeffreyfredrick.FooTest. But if some subset of those classes need a different setup then I'll move them into their own test class. I put my tests into a separate source directory but keep them in the same project.
The more important part is refactoring the tests.
Yes I try and refactor my tests as I go. The goal is to remove duplication while still remaining declarative and easy to read. This is true both within test classes and across test classes. Within a test class I might have a parametrized method for creating a test fake (mock or stub). My test fakes are usually inner classes within a test class but if I find there's need I'll pull them out for reuse across tests. I'll also create a TestUtil class with common methods when it seems appropriate.
I think refactoring yours tests is important to long term success of unit testing on large projects. Have you ever heard people complaining about how their tests are too brittle or preventing them from change? You don't want to be in a position where changing the behavior of a class means making dozens or even hundreds of changes to your tests. And just like with code, you achieve this through refactoring and keeping the tests clean.
Tests are code.
I write a unit test class for each class in the application, and keep the test classes organized in the same package structure as the classes under test.
Inside each test class I don't really have much organizational structure. Each one only has a handful of methods for each public method in the class under test, so I've never had any problem finding what I'm looking for.
For every class in the software, I maintain a unit test class. The unit test classes follow the same package hierarchy as the classes which are tested.
I keep my unit test code in a separate project. Some people also prefer to keep their test code in the same project under a separate source directory called 'test'. You could follow whatever feels comfortable to you.
I try to look at the unit tests as a project on their own. As with any project the organisation should follow some internal logic. It does not however have to be specific or formally defined - anything you're comfortable with is OK as long as it keeps your project well-organised and clean.
So for the unit tests I usually either follow the main project code structure or (sometimes when the situation calls of it) focus on the functional areas instead.
Leaving them in one heap is as you might imagine messy and difficult to maintain