I have a question about how to test 2 classes which have a dependency between each other, but cannot be unit tested together. I am not able to create a short enough code example so I will attempt to explain it with hypothetical classes. Sorry if this is too confusing, but I am still trying to come up with a succinct example.
In this example we are concerned with 4 different classes: ObjectCreator, ObjectUser1, ObjectUser2 and Object. In the unit tests for ObjectUser1 and ObjectUser2 each explicitly construct an instance of Object to be used. ObjectCreator also has unit tests to ensure the "correct" construction of an instance of Object. All the unit tests pass successfully. Now, a bug in ObjectUser1 is found which was not exposed in the unit tests. This bug is due to an incompatibility between how ObjectCreator creates the object and how ObjectUser1 makes use of it. This bug does not exist in ObjectUser2. The fix for the bug is to change the way ObjectCreator will construct the object. The problem I am facing is that when I change ObjectCreator all the unit tests pass again, but I know that the new change has broken ObjectUser2 (The unit tests pass for ObjectUser2 because of the way Object is constructed for the unit test). Let's assume that I do not want to use ObjectCreator in the unit tests for ObjectUser1 and ObjectUser2 for complexity reasons.
Is this a common problem in unit testing? What is the best way to test the dependency if they can't be unit tested together? Any help on this would be appreciated.
When testing class B, Mock A.
So you will get the results you want from A. But B will still depend on it.
You'll be able to unit test as you want. And you keep your dependency.
Please see: The Purpose of Mocking
It looks like you have not adequately split the dependency logic from the application logic as Object A creates Object B. You want to chop that out so A has a reference to a provider of some sort that creates Object Bs.
The route I would take would be to use Dependency Injection to separate the dependency construction logic from the application logic. However switching to a DI framework may be a large undertaking for your project. You have other options available to you, include a simple static factory method, as long as you add hooks into the method to force it to return what you need it to return.
Once the two have been separated it is a simple case of mocking Object B.
It sounds to me like your objects are too tightly coupled, and could do with being separated. Could you be reusing Object in a way that makes things difficult to decouple, and perhaps need two separate objects?
Related
How are mock objects better than real objects in unit testing?
When you create a mock object, what exactly it means? For example let's say you create
A a = Mockito.mock(A.class);
Does it create an object of class A? or something else?
When you create a mocked object and use it in calling a method, say a.method1(), how does the mocked object knows that it should call method1 itself?
Kindly let me know about these.
What is a unit-test
A unit-test suggests you only want to see your functionality in a small environment, typically a class. But this class often talks to other classes and those talk to other classes... which can mean a lot of work, just to get this one class going. Also, you just want to test your unit. This is, where mocks/stubs come in. I will explain the distinction later.
What you do, is - instead of recreating your whole app - to make assumptions. A mock is basically an assumption. Everything your class needs, is being mocked or stubbed. This can also be the case for already tested parts of your unit. You assume, then you test one specific (real) part of your unit. Then mock this one, if necessary, and so on.
Now what is the difference between mocks and stubs.
Both are assumptions, but mocks are also expectations. So, if you use a mock, your testing-framework should complain, if it hasn't been used/called.
A stub on the other hand, is just for the case. Like a random call to the Internet, lookups to database, which are not important for the test per se.
What your framework actually creates, depends on the framework and the language. But it will/should not call the original method, as explained above. Usually you'd have to tell this explicitly to the framework, which sometimes might make sense.
My class has a dependency which I mocked in my unit tests. I was getting a null reference exception in a place where it didn't make much sense to me.
I just realised it was because I didn't set up my mocked dependency. This dependency is tested but it doesn't connect to anything like file system or data sources.
I only wanted to test my new code in this new class but I guess in this case it is better not to mock at all.
Is this conclusion correct?
Correct. You should mock things that depend on anything persistent or external in order to prevent the test from depending on anything persistent or external.
If the dependency does not depend on anything persistent or external, the only benefit you gain from mocking it is that the test will work properly even if the dependency is wrong - but that's assuming the mock works properly. For that you need to either:
Write a mock that emulates the dependency completely.
Write a mock that emulates the dependency for the specific cases that will be used in the test.
The first option is outright ridiculous - why should your mock be any better than the original dependency? After all, it's safe to assume much more effort was invested in the original dependency than in the mock...
The second option means your mock knows the exact details of the implementation - otherwise you won't know how the implementation uses the dependency so you don't know how to mock those specific uses. This means the test does not serve one of the main purposes of unit tests - verifying that the code works properly after changes to the implementation.
The drawbacks of mocking are too big and the advantages are too small - especially considering that you can always run the dependency's tests to check if it works properly...
It sounds like it's working in your case, but it's not true in general that the only reason to stub or mock is to get external data sources and such out of your test's hair. Other reasons include
a method might be slow all by itself
it may be difficult to come up with the right parameters to pass to another class's method to get it to return the value that you need for a test of your method
the method you're testing needs an instance of another class which changes a lot, and you don't want your tests of the caller to break when the callee changes
the method you depend on is complicated enough to need its own tests, fully testing your method without stubbing or mocking means fully testing the methods that it calls (within your method's own tests), and this results in duplication between tests of your class and tests of the methods it calls
you're doing TDD from the outside in and you haven't written the callee yet, just designed its interface!
It depends: can you consider the dependency to actually just be a private implementation detail? If so, then mocking it only makes your test more brittle.
However, if it's a dependency that is actually injected into your SUT, then it absolutely should be replaced with a test double in your unit test.
The SUT should be the only interesting thing in your test. Everything else should be rote, boring, and constant, in order to ensure your SUT is operating in optimal conditions (for the scenario being tested).
Do I only have to mock out external dependencies in a unit test?
What if my method that I want to test, has a dependency on another class within the same assembly? Do I have to mock out the dependency for going sure to test only one thing and there for to make a unit test instead of an integration test?
Is an integration test a test that tests dependencies in general or do I have to difference between internal and external dependencies?
An example would be a method that has 2000 lines of code with 5 method invocations (all methods coming from the same assembly).
Generally a proper unit test is testing only that single piece of code. So a scenario like this is where you start to ask yourself about the coupling of these two classes. Does Class A internally depend on the implementation of Class B? Or does it just need to be supplied an instance of Type B (notice the difference between a class and a type)?
If the latter, then mock it because you're not testing Class B, just Class A.
If the former, then it sounds like creating the test has identified some coupling that can (perhaps even should) be re-factored.
Edit: (in response to your comment) I guess a key thing to remember while doing this (and retro-fitting unit tests into a legacy system is really, really difficult) is to mentally separate the concepts of a class and a type.
The unit tests are not for Class A, they are for Type A. Class A is an implementation of Type A which will either pass or fail the tests. Class A may have an internal dependency on Type B and need it to be supplied, but Type A might not. Type A is a contract of functionality, which is further expressed by its unit tests.
Does Type A specify in its contract that implementations will require an instance of Type B? Or does Class A resolve an instance of it internally? Does Type A need to specify this, or is it possible that different implementations of Type A won't need an instance of Type B?
If Type A requires an instance of Type B, then it should expose this externally and you'd supply the mock in your tests. If Class A internally resolves an instance of Type B, then you'd likely want to be using an IoC container where you'd bootstrap it with the mock of Type B before running the tests.
Either way, Type B should be a mock and not an implementation. It's just a matter of breaking that coupling, which may or may not be difficult in a legacy system. (And, additionally, may or may not have a good ROI for the business.)
Working with a code base you're describing isn't easy with multiple problems combined into something you don't know how to start changing. There are strong dependencies between classes as well as between problems and maybe even no overall design.
In my experience, this takes a lot of effort and time as well as skill in doing this kind of work. A very good resource to learn how to work with legacy code is Michael Feather's book: Working Effectively with Legacy Code.
In short, there are safe refactorings you can do without risking to break things, which might help you get started. There are also other refactorings which require tests to protect how things work. Tests are essential when refactoring code. This doesn't of course come with a 100% guarantee that things don't break, because there might be so many hidden "features" and complexity you cannot be aware of when you start. Depending on the code base the amount of work you need to do varies greatly, but for large code bases there is usually a lot of work.
You'll need to understand what the code does, either by simply knowing it or by finding out what the current code does. In either case, you start by writing "larger" tests which are not really unit tests, they just protect the current code. They might cover larger parts, more like integration/functional tests. These are your guards when you start to refactor the code. When you have such tests in place and you feel comfortable what the code does, you can start refactoring the parts the "larger" tests cover. For the smaller parts you change you write proper unit tests. Iterating doing various refactorings will at some point make the initial large tests unnecessary because you now have a much better code base and unit tests (or you simply keep them as functional test).
Now, coming back to your question.
I understand what you mean with your question, but I'd still like to change it slightly because there are more important aspects than external and internal. I believe a better question is to ask which dependencies do I need to break to get a better design and to write unit tests?
The answer to this question is you should break all dependencies you are not in control over, slow, non-deterministic or pulls in too much state for a single unit test. These are for sure all external (filesystem, printer, network etc.). Also note that multi-threading is not suitable for unit tests because this is not deterministic. For internal dependencies I assume you mean classes with members or functions calling other functions. The answer to this is maybe. You need to decide if you are in control and if the design is good. Probably in your case you are not in control and the code is not good, so here you need to refactor things to get things under control and into a better design. Michael Feather's book is great here, but you need to find how to apply the things on your code base of couse.
One very good technique for breaking dependencies is dependency injection. In short, it changes the design so that you pass in the members a class uses instead of letting the class itself instantiate them. For these you have an interface (abstract base class) for these dependencies you pass in, so you can easily change what you pass in. For instance, using this you can have different member implementations for a class in production and when you do unit test. This is a great technique and also leads to good design if use wisely.
Good luck and take your time! ;)
Generally speaking, a method with 2000 lines of code is just plain BAD. I usually start to look for reasons to make new classes -- not even methods, but classes -- when i have to use the pagedown key more than three or four times to browse through it (and collapsable regions doesn't count).
So, yes you do need to get rid of dependencies from outside and inside of the assembly, and you need to think of responsibility of the class. It sounds like this one has way too much weight on its shoulders, and it sounds like it is very close to impossible to write unittests for. If you think testability, you will automatically start to inject dependencies, and downsize your classes, and BAM!!!There you have it; nice and pretty code!! :-)
Regards,
Morten
I have a project I am trying to learn unit testing and TDD practices with. I'm finding that I'm getting to quite confusing cases where I am spending a long time setting up mocks for a utility class that's used practically everywhere.
From what I've read about unit testing, if I am testing MyClass, I should be mocking any other functionality (such as provided by UtilityClass). Is it acceptable (assuming that UtilityClass itself has a comprehensive set of tests) to just use the UtilityClass rather than setting up mocks for all the different test cases?
Edit: One of the things I am making a lot of setup for.
I am modelling a map, with different objects in different locations. One of the common methods on my utility class is GetDistanceBetween. I am testing methods that have effects on things depending on their individual properties, so for example a test that selects all objects within 5 units of a point and an age over 3 will need several tests (gets old objects in range, ignores old objects out of range, ignores young objects in range, works correctly with multiples of each case) and all of those tests need setup of the GetDistanceBetween method. Multiply that out by every method that uses GetDistanceBetween (almost every one) and the different results that the method should return in different circumstances, and it gets to be a lot of setup.
I can see as I develop this further, there may be more utility class calls, large numbers of objects and a lot of setup on those mock utility classes.
The rule is not "mock everything" but "make tests simple". Mocking should be used if
You can't create an instance with reasonable effort (read: you need a single method call but to create the instance, you need a working database, a DB connection, and five other classes).
Creation of the additional classes is expensive.
The additional classes return unstable values (like the current time or primary keys from a database)
TDD isn't really about testing. Its main benefit is to help you design clean, easy-to-use code that other people can understand and change. If its main benefit was to test then you would be able to write tests after your code, rather than before, with much of the same effect.
If you can, I recommend you stop thinking of them as "unit tests". Instead, think of your tests as examples of how you can use your code, together with descriptions of its behaviour which show why your code is valuable.
As part of that behaviour, your class may want to use some collaborating classes. You can mock these out.
If your utility classes are a core part of your class's behaviour, and your class has no value or its behaviour makes no sense without them, then don't mock them out.
Aaron Digulla's answer is pretty good; I'd rephrase each of his answers according to these principles as:
The behaviour of the collaborating class is complex and independent of the behaviour of the class you're interested in.
Creation of the collaborating class is not a valuable aspect of your class and does not need to be part of your class's responsibility.
The collaborating class provides context which changes the behaviour of your class, and therefore plays into the examples of how you can use it and what kind of behaviour you might expect.
Hope that makes sense! If you liked it, take a look at BDD which uses this kind of vocabulary far more than "test".
In theory you should try to mock all dependencies, but in reality it's never possible. E.g. you are not going to mock the basic classes from the standard library. In your case if the utility class just contains some basic helper methods I think I wouldn't bother to mock it.
If it's more complicated than that or connects to some external resources, you have to mock it. You could consider creating a dedicated mock builder class, that would create you a standard mock (with some standard stubs defined etc), so that you can avoid mocking code duplication in all test classes.
No, it is not acceptable because you are no longer testing the class in isolation which is one of the most important aspects of a unit test. You are testing it with its dependency to this utility even if the utility has its own set of tests. To simplify the creation of mock objects you could use a mock framework. Here are some popular choices:
Rhino Mocks
Moq
NSubstitute
Of course if this utility class is private and can only be used within the scope of the class under test then you don't need to mock it.
Yes, it is acceptable. What's important is to have the UtilityClass thoroughly unit tested and to be able to differentiate if a test is failing because of the Class under test or because of the UtilityClass.
Testing a class in isolation means testing it in a controlled environment, in an environment where one control how the objects behave.
Having to create too many objects in a test setup is a sign that the environment is getting too large and thus is not controlled enough. Time has come to revert to mock objects.
All the previous answers are very good and really match with my point of view about static utility classes and mocking.
You have two types of utilities classes, your own classes you write and the third party utility classes.
As the purpose of an utility class is to provide small set of helper methods, your utility classes or a third party utility classes should be very well tested.
First Case: the first condition to use your own utility class (even if static) without mocking, is to provide a set of valid unit tests for this class.
Second Case: if you use a third party utility library, you should have enough confidence to this library. Most of the time, those libraries are well tested and well maintained. You can use it without mocking its methods.
I have written my units tests, and where external resources are needed it is dealt with by using fakes.
All is good so far. Now i' am faced with the other test phases, mainly integration where i want to repeat the unit test methods against real external resources e.g The Database.
So, What are the recommendations for structuring test projects for Unit Vs Integration testing? I understand some people prefer separate assemblies for unit and Integration?
How would one share common test code between the two assemblies? Should i create a thrid assembly which contains all the Abstract Test Classes and let the unit and integration inherit? I am looking for maximum re-usability...
I hear alot of noise about Dependency Injection (StructureMap), How could one utilise such a tool in the given Unit + Integration test setup?
can anyone share some wisdom? Thanks
I don't think you should physically separate the two. A good solution is to put the Microsoft.TeamFoundation.PowerTools.Tasks.CategoryAttribute above your tests to identify regular and integration tests. When running tests (even with MSBuild) you can decide to run only the tests you're interested in.
Alternatively you can put them in seperate namespaces.
For code that will be executed in setup & teardown phases, the base class approach would work well. For integration tests, you can extract the functionality of your unit tests into well-parameterized non-test methods (preferably placed in another namespace) and call these "common" methods from both unit and integration tests. Putting unit tests, integration tests and common methods into separate namespaces would suffice, there would be no need for extra assemblies.
One approach would be to create a separate file with helper methods that would be used across multiple testing contexts, and then include that file in both your unit tests and your functional tests. For the parts that vary, you could use dependency injection - for example, by passing in different factories. In the unit tests, the factory could build a fake object, and in the functional tests it could insert a real object in to your test database.
Whether you split the tests into two projects or keep them in one might depend on the number of classes/tests you have. Too many classes in a single project would make it difficult to dig through. If you do split them out, helper/common methods could be thrown into a third assembly, or you could make them public in the unit test assembly, and let the integration assembly reference that one. Make things only as complex as you have to.
On our project we have both integration and unit tests together but in separate folders. Our project layout is such that we have separate assemblies for the main sections (Domain, Services, etc). Each assembly has a matching test assembly. Test assemblies are allowed to reference other test assemblies.
This means Services.Test can reference Domain.Test which makes sense to us because Services references Domain in the actual code.
In terms of reusable pieces we have
Builders - These provide a fluent interface for creating the most important/complex objects in our domain. These live in the main test folder for our domain. Our domain test assembly is referenced by all other test assemblies.
Mothers - These insert data into the database. They give back an Id for the inserted row which can be used to load the object if required. These live in the main test folder for our services.
Helpers - These are guys that do small things throughout our testing. For instance we prefer to allow access to collections via IEnuermable so we have a CollectionHelper.AssertCountIsEqualTo<_T>(int count, IEnumerable<_T> collection, string message) which wraps the IEnumerable in a List and asserts a count. Our Helpers all live in a common test which every other test references.
As for an IoC container if you can use one on your project they can be a huge help not only in testing (via auto mocking) but also in general development. With the overheard of registering everything with the contain though it might be a bit much for just testing.
After some experimenting this is how you can re-use test methods:
public abstract class TestBase
{
[TestMethod]
public void BaseTestMethod()
{
Assert.IsTrue(true);
}
}
[TestClass]
public class UnitTest : TestBase
{
}
[TestClass]
public class IntegrationTest : TestBase
{
}
The unit and integration test class will pick up the base class test methods and run them as two separate tests.
You should be able to create a parametised constructor on the base class to inject your mocks or resources.
I think this method can olny be used with classed within the same assembly. So it looks like the single assembly approach will have to do for now.
Thanks for the tips ppl!
If the only difference between many of your unit tests and the corresponding integration tests is that the latter use "real" ressources rather than fakes (mocks), one approach is the following:
Make a flag is_unit_test available to your test class from the outside
In the class setup, make fake or real resources available depending on the flag. For instance if you need to use a DB API that is either real (an instance of class DBreal) or fake (an instance of class DBfake), your initialization may look like if is_unit_test then this.dbapi = new DBfake else this.dbapi = new DBreal. (DBreal and DBfake need to conform to the same interface, let's call it DBapi.)
From the point of view of your test methods, step 2 amounts to (manual) Dependency Injection: The method does not know what class actually implements its dependency (the DB API). Rather, the dependency is injected into the method from the outside.
Where your test cases require the DB API, they use this.dbapi
Now you execute one and the same test class with the flag set for unit testing and without the flag set for integration testing. (How to make the flag available depends on your unit testing framework.)
Obviously, the same approach can be used if you need more than one resource in a test class.
Some people will find the explicit if in step 2 ugly. To make it more "elegant", you could employ an Inversion of Control (IoC) container (in Java for instance Spring or PicoContainer) to semi-automate the Dependency Injection instead. The initialization would then look like this.dbapi = myContainer.create(DBapi).
In simple cases, an IoC container will only complicate things, because configuring the container is not trivial, involves learning, opens the possibility of a new class of mistakes, and involves additional files.
In more complex cases however, the container makes things easier, because if the creation of your resources requires still other resources, the container will take care of their initialization as well and complexity would indeed go down. But unless you really get there, I suggest to KISS.
Unless you have an important reason for separate assemblies, they violate KISS. I suggest to wait for that reason first.
(Note that some people may tell you that Dependency Injection is only done at the class level.
I consider this unwarranted dogmatism. Injection simply means that a caller does not know the exact class it is calling, no matter how it obtained the object. It often becomes more useful when applied at the class level, but depending on your test framework this may make things overly complicated in the above case. Note that some test frameworks have their own injection capabilities, though.)