Related
I'm working on a personal project (meaning clean source code, no legacy dependencies) and attempting to follow best practices regarding unit testing, dependency management, etc.
My company's codebase is littered with code like this:
public Response chargeCard(CreditCard card, Money amount) {
if(Config.isInUnitTests()) {
throw new IllegalStateException("Cannot make credit card API calls in unit tests!");
}
return CreditCardPOS.connect().charge(card, amount);
}
The goal here is to actively prevent dangerous code / code with external dependencies from being executed during testing. I like the notion of failing fast if unit tests do something bad, but I don't like this implementation for a few reasons:
It leaves hidden dependencies to the static Config class scattered throughout our codebase.
It changes the control flow between testing and live behavior, meaning we're not necessarily testing the same code.
It adds an external dependency to a configuration file, or some other state-holding utility.
It looks ugly :)
A fair number of the places where we use this in my company's codebase could be avoided by better dependency awareness, which I'm attempting to do, but there are still some places where I'm still struggling to see away around implementing an isInUnitTests() method.
Using the credit card example above, I can avoid the isInUnitTests() check on every charge by properly wrapping this up inside a mock-able CardCharger class, or something similar, but while cleaner I feel like I've only moved the problem up one level - how do I prevent a unit test from constructing a real instance of CardCharger without putting a check in the constructor / factory method that creates it?
Is isInUnitTests() a code smell?
If so, how can I still enforce that unit tests don't reach external dependencies?
If not, what's the best way to implement such a method, and what are good practices for when to use / avoid it?
To clarify, I'm trying to prevent unit tests from accessing unacceptable resources, like the database or network. I'm all for test-friendly patterns like dependency injection, but good patterns are useless if they can be violated by a careless developer (i.e. me) - it seems important to me to fail-fast in cases where unit tests do things they aren't supposed to, but I'm not sure the best way to do that.
Is isInUnitTests() a code smell?
Yes, definitively, you have many ways to avoid coupling your code to unit tests. There is no valid reason to have something like that.
If so, how can I still enforce that unit tests don't reach external dependencies?
Your tests must only validate a unit of code and create mock or stubs for external dependencies.
Your code seems to be Java, which it's plenty of mature mock frameworks. Research a little about existing ones and pick the one that likes you more.
EDIT
how do I prevent a unit test from constructing a real instance of HTTPRequest without putting a check in the constructor / factory method that creates it
You're supposed to use a dependency injector to resolve your instance dependencies so you will never need to use a if to test if you're on a test or not since on your "real" code you inject full functional dependencies and on your test you inject mock or stubs.
Take a more serious example, like a credit card charger (a classic unit testing example) - it would seem like a very poor design if it were even possible for test code to access the real card charger without triggering a deluge of exceptions. I don't think it's enough in a case like that to trust the developer will always do the right thing
Again, you're supposed to inject external dependencies as a credit card charger so on your test you would inject a fake one. If some developer don't know this, I think the first thing your company needs is training for that developer and some pair programming to guide the process.
In any case, kind of I get your point and let me tell you a similar situation I experienced.
There was an application that sent a bunch of emails after some processing. These emails were not to be send on any other environment except live, otherwise it would be a big problem.
Before I started working on this application, it happened several times that developers "forgot" about this rule and emails were sent on test environments, causing a lot of problems. It's not a good strategy to depends on human memory to avoid this kind of problem.
What we did to avoid this to happen again was adding a config setting to indicate if send real emails or not. The problem was broader than executing things or not on unit tests as you can see.
Nothing replace communication though, a developer could set the incorrect value for this setting on his development environment. You're never 100% safe.
So in short:
First line of defense is communication and training.
You second line of defense should be dependency injection.
If you feel that this is not enough, you could add a third line of defense: a configuration setting to avoid executing real logic on test/development environments. Nothing wrong about that. But please don't name it IsInUnitTest as the problem is broader than that (you want also o avoid this logic to be executed on developer machine)
The AngularJS Unit Testing Documentation actually does a really fantastic job of describing the different ways that developers handle unit testing.
Of the four different methods they outline, the one they recommend is one that involves using dependency injection so that your production code flow is the same as your testing flow. The only differences are that the objects you pass into your code would be different. Note that Claudio used the term "Stub" to refer to objects you pass into a method to act as placeholders for those you'd use in production:
public Document getPage(String url, AbstractJsoup jsoup) {
return jsoup.connect(url).get();
}
My Java is a little rusty, so consider that you may need to actually do some funky stuff to make this work, like check the type of the jsoup object or cast it to your test Jsoup object or your production Jsoup object, which might just defeat the entire purpose of using dependency injection. Thus, if you're talking about a dynamic language like JavaScript or some other loosely-typed functional programming language, dependency injection would keep things really clean.
But in Java, unless you control the dependencies and can use a design pattern, like the strategy pattern, to pass in different concrete implementations, doing what you're doing may be easier, especially since you don't control the Jsoup code. However, you might check to see if they have available stubs you can use as placeholders, as some library developers do write stubs.
If you don't own the code, another option could be to use a Factory class to obtain the desired objects, depending on a flag you set when first instantiating it. This seems like a less desirable solution, since you're still sort of setting a global flag in that Factory object that may have an effect on things you might not be trying to test. The advantage of dependency injection for unit testing is that you must explicitly pass in a test stub when testing and explicitly pass in the production object when you wish the method to do what it was written to do. Once the method completes its execution, the test is over, and any production process that invokes it will automatically run it in production mode because it will inject production objects.
I've never seen a system where steps were taken to actively prevent code running under unit tests from accessing external resources. The problem just never came up. You have my deepest sympathy for working somewhere where it has.
Is there a way you could control the classpath used for unit tests, to make the libraries needed to access external resources unavailable? If there is no JSoup and no JDBC driver on the classpath, tests for code which attempts to use them will fail. You won't be able to exclude JDK-level classes like Socket and URLConnection this way, but it might still be useful.
If you were running tests using Gradle, that would be fairly straightforward. If you are using Maven, maybe not. I don't think there's any way to have different classpaths for compilation and testing in Eclipse.
Well, you can achieve the same clean way using Abstract Factory Pattern though (maybe not suitable to call it Abstract Factory Pattern).
Example in C#:
public class CardChargerWrapper{
public CardChargerWrapper(
NullCardCharger nullCharger
, TestUnitConfig config){
// assign local value
this.charger = new CardCharger();
}
CardCharger charger;
NullCardCharger nullCharger;
TestUnitConfig config;
public Response Charge(CreditCard card, Money amount){
if(!config.IsUnitTest()) { return charger.connect().charge(card, amount); }
else { return NullCardCharger.charge(card, amount); }
}
}
EDIT: Changed CardChargerWrapper to use hard-coded instance of CardCharger instead of injecting it.
Note: You can change NullCardCharger to something like MockCardCharger or OfflineCardCharger for logging purpose.
Note again: You can change the CardChargerWrapper's constructor to fit. Example, instead of constructor injecting the NullCardCharger, you can make it property injected. Same with TestUnitConfig.
EDIT: Regarding calling IsUnitTest() a good idea or not:
It really depends on your business perspective and how you are doing testings. As many people said, a code that has not yet been tested is not trusted for their correctness. It cannot be reliabled. On side note, I prefer IsChargeRealCard() will be more suitable than IsUnitTest().
Say that we take out the unit test in our context, at least you will still need to do integration testing in test environment. You probably want to test something like:
I want to test the credit card validation (is it real or not, etc).
I want to test the payment method and see whether the card being charged. As a process, but not as a real credit card payment.
For 2nd point, I think the best thing to do is to create a mock credit card charger to log the transaction. This is, to ensure that the charging is correct. And it will be conducted in both test and dev server.
So, How can the CardChargerWrapper help such situation?
Now with CardChargerWrapper, you can:
Switch the NullCardCharger to any mock card chargers to enhance your unit testing.
All class using CardChargerWrapper can be ensured that they are checking IsUnitTest first before charging real card.
Your developer need to use CardChargerWrapper instead of CardCharger, preventing development error for unit tests.
During code review, you can find whether CardCharger being used in other class instead of CardChargerWrapper. This is to ensure no leak of code.
I'm unsure, but seems like you can hide references of your main project to your real CardCharger. This will protect your code further.
If [isInUnitTest() is an antipattern], how can I still enforce that unit tests don't reach external dependencies?
I now have a solution I'm fairly satisfied with, which will ensure external dependencies cannot be used in test environments without explicitly enabling them. All classes which depend on external resources (HTTP requests, databases, credit card processors, etc.) take as one of their arguments a configuration class which contains the necessary settings to initialize these objects. In a real environment, a real Config object is passed in containing the data they need. In a test environment, a mock is passed in, and without explicitly configuring the mock, the object will fail to construct/connect.
For instance, I have an Http connection utility:
public class HttpOp {
public HttpOp(Config conf) {
if(!conf.isHttpAllowed()) {
throw new UnsupportedOperationException("Cannot execute HTTP requests in "+
getClass()+" if HTTP is disabled. Did you mean to mock this class?");
}
}
....
}
In a unit test, if you attempted to run code that constructs an HttpOp object, you'd raise an exception as the mocked Config will not return true unless explicitly set to do so. In a functional test where this is desired, you can do so explicitly:
#Test
public void connect() {
State.Http httpState = mock(State.Http.class);
when(httpState.isEnabled()).thenReturn(true);
RemoteDataProcessor rdp = new RemoteDataProcessor(new HttpOp(httpState));
...
}
Of course, this still depends on Config being properly mocked in test environments, but now we have exactly one danger point to look for and reviewers can quickly verify the Config object is mocked and trust that only explicitly enabled utilities will be accessible. Similarly, there's now only one gotcha new team-members need to be told ("always mock Config in tests") and they can now be confident they won't accidentally charge a credit card or send emails to clients.
Update
This was an idea I had shortly after posting this question, however I'm now convinced it's not a good plan. Leaving it here for posterity, but see my newer answer for what I ended up doing.
I'm far from certain this is the right thing to do, but one thought I had, which at least addresses my first and third objections, comes from Determine if code is running as part of a unit test:
You could avoid storing external state about whether you're in a unit test or not by directly examining the execution stack, like so:
/**
* Determines at runtime and caches per-thread if we're in unit tests.
*/
public static class IsInUnitTest extends ThreadLocal<Boolean> {
private static final ImmutableSet<String> TEST_PACKAGES =
ImmutableSet.of("org.testng");
#Override
protected Boolean initialValue() {
for(StackTraceElement ste : Thread.currentThread().getStackTrace()) {
for(String pkg : TEST_PACKAGES) {
if(ste.getClassName().startsWith(pkg)) {
return true;
}
}
}
return false;
}
}
The primary advantage here is we don't store any state; we simply check the stack trace - if the trace contains a test-framework package, we're in a unit test, otherwise we're not. It's not perfect - in particular it could potentially lead to false positives if you use the same testing framework for integration or other more permissive tests - but avoiding external state seems like at least a slight win.
Curious what others think of this idea.
I would like to start doing more unit testing in my applications, but it seems to me that most of the stuff I do is just no suitable to be unit tested. I know how unit tests are supposed to work in textbook examples, but in real world applications they do not seem of much use.
Some applications I write have very simple logic and complex interactions with things that are outside my control. For instance I would like to write a daemon which reacts to signals sent by some applications, and changes some user settings in the OS. I can see three difficulties:
first I have to be able to talk with the applications and be notified of their events;
then I need to interact with OS whenever I receive a signal, in order to change the appropriate user settings;
finally all of this should work as a daemon.
All these things are potentially delicate: I will have to browse possibly complex APIs and I may introduce bugs, say by misinterpreting some parameters. What can unit testing do for me? I can mock both the external application and the OS, and check that given a signal from the application, I will call the appropriate API method on the OS. This is... well, the trivial part of the application.
Actually most of the things I do involve interaction with databases, the filesystem or other applications, and these are the most delicate parts.
For another example look at my build tool PHPmake. I would like to refactor it, as it is not very well-written, but I fear to do this as I have no tests. So I would like to add some. The point is that the things which may be broken by refactoring may not be caught by unit tests:
One of things to do is deciding which things are to be built and which one are already up to date, and this depends on the time of last modification of the files. This time is actually changed by external processes, when some build command is fired.
I want to be sure that the output of external processes is displayed correctly. Sometimes the buikd commands require some input, and that should be also managed correctly. But I do not know a priori which processes will be ran - it may be anything.
Some logic is involved in pattern matching, and this may seem to be testable part. But the functions which do the pattern matching use (ni addition to their own logic) the PHP function glob, which works with the filesystem. If I just mock a tree in place of the actual filesystem, glob will not work.
I could go on with more examples, but the point is the following. Unless I have some delicate algorithms, most of what I do involves interaction with external resources, and this is not suitable for unit testing. More that this, often this interaction is actually the non-trivial part. Still many people see unit testing as a basic tool. What am I missing? How can I learn be a better tester?
I think you open a number of issues in your question.
Firstly, when your application integrates with external environments such as OS, other threads, etc. then you have to separate (1) the logic that is tied in with the external enviroment and (2) your business-code.. that is, the stuff your application does. This is no different to how you would separate GUI and SERVER in an application (or web application).
Secondly, you ask if you should test simple logic. I'd say, it depends. Often simple fetch/store functionality is nice to have tests for. It's like the foundation of your application.. hence its important. Other business stuff built upon your foundation that is very simple, you may easily find yourself both feeling that you are wasting your time, and mostly you are :-)
Thirdly, refactory an existing program and testing it in its existing state may be a problem. If your PHP program produces a set of files on the basis of some input, well, maybe thats your entry point to tests are. Sure the tests may be high-level, but it's an easy way to ensure that after the refactoring, your program produces the same output. Hence, aim for higher-level tests in that situation in the start phase of your refactoring efforts.
I'd like to recommend some literature, but I can only come up with one title. "Working Effectively with Legacy Code" By Micheal Feathers. It's a good start. Another would be "xUnit Test Patterns: Refactoring Test Code" by Gerard Meszaros (although that book is much more sloppy and FULL of copy paste text).
As regards your issue about existing code bases that aren't currently covered by tests in which you would like to start refactoring, I would suggest reading:
Working Effectively with Legacy Code By Micheal Feathers.
That book gives you techniques on how to deal with the issues you might be facing with PHPMake. It provides ways to introduce seams for testing, where there previously weren't any.
Additionally, with code that touches say the file systems, you can abstract the file system calls behind a thin wrapper, using the Adapter Pattern. The unit tests would be against a fake implementation of the abstract interface that the wrapping class implements.
At some point you get to a low enough level where a unit of code can't be isolated for unit testing as these depend on library or API calls (such as in the production implementation of the wrapper). Once this happens integration tests are really the only automated developer tests you can write.
I recommend this google tech-talk on unit testing.
The video boils down to
write your code so that it knows as little about how it will be used as possible. The less assumptions your code makes, the easier it is to test. Avoid complex logic in constructors, the use of singletons, static class members, and so on.
isolate your code from the external world (comms, databases, real time), and make sure that your code only talks to your isolation layer. Otherwise, writing tests will be a nightmare in terms of 'fake environment' setup.
unit tests should test stories; that is what we really understand and care for; given a class with a method foo(), testFoo() is uninformative. They actually recommend test names like itShouldCloseConnectionEvenWhenExceptionThrown(). Ideally, your stories should cover enough functionality that you can rebuild the spec from the stories.
NOTE: the video and this post use Java as an example; however, the main points stand for any language.
"Unit tests" tests one unit of your code. No external tools should be involved. This seems to be complicated for your first app (without knowing to much about it ;)) but the phpMake is unit-testable - I'm sure ... because ant, gradle and maven are unit-testable too ;)!
But of course you can test your first application automated too. There are several different layers one could test an application.
So the task for you is to find an automated way to test your app - be it integration testing or whatever.
E.g. you could write shell scripts, which asserts some output! With that you make sure your application behaves correctly ...
Tests of interactions with external resources are integration tests, not unit tests.
Tests of your code to see how it would behave if particular external interactions had occurred can be unit tests. These should be done by writing your code to use dependency injection, and then, in the unit test, injecting mock objects as dependencies.
For example, consider a piece of code that adds the results of a call to one service to the results of a call to another service:
public int AddResults(IService1 svc1, IService2 svc2, int parameter)
{
return svc1.Call(parameter) + svc2.Call(parameter);
}
You can test this by passing in mock objects for the two services:
private class Service1Returns1 : IService1
{
public int Call(int parameter){return 1;}
}
private class Service2Returns1 : IService2
{
public int Call(int parameter){return 1;}
}
public void Test1And1()
{
Assert.AreEqual(2, AddResults(new Service1Returns1(), new Service2Returns1(), 0));
}
First of all, if unit testing doesn't seem like it would be much use in your applications, why do you even want to start doing more of it? What is motivating you to care about it? It is definitely a waste of time if a) you do everything perfect the first time and nothing ever changes or b) you decide it's a waste of time and do it poorly.
If you do think that you really want to do unit testing, the answer to your questions are all the same: encapsulation. In your daemon example, you could create a ApplcationEventObeservationProxy with a very narrow interface that just implements pass through methods. The purpose of this class is to do nothing but completely encapsulate the rest of your code from the third-party event observing library (nothing means nothing -- no logic here). Do the same thing for OS settings. Then you can completely unit test the class that does actions based on events. I'd recommend have a separate class for the daemon-ness that just wraps your main class -- it will make the testing easier.
There are a couple of benefits to this approach outside of unit testing. One is that if you encapsulate the code that interacts directly with the OS, it's easier to switch it out. This kind of code is particularly prone to breakage outside of your control (i.e., MS patchsets). You will also probably want to support more than one OS, and if the OS specific logic is not tangled with the rest of your logic, it will be easier. The other benefit is that you'll be forced to realize that there is more business logic in your app than you think. :)
Finally, don't forget that unit testing is a foundation for a good product, but not the only ingredient. Having a set of tests that explore and verify the OS API calls you'll be using is a good strategy for the "hard" parts of this problem. You should also have end to end tests that ensure the events in your applications cause the OS setting changes.
As other answers suggested Working Effectively with Legacy Code By Micheal Feathers is a good read. If you have to deal with legacy code, and you want to make sure that the systems interaction work as expected, try writing integration tests first. And then it is more appropriate to write Unit Tests to test the behaviour of methods that are valued from the requirements point of view. You Tests serve a whole different purpose than the integration tests. Unit Tests are more likely to improve the design of your system than testing how everything hangs to gather.
During a recent interview I was asked why one would want to create mock objects. My answer went something like, "Take a database--if you're writing test code, you may not want that test hooked up live to the production database where actual operations will be performed."
Judging by response, my answer clearly was not what the interviewer was looking for. What's a better answer?
I'd summarize like this:
Isolation - You can test only a method, independently on what it calls. Your test becomes a real unit test (most important IMHO)
Decrease test development time - it is usually faster to use a mock then to create a whole class just for help you test
It lets you test even when you don't have implemented all dependencies - You don't even need to create, for instance, your repository class, and you'll be able to test a class that would use this repository
Keeps you away from external resources - helps in the sense you don't need to access databases, or to call web services, or to read files, or to send emails, or to charge a credit card, and so on...
In an interview, I'd recommend including that mocking is even better when developers use dependency injection, once it allows you to have more control, and build tests more easily.
When unit testing, each test is designed to test a single object. However most objects in a system will have other objects that they interact with. Mock Objects are dummy implementations of these other objects, used to isolate the object under test.
The benefit of this is that any unit tests that fail generally isolate the problem to the object under test. In some cases the problem will be with the mock object, but those problems should be simpler to identify and fix.
It might be an idea to write some simple unit tests for the mock objects as well.
They are commonly used to create a mock data access layer so that unit tests can be run in isolation from the data store.
Other uses might be to mock the user interface when testing the controller object in the MVC pattern. This allows better automated testing of UI components that can somewhat simulate user interaction.
An example:
public interface IPersonDAO
{
Person FindById(int id);
int Count();
}
public class MockPersonDAO : IPersonDAO
{
// public so the set of people can be loaded by the unit test
public Dictionary<int, Person> _DataStore;
public MockPersonDAO()
{
_DataStore = new Dictionary<int, Person>();
}
public Person FindById(int id)
{
return _DataStore[id];
}
public int Count()
{
return _DataStore.Count;
}
}
Just to add on to the fine answers here, mock objects are used in top-down or bottom-up structural programming (OOP too). They are there to provide data to upper-level modules (GUI, logic processing) or to act as out mock output.
Consider top-down approach: you develop a GUI first, but a GUI ought to have data. So you create a mock database which just return a std::vector<> of data. You have defined the 'contract' of the relationship. Who cares what goes on inside the database object - as long as my GUI list get a std::vector<> I'm happy. This can go to provide mock user login information, whatever you need to get the GUI working.
Consider a bottom-up approach. You wrote a parser which reads in delimited text files. How do you know if it is working? You write a mock 'data-sink' for those object and route the data there to verify (though usually) that the data are read correctly. The module on the next level up may require 2 data sources, but you have only wrote one.
And while defining the mock objects, you have also define the contract of how the relationship. This is often used in test-driven programming. You write the test cases, use the mock objects to get it working, and often than not, the mock object's interface becomes the final interface (which is why at some point you may want to separate out the mock object's interface into pure abstract class).
Hope this helps
Mock objects/functions can also be useful when working in a team. If you're working on a part of the code base that depends on a different part of the code base that some else is responsible for - which is still being written or hasn't been written yet - a mock object/function is useful in giving you an expected output so that you can carry on with you're work without being held up waiting for the other person to finish their part.
Here are the a few situations where mocking is indispensable:
When you are testing GUI interaction
When you are testing Web App
When you are testing the code that interacts with hardware
When you are testing legacy apps
I will go a different direction here. Stubbing/Faking does all of the things mentioned above, but perhaps the interviewers were thinking of mocks as a fake object that causes the test to pass or fail. I am basing this on the xUnit terminology. This could have lead to some discussion about state testing verses behavior / interaction testing.
The answer they may have been looking for is: That a mock object is different than a stub. A stub emulates a dependency for the method under test. A stub shouldn't cause a test to fail. A mock does this and also checks how and when it is called. Mocks cause a test to pass or fail based on underlying behavior. This has the benefit of being less reliant on data during a test, but tying it more closely to the implementation of the method.
Of course this is speculation, it is more likely they just wanted you to describe the benefits of stubbing and DI.
To take a slightly different approach (as I think mocks have been nicely covered above):
"Take a database--if you're writing test code, you may not want that test hooked up live to the production database where actual operations will be performed."
IMO a bad way of stating the example use. You would never "hook it up to the prod database" during testing with or without mocks. Every developer should have a developer-local database to test against. And then you would move on test environments database then maybe UAT and finally prod. You are not mocking to avoid using the live database, you are mocking in order that classes that are not directly dependent on a database do not require you to set up a database.
Finally (and I accept I might get some comments on this) IMO the developer local database is a valid thing to hit during unit tests. You should only be hitting it while testing code that directly interacts with the database and using mocks when you are testing code that indirectly access the database.
I have written my units tests, and where external resources are needed it is dealt with by using fakes.
All is good so far. Now i' am faced with the other test phases, mainly integration where i want to repeat the unit test methods against real external resources e.g The Database.
So, What are the recommendations for structuring test projects for Unit Vs Integration testing? I understand some people prefer separate assemblies for unit and Integration?
How would one share common test code between the two assemblies? Should i create a thrid assembly which contains all the Abstract Test Classes and let the unit and integration inherit? I am looking for maximum re-usability...
I hear alot of noise about Dependency Injection (StructureMap), How could one utilise such a tool in the given Unit + Integration test setup?
can anyone share some wisdom? Thanks
I don't think you should physically separate the two. A good solution is to put the Microsoft.TeamFoundation.PowerTools.Tasks.CategoryAttribute above your tests to identify regular and integration tests. When running tests (even with MSBuild) you can decide to run only the tests you're interested in.
Alternatively you can put them in seperate namespaces.
For code that will be executed in setup & teardown phases, the base class approach would work well. For integration tests, you can extract the functionality of your unit tests into well-parameterized non-test methods (preferably placed in another namespace) and call these "common" methods from both unit and integration tests. Putting unit tests, integration tests and common methods into separate namespaces would suffice, there would be no need for extra assemblies.
One approach would be to create a separate file with helper methods that would be used across multiple testing contexts, and then include that file in both your unit tests and your functional tests. For the parts that vary, you could use dependency injection - for example, by passing in different factories. In the unit tests, the factory could build a fake object, and in the functional tests it could insert a real object in to your test database.
Whether you split the tests into two projects or keep them in one might depend on the number of classes/tests you have. Too many classes in a single project would make it difficult to dig through. If you do split them out, helper/common methods could be thrown into a third assembly, or you could make them public in the unit test assembly, and let the integration assembly reference that one. Make things only as complex as you have to.
On our project we have both integration and unit tests together but in separate folders. Our project layout is such that we have separate assemblies for the main sections (Domain, Services, etc). Each assembly has a matching test assembly. Test assemblies are allowed to reference other test assemblies.
This means Services.Test can reference Domain.Test which makes sense to us because Services references Domain in the actual code.
In terms of reusable pieces we have
Builders - These provide a fluent interface for creating the most important/complex objects in our domain. These live in the main test folder for our domain. Our domain test assembly is referenced by all other test assemblies.
Mothers - These insert data into the database. They give back an Id for the inserted row which can be used to load the object if required. These live in the main test folder for our services.
Helpers - These are guys that do small things throughout our testing. For instance we prefer to allow access to collections via IEnuermable so we have a CollectionHelper.AssertCountIsEqualTo<_T>(int count, IEnumerable<_T> collection, string message) which wraps the IEnumerable in a List and asserts a count. Our Helpers all live in a common test which every other test references.
As for an IoC container if you can use one on your project they can be a huge help not only in testing (via auto mocking) but also in general development. With the overheard of registering everything with the contain though it might be a bit much for just testing.
After some experimenting this is how you can re-use test methods:
public abstract class TestBase
{
[TestMethod]
public void BaseTestMethod()
{
Assert.IsTrue(true);
}
}
[TestClass]
public class UnitTest : TestBase
{
}
[TestClass]
public class IntegrationTest : TestBase
{
}
The unit and integration test class will pick up the base class test methods and run them as two separate tests.
You should be able to create a parametised constructor on the base class to inject your mocks or resources.
I think this method can olny be used with classed within the same assembly. So it looks like the single assembly approach will have to do for now.
Thanks for the tips ppl!
If the only difference between many of your unit tests and the corresponding integration tests is that the latter use "real" ressources rather than fakes (mocks), one approach is the following:
Make a flag is_unit_test available to your test class from the outside
In the class setup, make fake or real resources available depending on the flag. For instance if you need to use a DB API that is either real (an instance of class DBreal) or fake (an instance of class DBfake), your initialization may look like if is_unit_test then this.dbapi = new DBfake else this.dbapi = new DBreal. (DBreal and DBfake need to conform to the same interface, let's call it DBapi.)
From the point of view of your test methods, step 2 amounts to (manual) Dependency Injection: The method does not know what class actually implements its dependency (the DB API). Rather, the dependency is injected into the method from the outside.
Where your test cases require the DB API, they use this.dbapi
Now you execute one and the same test class with the flag set for unit testing and without the flag set for integration testing. (How to make the flag available depends on your unit testing framework.)
Obviously, the same approach can be used if you need more than one resource in a test class.
Some people will find the explicit if in step 2 ugly. To make it more "elegant", you could employ an Inversion of Control (IoC) container (in Java for instance Spring or PicoContainer) to semi-automate the Dependency Injection instead. The initialization would then look like this.dbapi = myContainer.create(DBapi).
In simple cases, an IoC container will only complicate things, because configuring the container is not trivial, involves learning, opens the possibility of a new class of mistakes, and involves additional files.
In more complex cases however, the container makes things easier, because if the creation of your resources requires still other resources, the container will take care of their initialization as well and complexity would indeed go down. But unless you really get there, I suggest to KISS.
Unless you have an important reason for separate assemblies, they violate KISS. I suggest to wait for that reason first.
(Note that some people may tell you that Dependency Injection is only done at the class level.
I consider this unwarranted dogmatism. Injection simply means that a caller does not know the exact class it is calling, no matter how it obtained the object. It often becomes more useful when applied at the class level, but depending on your test framework this may make things overly complicated in the above case. Note that some test frameworks have their own injection capabilities, though.)
I have finally got in my mind what worried me about Dependency Injection and similar techniques that should make unit tests easier. Let's take this example:
public interface IRepository { void Item Find(); a lot of other methods here; }
[Test]
public void Test()
{
var repository = Mock<IRepository>();
repository.Expect(x => x.Find());
var service = new Service(repository);
service.ProcessWithItem();
}
Now, what's wrong with the code above? It's that our test roughly peeks into ProcessWithItem() implementation. What if it wants to do "from x in GetAll() where x..." - but no, our test knows what is going to happen there. And that's just a simple example. Imaging few calls that our test now is tied with, and when we want to change from GetAll() to a better GetAllFastWithoutStuff() inside the method... our test(s) are broken. Please change them. A lot of crappy work that happens so often without any real need.
And that's what often makes me to stop write tests. I just don't see how I can test without knowing implementation details. And knowing them, tests are now very fragile and pain to do.
Sure, it's not about interface (or DI) only. POCOs (and POJOs, why not) also suffer from the same thing, but they're now tied with the data, not with the interface. But the principle is the same - our final assertion is tightly coupled with our knowledge of what our SUT is going to do. "Yes you HAVE to provide this field, sir, and this better be of this value".
As a consequence, tests ARE going to fail - soon and often. This is pain. And the problem.
Are there any techniques to deal with this? AutoMockingContainer (which basically takes care all ALL methods and nested DI hierarchies) looks promising, but with its own drawback. Anything else?
Dependency Injection, per se, would let you inject an implementation of IRepository that accepts whatever calls are made on it, checks that the invariants and preconditions are satisfied, and returns results satisfying the postconditions. When you choose to inject a mock object that has very specific expectations for what methods will be called, then yes, you're doing highly implementation-specific testing -- but Dependency Injection is totally innocent in the matter, since it never dictates WHAT you should inject; rather, your beef appears to be with Mocking -- in fact, specifically the somewhat-automated mocking approach that you have chosen to use, which is one based on very specific expectations.
Mocking with very specific expectations IS indeed useful for white-box testing only. Depending on the tools / frameworks / libraries you're using (and you're not even specifying the exact programming language in a tag, so I assume your question is totally open ended) you may be able to specify the degrees of freedom allowed (these calls are allowed to come in any orders, these arguments must only satisfy the following preconditions, etc, etc). However, I don't know of an automated tool to perform exactly what you need for opaque-box testing, which is the "generic, tolerant implementation of yonder interface with all the ''programming by contract'' checks that are needed and no other".
What I tend to do over the life of a project is to build up a library of "not quite mocks" for the major interfaces needed. In some cases those may be somewhat obvious from the start, but in other cases they emerge incrementally as I'm considering some major refactoring, as follows (typical scenario)...:
The early stages of the refactoring break some aspect of the fragile strong-expectations mocking that I have cheaply put in place initially, I ponder whether to just tweak the expectations or go whole hog, if I decide it's not a one-off (i.e. the return in future refactorings and tests will justify the investment) then I hand-code a good "not quite mock" and stash it away in the project's specific bag of tricks -- actually often reusable across projects; such classes/packages as MockFilesystem, MockBigtable, MockDom, MockHttpClient, MockHttpServer, etc etc, go into a project-agnostic repository and get reused for testing all kinds of future projects (and in fact may be shared with other teams across the company, if several teams are using filesystem interfaces, bigtable interfaces, DOMs, http client/server interfaces, etc etc, that are uniform across the teams).
I acknowledge that the use of the word "mock" may be slightly inappropriate here if you take "mock" to refer specifically to the precise-expectation style of "fake implementation for testing purposes" of interfaces. Maybe Stub, Shim, Fake, Test, or some other prefix yet might be preferable (I do tend to use Mock for historical reasons, except when I remember to specifically call it Fake or the like;-).
If I was using languages with clear and precise way to express in the language itself the various design-by-contract specs in an interface, I imagine I'd get automatic tool support for most of this faking/shimming/etc; however I mostly code in other languages so I have to do a bit more manual work here. But I think that's a separate issue.
I read the excellent book http://www.manning.com/rainsberger/.
I would like to provide some insight I gained from it.
I believe several advice could help you to reduce the coupling between your tests and your implementation.
Edited: included in this coupling is the test asserting that the code under test calls some methods. Calling some method is never a functional need, it is an implementation concern. It relates to an interface other than the one being tested.
In many cases, the testing should be about the external behavior of an interface, and be completely black-box testing them.
The author gives the example that the test classes should be in a different package than the class to test. At first, I was sure this was wrong, because it makes it more difficult to test protected and package methods. But he argues that you should only test the external behavior of a system, that is the public methods. The non-public methods are implementation-details, and testing it results in coupling the test with the implementation. This was very insightful to me.
By the way, this book has so many excellent practical advice on how to design tests (say JUnit tests), that I would buy it on my own money if it wasn't provided by the company! ;-)
An excellent other advice from the book was to test at the functionality level, not the method level. For example, testing the add() method for a list requires trusted size() and get() methods, but they in turn require add() so we have a loop, we can't test safely. But testing the list's behavior globally (accross all methods) when adding involves testing the three methods at the same time, not proving that each is correct in isolation, but checking that together they provide the expected behavior. Often, when you try to test one of your methods in isolation, you cannot write a sensible test without using other methods, so you end up testing the implementation instead ; the consequence are coupling between test and implementation.
Only test functionalities, not methods.
Also, note that testing using external ressources (the database being the more common, but many others exist) is much slower, requires some access (IP, licence etc) from the executing machine, require a started container, may be sensitive to simultaneous access (a database can't run reliably multiple JUnit campaign at the same time), and has many other drawbacks. If all your tests use external resources, then you are in trouble, you can't run all your tests all the time, from any machine, from many machines at once, etc. So I understood (still from the book):
Test only once each external resource (database for example), in a dedicated test that is not a unit-test, but an integration test (although it can still use the same JUnit technology if appropriate).
Test enough dedicated tests to trust the resource is working. Then, other tests should never test it again, this is a waste, they should trust it.
Note that the current Maven best-practices give similar advice (see free book "Better builds with Maven"). I believe this is not a coincidence:
The JUnits in the test directory of a project are real unit tests. They run every time you do something with your project (except just compile).
The integration and functional tests should be provided in a different project, an integration-test project. They only run in a much later (optional) phase, after you have deployed your whole application in the container.
As a consequence, tests ARE going to
fail - soon and often. This is pain.
And the problem.
Well yes, unit tests can depend on internal implementation details. And sure, such "white box" tests are more brittle than "black box" tests which only rely on the externally published contract.
But I don't agree that this has to cause regular test failures. Think about how you arrived at testing with mocks in the first place: you've used dependency injection to limit the responsibilities of the class, to decrease coupling to other code, and to enable testing the class in isolation.
Are there any techniques to deal with
this?
A good unit test can only fail if you change the class under test, even if it depends on internal implementation details. And you can limit the responsibilities and coupling (to other classes) of your class, so that you will rarely have to change it.
In practice you'll have to be pragmatic; every now and then you'll write "unit tests" that are actually integration tests involving multiple classes or over-sized classes. Brittle tests depending on internal implementation details are more dangerous in that case. But for truly TDD-style classes, not so much.
Remember when you're writing a test you're not testing your repository, you're testing your Service class. In this specific example ProcessWithItem method. You create your expectations for repository object. By the way, you forgot to specify expected return for your x.Find method. That's the beauty of DI that you isolate everything from the code you about to write (I assume you do TDD).
To be honest I cannot relate to the problem you describe.
Yeah, that's one of the big problems with unit testing. That, and refactoring. And design changes that are a regular occurrence with Agile. And the inexperience of those creating the tests. And etc etc...
I think the only thing the average non-critical-systems developer can do is pick and choose your battles wisely. Early in development identify the truly critical paths and test those. Weigh the likelihood of that code changing before spending lots of time testing the rest of it.
If anybody figures it all out please let us know.