If created an interface which contains signatures for the static methods I would have a way to test the classes that were dependent on those static methods.
But how would I test the concrete wrapper class that implements the interface and wraps those static methods?
Do i just not test the concrete wrapper class?
It seems like wrapping the static methods has just moved the problem elsewhere. I still have some class, wrapper class, that is hard to test.
You're on the right track, but you're not moving the problem somewhere else -- you're isolating it to prevent that problem from leaking throughout your code base and other tests. I would only view this as a problem if you've made the wrong assumptions about the wrapped behavior or neglected to prove those assumptions. Some testing is required, but at varying degrees of difficulty. How far you choose to test those assumptions will be a trade-off you need to make.
For some scenarios, the wrapper can be tested but will require physical dependencies that you can control. For example, a static method that modifies a file will require that you take the time to setup the file system, or a helper class that downloads a file from a web-server. I like to think of these tests as "integration tests" instead of "unit tests" as they act upon their dependencies instead of running in isolation. As these tests may take longer to execute that standard tests, you may want to exclude them and run them less frequently than your faster unit tests.
For other scenarios, you may not be able to write a test without considerable effort. For example, you may have wrapped the logic that shows a dialog in a separate window -- testing this logic in code would be a lot work, but launching the application and verifying that it works may be enough. You'll take a hit on code-coverage but to me this is an acceptable trade-off.
The key challenge is understanding the assumptions of the wrapped behavior -- eg, what type of exception is thrown from a remote server if web-request is malformed or the network connection is lost? If you don't capture these assumptions correctly, those faults will bubble up into other areas of your code.
Related
I've recently started practising TDD and unit testing, with my main primers being the excellent GOOSGBT and a perusal of TDD-tagged questions here on SO.
Occasionally, the process I use creates a "controller" class - generally, a class which is a facade over a fairly complex subsystem where, as the number of features implemented in the subsystem grows, responsibilities are continually driven out into helper classes until the original class has essentially no responsibilities beyond making correct calls to a small set of collaborator classes and shunting the returned information (if any) to its other collaborator classes.
Originally, the tests for the soon-to-be controller classes were written at the level of intention of end-users of the class: "If I make this call, what should be the observable effects that I, as an end-user of the class, actually care about?". But as more and more responsibilities and tests for edge-cases were driven out into helper classes (which are replaced by Test Doubles in the tests for the controller class), these tests began to seem really ... vague and non-specific: they were invariably "happy-path" tests that didn't really seem to get to the heart of the matter. It's hard to explain what I mean, but reading the tests back left me with a kind of "So what? Why did you choose this particular happy-path test over any other? What is the significance? If someone breaks the code, will this test pinpoint the exact reason why the code is now broken?" As time went by, I was more and more strongly inclined to instead write the tests in terms of how the classes' collaborators were supposed to be used together: "the class should call this method on this collaborator, and pass the result to this other collaborator" which gave a much more focussed, descriptive and clearly-motivated set of tests (and generally, the resulting set of tests is small).
This obviously has its flaws: the tests are now strongly coupled to the specifics of the implementation of the controller class, rather than the much more flexible "what would an end-user of this class see that he would care about?". But really, the tests are already quite coupled to it by virtue of the fact that they must configure all of the Test Double collaborators to behave in the exact way required by the implementation to give the correct results from an end-user of the classes' point of view.
So my question is: do fellow TDD'ers find that a minority of classes do little but marshall their (small) set of collaborators around? Do you find keeping the tests for such classes to be written from an end-user of the classes' point of view to be imprecise and unsatisfactory and if so, is it acceptable to write tests for such classes explicitly in terms of how it calls and transfers data between their collaborators?
Hope it's reasonably clear what I'm driving at, here! :)
As a concrete example: one practise project I was working on was a TV listings downloader/ viewer (if you've ever seen "Digiguide", you'll know the kind of thing I mean), and I was implementing a core part of the app - the part that actually updates the listings over the net and integrates the newly downloaded listings into the current set of known TV programs. The interface to this (surprisingly complex when all requirements are taken on board) functionality was a class called ListingsUpdater, which had a method called "updateListings".
Now, end-users of ListingsUpdater only really care about a few things: after listingsUpdate has been called, is the database of TV listings now correct, and were the changes made to the database (adding TV programs, changing them if broadcast changes occurred etc) described to the provided change listeners? When the implementation was a very, very simple "fake it till you make it" type of deal, this worked fine: but as I progressively drove the implementation towards one that would work in the real-world, the "real work" got driven further and further away from ListingsUpdater, until it mainly just marshalled a few collaborators: a ListingsRequestPreparer for assessing the current state of the listings and building HTTP requests for a ListingsDownloader, and a ListingsIntegrator which unpacked the newly downloaded listings and incorporated them (it too delegating to collaborators) into the listings database. Now, note that in order to fulfil the contract of ListingsUpdater from a user's point of view, I must, in the test, instruct its ListingsIntegrator Test Double to populate the (fake) database with the correct data(!), which seems bizarre. It seems much more sensible to drop the "from the end-user of ListingsUpdater's point of view" tests and instead add a test that says "when the ListingsDownloader has downloaded the new listings ensure they are handed over to the ListingsIntegrator".
This obviously has its flaws: the tests are now strongly coupled to the specifics of the implementation of the controller class, rather than the much more flexible "what would an end-user of this class see that he would care about?". But really, the tests are already quite coupled to it by virtue of the fact that they must configure all of the Test Double collaborators to behave in the exact way required by the implementation to give the correct results from an end-user of the classes' point of view.
I'll repeat what I said in answer to another question:
I need to create either a mock a stub or a dummy object [a test double] for each dependency
This is commonly stated. But I think it is wrong. If a Car is associated with an Engine object, why not use a real Engine object when unit testing your Car class?
But, someone will declare, if you do that you are not unit testing your code; your test depends on both the Car class and the Engine class: two units, so an integration test rather than a unit test. But do those people mock the String class too? Or HashSet<String>? Of course not. The line between unit and integration testing is not so clear.
More philosophically, you can not create good mock objects [test doubles] in many cases. The reason is that, for most methods, the manner in which an object delegates to associated objects is undefined. Whether it does delegate, and how, is left by the contract as an implementation detail. The only requirement is that, on delegating, the method satisfies the preconditions of its delegate. In such a situation, only a fully functional (non-mock) delegate will do. If the real object checks its preconditions, failure to satisfy a precondition on delegating will cause a test failure. And debugging that test failure will be easy.
And I'll add in response to
they were invariably "happy-path" tests that didn't really seem to get to the heart of the matter
This is a more general testing problem, not specific to TDD or unit testing: how to you select a good set of test-cases, given that comprehensive testing is impossible? I rely on equivalence partitioning. When I start work on some code, I use equivalence partitioning to select the set of test-cases I want the code to pass, then work on each in turn in a TDD manner, but if passing one of the test-cases does not require a code change (because early work has created code that also satisfies that test case) I still add the test-case to my test-suite. My test suite therefore has better coverage of potential error paths.
I would like to start doing more unit testing in my applications, but it seems to me that most of the stuff I do is just no suitable to be unit tested. I know how unit tests are supposed to work in textbook examples, but in real world applications they do not seem of much use.
Some applications I write have very simple logic and complex interactions with things that are outside my control. For instance I would like to write a daemon which reacts to signals sent by some applications, and changes some user settings in the OS. I can see three difficulties:
first I have to be able to talk with the applications and be notified of their events;
then I need to interact with OS whenever I receive a signal, in order to change the appropriate user settings;
finally all of this should work as a daemon.
All these things are potentially delicate: I will have to browse possibly complex APIs and I may introduce bugs, say by misinterpreting some parameters. What can unit testing do for me? I can mock both the external application and the OS, and check that given a signal from the application, I will call the appropriate API method on the OS. This is... well, the trivial part of the application.
Actually most of the things I do involve interaction with databases, the filesystem or other applications, and these are the most delicate parts.
For another example look at my build tool PHPmake. I would like to refactor it, as it is not very well-written, but I fear to do this as I have no tests. So I would like to add some. The point is that the things which may be broken by refactoring may not be caught by unit tests:
One of things to do is deciding which things are to be built and which one are already up to date, and this depends on the time of last modification of the files. This time is actually changed by external processes, when some build command is fired.
I want to be sure that the output of external processes is displayed correctly. Sometimes the buikd commands require some input, and that should be also managed correctly. But I do not know a priori which processes will be ran - it may be anything.
Some logic is involved in pattern matching, and this may seem to be testable part. But the functions which do the pattern matching use (ni addition to their own logic) the PHP function glob, which works with the filesystem. If I just mock a tree in place of the actual filesystem, glob will not work.
I could go on with more examples, but the point is the following. Unless I have some delicate algorithms, most of what I do involves interaction with external resources, and this is not suitable for unit testing. More that this, often this interaction is actually the non-trivial part. Still many people see unit testing as a basic tool. What am I missing? How can I learn be a better tester?
I think you open a number of issues in your question.
Firstly, when your application integrates with external environments such as OS, other threads, etc. then you have to separate (1) the logic that is tied in with the external enviroment and (2) your business-code.. that is, the stuff your application does. This is no different to how you would separate GUI and SERVER in an application (or web application).
Secondly, you ask if you should test simple logic. I'd say, it depends. Often simple fetch/store functionality is nice to have tests for. It's like the foundation of your application.. hence its important. Other business stuff built upon your foundation that is very simple, you may easily find yourself both feeling that you are wasting your time, and mostly you are :-)
Thirdly, refactory an existing program and testing it in its existing state may be a problem. If your PHP program produces a set of files on the basis of some input, well, maybe thats your entry point to tests are. Sure the tests may be high-level, but it's an easy way to ensure that after the refactoring, your program produces the same output. Hence, aim for higher-level tests in that situation in the start phase of your refactoring efforts.
I'd like to recommend some literature, but I can only come up with one title. "Working Effectively with Legacy Code" By Micheal Feathers. It's a good start. Another would be "xUnit Test Patterns: Refactoring Test Code" by Gerard Meszaros (although that book is much more sloppy and FULL of copy paste text).
As regards your issue about existing code bases that aren't currently covered by tests in which you would like to start refactoring, I would suggest reading:
Working Effectively with Legacy Code By Micheal Feathers.
That book gives you techniques on how to deal with the issues you might be facing with PHPMake. It provides ways to introduce seams for testing, where there previously weren't any.
Additionally, with code that touches say the file systems, you can abstract the file system calls behind a thin wrapper, using the Adapter Pattern. The unit tests would be against a fake implementation of the abstract interface that the wrapping class implements.
At some point you get to a low enough level where a unit of code can't be isolated for unit testing as these depend on library or API calls (such as in the production implementation of the wrapper). Once this happens integration tests are really the only automated developer tests you can write.
I recommend this google tech-talk on unit testing.
The video boils down to
write your code so that it knows as little about how it will be used as possible. The less assumptions your code makes, the easier it is to test. Avoid complex logic in constructors, the use of singletons, static class members, and so on.
isolate your code from the external world (comms, databases, real time), and make sure that your code only talks to your isolation layer. Otherwise, writing tests will be a nightmare in terms of 'fake environment' setup.
unit tests should test stories; that is what we really understand and care for; given a class with a method foo(), testFoo() is uninformative. They actually recommend test names like itShouldCloseConnectionEvenWhenExceptionThrown(). Ideally, your stories should cover enough functionality that you can rebuild the spec from the stories.
NOTE: the video and this post use Java as an example; however, the main points stand for any language.
"Unit tests" tests one unit of your code. No external tools should be involved. This seems to be complicated for your first app (without knowing to much about it ;)) but the phpMake is unit-testable - I'm sure ... because ant, gradle and maven are unit-testable too ;)!
But of course you can test your first application automated too. There are several different layers one could test an application.
So the task for you is to find an automated way to test your app - be it integration testing or whatever.
E.g. you could write shell scripts, which asserts some output! With that you make sure your application behaves correctly ...
Tests of interactions with external resources are integration tests, not unit tests.
Tests of your code to see how it would behave if particular external interactions had occurred can be unit tests. These should be done by writing your code to use dependency injection, and then, in the unit test, injecting mock objects as dependencies.
For example, consider a piece of code that adds the results of a call to one service to the results of a call to another service:
public int AddResults(IService1 svc1, IService2 svc2, int parameter)
{
return svc1.Call(parameter) + svc2.Call(parameter);
}
You can test this by passing in mock objects for the two services:
private class Service1Returns1 : IService1
{
public int Call(int parameter){return 1;}
}
private class Service2Returns1 : IService2
{
public int Call(int parameter){return 1;}
}
public void Test1And1()
{
Assert.AreEqual(2, AddResults(new Service1Returns1(), new Service2Returns1(), 0));
}
First of all, if unit testing doesn't seem like it would be much use in your applications, why do you even want to start doing more of it? What is motivating you to care about it? It is definitely a waste of time if a) you do everything perfect the first time and nothing ever changes or b) you decide it's a waste of time and do it poorly.
If you do think that you really want to do unit testing, the answer to your questions are all the same: encapsulation. In your daemon example, you could create a ApplcationEventObeservationProxy with a very narrow interface that just implements pass through methods. The purpose of this class is to do nothing but completely encapsulate the rest of your code from the third-party event observing library (nothing means nothing -- no logic here). Do the same thing for OS settings. Then you can completely unit test the class that does actions based on events. I'd recommend have a separate class for the daemon-ness that just wraps your main class -- it will make the testing easier.
There are a couple of benefits to this approach outside of unit testing. One is that if you encapsulate the code that interacts directly with the OS, it's easier to switch it out. This kind of code is particularly prone to breakage outside of your control (i.e., MS patchsets). You will also probably want to support more than one OS, and if the OS specific logic is not tangled with the rest of your logic, it will be easier. The other benefit is that you'll be forced to realize that there is more business logic in your app than you think. :)
Finally, don't forget that unit testing is a foundation for a good product, but not the only ingredient. Having a set of tests that explore and verify the OS API calls you'll be using is a good strategy for the "hard" parts of this problem. You should also have end to end tests that ensure the events in your applications cause the OS setting changes.
As other answers suggested Working Effectively with Legacy Code By Micheal Feathers is a good read. If you have to deal with legacy code, and you want to make sure that the systems interaction work as expected, try writing integration tests first. And then it is more appropriate to write Unit Tests to test the behaviour of methods that are valued from the requirements point of view. You Tests serve a whole different purpose than the integration tests. Unit Tests are more likely to improve the design of your system than testing how everything hangs to gather.
When I write tests for certain types of objects, such as UI elements like Forms or UserControls, I often find myself altering my TDD pattern; instead of going test-first, I define and lay out the form's controls, to provide a "skeleton", then start writing behavioral tests (databinding/"unbinding", display-mode behavior, etc). In doing so, I find myself dealing with non-public members. I also run into the same concern with other behavioral methods; I may want to focus on and exercise logic in some private helper called by another method, before worrying about the usage in the other method and its behavior.
To me, making everything public (and sometimes virtual) just to be able to unit-test everything is a smell; I don't want other objects being able to call a helper, or directly access a textbox; but I need to know that the helper does its job and the textbox gets its value when the form loads.
The solution I arrived at some time ago is to create a "test proxy" for the actual object under test. The proxy derives from the object under test, and doesn't hide or override any behavior, but it does provide internally-visible getters, setters and/or methods that make calls to non-public members of the object under test, allowing me to tell the object to perform certain actions of which I can then view the results, without requiring the test to also depend on proper integration within the object or making the method or some other member of interest public in production code just for testing purposes.
Advantages I see:
Members' visibility is not determined by whether you want a unit test or not.
Finer control over what you can do with the object in a test allows for more flexible and extensible testing.
Disadvantages I see:
Class count increases, with an extra level to develop just for testing purposes.
Care must be taken not to somehow end up using the test proxy in production code (making the constructor or the entire class internal generally does the trick)
Not a "pure" unit test as you are, at some level, dependent on integration between the proxy and actual object under test.
The question is, is this a valid way to architect unit tests, or does the fact that I have to do this indicate a problem with the code or the testing strategy?
My first reaction to this pattern is that you're de-emphasizing the 'D's in TDD. Your code is tested but those tests are not driving your design so the code you end up with has a different structure than it would if you had written tests first. A structure with more inaccessible private state than necessary. In general I'll argue that if you can't test your class' behavior using the public interfaces then you are either writing a test which doesn't make sense (testing implementation details) or you have a poorly designed public interface.
However if you're working with view classes this becomes a bit more complicated since you have "public" inputs and outputs via your view which you want to test but which are not necessarily exposed to the code using this view component. In that case I think it makes sense to give your tests access to that user interface; either by exposing those normally private attributes to the test (your proxy is one option and others may be available depending on the language you are using) or by writing some form of functional test which can drive the UI (again tools available depend on your environment).
I would say it indicates a problem with your testing strategy or code. The reason is because your violating encapsulation which is going to couple your tests to the implementation rather than the interface. This will add to the overall work you do because refactors (for example) may no longer be free.
That being said I think there a good reasons to violate encapsulation, and they revolve around functions with side effects (which is often the case with UI programming). In many cases you need to assure that functions are being called in a particular order or that they are called at all. I think there are ways to mitigate how much you violate encapsulation.
If I'm writing tests for side effects I will usually separate them out into their own test. I will also stub/mock out the side effect functions and assert they are called according to my requirements (ie, order, timing, called or not, arguments, etc). This frees me from knowing the implementation details, but still allows me to assert that particular functions were called properly.
In certain languages it can be difficult to mock out the objects, or methods. In those cases I will use dependency injection to pass the object or functions with side effects. That way when testing I can pass my mocks for verification.
MSTest uses this method to test private methods. Visual Studio puts all the tests in a separate test project and creates an "Accessor" class. That class is a subclass where all the private members are made public. Since this subclass is in the test project it isn't available in the assembly under test. I think this is a viable pattern for testing private methods and could be manually implemented in a TDD environment if you aren't using Visual Studio and MSTest.
What unit tests generally tend to be hard to write and why? I am particularly interested in methods which don't need mocking.
Thanks
Two cases where unit testing is made difficult:
Methods that invoke static methods that belong to other classes, particularly when those other classes have static state, or do significant work. Being stuck trying to "unit" test a method that, through transitive closure, does database queries can suck.
Methods that create instances of other classes directly (i.e., via new), particularly when the constructor of the other class does itself requires static state, or when it does significant work in the constructor.
A great A to Z guide of testability concerns with side by side code examples of easy/hard to test code can be found in Misko's extensive testability guide.
Click on the "flaw #x" links (they look like plain text but they're separate links).
Big, complex methods that do lots of things at the same time that really should've been separated. (example: get something from a configuration object, create a URL based on some variables, encode the URL, send a request, do something with the response... you get the drill).
Everything static. Things created with New, although I haven't found a proper way to avoid it without spamming the entire application with factories.
It's almost always about dependencies.
Most code depends on external systems such as databases, file systems, email clients, networks, etc. It's also common to have dependencies on major internal systems (e.g, the spell checking module, or the recalc engine...).
If these dependences are not easily substitutable, then the system becomes hard to test.
Classes that call statics and singletons are the worst offenders, but any class that doesn't accept it's dependencies via constructor or properties will be hard to test.
There are some legitimate situations that are hard to test:
Concurrency
User Interface - this is why the trend is towards MVC architectures that create ViewModels which can be easily tested. The actual rendering is minimized - this is called the humble dialog or humble object pattern in the test literature.
I'm still in the learning stages regarding unit-testing and in particular regarding mocking (I'm using the PascalMock and DUnit frameworks). One thing I now stumbled over was that I couldn't find a way around hard-coding implementation details of the tested class/interface into my unit test and that just feels wrong...
For example: I want to test a class that implements a very simple interface for reading and writing application settings (basically name/value pairs). The interface that is presented to the consumer is completely agnostic to where and how the values are actually stored (e.g. registry, INI-file, XML, database, etc.). Naturally, the access layer is implemented by yet a different class that gets injected into the tested class on construction. I created a mock object for this access layer and I am now able to fully test the interface-implementing class without actually reading or writing anything to any registry/INI-file/whatever.
However, in order to ensure the mock behaves exactly like the real thing when accessed by the tested class, my unit tests have to set up the mock object by very explicitly defining expected method calls and the return values expected by the tested class. This means that if I should ever have to make changes to the interface of the access layer or to the way that the tested class uses that layer I will also have to change the unit tests for the class that internally uses that interface even though the interface of the class I'm actually testing hasn't changed at all. Is this something I will just have to live with when using mocks or is there a better way to design the class-dependencies that would avoid this?
to ensure the mock behaves exactly like the real thing when accessed by the tested class, my unit tests have to set up the mock object by very explicitly defining expected method calls and the return values expected by the tested class.
Correct.
changes to the interface of the access layer or to the way that the tested class uses that layer I will also have to change the unit tests
Correct.
even though the interface of the class I'm actually testing hasn't changed at all.
"Actually testing"? You mean the exposed interface class? That's fine.
The way the "tested" (interface) class uses the access layer means you've changed the internal interface to the access layer. Interface changes (even internal ones) require test changes and may lead to breakage if you've done something wrong.
Nothing wrong with this. Indeed, the whole point is that any change to the access layer must require changes to the mocks to assure that the change "works".
Testing is not supposed to be "robust". It's supposed to be brittle. If you make a change that alters internal behavior, then things can break. If your tests were too robust they wouldn't test anything -- they'd just work. And that's wrong.
Tests should only work for the exact right reason.
Is this something I will just have to
live with when using mocks or is there
a better way to design the
class-dependencies that would avoid
this?
A lot of times mocks (particularly sensitive frameworks like JMock) force you to account for details that don't relate directly to the behavior you're trying to test, and sometimes this can even be helpful by exposing suspect code that is doing too much and has too many calls/dependencies.
However in your case, if I read your description right, it sounds like you really don't have a problem. If you design the read/write layer correctly and with an appropriate level of abstraction, you shouldn't have to change it.
This means that if I should ever have
to make changes to the interface of
the access layer or to the way that
the tested class uses that layer I
will also have to change the unit
tests for the class that internally
uses that interface even though the
interface of the class I'm actually
testing hasn't changed at all.
Isn't the point of writing the abstracted access layer to avoid this? In general, following the Open/Closed principle, an interface of this sort shouldn't change and shouldn't break the contract with the class that consumes it, and by extension it won't break your unit tests either. Now if you change the order of the method calls, or have to make new calls to the abstracted layer, then, yes, particularly with some frameworks, your mock expectations will break. This is just part of the cost of using mocks, and it's perfectly acceptable. But the interface itself should, in general, remain stable.
Just to put some names to your example,
RegistryBasedDictionary implements the Role (interface) Dictionary.
RegistryBasedDictionary has a dependency on the Role RegistryAccessor, implemented by RegistryWinAPIWrapper.
You are currently interested in testing RegistryBasedDictionary. The unit tests would inject a mock dependency for the RegistryAccessor Role and would test the expected interaction with the dependencies.
The trick here to avoid unnecessary test-maintenance is to "Specify precisely what should happen.. and no more." (From the GOOS book (must-read for mock flavored TDD), so if order of dependency method calls doesn't matter, don't specify it in the test. That leaves you free to change the order of calls in the implementation.)
Design the Roles such that the they do not contain any leaks from the actual implementations - keep the Roles implementation-agnostic.
The only reason to change RegistryBasedDictionary tests, would be a change in the behavior of RegistryBasedDictionary and not in any of its dependencies. So if its interaction with its dependencies or the roles/contracts change, the tests would need to be updated. That is the price of interaction-based tests you need to pay, for the flexibility to test in isolation. However in practice, it doesn't that happen that often.