Related
I've recently started practising TDD and unit testing, with my main primers being the excellent GOOSGBT and a perusal of TDD-tagged questions here on SO.
Occasionally, the process I use creates a "controller" class - generally, a class which is a facade over a fairly complex subsystem where, as the number of features implemented in the subsystem grows, responsibilities are continually driven out into helper classes until the original class has essentially no responsibilities beyond making correct calls to a small set of collaborator classes and shunting the returned information (if any) to its other collaborator classes.
Originally, the tests for the soon-to-be controller classes were written at the level of intention of end-users of the class: "If I make this call, what should be the observable effects that I, as an end-user of the class, actually care about?". But as more and more responsibilities and tests for edge-cases were driven out into helper classes (which are replaced by Test Doubles in the tests for the controller class), these tests began to seem really ... vague and non-specific: they were invariably "happy-path" tests that didn't really seem to get to the heart of the matter. It's hard to explain what I mean, but reading the tests back left me with a kind of "So what? Why did you choose this particular happy-path test over any other? What is the significance? If someone breaks the code, will this test pinpoint the exact reason why the code is now broken?" As time went by, I was more and more strongly inclined to instead write the tests in terms of how the classes' collaborators were supposed to be used together: "the class should call this method on this collaborator, and pass the result to this other collaborator" which gave a much more focussed, descriptive and clearly-motivated set of tests (and generally, the resulting set of tests is small).
This obviously has its flaws: the tests are now strongly coupled to the specifics of the implementation of the controller class, rather than the much more flexible "what would an end-user of this class see that he would care about?". But really, the tests are already quite coupled to it by virtue of the fact that they must configure all of the Test Double collaborators to behave in the exact way required by the implementation to give the correct results from an end-user of the classes' point of view.
So my question is: do fellow TDD'ers find that a minority of classes do little but marshall their (small) set of collaborators around? Do you find keeping the tests for such classes to be written from an end-user of the classes' point of view to be imprecise and unsatisfactory and if so, is it acceptable to write tests for such classes explicitly in terms of how it calls and transfers data between their collaborators?
Hope it's reasonably clear what I'm driving at, here! :)
As a concrete example: one practise project I was working on was a TV listings downloader/ viewer (if you've ever seen "Digiguide", you'll know the kind of thing I mean), and I was implementing a core part of the app - the part that actually updates the listings over the net and integrates the newly downloaded listings into the current set of known TV programs. The interface to this (surprisingly complex when all requirements are taken on board) functionality was a class called ListingsUpdater, which had a method called "updateListings".
Now, end-users of ListingsUpdater only really care about a few things: after listingsUpdate has been called, is the database of TV listings now correct, and were the changes made to the database (adding TV programs, changing them if broadcast changes occurred etc) described to the provided change listeners? When the implementation was a very, very simple "fake it till you make it" type of deal, this worked fine: but as I progressively drove the implementation towards one that would work in the real-world, the "real work" got driven further and further away from ListingsUpdater, until it mainly just marshalled a few collaborators: a ListingsRequestPreparer for assessing the current state of the listings and building HTTP requests for a ListingsDownloader, and a ListingsIntegrator which unpacked the newly downloaded listings and incorporated them (it too delegating to collaborators) into the listings database. Now, note that in order to fulfil the contract of ListingsUpdater from a user's point of view, I must, in the test, instruct its ListingsIntegrator Test Double to populate the (fake) database with the correct data(!), which seems bizarre. It seems much more sensible to drop the "from the end-user of ListingsUpdater's point of view" tests and instead add a test that says "when the ListingsDownloader has downloaded the new listings ensure they are handed over to the ListingsIntegrator".
This obviously has its flaws: the tests are now strongly coupled to the specifics of the implementation of the controller class, rather than the much more flexible "what would an end-user of this class see that he would care about?". But really, the tests are already quite coupled to it by virtue of the fact that they must configure all of the Test Double collaborators to behave in the exact way required by the implementation to give the correct results from an end-user of the classes' point of view.
I'll repeat what I said in answer to another question:
I need to create either a mock a stub or a dummy object [a test double] for each dependency
This is commonly stated. But I think it is wrong. If a Car is associated with an Engine object, why not use a real Engine object when unit testing your Car class?
But, someone will declare, if you do that you are not unit testing your code; your test depends on both the Car class and the Engine class: two units, so an integration test rather than a unit test. But do those people mock the String class too? Or HashSet<String>? Of course not. The line between unit and integration testing is not so clear.
More philosophically, you can not create good mock objects [test doubles] in many cases. The reason is that, for most methods, the manner in which an object delegates to associated objects is undefined. Whether it does delegate, and how, is left by the contract as an implementation detail. The only requirement is that, on delegating, the method satisfies the preconditions of its delegate. In such a situation, only a fully functional (non-mock) delegate will do. If the real object checks its preconditions, failure to satisfy a precondition on delegating will cause a test failure. And debugging that test failure will be easy.
And I'll add in response to
they were invariably "happy-path" tests that didn't really seem to get to the heart of the matter
This is a more general testing problem, not specific to TDD or unit testing: how to you select a good set of test-cases, given that comprehensive testing is impossible? I rely on equivalence partitioning. When I start work on some code, I use equivalence partitioning to select the set of test-cases I want the code to pass, then work on each in turn in a TDD manner, but if passing one of the test-cases does not require a code change (because early work has created code that also satisfies that test case) I still add the test-case to my test-suite. My test suite therefore has better coverage of potential error paths.
Are we supposed to write tests for our getters and setters or is it overkill?
I would say no.
#Will said you should aim for 100% code coverage, but in my opinion that's a dangerous distraction. You can write unit tests that have 100% coverage, and yet test absolutely nothing.
Unit tests are there to test the behaviour of your code, in an expressive and meaningful way, and getters/setters are only a means to an end. If you tests use the getters/setters to achieve their goal of testing the "real" functionality, then that's good enough.
If, on the other hand, your getters and setters do more than just get and set (i.e. they're properly complex methods), then yes, they should be tested. But don't write a unit test case just to test a getter or setters, that's a waste of time.
Roy Osherove in his famous book 'The Art Of Unit Testing' says:
Properties (getters/setters in Java) are good examples of code that usually doesn’t contain any logic, and doesn’t require testing. But watch out: once you add any check inside the property, you’ll want to make sure that logic is being tested.
A resounding YES with TDD
Note: This answer keeps getting upvotes, albeit potentially a bad advice. To understand why, have a look at its little sister below.
Controversial alright, but I'd argue that anyone who answers 'no' to this question is missing a fundamental concept of TDD.
For me, the answer is a resounding yes if you follow TDD. If you aren't then no is a plausible answer.
The DDD in TDD
TDD is often quoted as having thee main benefits.
Defence
Ensuring the code may change but not its behaviour.
This allows the ever so important practice of refactoring.
You gain this TDD or not.
Design
You specify what something should do, how it should behaves before implementing it.
This often means more informed implementation decisions.
Documentation
The test suite should serve as the specification (requirements) documentation.
Using tests for such purpose mean that the documentation and implementation are always in consistent state - a change to one means a change to other. Compare with keeping requirements and design on separate word document.
Separate responsibility from implementation
As programmers, it is terribly tempting to think of attributes as something of significance and getters and setter as some sort of overhead.
But attributes are an implementation detail, while setters and getters are the contractual interface that actually make programs work.
It is far more important to spell that an object should:
Allow its clients to change its state
and
Allow its clients to query its state
then how this state is actually stored (for which an attribute is the most common, but not the only way).
A test such as
(The Painter class) should store the provided colour
is important for the documentation part of TDD.
The fact that the eventual implementation is trivial (attribute) and carries no defence benefit should be unknown to you when you write the test.
The lack of round-trip engineering...
One of the key problems in the system development world is the lack of round-trip engineering1 - the development process of a system is fragmented into disjointed sub-processes the artifacts of which (documentation, code) are often inconsistent.
1Brodie, Michael L. "John Mylopoulos: sewing seeds of conceptual modelling." Conceptual Modeling: Foundations and Applications. Springer Berlin Heidelberg, 2009. 1-9.
...and how TDD solves it
It is the documentation part of TDD that ensures that the specifications of the system and its code are always consistent.
Design first, implement later
Within TDD we write failing acceptance test first, only then write the code that let them pass.
Within the higher-level BDD, we write scenarios first, then make them pass.
Why should you exclude setters and getter?
In theory, it is perfectly possible within TDD for one person to write the test, and another one to implement the code that makes it pass.
So ask yourself:
Should the person writing the tests for a class mention getters and setter.
Since getters and setters are a public interface to a class, the answer is obviously yes, or there will be no way to set or query the state of an object. However, the way to do this is not necessarily by testing each method in isolation, see my other answer for more.
Obviously, if you write the code first, the answer may not be so clearcut.
tl;dr: Yes you should, and with OpenPojo it's trivial.
You should be doing some validation in your getters and setters so you should be testing that. For example, setMom(Person p) should not allow setting anyone younger than themselves as their mother.
Even if you aren't doing any of that now, odds are you will in the future, then this will be a good for regression analysis. If you want to allow setting mothers to null you should have a test for that should someone change that later on, this will reinforce your assumptions.
A common bug is void setFoo( Object foo ){ foo = foo; } where it should be void setFoo( Object foo ){ this.foo = foo; }. (In the first case the foo that is being written to is the parameter not the foo field on the object).
If you are returning an array or collection you should be testing whether or not the getter is going to be performing defensive copies of the data passed into the setter before returning.
Otherwise, if you have the most basic setters/getters then unit-testing them will add maybe about 10 minutes at most per-object, so what is the loss? If you add behaviour you already have a skeleton test and you get this regression testing for free. If you are using Java, you have no excuse since there is OpenPojo. There are an existing set of rules you can enable and then scan your entire project with them to make sure they are applied consistently within your code.
From their examples:
final Validator pojoValidator = ValidatorBuilder.create()
.with(
new NoPublicFieldsRule (),
new NoPrimitivesRule (),
new GetterMustExistRule (),
new SetterMustExistRule ()
)
.with(
new DefaultValuesNullTester (),
new SetterTester (),
new GetterTester ()
)
.build();
pojoValidator.validate( PojoClassFactory.getPojoClasses( "net.initech.app", new FilterPackageInfo() ) );
Yes, but not always in isolation
Allow me elaborate:
What is a unit test?
From Working effectively with legacy code1:
The term unit test has a long history in software development. Common to
most conceptions of unit tests is the idea that they are tests in isolation of individual
components of software. What are components? The definition varies,
but in unit testing, we are usually concerned with the most atomic behavioral units of a system. In procedural code, the units are often functions. In object oriented code, the units are classes.
Note that with OOP, where you find getters and setters, the unit is the class, not necessarily individual methods.
What is a good test?
All requirements and tests follow the form of Hoare logic:
{P} C {Q}
Where:
{P} is the precondition (given)
C is the trigger condition (when)
{Q} is the postcondition (then)
Then comes the maxim:
Test behaviour, not implementation
This means that you shouldn't test how C achieves the post-condition, you should check that {Q} is the result of C.
When it comes to OOP, C is a class. So you shouldn't test internal effects, only external effects.
Why not test bean getters and setters in isolation
Getters and setters may involve some logic, but so long this logic does not have external effect - making them bean accessors2) a test will have to look inside the object and by that not only violate encapsulation but also test for implementation.
So you shouldn't test bean getters and setters in isolation. This is bad:
Describe 'LineItem class'
Describe 'setVAT()'
it 'should store the VAT rate'
lineItem = new LineItem()
lineItem.setVAT( 0.5 )
expect( lineItem.vat ).toBe( 0.5 )
Although if setVAT would throw an exception, a corresponding test would be appropriate since now there is an external effect.
How should you test getters and setters?
There is virtually no point changing the internal state of an object if such change has no effect on the outside, even if such effect comes later on.
So a test for setters and getters should be concerned with the external effect of these methods, not the internal ones.
For example:
Describe 'LineItem class'
Describe 'getGross()'
it 'should return the net time the VAT'
lineItem = new LineItem()
lineItem.setNet( 100 )
lineItem.setVAT( 0.5 )
expect( lineItem.getGross() ).toBe( 150 )
You may think to yourself:
Wait a sec, we are testing getGross() here not setVAT().
But if setVAT() malfunction that test should fail all the same.
1Feathers, M., 2004. Working effectively with legacy code. Prentice Hall Professional.
2Martin, R.C., 2009. Clean code: a handbook of agile software craftsmanship. Pearson Education.
While there are justified reasons for Properties, there's a common Object Oriented Design belief that exposing member state via Properties is bad design. Robert Martin's article on the Open Closed Principle expands upon this by stating that Properties encourage coupling and therefore limit the ability to close a class from modification -- if you modify the property, all consumers of the class will need to change as well. He qualifies that exposing member variables isn't necessarily bad design, it might just be poor style. However, if properties are read-only, there's less chance of abuse and side-effects.
The best approach I can provide for unit testing (and this may seem odd) is to make as many properties as possible protected or internal. This will prevent coupling while discouraging writing silly tests for getters and setters.
There are obvious reasons where read/write Properties should be used, such as ViewModel properties that are bound to input fields, etc.
More practically, unit tests should drive functionality through public methods. If the code you're testing happens to use those properties, you get code-coverage for free. If it turns out that these properties never get highlighted by code-coverage there's a very strong possibility that:
You are missing tests that indirectly use the properties
The properties are unused
If you write tests for getters and setters, you get a false sense of coverage and will not be able to determine if the properties are actually used by functional behavior.
If the cyclomatic complexity of the getter and/or setter is 1 (which they usually are), then the answer is no, you shouldn't.
So unless you have a SLA that requires 100% code-coverage, don't bother, and focus on testing the important aspect of your software.
P.S. Remember to differentiate getters and setters, even in languages like C# where properties might seem like the same thing. The setter complexity can be higher than the getter, and thus validate a unit-test.
A humorous, yet wise take: The Way of Testivus
"Write the test you can today"
Testing getters/setters may be overkill if you're an experienced tester and this is a small project. However, if you're just getting started learning how to unit test or these getters/setters may contain logic (like #ArtB's setMom() example) then it would be a good idea to write tests.
This has actually been a recent topic between my team and I. We shoot for 80% code coverage. My team argues that getters and setters are auto implemented, and the compiler is generating some basic code behind the scenes. In this case, given the code generated is non-intrusive, it doesn't really make sense to test code the compiler creates for you. We also had this discussion about async methods and in that case the compiler generates a whole bunch of code behind the scenes. This is a different case and something we DO test. Long answer short, bring it up with your team and decide for yourselves if its worth testing.
Also, if you are using the code coverage report like us, one thing you can do is add the [ExcludeFromCodeCoverage] attribute. Our solution has been to use this for models that just have properties using getters and setters, or on the property itself. That way it won't affect the total code coverage % when the code coverage report is run, assuming that is what you are using to calculate your code coverage percentages. Happy testing!
I did a little analysis of the coverage achieved in the JUnit code itself.
One category of uncovered code is "too simple to test". This includes simple getters and setters, which the developers of JUnit do not test.
On the other hand, JUnit doesn't have any (non-deprecated) method longer than 3 lines that is not covered by any test.
In my opinion code coverage is a good way to see if you missed any functionality that you should cover.
When you inspect the coverage manually by it's pretty coloring then it can be argued that plain getters and setters do not need to be tested (although I always do).
When you only check the code coverage percentage on your project, then a test coverage percentage like 80% is meaningless. You can test all the none logical parts and forget some crucial parts. In this case only 100% means that you have tested all you vital code (and all the non-logical code as well). As soon as it is 99.9% you know that have forgotten something.
By the way: Code coverage is the final check to see if you have fully (unit) tested a class. But 100% code coverage not necessarily means that you have actually tested all the functionality of the class. So unit tests should always be implemented following the logic of the class. In the end you run coverage to see if you forgot anything. When you did it right, you hit 100% the first time.
One more thing: While recently working at a large bank in The Netherlands I noticed that Sonar indicated 100% code coverage. However, I knew something was missing. Inspecting the code coverage percentages per file it indicated a file at a lower percentage. The whole code base percentage was that large that the one file did not make the percentage be displayed as 99.9%. So you might want to look out for this...
I would say: YES
Errors in getter/setter methods can silently sneak in and cause some ugly bugs.
I have written a lib to make this and some other tests easier.
The only thing you have to write in your JUnit tests is this:
assertTrue(executor.execute(*TheClassIWantToTest.class*, Arrays.asList( new DefensiveCopyingCheck(),
new EmptyCollectionCheck(), new GetterIsSetterCheck(),
new HashcodeAndEqualsCheck(), new PublicVariableCheck())));
-> https://github.com/Mixermachine/base-test
Yes, especially if the item to get is an object of a class subclassed from an abstract class. Your IDE may or may not alert you that a certain property has not been initialized.
And then some apparently unrelated test crashes with a NullPointerException and it takes you a while to figure out that a gettable property is actually not there to get in the first place.
Though that still wouldn't be anywhere as bad as discovering the problem in production.
It might be a good idea to make sure all your abstract classes have constructors. If not, the test of a getter might alert you to a problem there.
As for getters and setters of primitives, the question might be: Am I testing my program or am I testing the JVM or CLR? Generally speaking, the JVM does not need to be tested.
Recently, i took ownership of some c++ code. I am going to maintain this code, and add new features later on.
I know many people say that it is usually not worth adding unit-tests to existing code, but i would still like to add some tests which will at least partially cover the code. In particular, i would like to add tests which reproduce bugs which i fixed.
Some of the classes are constructed with some pretty complex state, which can make it more difficult to unit-test.
I am also willing to refactor the code to make it easier to test.
Is there any good article you recommend on guidelines which help to identify classes which are easier to unit-test? Do you have any advice of your own?
While Martin Fowler's book on refactoring is a treasure trove of information, why not take a look at "Working Effectively with Legacy Code."
Also, if you're going to be dealing with classes where there's a ton of global variables or huge amounts of state transitions I'd put in a lot of integration checks. Separate out as much of the code which interacts with the code you're refactoring to make sure that all expected inputs in the order they are recieved continue to produce the same outputs. This is critical as it's very easy to "fix" a subtle bug that might have been addressed somewhere else.
Take notes too. If you do find that there is a bug which another function/class expects and handles properly you'll want to change both at the same time. That's difficult unless you keep thorough records.
Presumably the code was written for a purpose, and a unit test will check if the purpose is met, i.e. the pre-conditions and post-conditions hold for the methods.
If the public class methods are such that you can externally check the state it can be unit tested easily enough (black-box test). If the class state is invisible or if you have to test tricky private methods, your test class may need to be a friend (white-box test).
A class that is hard to unit test will be one that
Has enormous dependencies, i.e. tightly coupled
Is intended to work in a high-volume or multi-threaded environment. There you would use a system test rather than a unit test and the actual output may not be totally determinate.
I written a fair number of blog posts about unit testing, non-trivial, C++ code: http://www.lenholgate.com/blog/2004/05/practical-testing.html
I've also written quite a lot about adding tests to existing code: http://www.lenholgate.com/blog/testing/
Almost everything can and should be unit tested. If not directly, then by using mock classes.
Since you decided to refactor your classes, try to use BDD or TDD approach.
To prevent breaking existing functionality, the only way is to have good integration tests, but usually it takes time to execute them all for a complex system.
Without more details on what you do, it is not that easy to give more implementation details. Some are :
use MVP or presenter first for developing gui
use design patterns where appropriate
use function and member pointers, or observer design pattern to break dependencies
I think that if you're having to come up with some "measure" to test if a class is testable, you're already fscked. You should be able to tell just by looking at it: can you write an independent program that links to this class alone and makes sure it works?
If a class is too huge so that you can't be sure just by looking at it...chances are it probably isn't testable. People that don't know how to make small, distinct interfaces generally don't know how to adhere to any other principle either.
In the end though, the way to find out if a class is testable is to try to put it in a harness. If you end up having to pull in half your program to do it, try refactoring. If you find that you can't even perform the most basic refactor without having to rewrite the entire program, analyze the expense of doing so.
We at IPL published a paper It's testing Jim, but not as we know it which explores the practical problems of testing C++ and suggests some techniques to address them that may well be of use given your question. These techniques are also well supported in Cantata++ - our C/C++ unit and integration testing tool.
In Osherove's great book "The Art of Unit Testing" one of the test anti-patterns is over-specification which is basically the same as testing the internal state of the object instead of some expected output. To my experience, using Isolation frameworks can cause the same unwanted side effects as testing internal behavior because one tends to only implement the behavior necessary to make your stub interact with the object under test. Now if your implementation changes later on (but the contract remains the same), your test will suddenly break because you are expecting some data from the stub which was not implemented.
So what do you think is the best approach to counter this?
1) Implement your stubs/mocks fully, this has the negative side-effect of potentially making your test less readable and also specifying more than necessary to make your test pass.
2) Favor manual, fully implemented fakes.
3) Implement your stubs/fakes so that they make your test just pass, and then deal with the brittleness that this might introduce.
I do not think you should favor manual testing - unless you prefer to test instead of code.
Instead you have another option - if you test the functionality and not the implementation, try to avoid testing private methods (that can be refactored) and in general write less-fragile tests you'll see that using a mocking/isolation framework does not require you to over specify the system nor does it cause your tests to become more fragile.
In a nutshell - writing fragile tests can be done with or without fakes/mocks and vise-versa.
I tend to use mocks instead of stubbed/fake objects. I find them a lot less trouble and they are way better at keeping test code under control because it's not cluttered with all sorts of half baked implementations. They also help to clarify what is being tested.
Another advantage is that I only have to address where the class under test needs something specific from the mock. So I don't have to code where it's not important. As for verification, again I only have to very the calls from the class under test to the mock that I care about and consider important aspects of the test.
I think, the problem is always the same, although it comes in different flavours: If you have tests that somehow cover the internals of a class, then you will break the tests that cover this internal code.
IMHO there are two ways to deal with that:
Your tests only cover the public contract of a class - a test strategy which is widely adopted for that exact reason: You don't have to change your tests as long as the public contract remains constant. Unfortunately, this is not, what you will have when doing Test-driven development.
If your tests come from a TDD process, then they will regularly cover non-public code. This means that they will break if you change the code. The only way to keep things in sync here is to 'fix' the tests together with the code. This means more maintenance during development. There's no recipe to easily deal with that (other than throw away the test, of course...).
My personal 'way out' is think in terms of 'code elements' rather than just code. A code element consists of three parts: Documentation, test, code. So if you change one part of the element, you have to also adjust the other two - otherwise you leave a broken code element behind.
You may think this question is like this question asked on StackOverflow earlier. But I am trying to look at things differently.
In TDD, we write tests that include different conditions, criteria, verification code. If a class passes all these tests we are good to go. It is a way of making sure that the class actually does what it's supposed to do and nothing else.
If you follow Bertrand Meyers' book Object Oriented Software Construction word by word, the class itself has internal and external contracts, so that it only does what its supposed to do and nothing else. No external tests required because the to code to ensure contract is followed is the part of the class.
Quick example to make things clear
TDD
Create test to ensure that in all cases a value ranges from (0-100)
Create a class containing a method that passes the test.
DBC
Create a class, create a contract for that member var to range from (0-100), set contract for contract breach, define a method.
I personally like the DBC approach.
Is there a reason why pure DBC is not so popular? Is it the languages or tools or being Agile or is it just me who likes to have code responsible for itself?
If you think I am not thinking right, I would be more than willing to learn.
The main problem with DBC is that in the vast majority of cases, either the contract cannot be formally specified (at least not conveniently), or it cannot be checked with current static analysis tool. Until we get past this point for mainstream languages (not Eiffel), DBC will not give the kind of assurance that people need.
In TDD, tests are written by a human being based on the current natural-text specifications of the method that are (hopefully) well-documented. Thus, a human interprets correctness by writing the test and gets some assurance based on that interpretation.
If you read Sun's guide for writing JavaDocs, it says that the documentation should essentially lay out a contract sufficient to write a test plan. Hence, design by contract is not necessarily mutually exclusive with TDD.
TDD and DbC are two different strategies. DbC permits fail-fast at runtime while TDD act "at compile time" (to be exact it add a new step right after the compilation to run the unit tests).
That's a big advantage of TDD over DbC : it allows to get earlier feedback. When you write code the TDD way, you get the code and its unit-tests at the same time, you can verify it "works" according to what you thought it should, which you encoded in the test. With DbC, you get code with embedded tests, but you still have to exercise it. IMO,this certainly is one reason that Dbc is not so popular.
Other advantages : TDD creates an automatic test suite that allow detecting (read preventing) regressions and make Refactoring safe, so that you can grow your design incrementally. DbC does not offer this possibility.
Now, using DbC to fail-fast can be very helpful, especially when your code interfaced other components or has to rely on external sources, in which case testing the contract can save you hours.
First of all, I am an Eiffel software engineer, so I can speak to the matter from experience.
The premise of TDD vs DbC is incorrect
The two technologies are not at odds with each other, but complementary to each other. The complement has to do with the placement of assertions and purpose.
The purpose of TDD has both components and scope. The basic components of TDD are boolean assertions and object feature (e.g. method) execution. The steps are simple:
Create an object.
Execute some code in a feature.
Make assertions about the state of the data on the object.
Assertions that fail, fail the test. Passing all assertions is the goal.
Like TDD, the contracts of Design-by-Contract have purpose, scope, and components. While TDD is limited to unit-test-time, contracts can live through the entire SDLC (Software Development Life-cycle)! Within the scope of TDD, execution of object methods (features), will execute the contracts. In an Eiffel Studio Auto-test (TDD) setup, one creates an object, makes the call (just like TDD in other languages), but here is where likeness ends.
In Eiffel Studio with Auto-test and Eiffel code with contracts, the purpose changes somewhat. We want to test the Client-Supplier relationship. Our TDD code is pretending to be a Client of our Supplier method on its object. We create our objects and call the methods based on this purpose, and not just simplistic "TDD-ish method testing". Because the calls to our methods (features) have contracts, those contracts will execute as a part of our TDD-ish code in Auto-test. Because this is true, contract assertions (tests) that we place in our code do NOT have to appear in our TDD test code. Our job (as a programmer) is to simply ensure: A) The code + contracts are executed along all N-paths, and B) The code + contracts are executed using all reasonable data types and ranges.
There is perhaps more to write about the TDD-DbC complement relationship, but I won't be boorish with you on the matter. Suffice it to say that TDD and DbC are NOT at odds with other—not by a long shot!
The power of the contracts of DbC beyond where TDD can reach
Now, we can turn our attention to the power of the contracts of Design-by-Contract beyond where TDD can reach!
Contracts live in the code. They are not external to it, but internal. The most powerful bit (beyond their client-supplier contract relationship basis) about contracts is that the compiler is designed to know about them! They are NOT a bolt-on addition to Eiffel! Thus, they participate in every aspect of inheritance (both traditional vertical is-a inheritance and in lateral or horizontal Generics). Moreover, they reach to a place that TDD cannot reach—inside the method (feature).
While TDD can mimic pre-conditions and post-conditions with some ease, TDD cannot reach inside the code and perform loop-invariant contracts, nor can it do periodic spot-check "check" contracts along a block of code as it is executing. This is a powerful logical and qualitative paradigm, and a reality about how design-by-contract works.
Moreover, TDD cannot do class invariants but in the faintest of ways. I have tried my hardest to get my Auto-test code (which is really just Eiffel Studios version of applied-TDD) to do class invariant mimicry. It is not possible. To understand why you would have to know the in's-and-out's of how Eiffel class invariants work. So, for the moment, you will simply have to either take my word for it (or not) that TDD is incapable of this task, that DbC handles so easily, well, and elegantly!
The reach of DbC does not end with the above notions
We noted above that TDD lives at unit-test-time. Contracts, because they are applied in code under the supervision and control of the compiler, apply anywhere that the code can be executed:
Workbench: you, as a programmer, are using the code to see it work (e.g. before TDD-time or in conjunction with TDD-time).
Unit-test: your continuous integration testing, unit-testing, TDD, etc.
Alpha-test: your initial test users will trip over contracts as they run the executable
Beta-test: a wider audience of users will also trip over contracts.
Production: the final executable (or production system) can have continual testing applied through contracts (TDD cannot).
In each of the situations above, one will find that one has control over just which contracts run and from what sources! You can selectively and fine-grainly turn on and off various forms of contracts and control with extreme precision where and when they are applied by the compiler!
And if all of this was not enough, contracts (by design) can do something that no TDD assertion can ever do: tell you where in the call-stack and which client-supplier relationship is broken, and why (which also immediately suggests how to fix it). Why is this true?
TDD assertions are designed to tell you about the results of a code-run (execution) after the fact. TDD assertion can only see as far as the current state of the method under examination. What TDD assertions cannot do from their position on the outside of the code-base is to tell you precisely which call failed and why! You see—your initial TDD call to some method will trigger that method. Many times, that method will call another, and another, and another—sometimes, as the call-stack winds up and down and hither and yon, there is a breakage: Something writes data wrong, does not write it at all, or writes it when it ought not.
TDD is like the police showing up to the crime scene after the murder has already happened. All they have left is forensic clues that they hope will lead them to a suspect and a conviction. But what if we could be there as the crime was taking place? That is the difference between the placement of TDD assertions and contract assertions. Contracts are there to catch the crime in progress and they point directly at the offender as it is committing the offense!
Recap
Let's recap.
TDD is not at odds with DbC.
It is a complement and a cooperative set of technologies, but with different functions and purposes, as well as tools to work with them.
Contract reach further and reveal more about your code when it breaks.
TDD is one form of catalyst for contracts to be executed.
At the end of the day: I want both! After reading all of this (if you survived), I hope you do as well.
Design-by-contract and test-driven development are not mutually exclusive.
Bertrand Meyer's book Object Oriented Software Construction, 2nd Edition doesn't say that you never make mistakes. Indeed, if you look at the chapter "When the contract is broken", it discusses what happens when a function fails to accomplish what its contract states.
The simple fact that you use the DbC technique doesn't make your code correct. Design-by-contract establishes well-defined rules for your code and its users, in the form of contracts. It's helpful, but you can always mess things up anyway, only that you'll probably notice earlier.
Test-driven development will check, from the outside world, black box style, that the public interface of your class behaves correctly.
I think it is best to use both methods in conjunction rather than just one or the other.
It has always seemed to me that fully enforcing a contract within the class and its methods themselves can be impractical.
For example, if a function says it will hash a string by some method and return the hashed string as output, how does the function enforce that the string was hashed correctly? Hash it again and see if they match? Seems silly. Reverse the hash to see if you get the original? Not possible. Rather, you need a set of test cases to ensure that the function behaves correctly.
On the other hand, if your particular implementation requires that your input data be of a certain size, then establishing a contract and enforcing it in your code seems like the best approach.
In my mind TDD is more "inductive". You start with examples (test cases) and your code embodies the general solution to those examples.
DBC seems more "deductive", after gathering requirements you determine object behavior and contracts. You then code the specific implementation of those contracts.
Writing contracts is somewhat difficult, more so than tests that are concrete examples of behavior, this may be part of the reason TDD is more popular than DBC.
I see no reason why both cannot co-exist. It is wonderful to look at a method and know at a glance exactly what the contract is. It is also wonderful to know that I can run my unit tests and know that nothing was broken with my last change. The two techniques are not mutually exclusive. Why design by contract is not more popular is a mystery.
I've used both in the past and found DBC-style less "intrusive". The driver for DBC may be regular application running. For Unit Tests you have to take care of setup because you expect (validate) some responses. For DBC you don't have to. Rules are written in data-independent manner, so no need to setup and mocking around.
More on my experiences with DBC/Python: http://blog.aplikacja.info/2012/04/classic-testing-vs-design-by-contract/
I see Design By Contract as a specification for success/failure in ALL cases, whereas Test Driven Development targets ONE specific case. If the TDD case succeeds, then it is assumed a function is doing it's job, but it doesn't take into account other cases that could cause it to fail.
Design By Contract on the other hand doesn't necessary guarantee the desired answer, only that the answer is "correct." For example, if a function returns is supposed to return a non-null string, the only thing you can assume in the ENSURE is that it will not be null.
But maybe it doesn't not return the string that was expected. There is no way for a contract to be able to determine that, only a Test can show that it was performing according to the specification.
So the two are complementary.
Greg