Are we supposed to write tests for our getters and setters or is it overkill?
I would say no.
#Will said you should aim for 100% code coverage, but in my opinion that's a dangerous distraction. You can write unit tests that have 100% coverage, and yet test absolutely nothing.
Unit tests are there to test the behaviour of your code, in an expressive and meaningful way, and getters/setters are only a means to an end. If you tests use the getters/setters to achieve their goal of testing the "real" functionality, then that's good enough.
If, on the other hand, your getters and setters do more than just get and set (i.e. they're properly complex methods), then yes, they should be tested. But don't write a unit test case just to test a getter or setters, that's a waste of time.
Roy Osherove in his famous book 'The Art Of Unit Testing' says:
Properties (getters/setters in Java) are good examples of code that usually doesn’t contain any logic, and doesn’t require testing. But watch out: once you add any check inside the property, you’ll want to make sure that logic is being tested.
A resounding YES with TDD
Note: This answer keeps getting upvotes, albeit potentially a bad advice. To understand why, have a look at its little sister below.
Controversial alright, but I'd argue that anyone who answers 'no' to this question is missing a fundamental concept of TDD.
For me, the answer is a resounding yes if you follow TDD. If you aren't then no is a plausible answer.
The DDD in TDD
TDD is often quoted as having thee main benefits.
Defence
Ensuring the code may change but not its behaviour.
This allows the ever so important practice of refactoring.
You gain this TDD or not.
Design
You specify what something should do, how it should behaves before implementing it.
This often means more informed implementation decisions.
Documentation
The test suite should serve as the specification (requirements) documentation.
Using tests for such purpose mean that the documentation and implementation are always in consistent state - a change to one means a change to other. Compare with keeping requirements and design on separate word document.
Separate responsibility from implementation
As programmers, it is terribly tempting to think of attributes as something of significance and getters and setter as some sort of overhead.
But attributes are an implementation detail, while setters and getters are the contractual interface that actually make programs work.
It is far more important to spell that an object should:
Allow its clients to change its state
and
Allow its clients to query its state
then how this state is actually stored (for which an attribute is the most common, but not the only way).
A test such as
(The Painter class) should store the provided colour
is important for the documentation part of TDD.
The fact that the eventual implementation is trivial (attribute) and carries no defence benefit should be unknown to you when you write the test.
The lack of round-trip engineering...
One of the key problems in the system development world is the lack of round-trip engineering1 - the development process of a system is fragmented into disjointed sub-processes the artifacts of which (documentation, code) are often inconsistent.
1Brodie, Michael L. "John Mylopoulos: sewing seeds of conceptual modelling." Conceptual Modeling: Foundations and Applications. Springer Berlin Heidelberg, 2009. 1-9.
...and how TDD solves it
It is the documentation part of TDD that ensures that the specifications of the system and its code are always consistent.
Design first, implement later
Within TDD we write failing acceptance test first, only then write the code that let them pass.
Within the higher-level BDD, we write scenarios first, then make them pass.
Why should you exclude setters and getter?
In theory, it is perfectly possible within TDD for one person to write the test, and another one to implement the code that makes it pass.
So ask yourself:
Should the person writing the tests for a class mention getters and setter.
Since getters and setters are a public interface to a class, the answer is obviously yes, or there will be no way to set or query the state of an object. However, the way to do this is not necessarily by testing each method in isolation, see my other answer for more.
Obviously, if you write the code first, the answer may not be so clearcut.
tl;dr: Yes you should, and with OpenPojo it's trivial.
You should be doing some validation in your getters and setters so you should be testing that. For example, setMom(Person p) should not allow setting anyone younger than themselves as their mother.
Even if you aren't doing any of that now, odds are you will in the future, then this will be a good for regression analysis. If you want to allow setting mothers to null you should have a test for that should someone change that later on, this will reinforce your assumptions.
A common bug is void setFoo( Object foo ){ foo = foo; } where it should be void setFoo( Object foo ){ this.foo = foo; }. (In the first case the foo that is being written to is the parameter not the foo field on the object).
If you are returning an array or collection you should be testing whether or not the getter is going to be performing defensive copies of the data passed into the setter before returning.
Otherwise, if you have the most basic setters/getters then unit-testing them will add maybe about 10 minutes at most per-object, so what is the loss? If you add behaviour you already have a skeleton test and you get this regression testing for free. If you are using Java, you have no excuse since there is OpenPojo. There are an existing set of rules you can enable and then scan your entire project with them to make sure they are applied consistently within your code.
From their examples:
final Validator pojoValidator = ValidatorBuilder.create()
.with(
new NoPublicFieldsRule (),
new NoPrimitivesRule (),
new GetterMustExistRule (),
new SetterMustExistRule ()
)
.with(
new DefaultValuesNullTester (),
new SetterTester (),
new GetterTester ()
)
.build();
pojoValidator.validate( PojoClassFactory.getPojoClasses( "net.initech.app", new FilterPackageInfo() ) );
Yes, but not always in isolation
Allow me elaborate:
What is a unit test?
From Working effectively with legacy code1:
The term unit test has a long history in software development. Common to
most conceptions of unit tests is the idea that they are tests in isolation of individual
components of software. What are components? The definition varies,
but in unit testing, we are usually concerned with the most atomic behavioral units of a system. In procedural code, the units are often functions. In object oriented code, the units are classes.
Note that with OOP, where you find getters and setters, the unit is the class, not necessarily individual methods.
What is a good test?
All requirements and tests follow the form of Hoare logic:
{P} C {Q}
Where:
{P} is the precondition (given)
C is the trigger condition (when)
{Q} is the postcondition (then)
Then comes the maxim:
Test behaviour, not implementation
This means that you shouldn't test how C achieves the post-condition, you should check that {Q} is the result of C.
When it comes to OOP, C is a class. So you shouldn't test internal effects, only external effects.
Why not test bean getters and setters in isolation
Getters and setters may involve some logic, but so long this logic does not have external effect - making them bean accessors2) a test will have to look inside the object and by that not only violate encapsulation but also test for implementation.
So you shouldn't test bean getters and setters in isolation. This is bad:
Describe 'LineItem class'
Describe 'setVAT()'
it 'should store the VAT rate'
lineItem = new LineItem()
lineItem.setVAT( 0.5 )
expect( lineItem.vat ).toBe( 0.5 )
Although if setVAT would throw an exception, a corresponding test would be appropriate since now there is an external effect.
How should you test getters and setters?
There is virtually no point changing the internal state of an object if such change has no effect on the outside, even if such effect comes later on.
So a test for setters and getters should be concerned with the external effect of these methods, not the internal ones.
For example:
Describe 'LineItem class'
Describe 'getGross()'
it 'should return the net time the VAT'
lineItem = new LineItem()
lineItem.setNet( 100 )
lineItem.setVAT( 0.5 )
expect( lineItem.getGross() ).toBe( 150 )
You may think to yourself:
Wait a sec, we are testing getGross() here not setVAT().
But if setVAT() malfunction that test should fail all the same.
1Feathers, M., 2004. Working effectively with legacy code. Prentice Hall Professional.
2Martin, R.C., 2009. Clean code: a handbook of agile software craftsmanship. Pearson Education.
While there are justified reasons for Properties, there's a common Object Oriented Design belief that exposing member state via Properties is bad design. Robert Martin's article on the Open Closed Principle expands upon this by stating that Properties encourage coupling and therefore limit the ability to close a class from modification -- if you modify the property, all consumers of the class will need to change as well. He qualifies that exposing member variables isn't necessarily bad design, it might just be poor style. However, if properties are read-only, there's less chance of abuse and side-effects.
The best approach I can provide for unit testing (and this may seem odd) is to make as many properties as possible protected or internal. This will prevent coupling while discouraging writing silly tests for getters and setters.
There are obvious reasons where read/write Properties should be used, such as ViewModel properties that are bound to input fields, etc.
More practically, unit tests should drive functionality through public methods. If the code you're testing happens to use those properties, you get code-coverage for free. If it turns out that these properties never get highlighted by code-coverage there's a very strong possibility that:
You are missing tests that indirectly use the properties
The properties are unused
If you write tests for getters and setters, you get a false sense of coverage and will not be able to determine if the properties are actually used by functional behavior.
If the cyclomatic complexity of the getter and/or setter is 1 (which they usually are), then the answer is no, you shouldn't.
So unless you have a SLA that requires 100% code-coverage, don't bother, and focus on testing the important aspect of your software.
P.S. Remember to differentiate getters and setters, even in languages like C# where properties might seem like the same thing. The setter complexity can be higher than the getter, and thus validate a unit-test.
A humorous, yet wise take: The Way of Testivus
"Write the test you can today"
Testing getters/setters may be overkill if you're an experienced tester and this is a small project. However, if you're just getting started learning how to unit test or these getters/setters may contain logic (like #ArtB's setMom() example) then it would be a good idea to write tests.
This has actually been a recent topic between my team and I. We shoot for 80% code coverage. My team argues that getters and setters are auto implemented, and the compiler is generating some basic code behind the scenes. In this case, given the code generated is non-intrusive, it doesn't really make sense to test code the compiler creates for you. We also had this discussion about async methods and in that case the compiler generates a whole bunch of code behind the scenes. This is a different case and something we DO test. Long answer short, bring it up with your team and decide for yourselves if its worth testing.
Also, if you are using the code coverage report like us, one thing you can do is add the [ExcludeFromCodeCoverage] attribute. Our solution has been to use this for models that just have properties using getters and setters, or on the property itself. That way it won't affect the total code coverage % when the code coverage report is run, assuming that is what you are using to calculate your code coverage percentages. Happy testing!
I did a little analysis of the coverage achieved in the JUnit code itself.
One category of uncovered code is "too simple to test". This includes simple getters and setters, which the developers of JUnit do not test.
On the other hand, JUnit doesn't have any (non-deprecated) method longer than 3 lines that is not covered by any test.
In my opinion code coverage is a good way to see if you missed any functionality that you should cover.
When you inspect the coverage manually by it's pretty coloring then it can be argued that plain getters and setters do not need to be tested (although I always do).
When you only check the code coverage percentage on your project, then a test coverage percentage like 80% is meaningless. You can test all the none logical parts and forget some crucial parts. In this case only 100% means that you have tested all you vital code (and all the non-logical code as well). As soon as it is 99.9% you know that have forgotten something.
By the way: Code coverage is the final check to see if you have fully (unit) tested a class. But 100% code coverage not necessarily means that you have actually tested all the functionality of the class. So unit tests should always be implemented following the logic of the class. In the end you run coverage to see if you forgot anything. When you did it right, you hit 100% the first time.
One more thing: While recently working at a large bank in The Netherlands I noticed that Sonar indicated 100% code coverage. However, I knew something was missing. Inspecting the code coverage percentages per file it indicated a file at a lower percentage. The whole code base percentage was that large that the one file did not make the percentage be displayed as 99.9%. So you might want to look out for this...
I would say: YES
Errors in getter/setter methods can silently sneak in and cause some ugly bugs.
I have written a lib to make this and some other tests easier.
The only thing you have to write in your JUnit tests is this:
assertTrue(executor.execute(*TheClassIWantToTest.class*, Arrays.asList( new DefensiveCopyingCheck(),
new EmptyCollectionCheck(), new GetterIsSetterCheck(),
new HashcodeAndEqualsCheck(), new PublicVariableCheck())));
-> https://github.com/Mixermachine/base-test
Yes, especially if the item to get is an object of a class subclassed from an abstract class. Your IDE may or may not alert you that a certain property has not been initialized.
And then some apparently unrelated test crashes with a NullPointerException and it takes you a while to figure out that a gettable property is actually not there to get in the first place.
Though that still wouldn't be anywhere as bad as discovering the problem in production.
It might be a good idea to make sure all your abstract classes have constructors. If not, the test of a getter might alert you to a problem there.
As for getters and setters of primitives, the question might be: Am I testing my program or am I testing the JVM or CLR? Generally speaking, the JVM does not need to be tested.
Related
In my last project, we had Unit Testing with almost 100% cc, and as a result we almost didn’t have any bugs.
However, since Unit Testing must be White Box (you have to mock inner functions to get the result you want, so your tests need to know about the inner structure of your code) any time we changed the implementation of a function, we had to change the tests as well.
Note that we didn't change the logic of the functions, just the implementation.
It was very time-consuming and it felt as if we are working the wrong way.
Since we used all proper OOP guidelines (specifically Encapsulation), every time we changed the implementation we didn't have to change the rest of our code, but had to change the unit tests.
It felt as if we are serving the tests, instead of them serving us.
To prevent this, some of us argued that unit tests should be Black Box Testing.
That would be possible if we create one big mock of our entire Domain and create a stub for every function in every class in one place, and use it in every unit test.
Of course that if a specific test needs specific inner function to be called (Like making sure we write to the DB), we can override our stub.
So, every time we change the implementation of a function (like adding or replacing a call to a help function) we will only need to change our main big mock. Even if we do need to change some unit tests, it will still be much less than before.
Others argue that unit tests must be White Box, since not only do you want to make sure your app writes to the DB in a specific place, you want to make sure your app does not write to the DB anywhere else unless you specifically expect it to. While this is a valid point, I don't think it worth the time of writing White Box tests instead of Black Box tests.
So in conclusion, two questions:
What do you think about the concept of Black Box Unit Testing?
What do you think about the way we want to implement that concept? Do you have better ideas?
You need different types of tests.
Unit-tests which should be white-box testing, as you did
Integration tests (or system tests) which test the ability to use the actual implementations of your system and its communication with external layers (external systems, database, etc.) which should be black-box styled, but each one for a specific feature (CRUD tests for example)
Acceptance tests which should be completely black-box and are driven by functional requirements (as your users would phrase them). End-to-end as much as possible, and not knowing the internal of your chosen implementations. The textbook definition of black-box tests.
And remember code coverage is meaningless in most of the cases. You need a high lines coverage (or methods coverage, whatever your counting method is), but that's usually not sufficient. The concept you need to think about is functional coverage: making sure all your requirements and logical paths are covered.
and as a result we almost didn’t have any bugs
If you were really able to achieve this, then I don't think you should change anything.
Black box testing might sound appealing on paper, but truth is you almost always need to know parts of inner workings of a tested class. The provide input, verify output in reality works only for simple cases. Most of the times your tests need to have at least some knowledge of tested method - how it interacts with external collaborators, what methods it calls, in what order and so forth.
Whole idea behind mocking and SOLID design is to avoid situation where dependency implementation change causes other class test changes/failures. On contrary, if you change implementation details of tested method, so should change implementation details of it tests. That's nothing too uncommon.
Overall, if you were really able to achieve almost no bugs, then I would stick to that approach.
tl;dr version:
Black Box unit testing is exactly how unit testing should be done.
Black Box unit testing is exactly how unit testing should be done. Proper TDD practice does exactly this.
Full version.
There is absolutely no need in testing private methods of the objects. It'll have no impact on code coverage, also.
When you TDD a class, you write tests that check the behavior of that class. Behavior is expressed through the public methods of that class. You should never bother with how that methods are really implemented. Google people described that a lot better than I will ever be able to: http://googletesting.blogspot.ru/2013/08/testing-on-toilet-test-behavior-not.html
If you do the usual mistake and statically depend on other entity classes or worse, on classes from the different layer of application, it's inevitable that you will find yourself in a situation when you need to check a lot of things in your test and prepare a lot of stuff for it. For solving this the Dependency Injection principle and the Law of Demeter exist.
I think you should continue writing unit tests - just make them less fragile.
Unit tests should be low level but should test the result and not how things done. When implementation change cause a lot of test change it means that instead of testing requirements you're actually testing implementation.
There are several rules of the thumb - such as "don't test private methods" and use mock objects.
Mocking/simulating the entire domain usually result in the opposite of what you're trying to accomplish - when the code behavior change you need to update the tests to make sure that your "simulated objects" behaves the same - it becomes really hard really fast as the complexity of the project increase.
I suggest that you continue writing unit tests - just learn how to make them more robust and less fragile.
"as a result we almost didn’t have any bugs" -- so keep it that way.
Sole cause of frustration is necessity to maintain unit tests, which actually is not such a bad thing (alternative is much worse). Just make them more maintainable. "The art of Unit Testing" by Roy Osherove gave me a good start in this way.
So
1) Not an option. (The idea itself contradicts principles of TDD, for instance)
2) You'll have much more maintenance troubles with such approach. Unit testing philosophy is to chop out SUT from other system and test it using stubs as input and mocks as output (signals?) simulating real life situations (or mb I just dont catch the "one big mock of our entire Domain" idea).
For detailed information about black, white and grey box and decision tables refer to the following article, which explains everything.
Testing Web-based applications: The state of the art and future trends (PDF)
Since a few days ago I've started to feel interested in Unit Testing and TDD in C# and VS2010. I've read blog posts, watched youtube tutorials, and plenty more stuff that explains why TDD and Unit Testing are so good for your code, and how to do it.
But the biggest problem I find is, that I don't know what to check in my tests and what not to check.
I understand that I should check all the logical operations, problems with references and dependencies, but for example, should I create an unit test for a string formatting that's supossed to be user-input? Or is it just wasting my time while I just can check it in the actual code?
Is there any guide to clarify this problem?
In TDD every line of code must be justified by a failing test-case written before the code.
This means that you cannot develop any code without a test-case. If you have a line of code (condition, branch, assignment, expression, constant, etc.) that can be modified or deleted without causing any test to fail, it means this line of code is useless and should be deleted (or you have a missing test to support its existence).
That is a bit extreme, but this is how TDD works. That being said if you have a piece of code and you are wondering whether it should be tested or not, you are not doing TDD correctly. But if you have a string formatting routine or variable incrementation or whatever small piece of code out there, there must be a test case supporting it.
UPDATE (use-case suggested by Ed.):
Like for example, adding an object to a list and creating a test to see if it is really inside or there is a duplicate when the list shouldn't allow them.
Here is a counterexample, you would be surprised how hard it is to spot copy-paste errors and how common they are:
private Set<String> inclusions = new HashSet<String>();
private Set<String> exclusions = new HashSet<String>();
public void include(String item) {
inclusions.add(item);
}
public void exclude(String item) {
inclusions.add(item);
}
On the other hand testing include() and exclude() methods alone is an overkill because they do not represent any use-cases by themselves. However, they are probably part of some business use-case, you should test instead.
Obviously you shouldn't test whether x in x = 7 is really 7 after assignment. Also testing generated getters/setters is an overkill. But it is the easiest code that often breaks. All too often due to copy&paste errors or typos (especially in dynamic languages).
See also:
Mutation testing
Your first few TDD projects are going to probably result in worse design/redesign and take longer to complete as you are learning (at least in my experience). This is why you shouldn't jump into using TDD on a large critical project.
My advice is to use "pure" TDD (acceptance/unit test everything test-first) on a few small projects (100-10,000 LOC). Either do the side projects on your own or if you don't code in your free time, use TDD on small internal utility programs for your job.
After you do "pure" TDD on about 6-12 projects, you will start to understand how TDD affects design and learn how to design for testability. Once you know how to design for testability, you will need to TDD less and maximize the ROI of unit, regression, acceptance, etc. tests rather than test everything up front.
For me, TDD is more of teaching method for good code design than a practical methodology. However, I still TDD logic code and unit test instead of debug.
There is no simple answer to this question. There is the law of diminishing returns in action, so achieving perfect coverage is seldom worth it. Knowing what to test is a thing of experience, not rules. It’s best to consciously evaluate the process as you go. Did something break? Was it feasible to test? If not, is it possible to rewrite the code to make it more testable? Is it worth it to always test for such cases in the future?
If you split your code into models, views and controllers, you’ll find that most of the critical code is in the models, and those should be fairly testable. (That’s one of the main points of MVC.) If a piece of code is critical, I test it, even if it means that I would have to rewrite it to make it more testable. If a piece of code is easy to get wrong or get broken by future updates, it gets a test. I seldom test controllers and views, as it’s not proving worth the trouble for me.
The way I see it all of your code falls into one of three buckets:
Code that is easy to test: This includes your own deterministic public methods.
Code that is difficult to test: This includes GUI, non-deterministic methods, private methods, and methods with complex setup.
Code that you don't want to test: This includes 3rd party code, and code that is difficult to test and not worth the effort.
Of the three, you should focus on testing the easy code. The difficult to test code should be refactored so that into two parts: code that you don't want to test and easy code. And of course, you should test the refactored easy code.
I think you should only unit test entry points to behavior of the system. This include public methods, public accessors and public fields, but not constants (constant fields, enums, methods, etc.). It also includes any code which directly deals with IO, I explain why further below.
My reasoning is as follows:
Everything that's public is basically an entry point to a behavior of the system. A unit test should therefore be written that guarantees that the expected behavior of that entry point works as required. You shouldn't test all possible ways of calling the entry point, only the ones that you explicitly require. Your unit tests are therefore also the specs of what behavior your system supports and your documentation of how to use it.
Things that are not public can basically be deleted/re-factored at will with no impact to the behavior of the system. If you were to test those, you'd create a hard dependency from your unit test to that code, which would prevent you from doing refactoring on it. That's why you should not test anything else but public methods, fields and accessors.
Constants by design are not behavior, but axioms. A unit test that verifies a constant is itself a constant, so it would only be duplicated code and useless effort to write a test for constants.
So to answer your specific example:
should I create an unit test for a string formatting that's supossed
to be user-input?
Yes, absolutely. All methods which receive or send external input/output (which can be summed up as receiving IO), should be unit tested. This is probably the only case where I'd say non-public things that receive IO should also be unit tested. That's because I consider IO to be a public entry. Anything that's an entry point to an external actor I consider public.
So unit test public methods, public fields, public accessors, even when those are static constructs and also unit test anything which receives or sends data from an external actor, be it a user, a database, a protocol, etc.
NOTE: You can write temporary unit tests on non public things as a way for you to help make sure your implementation works. This is more of a way to help you figure out how to implement it properly, and to make sure your implementation works as you intend. After you've tested that it works though, you should delete the unit test or disable it from your test suite.
Kent Beck, in Extreme Programming Explained, said you only need to test the things that need to work in production.
That's a brusque way of encapsulating both test-driven development, where every change in production code is supported by a test that fails when the change is not present; and You Ain't Gonna Need It, which says there's no value in creating general-purpose classes for applications that only deal with a couple of specific cases.
I think you have to change your point of view.
In a pure form TDD requires the red-green-refactor workflow:
write test (it must fail) RED
write code to satisfy test GREEN
refactor your code
So the question "What I have to test?" has a response like: "You have to write a test that correspond to a feature or a particular requirements".
In this way you get must code coverage and also a better code design (remember that TDD stands also for Test Driven "Design").
Generally speaking you have to test ALL public method/interfaces.
should I create an unit test for a string formatting that's supossed
to be user-input? Or is it just wasting my time while I just can check
it in the actual code?
Not sure I understand what you mean, but the tests you write in TDD are supposed to test your production code. They aren't tests that check user input.
To put it another way, there can be TDD unit tests that test the user input validation code, but there can't be TDD unit tests that validate the user input itself.
In Osherove's great book "The Art of Unit Testing" one of the test anti-patterns is over-specification which is basically the same as testing the internal state of the object instead of some expected output. To my experience, using Isolation frameworks can cause the same unwanted side effects as testing internal behavior because one tends to only implement the behavior necessary to make your stub interact with the object under test. Now if your implementation changes later on (but the contract remains the same), your test will suddenly break because you are expecting some data from the stub which was not implemented.
So what do you think is the best approach to counter this?
1) Implement your stubs/mocks fully, this has the negative side-effect of potentially making your test less readable and also specifying more than necessary to make your test pass.
2) Favor manual, fully implemented fakes.
3) Implement your stubs/fakes so that they make your test just pass, and then deal with the brittleness that this might introduce.
I do not think you should favor manual testing - unless you prefer to test instead of code.
Instead you have another option - if you test the functionality and not the implementation, try to avoid testing private methods (that can be refactored) and in general write less-fragile tests you'll see that using a mocking/isolation framework does not require you to over specify the system nor does it cause your tests to become more fragile.
In a nutshell - writing fragile tests can be done with or without fakes/mocks and vise-versa.
I tend to use mocks instead of stubbed/fake objects. I find them a lot less trouble and they are way better at keeping test code under control because it's not cluttered with all sorts of half baked implementations. They also help to clarify what is being tested.
Another advantage is that I only have to address where the class under test needs something specific from the mock. So I don't have to code where it's not important. As for verification, again I only have to very the calls from the class under test to the mock that I care about and consider important aspects of the test.
I think, the problem is always the same, although it comes in different flavours: If you have tests that somehow cover the internals of a class, then you will break the tests that cover this internal code.
IMHO there are two ways to deal with that:
Your tests only cover the public contract of a class - a test strategy which is widely adopted for that exact reason: You don't have to change your tests as long as the public contract remains constant. Unfortunately, this is not, what you will have when doing Test-driven development.
If your tests come from a TDD process, then they will regularly cover non-public code. This means that they will break if you change the code. The only way to keep things in sync here is to 'fix' the tests together with the code. This means more maintenance during development. There's no recipe to easily deal with that (other than throw away the test, of course...).
My personal 'way out' is think in terms of 'code elements' rather than just code. A code element consists of three parts: Documentation, test, code. So if you change one part of the element, you have to also adjust the other two - otherwise you leave a broken code element behind.
This question already has answers here:
When should I mock?
(4 answers)
Closed 9 years ago.
Classes that use other classes (as members, or as arguments to methods) need instances that behave properly for unit test. If you have these classes available and they introduce no additional dependencies, isn't it better to use the real thing instead of a mock?
I say use real classes whenever you can.
I'm a big believer in expanding the boundaries of "unit" tests as much as possible. At this point they aren't really unit tests in the traditional sense, but rather just an automated regression suite for your application. I still practice TDD and write all my tests first, but my tests are a little bigger than most people's and my green-red-green cycles take a little longer. But now that I've been doing this for a little while I'm completely convinced that unit tests in the traditional sense aren't all they're cracked up to be.
In my experience writing a bunch of tiny unit tests ends up being an impediment to refactoring in the future. If I have a class A that uses B and I unit test it by mocking out B, when I decide to move some functionality from A to B or vice versa all of my tests and mocks have to change. Now if I have tests that verify that the end to end flow through the system works as expected then my tests actually help me to identify places where my refactorings might have caused a change in the external behavior of the system.
The bottom line is that mocks codify the contract of a particular class and often end up actually specifying some of the implementation details too. If you use mocks extensively throughout your test suite your code base ends up with a lot of extra inertia that will resist any future refactoring efforts.
It is fine to use the "real thing" as long as you have absolute control over the object. For example if you have an object that just has properties and accessors you're probably fine. If there is logic in the object you want to use, you could run into problems.
If a unit test for class a uses an instance of class b and an change introduced to b breaks b, then the tests for class a are also broken. This is where you can run into problems where as with a mock object you could always return the correct value. Using "the real thing" Can kind of convolute tests and hide the real problem.
Mocks can have downsides too, I think there is a balance with some mocks and some real objects you will have to find for yourself.
There is one really good reason why you want to use stubs/mocks instead of real classes. I.e. to make your unit test's (pure unit test) class under test isolated from everything else. This property is extremely useful and the benefits for keeping tests isolated are plentiful:
Tests run faster because they don't need to call the real class implementation. If the implementation is to run against file system or relational database then the tests will become sluggish. Slow tests make developers not run unit tests as often. If you're doing Test Driven Development then time hogging tests are together a devastating waste of developers time.
It will be easier to track down problems if the test is isolated to the class under test. In contrast to a system test it will be much more difficult to track down nasty bugs that are not apparently visible in stack traces or what not.
Tests are less fragile on changes done on external classes/interfaces because you're purely testing the class that is under test. Low fragility is also an indication of low coupling, which is a good software engineering.
You're testing against external behaviour of a class rather than the internal implementation which is more useful when deciding code design.
Now if you want to use real class in your test, that's fine but then it is NOT a unit test. You're doing a integration test instead, which is useful for the purpose of validating requirements and overall sanity check. Integration tests are not run as often as unit tests, in practice it is mostly done before committing to favorite code repository, but is equally important.
The only thing you need to have in mind is the following:
Mocks and stubs are for unit tests.
Real classes are for integration/system tests.
Extracted and extended from an answer of mine How do I unit-test inheriting objects?">here:
You should always use real objects where possible.
You should only use mock objects if the real objects do something you dont want to set up (like use sockets, serial ports, get user input, retrieve bulky data etc). Essentially, mock objects are for when the estimated effort to implement and maintain a test using a real object is greater than that to implement and maintain a test using a mock object.
I dont buy into the "dependant test failure" argument. If a test fails because a depended-on class broke, the test did exactly what it should have done. This is not a smell! If a depended-on interface changes, I want to know!
Highly mocked testing environments are very high-maintenance, particularly early in a project when interfaces are in flux. Ive always found it better to start integration testing ASAP.
I always use a mock version of a dependency if the dependency accesses an external system like a database or web service.
If that isn't the case, then it depends on the complexity of the two objects. Testing the object under test with the real dependency is essentially multiplying the two sets of complexities. Mocking out the dependency lets me isolate the object under test. If either object is reasonably simple, then the combined complexity is still workable and I don't need a mock version.
As others have said, defining an interface on the dependency and injecting it into the object under test makes it much easier to mock out.
Personally, I'm undecided about whether it's worth it to use strict mocks and validate every call to the dependency. I usually do, but it's mostly habit.
You may also find these related questions helpful:
What is object mocking and when do I need it?
When should I mock?
How are mocks meant to be used?
And perhaps even, Is it just me, or are interfaces overused?
Use the real thing only if it has been unit tested itself first. If it introduces dependencies that prevent that (circular dependencies or if it requires certain other measures to be in place first) then use a 'mock' class (typically referred to as a "stub" object).
If your 'real things' are simply value objects like JavaBeans then thats fine.
For anything more complex I would worry as mocks generated from mocking frameworks can be given precise expectations about how they will be used e.g. the number of methods called, the precise sequence and the parameters expected each time. Your real objects cannot do this for you so you risk losing depth in your tests.
I've been very leery of mocked objects since I've been bitten by them a number of times. They're great when you want isolated unit tests, but they have a couple of issues. The major issue is that if the Order class needs a a collection of OrderItem objects and you mock them, it's almost impossible to verify that the behavior of of the mocked OrderItem class matches the real-world example (duplicating the methods with appropriate signatures is generally not enough). More than once I've seen systems fail because the mocked classes don't match the real ones and there weren't enough integration tests in place to catch the edge cases.
I generally program in dynamic languages and I prefer merely overriding the specific methods which are problematic. Unfortunately, this is sometimes hard to do in static languages. The downside of this approach is that you're using integration tests rather than unit tests and bugs are sometimes harder to track down. The upside is that you're using the actual code that is written, rather than a mocked version of that code.
If you don't care for verifying expectations on how your UnitUnderTest should interact with the Thing, and interactions with the RealThing have no other side-effects (or you can mock these away) then it is in my opinion perfectly fine to just let your UnitUnderTest use the RealThing.
That the test then covers more of your code base is a bonus.
I generally find it is easy to tell when I should use a ThingMock instead of a RealThing:
When I want to verify expectations in the interaction with the Thing.
When using the RealThing would bring unwanted side-effects.
Or when the RealThing is simply too hard/troublesome to use in a test setting.
If you write your code in terms of interfaces, then unit testing becomes a joy because you can simply inject a fake version of any class into the class you are testing.
For example, if your database server is down for whatever reason, you can still conduct unit testing by writing a fake data access class that contains some cooked data stored in memory in a hash map or something.
It depends on your coding style, what you are doing, your experience and other things.
Given all that, there's nothing stopping you from using both.
I know I use the term unit test way too often. Much of what I do might be better called integration test, but better still is to just think of it as testing.
So I suggest using all the testing techniques where they fit. The overall aim being to test well, take little time doing it and personally have a solid feeling that it's right.
Having said that, depending on how you program, you might want to consider using techniques (like interfaces) that make mocking less intrusive a bit more often. But don't use Interfaces and injection where it's wrong. Also if the mock needs to be fairly complex there is probably less reason to use it. (You can see a lot of good guidance, in the answers here, to what fits when.)
Put another way: No answer works always. Keep your wits about you, observe what works what doesn't and why.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
The thing I've found about TDD is that its takes time to get your tests set up and being naturally lazy I always want to write as little code as possible. The first thing I seem do is test my constructor has set all the properties but is this overkill?
My question is to what level of granularity do you write you unit tests at?
..and is there a case of testing too much?
I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don't typically make a kind of mistake (like setting the wrong variables in a constructor), I don't test for it. I do tend to make sense of test errors, so I'm extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.
Different people will have different testing strategies based on this philosophy, but that seems reasonable to me given the immature state of understanding of how tests can best fit into the inner loop of coding. Ten or twenty years from now we'll likely have a more universal theory of which tests to write, which tests not to write, and how to tell the difference. In the meantime, experimentation seems in order.
Write unit tests for things you expect to break, and for edge cases. After that, test cases should be added as bug reports come in - before writing the fix for the bug. The developer can then be confident that:
The bug is fixed;
The bug won't reappear.
Per the comment attached - I guess this approach to writing unit tests could cause problems, if lots of bugs are, over time, discovered in a given class. This is probably where discretion is helpful - adding unit tests only for bugs that are likely to re-occur, or where their re-occurrence would cause serious problems. I've found that a measure of integration testing in unit tests can be helpful in these scenarios - testing code higher up codepaths can cover the codepaths lower down.
Everything should be made as simple as
possible, but not simpler. - A. Einstein
One of the most misunderstood things about TDD is the first word in it. Test. That's why BDD came along. Because people didn't really understand that the first D was the important one, namely Driven. We all tend to think a little bit to much about the Testing, and a little bit to little about the driving of design. And I guess that this is a vague answer to your question, but you should probably consider how to drive your code, instead of what you actually are testing; that is something a Coverage-tool can help you with. Design is a quite bigger and more problematic issue.
To those who propose testing "everything": realise that "fully testing" a method like int square(int x) requires about 4 billion test cases in common languages and typical environments.
In fact, it's even worse than that: a method void setX(int newX) is also obliged not to alter the values of any other members besides x -- are you testing that obj.y, obj.z, etc. all remain unchanged after calling obj.setX(42);?
It's only practical to test a subset of "everything." Once you accept this, it becomes more palatable to consider not testing incredibly basic behaviour. Every programmer has a probability distribution of bug locations; the smart approach is to focus your energy on testing regions where you estimate the bug probability to be high.
The classic answer is "test anything that could possibly break". I interpret that as meaning that testing setters and getters that don't do anything except set or get is probably too much testing, no need to take the time. Unless your IDE writes those for you, then you might as well.
If your constructor not setting properties could lead to errors later, then testing that they are set is not overkill.
I write tests to cover the assumptions of the classes I will write. The tests enforce the requirements. Essentially, if x can never be 3, for example, I'm going to ensure there is a test that covers that requirement.
Invariably, if I don't write a test to cover a condition, it'll crop up later during "human" testing. I'll certainly write one then, but I'd rather catch them early. I think the point is that testing is tedious (perhaps) but necessary. I write enough tests to be complete but no more than that.
Part of the problem with skipping simple tests now is in the future refactoring could make that simple property very complicated with lots of logic. I think the best idea is that you can use Tests to verify requirements for the module. If when you pass X you should get Y back, then that's what you want to test. Then when you change the code later on, you can verify that X gives you Y, and you can add a test for A gives you B, when that requirement is added later on.
I've found that the time I spend during initial development writing tests pays off in the first or second bug fix. The ability to pick up code you haven't looked at in 3 months and be reasonably sure your fix covers all the cases, and "probably" doesn't break anything is hugely valuable. You also will find that unit tests will help triage bugs well beyond the stack trace, etc. Seeing how individual pieces of the app work and fail gives huge insight into why they work or fail as a whole.
In most instances, I'd say, if there is logic there, test it. This includes constructors and properties, especially when more than one thing gets set in the property.
With respect to too much testing, it's debatable. Some would say that everything should be tested for robustness, others say that for efficient testing, only things that might break (i.e. logic) should be tested.
I'd lean more toward the second camp, just from personal experience, but if somebody did decide to test everything, I wouldn't say it was too much... a little overkill maybe for me, but not too much for them.
So, No - I would say there isn't such a thing as "too much" testing in the general sense, only for individuals.
Test Driven Development means that you stop coding when all your tests pass.
If you have no test for a property, then why should you implement it? If you do not test/define the expected behaviour in case of an "illegal" assignment, what should the property do?
Therefore I'm totally for testing every behaviour a class should exhibit. Including "primitive" properties.
To make this testing easier, I created a simple NUnit TestFixture that provides extension points for setting/getting the value and takes lists of valid and invalid values and has a single test to check whether the property works right. Testing a single property could look like this:
[TestFixture]
public class Test_MyObject_SomeProperty : PropertyTest<int>
{
private MyObject obj = null;
public override void SetUp() { obj = new MyObject(); }
public override void TearDown() { obj = null; }
public override int Get() { return obj.SomeProperty; }
public override Set(int value) { obj.SomeProperty = value; }
public override IEnumerable<int> SomeValidValues() { return new List() { 1,3,5,7 }; }
public override IEnumerable<int> SomeInvalidValues() { return new List() { 2,4,6 }; }
}
Using lambdas and attributes this might even be written more compactly. I gather MBUnit has even some native support for things like that. The point though is that the above code captures the intent of the property.
P.S.: Probably the PropertyTest should also have a way of checking that other properties on the object didn't change. Hmm .. back to the drawing board.
I make unit test to reach the maximum feasible coverage. If I cannot reach some code, I refactor until the coverage is as full as possible
After finished to blinding writing test, I usually write one test case reproducing each bug
I'm used to separate between code testing and integration testing. During integration testing, (which are also unit test but on groups of components, so not exactly what for unit test are for) I'll test for the requirements to be implemented correctly.
So the more I drive my programming by writing tests, the less I worry about the level of granuality of the testing. Looking back it seems I am doing the simplest thing possible to achieve my goal of validating behaviour. This means I am generating a layer of confidence that my code is doing what I ask to do, however this is not considered as absolute guarantee that my code is bug free. I feel that the correct balance is to test standard behaviour and maybe an edge case or two then move on to the next part of my design.
I accept that this will not cover all bugs and use other traditional testing methods to capture these.
Generally, I start small, with inputs and outputs that I know must work. Then, as I fix bugs, I add more tests to ensure the things I've fixed are tested. It's organic, and works well for me.
Can you test too much? Probably, but it's probably better to err on the side of caution in general, though it'll depend on how mission-critical your application is.
I think you must test everything in your "core" of your business logic. Getter ans Setter too because they could accept negative value or null value that you might do not want to accept. If you have time (always depend of your boss) it's good to test other business logic and all controller that call these object (you go from unit test to integration test slowly).
I don't unit tests simple setter/getter methods that have no side effects. But I do unit test every other public method. I try to create tests for all the boundary conditions in my algorthims and check the coverage of my unit tests.
Its a lot of work but I think its worth it. I would rather write code (even testing code) than step through code in a debugger. I find the code-build-deploy-debug cycle very time consuming and the more exhaustive the unit tests I have integrated into my build the less time I spend going through that code-build-deploy-debug cycle.
You didn't say why architecture you are coding too. But for Java I use Maven 2, JUnit, DbUnit, Cobertura, & EasyMock.
The more I read about it the more I think some unit tests are just like some patterns: A smell of insufficient languages.
When you need to test whether your trivial getter actually returns the right value, it is because you may intermix getter name and member variable name. Enter 'attr_reader :name' of ruby, and this can't happen any more. Just not possible in java.
If your getter ever gets nontrivial you can still add a test for it then.
Test the source code that makes you worried about it.
Is not useful to test portions of code in which you are very very confident with, as long as you don't make mistakes in it.
Test bugfixes, so that it is the first and last time you fix a bug.
Test to get confidence of obscure code portions, so that you create knowledge.
Test before heavy and medium refactoring, so that you don't break existing features.
This answer is more for figuring out how many unit tests to use for a given method you know you want to unit test due to its criticality/importance. Using Basis Path Testing technique by McCabe, you could do the following to quantitatively have better code coverage confidence than simple "statement coverage" or "branch coverage":
Determine Cyclomatic Complexity value of your method that you want to unit test (Visual Studio 2010 Ultimate for example can calculate this for you with static analysis tools; otherwise, you can calculate it by hand via flowgraph method - http://users.csc.calpoly.edu/~jdalbey/206/Lectures/BasisPathTutorial/index.html)
List the basis set of independent paths that flow thru your method - see link above for flowgraph example
Prepare unit tests for each independent basis path determined in step 2