In TDD or BDD, we start from failing our unit tests, then fix the methods under test to let the unit tests pass.
Often times, at a new job, we need to write unit tests for existing methods. Probably not a good practice, but this does happen. That's the situation I am in now.
So, here is my question: Should I let my unit tests for existing methods fail? Thank you.
You're not dealing with TDD when you're adding unit tests for working code. However, it is still a good idea to make the tests fail when you first write them (for example, in an extremely simple case, if the actual output will be abc, you write the test to expect abd) so that you know that the tests do fail when the output is different from what the test says is expected. Once you've proved that the tests can fail, you can make them pass by fixing the expected output.
The worst situation to be in is to add unit tests to working code that pass when they're first written. Then you modify the code, changing the output, and the unit tests still pass — when they shouldn't. So, make sure your new unit tests do detect problems — which really does mean writing them so they fail at first (but that may mean you have to write them with known-to-be-bogus expected results).
No, you should not try to make your tests fail.
Why does TDD make the test fail first?
The first reason is to ensure that you really are writing your tests before your code. If you write a test and it passes right away, you are writing code before your test. We write failing tests to be sure that we really are writing the tests first. In your case, its too late for that, so this reason doesn't apply.
A second reason is to verify that the tests are correct. The danger of a test is that it will not be testing the functionality that you think it is. Having the test fail for the correct reason gives confidence that the test is actually working. However, you cannot have the test fail for the correct reason. The code works, and the test is supposed to detect whether or not the good is working. So there is no way to write a test that actually fails for the correct reason.
You can, as the other answer suggested, write test code that is wrong, watch if fail and then correct it. But that fails because the test is wrong, its not really showing that your test correctly catches actual errors. At best it really shows that your assertions work. But generally we are pretty confident that the assertions work, and we don't need to constantly retest our assertions functions.
I don't think you gain much by trying to get your tests to fail when you are adding tests to already working code. So I wouldn't do it.
Related
Please note: I'm not asking for your opinion. I'm asking about conventions.
I was just wondering whether I should have both passing and failing tests with appropriate method names such as, Should_Fail_When_UsageQuantityIsNegative() , Should_Fail_When_UsageQuantityMoreThan50() , Should_Pass_When_UsageQuantityIs50().
Or instead, should I code them to pass and keep all the tests in Passed condition?
When you create unit tests, they should all pass. That doesn't mean that you shouldn't test the "failing" cases. It just means that the test should pass when it "fails."
This way, you don't have to go through your (preferably) large number of tests and manually check that the correct ones passed and failed. This pretty much defeats the purpose of automation.
As Mark Rotteveel points out in the comments, just testing that something failed isn't always enough. Make sure that the failure is the correct failure. For example, if you are using error codes and error_code being equal to 0 indicates a success and you want to make sure that there is a failure, don't test that error_code != 0; instead, test for example that error_code == 19 or whatever the correct failing error code is.
Edit
There is one additional point that I would like to add. While the final version of your code that you deploy should not have failing tests, the best way to make sure that you are writing correct code is to write your tests before you write the rest of the code. Before making any change to your source code, write a unit test (or ideally, a few unit tests) that should fail (or fail to compile) now, but pass after your change has been made. That's a good way to make sure that the tests that you write are testing the correct thing. So, to summarize, your final product should not have failing unit tests; however, the software development process should include periods where you have written unit tests that do not yet pass.
You should not have failing tests unless your program is acting in a way that it is not meant to.
If the intended behavior of your program is for something to fail, and it fails, that should trigger the test to pass.
If the program passes in a place where it should be failing, the test for that portion of code should fail.
In summary, a program is not working properly unless all tests are passing.
You should never have failing tests, as others have pointed out, this defeats the purpose of automation. What you might want are tests that verifies your code works as expected when inputs are incorrect. Looking at your examples Should_Fail_When_UsageQuantityIsNegative() is a test that should pass, but the assertions you make depend on what fail means. For example, if your code should throw an IllegalArgumentException when usage quantity is negative then you might have a test like this:
#Test(expected = IllegalArgumentException.class)
public void Should_Fail_When_UsageQuantityIsNegative() {
// code to set usage quantity to a negative value
}
There's a few different ways to interpret the question if tests should fail.
A test like Should_Fail_When_UsageQuantityMoreThan50() should instead be a passing test which checks the appropriate error is thrown. Throws_Exception_When_UsageQuantityMoreThan50() or the like. Many test suites have special facilities for testing exceptions: JUnit's expected parameter and Perl modules such as Test::Exception and can even test for warnings.
Tests should fail during the course of development, it means they're doing their job. You should be suspicious of a test suite which never fails, it probably has bad coverage. The failing tests will catch changes to public behavior, bugs, and other mistakes by the developer or the tests or the code. But when committed and pushed, the tests should be returned to passing.
Finally, there are legitimate cases where you have a known bug or missing feature which cannot at this time be fixed or implemented. Sometimes bugs are incidentally fixed, so it's good to write a test for it. When it passes, you know the bug has been fixed, and you want a notice when it starts passing. Different testing systems allow you to write tests which are expected to fail, and will only be visible if they pass. In Perl this is the TODO or expected failure. POSIX has a number of results such as UNRESOLVED, UNSUPPORTED and UNTESTED to cover this case.
Given you can't write tests for dynamic content, outside of this, is there ever a reason you should not add a unit test? Maybe a test project integrity might be considered not necessary, but I could argue this both ways.
Example:
Objective-C/xCode = Write a test to make sure fonts are listed in your constants are also listed in your projects info.plist UIAppFonts array.
Technically, tests are supposed to adhere to a few key metrics: they should be fast, easy to read and interpret, consistent results, etc.
If any of these (and more) qualities of a good unit test are not met, you will end up with a cost. If the unit tests are slow then you spend time twindling your thumbs or if they are too hard you're spending time interpreting tests instead of writing new ones/code, same goes for tests that have inconsistent results.
Therefore we can say that bad unit tests exist.
However if we look into your concrete example of "should we test X" then that is a lot more subjective.
If something is easy to test like a getter/setter (aka: trivial code) then some might not find it worth their time while others consider it no problem: by adding these quick, small tests you will never encounter an unexpected problem just because someone added logic to their getter/setter and there are no tests to catch any mistakes.
I have no knowledge about Objective-C but at first glance that seems like a reasonable concept to test.
General rule: unless you have an explicit reason not to test something, test it.
Unit tests are really just a tool to create a lower watermark for quality of your code.
If you're 100% confident that your code works as intended, then you have enough unit tests. Adding more tests in this case is just a waste of time.
Think "hello world". How many unit tests would you write for that? 1 or 0?
If you're unsure about something, then you need more unit tests. Reasons for this feeling can be:
You or someone else just found a bug. Always write unit tests for bugs.
Someone asked for a new feature and you're not confident how to implement -> write tests to design the API and to be sure the final result will meet the expectation (and to make sure that everyone knows and agrees on expectations).
You are using a new technology and want to document a) how it works and b) how you use it. These tests work as a kind of template when you wonder later "how did I do this?"
You just found a bug in a library that you use. When you fix the bug, you should also add a test case that tells you "this bug has now been fixed!" so you don't hunt in the wrong place later.
Examples for bad unit tests:
Integration test hiding inside of a unit test
Testing setters and getters
Disabled unit tests
Commented out unit tests
Unit tests that break one per day or week (they erode your confidence and your willingness to write unit tests)
Any test that takes more than 10s to execute
Unit tests that are longer than 50 lines (incl. all the setup code)
My answer would be yes, writing tests is still writing code and the best way to avoid bugs is to not write the code in the first place.
IMHO, writing good tests is generally harder than writing good code. You can't write a useable test until you understand the problem, both in how it should work and how it can fail. The former is generally much easier to understand than the latter.
Having said all that, you have to start somewhere and sometimes it's easiest to write the simplest tests first, even if then don't really test anything useful.
However, you should winnow those tests out as you work through the TDD process. Work towards having a test set that documents just the external interfaces of an object. This is important, as when you come back to the object for later refactoring, you want a set of tests that defines the responsibilities of the object to the rest of the program, not the responsibilities of the object to itself.
(i.e. you want to test the inputs and outputs of the object as a "black box", not the internal wiring of the object. This allows you as much freedom to change w/o causing damage outside of the object. )
I finished an app and after that I'm trying to write unit test to cover all methods.
The thing is that Im seeing that I'm testing my code with the knowledge of how it works.
For me is a bit stupid because I know how the code works and I'm testing what is my code actually doing.
My question is:
Is this useless? Im testing what it does, not what it is supposing to do. My code works but I can improve it. Then:
Should I complete all my tests and then try to refactor my code changing my test to "How the code should work" and making the changes to the app to pass the test?
Thank you.
You need to test "How the code should work". That is, you have a clear idea of what a particular method should behave like, then create the set of tests that covers that behaviour. If your code fails the test then you can fix it.
Even though your code is working now you still need the tests for regression testing. Later when you modify your code or add new extensions to it you need to ensure that you did not break the existing functionality. The set of tests that you derive today will be able to tell you how well you did.
While I don't always do Test-Driven Development (write tests before implementing the class), I do always write tests after I implement the code. Even some days after. At least I try to follow a path coverage strategy (that is, every route flow from the beginning to the method up to when it returns). Also for unexpected parameter values. This is useful for enhancing your confidence of correct behaviour of your functions, also when you change the code.
Quite always I find unexpected behaviors or little bugs :) So it works
It will be extremely useful if you ever change the code.
Pretty useless, I'd say. It is called "test driven" for a reason. Every line of code is motivated by a test.
That means the code line is also protected against changes.
Test driven development requires a minimalistic approach and tons of discipline. Add a test to extend the functionality a little. Implement the functionality so it just makes the light green but nothing more. Make foolish implementations until the tests force you to make serious ones.
I have tried to add tests to existing code and found it difficult. The tendency is that only some of the functionality becomes tested. This is dangerous since the test suite will make people think the code is protected. One method is to go through the code line by line and ensure there is a test for it. Make foolish changes to the code until a test forces you to back to the original version. In the end, any change to the code should break the tests.
However, there is no way you can derive the requirements from the code. No test suite that is the result of reverse-engineering will be complete.
You should at least try to build the tests on how its supposed to work. Therefor it is better to build the tests in advance, but it is still not useless to build the tests now.
The reason: you don't build the tests to test your current code, but you're building them to test future modifications. Unit tests are especially useful to test if a modification didn't break earlier code.
Better late than never!
Of course, the best option is to do the unit tests before you start implementing so that you can profit from them even more, but having a good set of unit tests will prevent you from introducing many bugs when you refactor or implement new features, regardless of whether you implemented the unit tests before or after the implementation.
One thing that I've always noticed with my unit tests is that they get to be kind of verbose; seeing as they could also be not verbose enough, how do you get a sense of when your unit tests are the right size?
I know of a good quote for this and it's:
"Perfection is achieved, not when
there is nothing left to add, but when
there is nothing left to remove."
- Antoine de Saint-Exupery.
One reason why them become verbose is that they're testing multiple things. I try to make each unit test test one thing, and one thing only. So I end up with a lot of tests, each asserting one piece of functionality. It works well since a failure won't disguise another failure by bailing out of the unit test before other assertions are made.
Writing fine-grained tests like this means that if you're not careful, you'll duplicate a lot of setup/teardown code, and that's the stuff you need to spend time on, abstracting it out (in whatever way is suitable).
I know a unit test is done when one thing changing in the code could cause the test to fail. If it's testing too many things, that might be good right now, but doesn't help me in the future when I run the test again. I want a failing test to point me right to the thing that caused the failure, not a whole list of possibilities.
I've never worried too much about the unit tests being really clean, short and concise code. Sometimes I have 100 line functions which are lots of copy and paste in unit tests, and I don't see that as too large a problem considering most unit tests are setting up some dummy data, and then testing for the correct results based on that.
However, I would say I know I have enough unit tests when I can't think of a single thing in the code which could be wrong.
Edit
In response to your edit: I don't look for perfection in unit tests.
If one has a project that has tests that are executed as part of the build procedure on a build machine, if a set tests fail, should the entire build fail?
What are the things one should consider when answering that question? Does it matter which tests are failing?
Background information that prompted this question:
Currently I am working on a project that has NUnit tests that are done as part of the build procedure and are executed on our cruise control .net build machine.
The project used to be setup so that if any tests fail, the build fails. The reasoning being if the tests fail, that means the product is not working/not complete/it is a failure of a project, and hence the build should fail.
We have added some tests that although they fail, they are not crucial to the project (see below for more details). So if those tests fail, the project is not a complete failure, and we would still want it to build.
One of the tests that passes verifies that incorrect arguments result in an exception, but the test does not pass is the one that checks that all the allowed arguments do not result in an exception. So the class rejects all invalid cases, but also some valid ones. This is not a problem for the project, since the rejected valid arguments are fringe cases, on which the application will not rely.
If it's in any way doable, then do it. It greatly reduces the broken-window-problem:
In a system with no (visible) flaws, introducing a small flaw is usually seen as a very bad idea. So if you've got a project with a green status (no unit test fails) and you introduce the first failing test, then you (and/or your peers) will be motivated to fix the problem.
If, on the other side, there are known-failing tests, then adding just another broken test is seen as keeping the status-quo.
Therefore you should always strive to keep all tests running (and not just "most of them"). And treating every single failing test as reason for failing the build goes a long way towards that goal.
If a unit test fails, some code is not behaving as expected. So the code shouldn't be released.
Although you can make the build for testing/bugfixing purposes.
If you felt that a case was important enough to write a test for, then if that test is failing, the software is failing. Based on that alone, yes, it should consider the build a failure and not continue. If you don't use that reasoning, then who decides what tests are not important? Where is the line between "if this fails it's ok, but if that fails it's not"? Failure is failure.
I think a nice setup like yours should always build successfully, including all unit tests passed.
Like Gamecat said, the build itself is succeeded, but this code should never go to production.
Imagine one of your team members introducing a bug in the code which that one unit test (which always fails) covers. It won't be discovered by the test since you allow that one test to always fail.
In our team we have a simple policy: if all tests don't pass, we don't go to production with the code. This is also a very simple to understand for our project manager.
In my opinion it really depends on your Unit Tests,...
if your Unit tests are really UNIT tests (like they should be => "reference to endless books ;)" )
then the build should fail, because something is not behaving as should...
but most often (unfortunately to often seen), in so many projects these unit tests only cover some 'edges' and/or are integration tests, then the build should go on
(yes, this is a subjective answer ;)
in short:
do you know the unit tests to be fine: fail;
else: build on
The real problem is with your failing tests. You should not have a unit test where it's OK to fail because it's an edge case. Decide whether the edge case is important or not - if not then delete the unit test; if yes then fix the code.
Like some of the other answers implied, it's definitely a code smell when unit tests fail. If you live with the smell, then you're less likely to spot the next problem
All the answers have been great, here is what I decided to do:
Make the tests (or if need be split a failing test) that are not crucial be ignored by NUnit (I remembered this feature after asking the question). This allows:
The build can fail if any tests fail, hence reducing the smelliness
The tests that are ignored have to be defended to project manager (whomever is in charge)
Any tests that are ignored are marked in a special way
I think that is the best compromise, forcing people to fix the tests, but not necessarily right away (but they have to defend their decision of not fixing it now since everyone knows what they did).
What I actually did: fixed the broken tests.