Related
Given you can't write tests for dynamic content, outside of this, is there ever a reason you should not add a unit test? Maybe a test project integrity might be considered not necessary, but I could argue this both ways.
Example:
Objective-C/xCode = Write a test to make sure fonts are listed in your constants are also listed in your projects info.plist UIAppFonts array.
Technically, tests are supposed to adhere to a few key metrics: they should be fast, easy to read and interpret, consistent results, etc.
If any of these (and more) qualities of a good unit test are not met, you will end up with a cost. If the unit tests are slow then you spend time twindling your thumbs or if they are too hard you're spending time interpreting tests instead of writing new ones/code, same goes for tests that have inconsistent results.
Therefore we can say that bad unit tests exist.
However if we look into your concrete example of "should we test X" then that is a lot more subjective.
If something is easy to test like a getter/setter (aka: trivial code) then some might not find it worth their time while others consider it no problem: by adding these quick, small tests you will never encounter an unexpected problem just because someone added logic to their getter/setter and there are no tests to catch any mistakes.
I have no knowledge about Objective-C but at first glance that seems like a reasonable concept to test.
General rule: unless you have an explicit reason not to test something, test it.
Unit tests are really just a tool to create a lower watermark for quality of your code.
If you're 100% confident that your code works as intended, then you have enough unit tests. Adding more tests in this case is just a waste of time.
Think "hello world". How many unit tests would you write for that? 1 or 0?
If you're unsure about something, then you need more unit tests. Reasons for this feeling can be:
You or someone else just found a bug. Always write unit tests for bugs.
Someone asked for a new feature and you're not confident how to implement -> write tests to design the API and to be sure the final result will meet the expectation (and to make sure that everyone knows and agrees on expectations).
You are using a new technology and want to document a) how it works and b) how you use it. These tests work as a kind of template when you wonder later "how did I do this?"
You just found a bug in a library that you use. When you fix the bug, you should also add a test case that tells you "this bug has now been fixed!" so you don't hunt in the wrong place later.
Examples for bad unit tests:
Integration test hiding inside of a unit test
Testing setters and getters
Disabled unit tests
Commented out unit tests
Unit tests that break one per day or week (they erode your confidence and your willingness to write unit tests)
Any test that takes more than 10s to execute
Unit tests that are longer than 50 lines (incl. all the setup code)
My answer would be yes, writing tests is still writing code and the best way to avoid bugs is to not write the code in the first place.
IMHO, writing good tests is generally harder than writing good code. You can't write a useable test until you understand the problem, both in how it should work and how it can fail. The former is generally much easier to understand than the latter.
Having said all that, you have to start somewhere and sometimes it's easiest to write the simplest tests first, even if then don't really test anything useful.
However, you should winnow those tests out as you work through the TDD process. Work towards having a test set that documents just the external interfaces of an object. This is important, as when you come back to the object for later refactoring, you want a set of tests that defines the responsibilities of the object to the rest of the program, not the responsibilities of the object to itself.
(i.e. you want to test the inputs and outputs of the object as a "black box", not the internal wiring of the object. This allows you as much freedom to change w/o causing damage outside of the object. )
In TDD or BDD, we start from failing our unit tests, then fix the methods under test to let the unit tests pass.
Often times, at a new job, we need to write unit tests for existing methods. Probably not a good practice, but this does happen. That's the situation I am in now.
So, here is my question: Should I let my unit tests for existing methods fail? Thank you.
You're not dealing with TDD when you're adding unit tests for working code. However, it is still a good idea to make the tests fail when you first write them (for example, in an extremely simple case, if the actual output will be abc, you write the test to expect abd) so that you know that the tests do fail when the output is different from what the test says is expected. Once you've proved that the tests can fail, you can make them pass by fixing the expected output.
The worst situation to be in is to add unit tests to working code that pass when they're first written. Then you modify the code, changing the output, and the unit tests still pass — when they shouldn't. So, make sure your new unit tests do detect problems — which really does mean writing them so they fail at first (but that may mean you have to write them with known-to-be-bogus expected results).
No, you should not try to make your tests fail.
Why does TDD make the test fail first?
The first reason is to ensure that you really are writing your tests before your code. If you write a test and it passes right away, you are writing code before your test. We write failing tests to be sure that we really are writing the tests first. In your case, its too late for that, so this reason doesn't apply.
A second reason is to verify that the tests are correct. The danger of a test is that it will not be testing the functionality that you think it is. Having the test fail for the correct reason gives confidence that the test is actually working. However, you cannot have the test fail for the correct reason. The code works, and the test is supposed to detect whether or not the good is working. So there is no way to write a test that actually fails for the correct reason.
You can, as the other answer suggested, write test code that is wrong, watch if fail and then correct it. But that fails because the test is wrong, its not really showing that your test correctly catches actual errors. At best it really shows that your assertions work. But generally we are pretty confident that the assertions work, and we don't need to constantly retest our assertions functions.
I don't think you gain much by trying to get your tests to fail when you are adding tests to already working code. So I wouldn't do it.
I've joined a new team, and I've had a problem understanding how they are doing unit tests. When I asked where the unit tests are written, they explained they don't do their unit tests that way.
They explained that what they're calling unit tests is when they actually check the code they wrote locally, and that all of the points are being connected. To me, this is integration testing and just testing your code locally.
I was under the impression that unit tests are code written to verify behavior in a small section of a code. For example, you may write a unit test to make sure it returns the right value, and make the appropriate calls to the database. use a framework like NUnit or MbUnit to help you out in your assertions.
Unit testing to me is supposed to be fast and quick. To me, you want these so you can automate it, and have a huge suite of tests for your application to make sure that it behaves AS YOU EXPECT.
Can someone provide clarification in my or their misunderstandings?
I have worked places that did testing that way and called it unit testing. It reminded me of a quote attributed to Abe Lincoln:
Lincoln: How many legs does a dog have?
Other Guy: 4.
Lincoln: What if we called the tail a leg?
Other Guy: Well, then it would have 5.
Lincoln: No, the answer is still 4. Calling a tail a leg doesn't make it so.
They explained that what they're calling unit tests is when they
actually check the code they wrote locally, and that all of the points
are being connected.
That is not a unit test. That is a code review. Code reviews are good, but without actual unit tests things will break.
Unit tests involve writing code. Specifically, a unit test operate on one unit, which is just a class or component of your software.
If a class under test depends on another class, and you test both classes together, you have an integration test. Integration tests are good. Depending on the language/framework you might use the same testing framework (e.g. junit for java) for both unit and integration tests. If you have a dependency but mock or stub that dependency, then you have a pure unit test.
Unit testing to me is supposed to be fast and quick. To me, you want
these so you can automate it, and have a huge suite of tests for your
application to make sure that it behaves AS YOU EXPECT.
That is essentially correct. How 'fast and quick' developing unit tests is depends on the complexity of what is being tested and the skill of the developer writing the test. You definitely want to build up a suite of tests over time, so you know when something breaks as a codebase becomes more complex. That is how testing makes your codebase more maintainable, by telling you what ceases to function as you make changes.
Your team-mates are not doing unit testing. They are doing "fly by the seat of your pants" development.
Your assumptions are correct.
Doing a project without unit-tests (as they do, don't be fooled) might seem nice for the first few weeks: less code to write, less architecture to think about, less problems to worry about. And you can see the code is working correctly, right?
But as soon as someone (someone else, or even the original coder) comes back to an existing piece of code to modify it, add feature, or simply understand how it worked and what it exactly did, things will become a lot more problematic. And before you realize it, you'll spend your nights browsing through log files and debugging what seemed like a small feature just because it needs to integrate with other code that nobody knows exactly how it works. ANd you'll hate your job.
If it's not worth testing it (with actual unit-tests), then it's not worth writing the code in the first place. Everyone who tried coding without and with unit tests know that. Please, please, make them change their mind. Every time a piece of untested code is checked in somewhere, a puppy dies horribly.
Also, I should say, it's a lot (A LOT) harder to add tests later to a project that was done without testing in mind, than to build the test and production code side-to-side from the very start. Testing not only help you make sure your code works fine, it improves your code quality by forcing you to make good decisions (i.e. coding on interfaces, loose coupling, inversion of control, etc.)
"Unit testing" != "unit tests".
Writing unit tests is one specific method of performing unit testing. It is a very good one, and if your unit tests are written well, it can give you good value over a long time. But what they're doing is indeed unit testing. It's just the kind of unit testing that doesn't help you at all the next time you need to carve on the same code. And that's wasteful.
To add my two cents, yes, that is indeed not Unit testing. IMHO, the main features of unit tests are that it should be fast, automated and isolated. You can using a mocking framework such as RhinoMocks to isolate external dependencies.
Unit tests also have to be very simple and short. Ideally no more than a screen length. It is also one of the few places in software engineering where copy and pasting code might be a better solution than creating highly reusable and highly abstract functions. The reason simplicity is given such a high priority is to avoid the "Who watches the Watchers" problem. You really don't want to be in a situation where you have complex bugs in your unit tests, because they themselves aren't being tested. Here you are relying on the extreme simplicity and tiny size of the tests to avoid bugs.
The names of the unit tests also should be very descriptive, again following the simplicity and self documenting paradigm. I should be able to read the name of the test method and know exactly what it is doing. A quick glance at the code should show me exactly what functionality is being tested and if any external dependencies are being mocked.
The descriptive test names also make you think about the application as a whole. If I look at the entire test run, ideally just by looking at the names of all the tests that were run, I should have a fairly good idea of what the application does.
At times I find a very brittle test to be a good thing because when I change the intent of the code under test I want to make sure my unit test breaks so that I'm forced to refactor ... is this approach not recommended when building a large suite of regression tests?
Unit tests must be brittle -- it must be easy to break them. If they don't break, then they're not unit tests at all; they're code comments.
...
or am I missing the point of the question?
Edit: I should clarify my earlier answer.
I was being a bit pedantic about the language. "brittle" just means "easy to break". A unit test should be easy to break. The term "brittle tests" should really be "overly-brittle tests"; tests that break when they shouldn't. Even so, it's much, much easier to fix an over-brittle test than to fix a bug that slipped through an under-brittle test, so go ahead and write your brittle tests!
The general statement admonishing brittle unit tests applies mostly to shops which haven't fully embraced unit testing. For instance, when trying to convert from having no tests to having a full suite of unit tests, or when your project is the unit testing pilot project. In these cases developers get used to false positives from unit tests and begin to ignore them. Then the unit tests fall behind the production code and either get left behind or require a major effort to update.
I would say you should always aim for the least brittle tests you can that fully test your function/module, but if you have 1 or 2 that are brittle you should be okay in most cases.
IMO, as long as your tests make sure that your app code does what it should do, and if changed, the tests fail, then your tests are fine. Could you define what exactly you mean by "brittle"?
Just make sure that your tests really cover every aspect of your app code. (Within reason).
Yes, brittleness of tests is always a bad thing. But it seems to be one of those things that we have to live with in order to fully test our classes. Many classes can't be tested like black boxes that take some input and return some output, like you might see with Math.cos(). Most of them have side effects on other classes or entities in the system and you have to test that those entities were manipulated properly by the class. That means the test has to know implementation details about the class being tested, which creates brittle classes.
Brittle tests are like proctology exams. They are definitely bad, unpleasant things, but we must put up with them because we have no better choice.
As dysfunctor points out, unit tests should be brittle in that they are easy to break. However, I would add that they should not be brittle in that they pass or fail randomly.
This happens a lot in tests that involve threads and sockets. Tests should make use of mutexes and other "wait" devices to avoid the tests failing under uncontrollable circumstances, such as high processor load.
A definite "smell" of a randomly-brittle test is the use of a sleep() function in a test.
When a code change meant to be "internal" to a class (i.e. not changing the API), causes a test to fail, there are two possibilities:
The code has a new bug, OR
The code is correct, the test needs fixing
Try to reduce #2. Those are the 'brittle' tests.
Unit tests are by definition brittle. They break when related code (the system under test) changes. That's by design. That's how unit tests provide value.
What we want to avoid is tests which break when the code changes but the logic does not. For example, having the majority of existing tests break when adding a new requirement is undesirable.
Unfortunately, avoiding such brittleness is not easy. In all but the simplest of cases, the unit test will have some knowledge about the implementation of the system under test (eg. mocked objects). As long as that is true, test will be brittle.
The best way to avoid that problem is to avoid writing classes that need to change. This is actually easier than it sounds when adhering to SOLID principles.
I've only done minor unit testing at various points in my career. Whenever I start diving into it again, it always troubles me how to prove that my tests are correct. How can I tell that there isn't a bug in my unit test? Usually I end up running the app, proving it works, then using the unit test as a sort of regression test. What is the recommended approach and/or what is the approach you take to this problem?
Edit: I also realize that you could write small, granular unit tests that would be easy to understand. However, if you assume that small, granular code is flawless and bulletproof, you could just write small, granular programs and not need unit testing.
Edit2: For the arguments "unit testing is for making sure your changes don't break anything" and "this will only happen if the test has the exact same flaw as the code", what if the test overfits? It's possible to pass both good and bad code with a bad test. My main question is what good is unit testing since if your tests can be flawed you can't really improve your confidence in your code, can't really prove your refactoring worked, and can't really prove that you met the specification?
The unit test should express the "contract" of whatever you are testing. It's more or less the specification of the unit put into code. As such, given the specs, it should be more or less obvious whether the unit tests are "correct".
But I would not worry too much about the "correctness" of the unit tests. They are part of the software, and as such, they could well be incorrect as well. The point of unit tests - from my POV - is that they ensure the "contract" of your software is not broken by accident. That is what makes unit tests so valuable: You can dig around in the software, refactor some parts, change the algorithms in others, and your unit tests will tell you if you broke anything. Even incorrect unit tests will tell you that.
If there is a bug in your unit tests, you will find out - because the unit test fails while the tested code turns out to be correct. Well then, fix the unit test. No big deal.
Well, Dijkstra famously said:
"Testing shows the presence, not the
absence of bugs"
IOW, how would you write a unit test for the function add(int, int)?
IOW, it's a tough one.
There are two ways to help ensure the correctness of your unit tests:
TDD: Write the test first, then write the code it's meant to test. That means you get to see them fail. If you know that it detects at least some classes of bugs (such as "I haven't implemented any functionality in the function I want to test yet"), then you know that it's not completely useless. It may still let some other bugs slip past, but we know that the test is not completely incorrect.
Have lots of tests. If one test lets some bugs slip past, they'll most likely cause errors further down the line, causing other tests to fail. As you notice that, and fix the offending code, you get a chance to examine why the first test didn't catch the error as expected.
And finally, of course, keep the unit tests so simple that they're unlikely to contain bugs.
For this to be a problem your code would have to be buggy in a way that coincidentally causes your tests to pass. This happened to me recently, where I was checking that a given condition (a) caused a method to fail. The test passed (i.e. the method failed), but it passed because another condition (b) caused a failure. Write your tests carefully, and make sure that unit tests test ONE thing.
Generally though, tests cannot be written to prove code is bug free. They're a step in the right direction.
The complexity of the unit test code is (or should be) less (often orders of magnitude less) than the real code
The chance of your coding a bug in your unit test that exactly matches a bug in your real code is much less than just coding the bug in your real code (if you code a bug in your unit test that doesn't match a bug in your real code it should fail). Of course if you have made incorrect assumptions in your real code you are likely to make the same assumption again - although the mind set of unit testing should still reduce even that case
As already alluded to, when you write a unit test you have (or should have) a different mind set. When writing real code you're thinking "how do I solve this problem". When writing a unit test you're thinking, "how do I test every possibly way this could break"
As others have already said, it's not about whether you can prove that the unit tests are correct and complete (although that's almost certainly much easier with test code), as it is reducing the bug count to a very low number - and pushing it lower and lower.
Of course there has to come a point where your confident in your unit tests enough to rely on them - for example when doing refactorings. Reaching this point is usually just a case of experience and intuition (although there are code coverage tools that help).
I had this same question, and having read the comments, here's what I now think (with due credit to the previous answers):
I think the problem may be that we both took the ostensible purpose of unit tests -- to prove the code is correct -- and applied that purpose to the tests themselves. That's fine as far as it goes, except the purpose of unit tests is not to prove that the code is correct.
As with all nontrivial endeavors, you can never be 100% sure. The correct purpose of unit tests is to reduce bugs, not eliminate them. Most specifically, as others have noted, when you make changes later on that might accidentally break something. Unit tests are just one tool to reduce bugs, and certainly should not be the only one. Ideally you combine unit testing with code review and solid QA in order to reduce bugs to a tolerable level.
Unit tests are much simpler than your code; it's not possible to make your code as simple as a unit test if your code does anything significant. If you write "small, granular" code that's easy to prove correct, then your code will consist of a huge number of small functions, and you're still going to have to determine whether they all work correctly in concert.
Since unit tests are inevitably simpler than the code they're testing, they're less likely to have bugs. Even if some of your unit tests are buggy, overall they're still going to improve the quality of your main codebase. (If your unit tests are so buggy that this isn't true, then likely your main codebase is a steaming pile as well, and you're completely screwed. I think we're all assuming a basic level of competence.)
If you DID want to apply a second level of unit testing to prove your unit tests correct, you could do so, but it's subject to diminishing returns. To look at it faux-numerically:
Assume that unit testing reduces the number of production bugs by 50%. You then write meta-unit tests (unit tests to find bugs in the unit tests). Say that this finds problems with your unit tests, reducing the production bug rate to 40%. But it took 80% as long to write the meta-unit tests as it did to write the unit tests. For 80% of the effort you only got another 20% of the gain. Maybe writing meta-meta-unit tests gives you another 5 percentage points, but now again that took 80% of the time it took to write the meta-unit tests, so for 64% of the effort of writing the unit tests (which gave you 50%) you got another 5%. Even with substantially more liberal numbers, it's not an efficient way to spend your time.
In this scenario it's clear that going past the point of writing unit tests isn't worth the effort.
I guess writing the test first (before writing the code) is a pretty good way of being sure your test is valid.
Or you could write tests for your unit tests... :P
You don't tell. Generally, the tests will be simpler than the code they're testing, so the idea is simply that they'll be less likely to have bugs than the real code will.
First let me start by saying that unit testing is NOT only about testing. It is more about the design of the application. To see this in action you should put a camera with your display and record your coding while writing unit testing. You will realize that you are making a lot of design decisions when writing unit tests.
How to know if my unit tests are good?
You cannot test the logical part period! If your code is saying that 2+2 = 5 and your test is making sure that 2+2 = 5 then for you 2+2 is 5. To write good unit tests you MUST have good understanding of the domain you are working with. When you know what you are trying to accomplish you will write good tests and good code to accomplish it. If you have many unit tests and your assumptions are wrong then sooner or later you will find out your mistakes.
This is one of the advantages of TDD: the code acts as a test for the tests.
It is possible that you'll make equivalent errors, but it is uncommon in my experience.
But I have certainly had the case where I write a test that should fail only to have it pass, which told me my test was wrong.
When I was first learning unit testing, and before I was doing TDD, I would also deliberately break the code after writing the test to ensure that it failed as I expected. When I didn't I knew the test was broken.
I really like Bob Martin's description of this as being equivalent to double entry bookkeeping.
As above, the best way is to write the test before the actual code. Find real life examples of the code your testing also if applicable (mathematical formula or similar), and compare the unit test and expected output to that.
This is something that bugs everyone that uses unit tests. If I would have to give you a short answer I 'd tell you to always trust your unit tests. But I would say that this should be backed up with your previous experience:
Did you have any defects that were reported from manual testing and the unit test didn't catch (although it was responsible to) because there was a bug in your test?
Did you have false negatives in the past?
Are your unit tests simple enough?
Do you write them before new code or at least in parallel?
You can't prove tests are correct, and if you're trying to, you're Doing It Wrong.
Unit tests are a first screen - a smoke test - like all automated testing. They are primarily there to tell you if a change you make later on breaks stuff. They are not designed to be a proof of quality, even at 100% coverage.
The metric does make management feel better, though, and that is useful in itself sometimes!
Dominic mentioned that "For this to be a problem your code would have to be buggy in a way that coincidentally causes your tests to pass.". One technique you can use to see if this is a problem is mutation testing. It makes changes to your code, and see if it causes the unit tests to fail. If it doesn't, then it may indicate areas where the testing isn't 100% thorough.
Unit tests are your requirements concretized. I don't know about you but I like having the requirements specified before starting to code (TDD). By writing them and treating them like any other piece of your code you'll start to feel confident introducing new features without breaking old functionality. To ensure that all your code is needed and that the tests actually tests the code I use pitest (other variants for mutation testing exists for other languages). For me, untested code, is buggy code, however clean it may be.
If the test tests complex code and is complex itself I often write tests for my tests (example).
Edit: I also realize that you could write small, granular unit tests that would be easy to understand. However, if you assume that small, granular code is flawless and bulletproof, you could just write small, granular programs and not need unit testing.
The idea of unit testing is to test the most granular things, then stack together tests to prove the larger case. If you're writing large tests, you lose a bit of the benefits there, although it's probably quicker to write larger tests.