Related
Given you can't write tests for dynamic content, outside of this, is there ever a reason you should not add a unit test? Maybe a test project integrity might be considered not necessary, but I could argue this both ways.
Example:
Objective-C/xCode = Write a test to make sure fonts are listed in your constants are also listed in your projects info.plist UIAppFonts array.
Technically, tests are supposed to adhere to a few key metrics: they should be fast, easy to read and interpret, consistent results, etc.
If any of these (and more) qualities of a good unit test are not met, you will end up with a cost. If the unit tests are slow then you spend time twindling your thumbs or if they are too hard you're spending time interpreting tests instead of writing new ones/code, same goes for tests that have inconsistent results.
Therefore we can say that bad unit tests exist.
However if we look into your concrete example of "should we test X" then that is a lot more subjective.
If something is easy to test like a getter/setter (aka: trivial code) then some might not find it worth their time while others consider it no problem: by adding these quick, small tests you will never encounter an unexpected problem just because someone added logic to their getter/setter and there are no tests to catch any mistakes.
I have no knowledge about Objective-C but at first glance that seems like a reasonable concept to test.
General rule: unless you have an explicit reason not to test something, test it.
Unit tests are really just a tool to create a lower watermark for quality of your code.
If you're 100% confident that your code works as intended, then you have enough unit tests. Adding more tests in this case is just a waste of time.
Think "hello world". How many unit tests would you write for that? 1 or 0?
If you're unsure about something, then you need more unit tests. Reasons for this feeling can be:
You or someone else just found a bug. Always write unit tests for bugs.
Someone asked for a new feature and you're not confident how to implement -> write tests to design the API and to be sure the final result will meet the expectation (and to make sure that everyone knows and agrees on expectations).
You are using a new technology and want to document a) how it works and b) how you use it. These tests work as a kind of template when you wonder later "how did I do this?"
You just found a bug in a library that you use. When you fix the bug, you should also add a test case that tells you "this bug has now been fixed!" so you don't hunt in the wrong place later.
Examples for bad unit tests:
Integration test hiding inside of a unit test
Testing setters and getters
Disabled unit tests
Commented out unit tests
Unit tests that break one per day or week (they erode your confidence and your willingness to write unit tests)
Any test that takes more than 10s to execute
Unit tests that are longer than 50 lines (incl. all the setup code)
My answer would be yes, writing tests is still writing code and the best way to avoid bugs is to not write the code in the first place.
IMHO, writing good tests is generally harder than writing good code. You can't write a useable test until you understand the problem, both in how it should work and how it can fail. The former is generally much easier to understand than the latter.
Having said all that, you have to start somewhere and sometimes it's easiest to write the simplest tests first, even if then don't really test anything useful.
However, you should winnow those tests out as you work through the TDD process. Work towards having a test set that documents just the external interfaces of an object. This is important, as when you come back to the object for later refactoring, you want a set of tests that defines the responsibilities of the object to the rest of the program, not the responsibilities of the object to itself.
(i.e. you want to test the inputs and outputs of the object as a "black box", not the internal wiring of the object. This allows you as much freedom to change w/o causing damage outside of the object. )
I'm currently working on a project with some fairly significant business rules where the problem space is being 'discovered' as we are writing the solution (pretty typical chaotic project management kind of thing). We have decent test coverage & rely on them quite a bit to make sure our signficant changes don't blow anything up. This scenario is the kind of thing that unit test zealots highlight as a prime example of testing helping software be at once easily modified with less defects and quicker completion then if you aren't using unit tests. I shudder to think about how I'd be able to cope without the test suite.
My question is that while I certainly believe in the value of unit testing (this project is actually TDD but it's not really germane to the question), I'm wondering, as have others, about the classic unit test problem of having so much more code to grok and maintain (i.e. the tests themselves). Again. there is no doubt in my mind that this particular project is much better off with the unit test cruft then without it, I'm also concerned about the long term maintainability of the tests.
There are a few techniques that I've used following the advice of others to help with this problem. In general,
we create test lists that are either in a 'dependent' vs 'independent' bucket. independent tests don't require anything that is not in source control. So any calls to our data access layer are either mocked or getting data from an xml file instead of the real db for example. Dependent tests as the name suggest depend on something like a config file or a db or a networkie thing that might not be correctly configured/available when running the test. Splitting the tests into to groups like this has been extremembly valuable in allowing us to write dependent 'throw away' tests for early development and independent mission critical tests that can be relied upon and reist test rot. It also makes the CI server easy to manage since it doesn't have to be setup and maintained w/ db connections and the like.
We target different levels of our code. For example, we have tests hitting the 'main' and tests hitting all of the methods that 'main' would call. This gives us the ability to target details of the system and the overarching goals. The 'main' tests are difficult to debug if they break, but they typically aren't the only thing that breaks (detailed tests also break). It's easier to follow the detailed tests and debug them if they break but they are insufficient to know if a refactor kills the system (that's what the 'main' tests are for).
The 'main' tests have been critical to feeling comfortable that a refactor hasn't hosed the codebase. so a 'main' test would be like many tests to a single method called with different args that map to use cases. It's basically the entrypoint into our code at the highest level and as such are arguably not really 'unit' tests. However, I find that I really need the higher level tests in order to feel comfortable that a refactor didn't blow up the codebase. The lower level tests (the ones that are truly 'unit' of work) are not sufficient.
All this to get to the question. As the project moves forward and I find I need to implement changes (sometimes quite significant, sometimes trivial) to the codebase, I find that when changes cause tests to fail, there is ratio between the test failing and actual regressive business logic failure and unit test invalidity. In other words, sometimes the test failure is because of a regression bug in the actual codebase and sometimes it is because the unit test assertions are not valid any longer and it is the assertions that need to be changed. It seems roughly that when tests fail it's been at about even (50%) for this particular project.
Has anyone tracked this ratio on their projects, and if so, what kinds of things have you learned (if any) regarding this ratio? I'm not sure that it even indicates anything, but I have noticed that about half the time that test failures lead me to adjusting test asserts rather than actually fixing regression bugs in the real codebase. Whenever this occurs, it makes me feel like I just wasted x hours of my day & I wonder if I could be more efficient somehow with my testing approach. It often takes longer to resolve test-assert failures than actual regression bugs which is both counterintuative and frustrating.
EDIT
Note this question is about exploring what this ratio means and your experience with this ratio. When is it 'smelly'??
"test failures lead me to adjusting test asserts rather than actually fixing regression bugs in the real codebase."
Correct. Your requirements changed. Your test assertions must change.
"it makes me feel like I just wasted x hours of my day"
Why? How else are you going track requirements changes?
"It often takes longer to resolve test-assert failures than actual regression bugs"
No kidding. When your requirements are in a state of flux, it takes a lot of time and effort to map requirements changes to test result changes.
"which is ... counterintuative". Depends on your intuition. My intuition (after 18 months of TDD) is that requirements change lead to design changes, lots of complex test changes to reflect the design changes.
Sometimes, very little (or no) code changes.
If your code is really good, it doesn't change much. When you're spending more time on testing than code, it means you wrote good code.
Go home happy.
The code smell shows up when you're spending more time trying to get code to pass a set of tests which never change. Think about what that means. You wrote the tests, but you just can't get code to pass. That's horrible.
If you spend an hour writing tests and 4 hours trying to get code to pass the tests, you've either got a really complex algorithm (and should have broken it down into more testable pieces) or you're a terrible application programmer.
If you spend 1 hour writing tests and 1 hour getting code the pass the tests, that's good.
If you spend 2 hours fixing tests because of a requirements change, and 1 hour getting code to pass the revised tests, that means your code wasn't very resilient against change.
If you spend 2 hours fixing tests because of a requirements change, and 1/2 hour tweaking code to pass those tests, you've written some really good code.
I have noticed that about half the time that test failures lead me to adjusting test asserts rather than actually fixing regression bugs in the real codebase.
When a test fails, there are three options:
the implementation is broken and should be fixed,
the test is broken and should be fixed, or
the test is not anymore needed (because of changed requirements) and should be removed.
It's important to identify correctly that which of those three options it is. The way that I write my tests, I document in the name of the test the behaviour which the test specifies/tests, so that when the test fails, I will easily find out why the test was originally written. I have written more about it here: http://blog.orfjackal.net/2010/02/three-styles-of-naming-tests.html
In your case,
if you need to change the tests because of changed requirements and only a couple of tests at a time need to be changed, then everything is normal (the tests are well isolated, so that each piece of behaviour is specified by only one test).
If you need to change the tests because of changed requirements and many tests at a time need to be changed, then it's a test smell that you have lots of tests testing the same thing (the tests are not well isolated). The tests might be testing more than one interesting behaviour. The solution is to write more focused tests and better decoupled code.
If the tests need to be changed when refactoring, then it's a test smell that the tests are too tightly coupled with implementation details. Try to write tests which are centered around the behaviour of the system, instead of its implementation. The article which I linked earlier should give you some ideas.
(An interesting side point is, that if you find yourself mostly rewriting classes, instead of changing them, when requirements change, then it can be an indication that the code is following well SRP, OCP and other design principles.)
I definitely second #S.Lott's answer. I would just point out that what happens when the spec is established on piles of dead trees is that when requirements change, the dead trees (or word processor files) don't yell at you the way the tests do, so everything runs along just fine, except that you have this pile of dead trees that everyone looks at and says "the documentation is fiction."
That being said, there are cases where the tests are just not well written or useful, and probably should be dropped. I find that especially with TDD, where the tests teased out the design and were really incremental, and now that the design and functionality is further along some of those original tests are really not relevant anymore.
If you regard fixing a bunch of tests as "wasted x hours of my day" then when you move on to the second test you don't think about the first after it passes. That is going to make the cost of change higher. It is probably the correct decision, but there is nothing wrong with looking at a test and saying that it was overcome by subsequent events and just dropping it, just don't use that as a cheap way out.
S. Lott pretty much said it all. I think the only thing a ratio of test assertion changes (T) vs. regression fixes (R) will tell you is a combination of how volatile your requirements are (which will make T higher) versus how successful the application code is at passing the test (which will affect the value of R). Those two factors could vary independently according to the quality of your requirements and development processes.
One of the primary benefits of unit testing is to provide confidence that when one needs to later alter the code one is not breaking it. However, what benefits does unit testing provide for code that is literally used as one-off throwaway code? This throwaway code is most certainly used in production, but is never actually altered once it's deployed. Does unit testing still make sense in this situation and if so how specifically?
UPDATE: The throwaway code actually is functional-tested before hitting production. Normally, for non-throwaway code it still makes sense to have unit tests despite functional testing occurring. The question here is whether or not it makes sense to also have the unit tests also in the case of throwaway code.
UPDATE 2: The reason why throwaway code is in production in the first place is that this code is literally used for one client, one time only. It's never subject to revision. It is used by a used a single time for a few days. It's very specific to a single client. It's not ever used after that for any other purpose, including the same client. Is there still value in writing unit tests in this case, despite functional tests occurring?
I'm not sure how code can be both production code and throwaway code. The fact that it is being used in production means that it is not throwaway. What makes you so sure that just because you don't plan on changing it now, someone else might not come around and reuse it or alter it at a later date?
Regardless, there are many benefits to unit testing beyond protection when refactoring. Unit tests (like all tests) help prove that the code is doing what is supposed to do. You will need to test at some level before you go into production to prove that, and unit tests are an easy way to automate some of that.
The process of writing unit tests often exposes bugs. It forces you to think about cases that you may not have thought of when you wrote the happy day case.
I'd be willing to bet the time you spent asking the question and coming back to read the answers would have been more than enough time to write a few unit tests. Which was the better use of your time?
It's simple: if you care whether the code is correct or not, you need to test it. Unit test, acceptance test, service test, whatever you call it, it needs to be tested one way or another.
Tests provide benefits in three ways:
When code is written test-first, tests help drive design and ensure you write testable code. If this is truly throwaway code, then this doesn't matter to you.
Tests provide a safety net for refactoring. Again, in your situation you may not care.
Tests prove the code does what it is supposed to do. This DOES apply to throwaway code, especially if the code is written in a test-first manner. I'd much rather write tests as I go, and be confident that I'm shipping something solid to QA, than skip the tests and wait for a QA person to tell me about defects. What if a defect is reported and you have to make a significant change to the code? With automated tests, it's easy to tell if your "fix" broke anything else. Without automated tests you have to repeat the full suite of manual tests, which might be expensive.
To summarize, if the code is truly "throwaway" AND is small or very simple, then you may not get a lot of benefit. If the code is complex, or is relatively large, then tests may be worth something. (Although, I'd focus on writing high value tests only, targeting features that are hard or costly to test by hand)
You should be testing this code somehow before putting it into production. Unit testing can be a valuable part here, regardless of the need to do automated regression tests later.
Unit testing can also be used in test-driven development, where you sketch out what the code is supposed to do before actually writing the code. In this scenario, the unit tests help speed up the development process (precisely because you have automated tests at a time when the code still does change).
If I only had a dollar for every piece of "thorwaway" code I ever needed to maintain...
I don't understand how an unit test could possibly benefit.
Isn't it sufficient for a tester to test the entire output as a whole rather than doing unit tests?
Thanks.
What you are describing is integration testing. What integration testing will not tell you is which piece of your massive application is not working correctly when your output is no longer correct.
The advantage to unit testing is that you can write a test for each business assumption or algorithm step that you need your program to perform. When someone adds or changes code to your application, you immediately know exactly which step, which piece, and maybe even which line of code is broken when a bug is introduced. The time savings on maintenence for that reason alone makes it worthwhile, but there is an even bigger advantage in that regression bugs cannot be introduced (assuming your tests are running automatically when you build your software). If you fix a bug, and then write a test specifically to catch that bug in the future, there is no way someone could accidentally introduce it again.
The combination of integration testing and unit testing can let you sleep much easier at night, especially when you've checked in a big piece of code that day.
The earlier you catch bugs, the cheaper they are to fix. A bug found during unit testing by the coder is pretty cheap (just fix the darn thing).
A bug found during system or integration testing costs more, since you have to fix it and restart the test cycle.
A bug found by your customer will cost a lot: recoding, retesting, repackaging and so forth. It may also leave a painful boot print on your derriere when you inform management that you didn't catch it during unit testing because you didn't do any, thinking that the system testers would find all the problems :-)
How much money would it cost GM to recall 10,000 cars because the catalytic converter didn't work properly?
Now think of how much it would cost them if they discovered that immediately after those converters were delivered to them, but before they were put into those 10,000 cars.
I think you'll find the latter option to be quite a bit cheaper.
That's one reason why test driven development and continuous integration are (sometimes) a good thing - testing is done all the time.
In addition, unit tests don't check that the program works as a whole, just that each little bit performs as expected. That's often quite a lot more than higher level tests would check.
From my experience:
Integration and functional testing tend to be more indicative of the overall quality of the system, than unit test suit is.
High level testing (functional, acceptance) is a QA tool.
Unit testing is a development tool. Especially in a TDD context, where unit test becomes more of a design implement, rather than that of a quality assurance.
As a result of better design, quality of the entire system improves (indirectly).
Passing unit test suite is meant to ensure that a single component conforms to the developer's intentions (correctness). Acceptance test is the level that covers validity of the system (i.e. system does what user want it to do).
Summary:
Unit test is meant as a development tool first, QA tool second.
Acceptance test is meant as a QA tool.
There is still a need for a certain level of manual testing to be performed but unit testing is used to decrease the number of defects that make it to that stage. Unit testing tests the smallest parts of the system and if they all work the chances of the application as a whole working correctly are increased significantly.
It also assists when adding new features since regression testing can be performed quickly and automatically.
For a complex enough application, testing the entire output as a whole may not cover enough different possibilities. For example, any given application has a huge number of different code paths that can be followed depending on input. In typical testing, there may be many parts of your code that are simply never encountered, because they are only used in certain circumstances, so you can't be sure that any code that isn't run in your test situation, actually works. Also, errors in one section of code may be masked a majority of the time by something else in another section of code, so you may never discover some errors.
It is better to test each function or class separately. That way, the test is easier to write, because you are only testing a certain small section of the code. It's also easier to cover every possible code path when testing, and if you test each small part separately then you can detect errors even when those errors would often be masked by other parts of your code when run in your application.
Do yourself a favor and try out unit testing first. I was quite the skeptic myself until I realized just how darned helpful/powerful unit-tests can be. If you think about it, they aren't really there to add to your workload. They are there to provide you with peace of mind and allow you to continue extending your application while ensuring that your code is solid. You get immediate feedback as to when you may have broke something and this is something of extraordinary value.
To your question regarding why to test small sections of code consider this: Suppose your giant app uses a cool XOR encryption scheme that you wrote and eventually product management changes the requirements of how you generate these encrypted strings. So you say: "Heck, I wrote the the encryption routine so I'll go ahead and make the change. It'll take me 15 minutes and we'll all go home and have a party." Well, perhaps you introduced a bug during this process. But wait!!! Your handy dandy TestXOREncryption() test method immediately tells you that the expected output did not match the input. Bingo, this is why you broke down your unit tests ahead of time into small "units" to test for because in your big giant application you would not have figured this out nearly as fast.
Also, once you get into the frame of mind of regularly writing unit tests you'll realize that although you pay an upfront cost in the beginning in terms of time, you'll get that back 10 fold later in the development cycle when you can quickly identify areas in your code that have introduced problems.
There is no magic bullet with unit tests because your ability to identify problems is only as good as the tests you write. It boils down to delivering a better product and relieving yourself of stress and headaches. =)
Agree with most of the answers. Let's drill down on the topic of speed. Here are some real numbers:
Unit test results in 1 or 2 minutes from a
fresh compile. As true unit tests
(no interaction with external
systems like dbs) they can cover a
lot of logic really fast.
Automated functional test results in 1 or 2 hours. These run on a simplified platform, but sometimes cover multiple systems and the database - which really kills the speed.
Automated integration test results once a day. These exercise the full meal deal, but are so heavy and slow, we can only execute them once a day and it takes a few hours.
Manual regression results come in after a few weeks. We get stuff over to testers a few times a day, but your change isn't realistically regressed for week or two at best.
I want to find out what I broke in 1 or 2 minutes, not a few weeks, not even a few hours. That's where the 10fold ROI on unit tests that people talk about comes from.
This is a tough question to approach because it questions something of such enormous breadth. Here's my short answer, however:
Test Driven Development (or TDD) seeks to prove that every logical unit of an application (or block of code) functions exactly as it should. By making tests as automated as possible for productivity's sake, how could this really be harmful?
By testing every logical piece of code, you can trust the usage of the code up some hierarchy. Say I build an application that relies on a thread-safe stack implementation. Shouldn't the stack be guaranteed to work up at every stage before I build on it?
The key is that if something in the whole application breaks, meaning just looking at the total output/outcome, how do you know where it came from? Well, debugging, of course! Which puts you back where you started. TDD allows you to -hopefully- bypass this most painful stage in development.
Testers generally test end to end functionality. Obviously this is geared for going at user scenarios and has incredible value.
Unit Tests serve a different functionality. The are the developers way of verifying the components they write work correctly in the absence of other features or in combination with other features. This offers a range of value including
Provides un-ignorable documentation
Ability to isolate bugs to specific components
Verify invariants in the code
Provide quick, immediate feedback to changes in the code base.
One place to start is regression testing. Once you find a bug, write a small test that demonstrates the bug, fix it, then make sure the test now passes. In future you can run that test before each release to ensure that the bug has not been reintroduced.
Why do that at a unit level instead of a whole-program level? Speed. In good code it's much faster to isolate a small unit and write a tiny test than to drive a complex program through to the bug point. Then when testing a unit test will generally run significantly faster than an integration test.
Very simply: Unit tests are easier to write, since you're only testing a single method's functionality. And bugs are easier to fix, since you know exactly what method is broken.
But like the other answerers have pointed out, unit tests aren't the end-all-be-all of testing. They're just the smallest piece of the equation.
Probably the single biggest difficulty with software is the sheer number of interacting things, and the most useful technique is to reduce the number of things that have to be considered.
For example, using higher-level languages rather than lower-level improves productivity, because one line is a separate thing, and being able to write a program in fewer lines reduces the number of things.
Procedural programming came about as an attempt to reduce complexity by making it possible to treat a function as a thing. In order to do that, though, we have to be able to think about what the function does in a coherent manner, and with confidence that we're right. (Object-oriented programming does a similar thing, on a larger scale.)
There are several ways to do this. Design-by-contract is a way of exactly specifying what the function does. Using function parameters rather than global variables to call the function and get results reduces the complexity of the function.
Unit testing is one way to verify that the function does what it is supposed to. It's usually possible to test all the code in a function, and sometimes all the execution paths. It is a way to tell if the function works as it should or not. If the function works, we can think about it as a single thing, rather than as multiple things we have to keep track of.
It serves other purposes. Unit tests are usually quick to run, and so can catch bugs quickly, when they're easy to fix. If developers make sure a function passes the tests before being checked in, then the tests are a form of documenting what the function does that is guaranteed correct. The act of creating the tests forces the test writer to think about what the function should be doing. After that, whoever wanted the change can look at the tests to see if he or she was properly understood.
By way of contrast, larger tests are not exhaustive, and so can easily miss lots of bugs. They're bad at localizing bugs. They are usually performed at fairly long intervals, so they may detect a bug some time after it's made. They define parts of the total user experience, but provide no basis to reason about any part of the system. They should not be neglected, but they are not a substitute for unit tests.
As others have stated, the length of the feedback loop and isolation of the problem to a specific component are key benefits of Unit Tests.
Another way that they are complementary to functional tests is how coverage is tracked in some organizations:
Unit tests on code coverage
Functional tests on requirements coverage
Functional tests might miss features that were implemented but are not in the spec.
Being based on the code, Unit tests might miss that a certain feature wasn't implemented, which is where requirements based coverage analysis of Functional testing comes in.
A final point : there are some things that are easier/faster to test at the unit level, especially around error scenarios.
Unit testing will help you identify the source of your bug more clearly and let you know that you have a problem earlier. Both are good to have, but they are different, and unit testing does have benefits.
The software you test is a system. When you are testing it as a whole you are black box testing since you primarily deal with inputs and outputs. Black box testing is great when you have no means of getting inside of the system.
But since you usually do, you create a lot of unit tests that actually test your system as a white box. You can slice system open in many ways and organize your tests depending on system internal structure. White box testing provides you with many more ways of testing and analyzing systems. It's clearly complimentary to Black box testing and should not be considered as an alternative or competing methodology.
I've only done minor unit testing at various points in my career. Whenever I start diving into it again, it always troubles me how to prove that my tests are correct. How can I tell that there isn't a bug in my unit test? Usually I end up running the app, proving it works, then using the unit test as a sort of regression test. What is the recommended approach and/or what is the approach you take to this problem?
Edit: I also realize that you could write small, granular unit tests that would be easy to understand. However, if you assume that small, granular code is flawless and bulletproof, you could just write small, granular programs and not need unit testing.
Edit2: For the arguments "unit testing is for making sure your changes don't break anything" and "this will only happen if the test has the exact same flaw as the code", what if the test overfits? It's possible to pass both good and bad code with a bad test. My main question is what good is unit testing since if your tests can be flawed you can't really improve your confidence in your code, can't really prove your refactoring worked, and can't really prove that you met the specification?
The unit test should express the "contract" of whatever you are testing. It's more or less the specification of the unit put into code. As such, given the specs, it should be more or less obvious whether the unit tests are "correct".
But I would not worry too much about the "correctness" of the unit tests. They are part of the software, and as such, they could well be incorrect as well. The point of unit tests - from my POV - is that they ensure the "contract" of your software is not broken by accident. That is what makes unit tests so valuable: You can dig around in the software, refactor some parts, change the algorithms in others, and your unit tests will tell you if you broke anything. Even incorrect unit tests will tell you that.
If there is a bug in your unit tests, you will find out - because the unit test fails while the tested code turns out to be correct. Well then, fix the unit test. No big deal.
Well, Dijkstra famously said:
"Testing shows the presence, not the
absence of bugs"
IOW, how would you write a unit test for the function add(int, int)?
IOW, it's a tough one.
There are two ways to help ensure the correctness of your unit tests:
TDD: Write the test first, then write the code it's meant to test. That means you get to see them fail. If you know that it detects at least some classes of bugs (such as "I haven't implemented any functionality in the function I want to test yet"), then you know that it's not completely useless. It may still let some other bugs slip past, but we know that the test is not completely incorrect.
Have lots of tests. If one test lets some bugs slip past, they'll most likely cause errors further down the line, causing other tests to fail. As you notice that, and fix the offending code, you get a chance to examine why the first test didn't catch the error as expected.
And finally, of course, keep the unit tests so simple that they're unlikely to contain bugs.
For this to be a problem your code would have to be buggy in a way that coincidentally causes your tests to pass. This happened to me recently, where I was checking that a given condition (a) caused a method to fail. The test passed (i.e. the method failed), but it passed because another condition (b) caused a failure. Write your tests carefully, and make sure that unit tests test ONE thing.
Generally though, tests cannot be written to prove code is bug free. They're a step in the right direction.
The complexity of the unit test code is (or should be) less (often orders of magnitude less) than the real code
The chance of your coding a bug in your unit test that exactly matches a bug in your real code is much less than just coding the bug in your real code (if you code a bug in your unit test that doesn't match a bug in your real code it should fail). Of course if you have made incorrect assumptions in your real code you are likely to make the same assumption again - although the mind set of unit testing should still reduce even that case
As already alluded to, when you write a unit test you have (or should have) a different mind set. When writing real code you're thinking "how do I solve this problem". When writing a unit test you're thinking, "how do I test every possibly way this could break"
As others have already said, it's not about whether you can prove that the unit tests are correct and complete (although that's almost certainly much easier with test code), as it is reducing the bug count to a very low number - and pushing it lower and lower.
Of course there has to come a point where your confident in your unit tests enough to rely on them - for example when doing refactorings. Reaching this point is usually just a case of experience and intuition (although there are code coverage tools that help).
I had this same question, and having read the comments, here's what I now think (with due credit to the previous answers):
I think the problem may be that we both took the ostensible purpose of unit tests -- to prove the code is correct -- and applied that purpose to the tests themselves. That's fine as far as it goes, except the purpose of unit tests is not to prove that the code is correct.
As with all nontrivial endeavors, you can never be 100% sure. The correct purpose of unit tests is to reduce bugs, not eliminate them. Most specifically, as others have noted, when you make changes later on that might accidentally break something. Unit tests are just one tool to reduce bugs, and certainly should not be the only one. Ideally you combine unit testing with code review and solid QA in order to reduce bugs to a tolerable level.
Unit tests are much simpler than your code; it's not possible to make your code as simple as a unit test if your code does anything significant. If you write "small, granular" code that's easy to prove correct, then your code will consist of a huge number of small functions, and you're still going to have to determine whether they all work correctly in concert.
Since unit tests are inevitably simpler than the code they're testing, they're less likely to have bugs. Even if some of your unit tests are buggy, overall they're still going to improve the quality of your main codebase. (If your unit tests are so buggy that this isn't true, then likely your main codebase is a steaming pile as well, and you're completely screwed. I think we're all assuming a basic level of competence.)
If you DID want to apply a second level of unit testing to prove your unit tests correct, you could do so, but it's subject to diminishing returns. To look at it faux-numerically:
Assume that unit testing reduces the number of production bugs by 50%. You then write meta-unit tests (unit tests to find bugs in the unit tests). Say that this finds problems with your unit tests, reducing the production bug rate to 40%. But it took 80% as long to write the meta-unit tests as it did to write the unit tests. For 80% of the effort you only got another 20% of the gain. Maybe writing meta-meta-unit tests gives you another 5 percentage points, but now again that took 80% of the time it took to write the meta-unit tests, so for 64% of the effort of writing the unit tests (which gave you 50%) you got another 5%. Even with substantially more liberal numbers, it's not an efficient way to spend your time.
In this scenario it's clear that going past the point of writing unit tests isn't worth the effort.
I guess writing the test first (before writing the code) is a pretty good way of being sure your test is valid.
Or you could write tests for your unit tests... :P
You don't tell. Generally, the tests will be simpler than the code they're testing, so the idea is simply that they'll be less likely to have bugs than the real code will.
First let me start by saying that unit testing is NOT only about testing. It is more about the design of the application. To see this in action you should put a camera with your display and record your coding while writing unit testing. You will realize that you are making a lot of design decisions when writing unit tests.
How to know if my unit tests are good?
You cannot test the logical part period! If your code is saying that 2+2 = 5 and your test is making sure that 2+2 = 5 then for you 2+2 is 5. To write good unit tests you MUST have good understanding of the domain you are working with. When you know what you are trying to accomplish you will write good tests and good code to accomplish it. If you have many unit tests and your assumptions are wrong then sooner or later you will find out your mistakes.
This is one of the advantages of TDD: the code acts as a test for the tests.
It is possible that you'll make equivalent errors, but it is uncommon in my experience.
But I have certainly had the case where I write a test that should fail only to have it pass, which told me my test was wrong.
When I was first learning unit testing, and before I was doing TDD, I would also deliberately break the code after writing the test to ensure that it failed as I expected. When I didn't I knew the test was broken.
I really like Bob Martin's description of this as being equivalent to double entry bookkeeping.
As above, the best way is to write the test before the actual code. Find real life examples of the code your testing also if applicable (mathematical formula or similar), and compare the unit test and expected output to that.
This is something that bugs everyone that uses unit tests. If I would have to give you a short answer I 'd tell you to always trust your unit tests. But I would say that this should be backed up with your previous experience:
Did you have any defects that were reported from manual testing and the unit test didn't catch (although it was responsible to) because there was a bug in your test?
Did you have false negatives in the past?
Are your unit tests simple enough?
Do you write them before new code or at least in parallel?
You can't prove tests are correct, and if you're trying to, you're Doing It Wrong.
Unit tests are a first screen - a smoke test - like all automated testing. They are primarily there to tell you if a change you make later on breaks stuff. They are not designed to be a proof of quality, even at 100% coverage.
The metric does make management feel better, though, and that is useful in itself sometimes!
Dominic mentioned that "For this to be a problem your code would have to be buggy in a way that coincidentally causes your tests to pass.". One technique you can use to see if this is a problem is mutation testing. It makes changes to your code, and see if it causes the unit tests to fail. If it doesn't, then it may indicate areas where the testing isn't 100% thorough.
Unit tests are your requirements concretized. I don't know about you but I like having the requirements specified before starting to code (TDD). By writing them and treating them like any other piece of your code you'll start to feel confident introducing new features without breaking old functionality. To ensure that all your code is needed and that the tests actually tests the code I use pitest (other variants for mutation testing exists for other languages). For me, untested code, is buggy code, however clean it may be.
If the test tests complex code and is complex itself I often write tests for my tests (example).
Edit: I also realize that you could write small, granular unit tests that would be easy to understand. However, if you assume that small, granular code is flawless and bulletproof, you could just write small, granular programs and not need unit testing.
The idea of unit testing is to test the most granular things, then stack together tests to prove the larger case. If you're writing large tests, you lose a bit of the benefits there, although it's probably quicker to write larger tests.