Related
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I have read a lot of threads about the joys and awesome points of unit testing. Is there have a good argument against unit testing?
In the places I have previously worked, unit testing is usually used as a reason to run with a smaller testing department; the logic is "we have UNIT TESTS!! Our code can't possibly fail!! Because we have unit tests, we don't need real testers!!"
Of course that logic is flawed. I have seen many cases where you cannot trust the tests. I have also seen many cases where the tests become out of date due to tight time schedules - when you have a week to do a big job, most developers would spend the week doing the real code and shipping the product, rather than refactoring the unit tests for that first week, then pleading for at least another week to do the real code, and then spending a final week bringing the unit tests up to date with what they actually wrote.
I have also seen cases where the business logic involved in the unit test was more monstrous and hard to understand than the logic buried in the application. When these tests fail, you have to spend twice as long trying to work out the problem - is the test flawed, or the real code?
Now the zealots are not going to like this bit: the place where I work has largely escaped using unit tests because the developers are of a high enough calibre that it is hard to justify the time and resource to write unit tests (not to mention we use REAL testers). For real. Writing unit tests would have only given us minimal value, and the return on investment is just not there. Sure it would give people a nice warm fuzzy feeling - "I can sleep at night because my code is PROTECTED by unit tests and consequently the universe is at a nice equilibrium", but the reality is we are in the business of writing software, not giving managers warm fuzzy feelings.
Sure, there absolutely are good reasons for having unit tests. The trouble with unit testing and TDD is:
Too many people bet the family farm on it.
Too many people use it as a religion rather than just a tool or another methodology.
Too many people have tried to make money out of it, which has skewed how it should be used.
In reality, it should be used as one of the tools or methodologies that you use on a day to day basis, and it should never become the single methodology.
It's important to understand that it's not free. Tests require effort to write - and, more importantly, maintain.
Project managers and development teams need to be aware of this.
Virtually none of my bugs would have been found by unit testing. My bugs are mostly integration or unexpected-use-case bugs, which in order to have found them earlier, more extensive (and ideally automated) system tests would have been the best bet.
I'm waiting for more evidence-based and less religion-based arguments for unit testing, as dummymo said. And I don't mean some experiment in some academic setting; I mean an argument that for my development scenario and programming ability, cost-benefit would be positive.
So, to agree with other answers to the OP: because they cost time and cost-benefit is not shown.
You have a data access layer that isn't easy adapted for mocking.
The simple truth is, when you write some code, you have to make sure it works before you say it's done. Which means you exercise it - build some scaffolding to call the function, passing some arguments, checking to make sure it returns what you expect. Is it so much extra work, to keep the scaffolding around, so you can run the tests again?
Well yes, actually, it can be. More often than not the tests will fail, even when the code is right, because the data you were using is no longer consistent, etc.
But if you have a unit testing framework in place, the cost of keeping the test code around can be only marginally more work than throwing it away. And while yes, you'll find that many of your test cases will fail because of problems with the data you are using, instead of problems with the code, that will happen less as you learn how to structure your tests so as to minimize the problem.
True, passing your unit tests does not guarantee that your system works. But it does provide some assurance that certain subsystems are working, which isn't nothing. And the test cases provide useful examples of how the functions were meant to be called.
It's not a panacea, but it's a useful practice.
Formal verification.
If you can formally prove the correctness of code, there is no reason to unit test it unless the test condition brings in new variables, in which case, you'd still only have a small amount of unit tests (or prove for the new variables).
Unit tests will tell you whether one specific class method sets a variable correctly (or some variation on that). That does not, in any way, shape, or form, indicate that your application will behave properly or that it will handle the circumstances it will need to handle.
Any problem you can think to write a test for, you are going to handle in your code, and that problem is never going to show up. So then you have 300 tests passing but real-world scenarios you just didn't think to test for. The effort required to create and maintain the tests, then, isn't necessarily worth it.
It's the usual cost/benefit analysis.
Cost: You need to spend time developing and maintaining the tests, and put resources into actually running them.
Benefits are well known (mostly cheaper maintenance/refactoring with less bugs).
So, you balance one against the other in the context of the project.
If it's a throwaway quick hack you know will never be re-used, unit tests might not make sense. Although to be honest, if I had a dollar for every throwaway quick hack that I saw running years later or worse, had to maintaing/refactor years later, I'd probably be able to be one of venture capitalists investing into SO :)
Non-deterministic outcomes.
In simple cases you can seed the random generator(s) (or mock them somehow) to get reproducible results but when the algorithm is complex this becomes impossible as code changes will alter the demand for random numbers and thus alter the results.
This would rarely be encountered in a business situation but it's quite possible in games.
Testing is like insurance.
You don't put all your money in to it.
But you don't avoid your life insurance. (People form US should still be remembering the Health Insurance Bill).
Insurance is ESSENTIAL evil.
BUT BUT BUT...
You don't get insured expecting a Fatal accident to recover all the money you put into your insurance plan.
In summary,
There is SOME reason to write Tests.
Unit tests are sometimes One of the many ways to go forward
BUT There is NO REASON to just to focus entirely on writing (Unit) Tests.
It can discourage experimenting with several variations, especially in early stages of a project. (But it can also encourage experimenting in later stages!)
It can't replace system testing, because it doesn't cover the relationship between components. So if you have to split up the available testing time between system testing and unit testing, then too much unit testing can have a negative impact on the amount of system tests.
I want to add, that I usually encourage unit testing!
It is impossible to generalize where unit tests are going to provide cost-benefit and where they are not. I see a lot of people arguing strongly in favor of unit testing and blaming people who don't for not using TDD enough, while completely ignoring the fact that applications can differ as much as the real world does.
For instance, it is incredibly hard to get anything useful out of unit tests when you have a lot of integration points, either between systems, and/or between processes and threads of your own application.
If all you ever did were websites like Stackoverflow, where problem domain is well understood, and most solutions are fairly trivial, then yes, writing unit tests have a lot of benefits, but there are lots of applications out there that simply can't be unit tested properly, as they lack, well, "units".
Laziness; sometimes I'm lazy and just don't want to do it!
But seriously, unit testing is great, but if I'm just coding for my own enjoyment I generally don't do it, because the projects are short lived, I'm the only one working on it, and they're not that big.
There is never a reason to never write unit tests.
There are good reasons to not write specific unit tests. (Especially if you use code generation. Of course you could code generate the unit tests to make sure nobody has mucked with the generated code. But that is dependent upon trusting the team.)
*Edit
Oh. And from what I understand some things in functional programming either compile thus work or don't compile.
Would those things need unit tests?
I agree with the notion that there are no good arguments against unit testing in general. There are some specific situations, however, where unit testing may not be a viable option or is at least problematic and/or poses a difficult return-on-investment proposition for the level of effort involved to create and maintain tests.
Here are some examples:
Real-time-dependent behavior in response to external conditions. Some purists may argue that this is not unit testing but rather involves scenarios at an integration or system testing level. However, I've written code for simple, low-level functionality for quasi-embedded applications where it would be useful to at least partially test real-time response via a unit-testing framework for build/regression testing purposes.
Testing behaviorial and/or policy-level functionality that requires a complex data description of the environmental state to which the tested code module is responding. This is related to an earlier poster's comment regarding the difficulty of doing unit testing involving a data access layer that isn't easily adapted for mocking. Although the behavior/policy being tested may be relatively simple, it needs to be tested across a complex state description. The value of doing unit testing here is in assuring that rare yet key conditions are handled correctly for a mission-critical application. One wants to mock the latter and create a simulated environment/state, but the cost of doing so may be high.
There are at least two alternatives to unit testing for the above scenarios:
For real-time or quasi-real-time functionality testing, extensive system testing can be done to try to compensate for the lack of good unit testing. For some applications this may be the only option, e.g., embedded systems involving hardware and/or physical systems.
Create a test harness or system-level simulator that facilitates extensive testing across a range of randomly simulated conditions. This can be useful for testing the policy/behavior scenarios described earlier involving a complex environmental state. Although significant work may be involved in creating the test harness or simulator, the return-on-investment may be a much better value than for isolated unit tests since a much broader range of conditions can be tested.
Since the test environment involves random rather than specific conditions, this approach may not offer quite the level of assurance desired for some mission-critical scenarios. Conducting extensive tests may help make up for this. Alternatively, creating a test harness or system simulator for random conditions may also help with reducing the overall cost of testing specific complex state scenarios since the development cost is now shared across a broader range of testing needs.
In the end, how to best approach testing any given application comes down to cost vs. value. Unit testing is one of the best options and should always be used where feasible, but it is not universally applicable to all scenarios. Like many things in software, sometimes it will just be a judgment call one has to make and then be prepared to make adjustments based on the outcome.
Unit testing is a trade-off. I see two problems:
It takes time. Not only to write the tests, but also (and this is the most annoying) if you want to make an important change, now you have two places you need to modify. In the worst case it could possibly discourage you to re-architect your codebase.
It only protects against problems that you think could arise, and you mostly cannot test against side-effects. This can lead to a false sense of security as mentioned before.
I agree unit testing is a valuable tool for increasing the reliability of enterprise software with a relatively stable codebase. But for personal projects or infant projects, I think a generous use of asserts in your code is a much better trade-off.
I wouldn't say it's an argument against it, but for us we have a legacy application with a TON of code, and written in COBOL. It is virtually impossible at this point to say we want to implement unit testing and do it with any degree of accuracy or within a reasonable time frame for business as pointed out by duffymo.
So I guess to add onto that, maybe one argument would be the inability (in some cases) of trying to implement unit tests after development has been completed (and maintained for years).
Instead of completely getting rid of them, we write unit tests only for core functionality such as payment authorization, user authentication, etc etc. It is very useful as there will always be some touch points that are very sensitive to code changes in a large code base and you would want some way to verify those touch points work without failing in QA.
For writing unit tests in general, learning curve is the biggest reason I know of to not bother. I have been trying to learn good unit testing for about 1.5 years now, and I feel like I'm just getting good at it (writing audit log spies, mocking, testing 1 constraint per test, etc.), although I feel it has sped up development for me for about 1 year of that time. So call it 6 months of struggling through it before it really started paying off. (I was still doing "real" work during that time, of course.)
Most of the pain experienced during that time was due to failure to follow the guidelines of good unit testing.
For a variety of specific cases, ability to unit test may be blocked; others have commented on some of those.
In Test Driven Development, the unit tests are actually more importantly a way to design your code to be testable to begin with. As it turns out your code tends to be more modular and writing the tests helps to flesh out APIs and so forth.
Too often though, you find yourself developing the code then writing the tests, commenting out the code you just wrote to ensure that the tests fail, then selectively removing the comment tokens to make the tests pass. Why? Well because it's much harder to write tests than it is to write code in some cases. It's also often much more difficult to write code that can be tested in a completely automated way. Think about user interfaces, code that generates images or pdfs, database transactions.
So unit tests do help a lot, but expect to write about 3 times as much code for the tests than you will write for the actual code. Plus all of this will need to be maintained. Significant changes to the application will invalidate huge chunks of tests - a good measure of the impact to the system but still... Are you prepared for that? Are you on a small team where you are doing the job of 5 developers? If so then automated TDD development just won't fly - you won't have time to get stuff done fast enough. So then you end up relying on you're own manual testing, QA testing stuff as well and just living with bugs slipping through and then fixing things up ASAP. It's unfortunate, high pressure and exasperating but it's reality in small companies that don't hire enough developers for the work that needs to be done.
No, really I don't. In my experience people who do are presenting a straw man's argument or just don't know how to unit test things that are not obvious how to unit test.
#Khorkak - If you change a feature in your production code, only a handfull of your unit tests should be affected. If you don't that means that you are not decoupling your production code and testing in isolation, but instead excecising large chunks of your production code in integration. That's just poor unit testing skills, and yes it's a BIG problem. Not only because your unit test code base will become hard to maintain, but also because your production code base will suck and have the same problem.
Unit tests make no sense for Disposable Code: If the code is a Q&D proof of concept, something created during a spike to investigate various approaches, or anything else you are SURE will almost always be thrown away, then doing unit tests won't bring much return on the investment. In fact they could hurt you as the time spent there is time spent not trying a different approach etc (alternative cost)
The key is being sure that's the case, or having enough of an understanding with teammates etc that if someone says 'that's great, use that one' that you then invest the time bring the code up to the standards for NON disposable code.
For those who asked for better proof I'd refer you to this page. http://biblio.gdinwiddie.com/biblio/StudiesOfTestDrivenDevelopment
note that many of these are studies by academic types, but done against groups doing real production software work, so personally it seems like they have a pretty good amount of validity.
I have a largish complex app around 27k lines. Its essentially a rule drive multithreaded processing engine, without giving too much away Its been partially tested as it's been built, certain components.
Question I have, is what is the pro's and con's of doing unit testing on after the fact, so to speak, after its been implemented. It is clear that traditional testing is going to take 2-3+ months to test every facet, and it all needs to work, and that time is not available really.
I've done a fair bit of unit testing in the past, but generally it's been on desktop automation or LOB apps, which are fairly simple. The app is itself is highly componentized internally, interface driven really. I've not decided on what particular framework to use. Any advice would be appreciated.
What say you.
I think there are several advantages to unit testing existing code
Regression management
Better understanding of the code. Testing it will reveal cases you did not anticipate and will help define the behavior of the code
It will point out design deficiencies in the code as you stuggle to test poorly defined methods.
But I think it's more interesting to consider the cons of unit testing code. AFAIK, there are no cons. All of the time spent adding tests will pay for themselves even in everything but the shortest of time cycles.
There are many reasons to unit test code. The main reason I would advocate unit testing after the fact is simple. Your code is broken, you just don't know it yet.
There is a very simple rule in software. If the code is not tested, it's broken. This may not be immediately obvious at first, but as you begin testing, you will find bugs. It's up to you to determine how much you care about finding these bugs.
Besides this, there are several other important benefits of unit testing,
regression testing will be made simpler
other developers, that are less knowledgeable, can't break your desired behavior
the tests are a form of self documentation
can reduce time in future modifications (no more manual testing?, less bugs?)
The list can go on and on. The only real drawback is the time it takes to write these tests. I believe that drawback will always be offset by the time it takes you to debug
problems you could have found while unit testing!
Depending on how many bugs "manual testing" turns up, you could simply do test-driven bug fixing which in my experience is far more effective than simply driving up code coverage by writing "post-mortem" unit tests.
(Which is not to say writing unit tests afterwards is a bad idea, it's just that TDD is almost always a better idea.)
Here's a few of each to my mind:
Pro:
Time is saved in not having to test methods that have been removed as the design evolved over time. What is left is what really has to get tested.
By adding tests, this allows an opportunity to review all the aspects in the app and determine what other optimizations one could add now that a working prototype is ready.
Con:
Large time investment to get the tests written, new functionality may be delayed for some time to generate all the tests.
Bugs may have been introduced that the tests will discover that may cause this to be longer than initially planned.
The main point would be that adding unit tests allows for refactoring and putting more polish on the application.
I think one of the biggest con of testing "after the fact" is that you will probably have a harder time testing. If you write code without tests, you usually don't have testability in mind and end up writing code that is hard to test.
But, after you spent this extra time writing tests and changing your code for better testability, you'll be much more confident about making changes, once you won't need a lot of time to debug and check if everything is ok.
Finally, you might find new bugs which weren't caught before, and spend some time fixing it. But hey, that's what tests are for =)
Pro post facto unit testing:
Get documentation you can trust.
Improve understanding of the code.
Push toward refactoring and improving the code itself.
Fix bugs that lurk in the code.
Con post facto unit testing:
Waste time fixing bugs you can live with. (If you wrote 27KLOC, we hope it does something, right?)
Spend time understanding and refactoring code you don't need to understand.
Lose time that could go into the next project.
The unasked question is just how important an asset is this code to your organization, long term? The answer to this question determines how much you should invest. I have plenty of (successful) competitors where the major purpose of their code is to get out numbers to evaluate some new technique or idea. Once they have the numbers, the code is of little marginal value. They (rightly) test very carefully to make sure the numbers are meaningful. After that, if there are fifty open bugs that don't affect the numbers, they don't care. And why should they? The code has served its purpose.
If you are doing any refactoring, those tests will help you detect any bugs that will appear in the process.
Unit testing "after the fact" is still valuable, and provides most of the same advantages of unit testing during development.
That being said, I find it's more work to test after the fact (if you want to get the same level of testing). It's still valuable, and still worth while.
Personally, when trying to tackle something with limited time, I try to focus my testing efforts as much as possible. Any time you fix a bug, add tests to help prevent it in the future. Any time you're going to refactor, try to put enough testing in place to feel confident you're not going to break something.
The only con of adding unit testing is that it does take some development time. Personally, I find that the development time spent on testing is far outweighed by the time saved in maintenance, but this is something you need determine on your own.
Unit testing is still definitely useful. Check out http://en.wikipedia.org/wiki/Unit_testing for a full list and explanation of the benefits.
The main benefits you will gain are documentation, making change easier, and it simplifies future integration.
There are really no costs to adding unit testing except your time. Realize though that the time you spend adding unit testing will reduce the amount of time you will need to spend in other areas of development by at least the same amount and most likely more.
Unit testing doesn't prove that a system works. It proves that each unit works as an independent unit. It doesn't prove that the integrated system will work
Unit testing "after the fact" is useful for two things - finding bugs that you've missed so far and won't find using any other kind of testing (especially for rare conditions - there's huge numbers of rare conditions that can happen in particular units for any real world system), and as regression tests during maintenance.
Neither of these is going to help much in your situation - you need to do other forms of testing either way. If you don't have time to do what you need to do, taking on even more work is unlikely to help.
That said, without unit testing, I guarantee you will have nasty surprises when the customers start using the code. It's all those rare conditions - there's so many of them that some of them are bound to occur soon. Black-box testers tend to get into habitual patterns, which mean they only test so many rare cases - and they have no way of knowing what rare cases there are in particular units and how to trigger them anyway. More users means more variations in usage patterns.
I'm with those who say unit tests should be written as part of the programming process - one of the programmers responsibilities. As a rule, code gets written faster that way, as you get fewer and less complex bugs to track down as you go, and you tend to find out about them when you're still familiar with the code that has the bug.
If development is "done" I would say that there is not too much point in unit testing.
This is one of these difficult value judgement types of questions.
I would mostly agree with Epaga, that writing new tests as you fix bugs (perhaps with a couple of extra tests thrown in) is a good approach.
I would add two further comments:
Doing backed-off black box testing to a unit before making large changes can be a good idea
Consistency testing isn't unit testing, but certain types of program lend themselves to the easy generation of consistency tests. This might be one approach to making sure you don't break things.
Do your unit tests constitute 100% code coverage? Yes or no, and why or why not.
No for several reasons :
It is really expensive to reach the 100% coverage, compared to the 90% or 95% for a benefit that is not obvious.
Even with 100% of coverage, your code is not perfect. Take a look at this method (in fact it depends on which type of coverage you are talking about - branch coverage, line coverage...):
public static String foo(boolean someCondition) {
String bar = null;
if (someCondition) {
bar = "blabla";
}
return bar.trim();
}
and the unit test:
assertEquals("blabla", foo(true));
The test will succeed, and your code coverage is 100%. However, if you add another test:
assertEquals("blabla", foo(false));
then you will get a NullPointerException. And as you were at 100% with the first test, you would have not necessarily write the second one!
Generally, I consider that the critical code must be covered at almost 100%, while the other code can be covered at 85-90%
To all the 90% coverage tester:
The problem with doing so is that the 10% hard to test code is also the not-trivial code that contains 90% of the bug! This is the conclusion I got empirically after many years of TDD.
And after all this is pretty straightforward conclusion. This 10% hard to test code, is hard to test because it reflect tricky business problem or tricky design flaw or both. These exact reasons that often leads to buggy code.
But also:
100% covered code that decreases with time to less than 100% covered often pinpoints a bug or at least a flaw.
100% covered code used in conjunction with contracts, is the ultimate weapon to lead to live close to bug-free code. Code Contracts and Automated Testing are pretty much the same thing
When a bug is discovered in 100% covered code, it is easier to fix. Since the code responsible for the bug is already covered by tests, it shouldn't be hard to write new tests to cover the bug fix.
No, because there is a practical trade-off between perfect unit tests and actually finishing a project :)
It is seldom practical to get 100% code coverage in a non-trivial system. Most developers who write unit tests shoot for the mid to high 90's.
An automated testing tool like Pex can help increase code coverage. It works by searching for hard-to-find edge cases.
Yes we do.
It depends on what language and framework you're using as to how easy that is to achieve though.
We're using Ruby on Rails for my current project. Ruby is very "mockable" in that you can stub/mock out large chunks of your code without having to build in overly complicated class composition and construction designs that you would have to do in other languages.
That said, we only have 100% line coverage (basically what rcov gives you). You still have to think about testing all the required branches.
This is only really possible if you include it from the start as part of your continuous integration build, and break the build if coverage drops below 100% - prompting developers to immediately fix it. Of course you could choose some other number as a target, but if you're starting fresh, there isn't much difference for the effort to get from 90% to 100%
We've also got a bunch of other metrics that break the build if they cross a given threshold as well (cyclomatic complexity, duplication for example) these all go together and help reinforce each other.
Again, you really have to have this stuff in place from the start to keep working at a strict level - either that or set some target you can hit, and gradually ratchet it up till you get to a level you're happy with.
Does doing this add value? I was skeptical at first, but I can honestly say that yes it does. Not primarily because you have thoroughly tested code (although that is definitely a benefit), but more in terms of writing simple code that is easy to test and reason about. If you know you have to have 100% test coverage, you stop writing overly complex if/else/while/try/catch monstrosities and Keep It Simple Stupid.
What I do when I get the chance is to insert statements on every branch of the code that can be grepped for and that record if they've been hit, so that I can do some sort of comparison to see which statements have not been hit. This is a bit of a chore, so I'm not always good about it.
I just built a small UI app to use in charity auctions, that uses MySQL as its DB. Since I really, really didn't want it to break in the middle of an auction, I tried something new.
Since it was in VC6 (C++ + MFC) I defined two macros:
#define TCOV ASSERT(FALSE)
#define _COV ASSERT(TRUE)
and then I sprinkled
TCOV;
throughout the code, on every separate path I could find, and in every routine.
Then I ran the program under the debugger, and every time it hit a TCOV, it would halt.
I would look at the code for any obvious problems, and then edit it to _COV, then continue. The code would recompile on the fly and move on to the next TCOV.
In this way, I slowly, laboriously, eliminated enough TCOV statements so it would run "normally".
After a while, I grepped the code for TCOV, and that showed what code I had not tested. Then I went back and ran it again, making sure to test more branches I had not tried earlier.
I kept doing this until there were no TCOV statements left in the code.
This took a few hours, but in the process I found and fixed several bugs. There is no way I could have had the discipline to make and follow a test plan that would have been that thorough.
Not only did I know I had covered all branches, but it had made me look at every branch while it was running - a very good kind of code review.
So, whether or not you use a coverage tool, this is a good way to root out bugs that would otherwise lurk in the code until a more embarrasing time.
I personally find 100% test coverage to be problematic on multiple levels. First and foremost, you have to make sure you are gaining a tangible, cost-saving benefit from the unit tests you write. In addition, unit tests, like any other code, are CODE. That means it, just like any other code, must be verified for correctness and maintained. That additional time verifying additional code for correctness, and maintaining it and keeping those tests valid in response to changes to business code, adds cost. Achieving 100% test coverage and ensuring you test you're code as thoroughly as possible is a laudable endeavor, but achieving it at any cost...well, is often too costly.
There are many times when covering error and validity checks that are in place to cover fringe or extremely rare, but definitely possible, exceptional cases are an example of code that does not necessarily need to be covered. The amount of time, effort (and ultimately money) that must be invested to achieve coverage of such rare fringe cases is often wasteful in light of other business needs. Properties are often a part of code that, especially with C# 3.0, do not need to be tested as most, if not all, properties behave exactly the same way, and are excessively simple (single-statement return or set.) Investing tremendous amounts of time wrapping unit tests around thousands of properties could quite likely be better invested somewhere else where a greater, more valuable tangible return on that investment can be realized.
Beyond simply achieving 100% test coverage, there are similar problems with trying to set up the "perfect" unit. Mocking frameworks have progressed to an amazing degree these days, and almost anything can be mocked (if you are willing to pay money, TypeMock can actually mock anything and everything, but it does cost a lot.) However, there are often times when dependencies of your code were not written in a mock-able way (this is actually a core problem with the vast bulk of the .NET framework itself.) Investing time to achieve the proper scope of a test is useful, but putting in excessive amounts of time to mock away everything and anything under the face of the sun, adding layers of abstraction and interfaces to make it possible, is again most often a waste of time, effort, and ultimately money.
The ultimate goal with testing shouldn't really be to achieve the ultimate in code coverage. The ultimate goal should be achieving the greatest value per unit time invested in writing unit tests, while covering as much as possible in that time. The best way to achieve this is to take the BDD approach: Specify your concerns, define your context, and verify the expected outcomes occur for any piece of behavior being developed (behavior...not unit.)
On a new project I practice TDD and maintain 100% line coverage. It mostly occurs naturally through TDD. Coverage gaps are usually worth the attention and are easily filled. If the coverage tool I'm using provided branch coverage or something else I'd pay attention to that, although I've never seen branch coverage tell me anything, probably because TDD got there first.
My strongest argument for maintaining 100% coverage (if you care about coverage at all) is that it's much easier to maintain 100% coverage than to manage less than 100% coverage. If you have 100% coverage and it drops, you immediately know why and can easily fix it, because the drop is in code you've just been working on. But if you settle for 95% or whatever, it's easy to miss coverage regressions and you're forever re-reviewing known gaps. It's the exact reason why current best practice requires one's test suite to pass completely. Anything less is harder, not easier, to manage.
My attitude is definitely bolstered by having worked in Ruby for some time, where there are excellent test frameworks and test doubles are easy. 100% coverage is also easy in Python. I might have to lower my standards in an environment with less amenable tools.
I would love to have the same standards on legacy projects, but I've never found it practical to bring a large application with mediocre coverage up to 100% coverage; I've had to settle for 95-99%. It's always been just too much work to go back and cover all the old code. This does not contradict my argument that it's easy to keep a codebase at 100%; it's much easier when you maintain that standard from the beginning.
No because I spent my time adding new features that help the users rather than tricky to write obscure tests that deliver little value. I say unit test the big things, subtle things and things that are fragile.
I generally write unit tests just as a regression-prevention method. When a bug is reported that I have to fix, I create a unit test to ensure that it doesn't re-surface in the future. I may create a few tests for sections of functionality I have to make sure stay intact (or for complex inter-part interactions), but I usually want for the bug fix to tell me one is necessary.
I usually manage to hit 93..100% with my coverage but I don't aim for 100% anymore. I used to do that and while it's doable, it's not worth the effort beyond a certain point because testing blindly obvious usually isn't needed. Good example of this could be the true evaluation branch of the following code snipped
public void method(boolean someBoolean) {
if (someBoolean) {
return;
} else {
/* do lots of stuff */
}
}
However what's important to achieve is to as close to 100% coverage on functional parts of the class as possible since those are the dangerous waters of your application, the misty bog of creeping bugs and undefined behaviour and of course the money-making flea circus.
From Ted Neward blog.
By this point in time, most developers have at least heard of, if not considered adoption of, the Masochistic Testing meme. Fellow NFJS'ers Stuart Halloway and Justin Gehtland have founded a consultancy firm, Relevance, that sets a high bar as a corporate cultural standard: 100% test coverage of their code.
Neal Ford has reported that ThoughtWorks makes similar statements, though it's my understanding that clients sometimes put accidental obstacles in their way of achieving said goal. It's ambitious, but as the ancient American Indian proverb is said to state,
If you aim your arrow at the sun, it will fly higher and farther than if you aim it at the ground.
In many cases it's not worth getting 100% statement coverage, but in some cases, it is worth it. In some cases 100% statement coverage is far too lax a requirement.
The key question to ask is, "what's the impact if the software fails (produces the wrong result)?". In most cases, the impact of a bug is relatively low. For example, maybe you have to go fix the code within a few days and rerun something. However, if the impact is "someone might die in 120 seconds", then that's a huge impact, and you should have a lot more test coverage than just 100% statement coverage.
I lead the Core Infrastructure Initiative Best Practices Badge for the Linux Foundation. We do have 100% statement coverage, but I wouldn't say it was strictly necessary. For a long time we were very close to 100%, and just decided to do that last little percent. We couldn't really justify the last few percent on engineering grounds, though; those last few percent were added purely as "pride of workmanship". I do get a very small extra piece of mind from having 100% coverage, but really it wasn't needed. We were over 90% statement coverage just from normal tests, and that was fine for our purposes. That said, we want the software to be rock-solid, and having 100% statement coverage has helped us get there. It's also easier to get 100% statement coverage today.
It's still useful to measure coverage, even if you don't need 100%. If your tests don't have decent coverage, you should be concerned. A bad test suite can have good statement coverage, but if you don't have good statement coverage, then by definition you have a bad test suite. How much you need is a trade-off: what are the risks (probability and impact) from the software that is totally untested? By definition it's more likely to have errors (you didn't test it!), but if you and your users can live with those risks (probability and impact), it's okay. For many lower-impact projects, I think 80%-90% statement coverage is okay, with better being better.
On the other hand, if people might die from errors in your software, then 100% statement coverage isn't enough. I would at least add branch coverage, and maybe more, to check on the quality of your tests. Standards like DO-178C (for airborne systems) take this approach - if a failure is minor, no big deal, but if a failure could be catastrophic, then much more rigorous testing is required. For example, DO-178C requires MC/DC coverage for the most critical software (the software that can quickly kill people if it makes a mistake). MC/DC is way more strenuous than statement coverage or even branch coverage.
I only have 100% coverage on new pieces of code that have been written with testability in mind. With proper encapsulation, each class and function can have functional unit tests that simultaneously give close to 100% coverage. It's then just a matter of adding some additional tests that cover some edge cases to get you to 100%.
You shouldn't write tests just to get coverage. You should be writing functional tests that test correctness/compliance. By a good functional specification that covers all grounds and a good software design, you can get good coverage for free.
Yes, I have had projects that have had 100% line coverage. See my answer to a similar question.
You can get 100% line coverage, but as others have pointed out here on SO and elsewhere on the internet its maybe only a minimum. When you consider path and branch coverage, there's a lot more work to do.
The other way of looking at it is to try to make your code so simple that its easy to get 100% line coverage.
There's a lot of good information here, I just wanted to add a few more benefits that I've found when aiming for 100% code coverage in the past
It helps reduce code complexity
Since it is easier to remove a line than it is to write a test case, aiming for 100% coverage forces you to justify every line, every branch, every if statement, often leading you to discover a much simpler way to do things that requires fewer tests
It helps develop good test granularity
You can achieve high test coverage by writing lots of small tests testing tiny bits of implementation as you go. This can be useful for tricky bits of logic but doing it for every piece of code no matter how trivial can be tedious, slow you down and become a real maintenance burden also making your code harder to refactor. On the other hand, it is very hard to achieve good test coverage with very high level end to end behavioural tests because typically the thing you are testing involves many components interacting in complicated ways and the permutations of possible cases become very large very quickly. Therefore if you are practical and also want to aim for 100% test coverage, you quickly learn to find a level of granularity for your tests where you can achieve a high level of coverage with a few good tests. You can achieve this by testing components at a level where they are simple enough that you can reasonably cover all the edge cases but also complicated enough that you can test meaningful behaviour. Such tests end up being simple, meaningful and useful for identifying and fixing bugs. I think this is a good skill and improves code quality and maintainability.
A while ago I did a little analysis of coverage in the JUnit implementation, code written and tested by, among others, Kent Beck and David Saff.
From the conclusions:
Applying line coverage to one of the best tested projects in the world, here is what we learned:
Carefully analyzing coverage of code affected by your pull request is more useful than monitoring overall coverage trends against thresholds.
It may be OK to lower your testing standards for deprecated code, but do not let this affect the rest of the code. If you use coverage thresholds on a continuous integration server, consider setting them differently for deprecated code.
There is no reason to have methods with more than 2-3 untested lines of code.
The usual suspects (simple code, dead code, bad weather behavior, …) correspond to around 5% of uncovered code.
In summary, should you monitor line coverage? Not all development teams do, and even in the JUnit project it does not seem to be a standard practice. However, if you want to be as good as the JUnit developers, there is no reason why your line coverage would be below 95%. And monitoring coverage is a simple first step to verify just that.
I tried looking through all the pages about unit tests and could not find this question. If this is a duplicate, please let me know and I will delete it.
I was recently tasked to help implement unit testing at my company. I realized that I could unit test all the Oracle PL/SQL code, Java code, HTML, JavaScript, XML, XSLT, and more.
Is there such a thing as too much unit testing? Should I write unit tests for everything above or is that overkill?
This depends on the project and its tolerance for failure. There is no single answer. If you can risk a bug, then don't test everything.
When you have tons of tests, it is also likely you will have bugs in your tests. Adding to your headaches.
test what needs testing, leave what does not which often leaves the fairly simple stuff.
Is there such as thing as too much unit testing?
Sure. The problem is finding the right balance between enough unit testing to cover the important areas of functionality, and focusing effort on creating new value for your customers in the terms of system functionality.
Unit testing code vs. leaving code uncovered by tests both have a cost.
The cost of excluding code from unit testing may include (but aren't limited to):
Increased development time due to fixing issues you can't automatically test
Fixing problems discovered during QA testing
Fixing problems discovered when the code reaches your customers
Loss of revenue due to customer dissatisfaction with defects that made it through testing
The costs of writing a unit test include (but aren't limited to):
Writing the original unit test
Maintaining the unit test as your system evolves
Refining the unit test to cover more conditions as you discover them in testing or production
Refactoring unit tests as the underlying code under test is refactored
Lost revenue when it takes longer for you application to reach enter the market
The opportunity cost of implementing features that could drive sales
You have to make your best judgement about what these costs are likely to be, and what your tolerance is for absorbing such costs.
In general, unit testing costs are mostly absorbed during the development phase of a system - and somewhat during it's maintenance. If you spend too much time writing unit tests you may miss a valuable window of opportunity to get your product to market. This could cost you sales or even long-term revenue if you operate in a competitive industry.
The cost of defects is absorbed during the entire lifetime of your system in production - up until the point the defect is corrected. And potentially, even beyond that, if they defect is significant enough that it affects your company's reputation or market position.
Kent Beck of JUnit and JUnitMax fame answered a similar question of mine.
The question has slightly different semantics but the answer is definitely relevant
The purpose of Unit tests is generally to make it possibly to refector or change with greater assurance that you did not break anything. If a change is scary because you do not know if you will break anything, you probably need to add a test. If a change is tedious because it will break a lot of tests, you probably have too many test (or too fragile a test).
The most obvious case is the UI. What makes a UI look good is something that is hard to test, and using a master example tends to be fragile. So the layer of the UI involving the look of something tends not to be tested.
The other times it might not be worth it is if the test is very hard to write and the safety it gives is minimal.
For HTML I tended to check that the data I wanted was there (using XPath queries), but did not test the entire HTML. Similarly for XSLT and XML. In JavaScript, when I could I tested libraries but left the main page alone (except that I moved most code into libraries). If the JavaScript is particularly complicated I would test more. For databases I would look into testing stored procedures and possibly views; the rest is more declarative.
However, in your case first start with the stuff that worries you the most or is about to change, especially if it is not too difficult to test. Check the book Working Effectively with Legacy Code for more help.
Yes, there is such a thing as too much unit testing. One example would be unit testing in a whitebox manner, such that you're effectively testing the specific implementation; such testing would effectively slow down progress and refactoring by requiring compliant code to need new unit tests (because the tests were dependent upon specific implementation details).
I suggest that in some situations you might want automated testing, but no 'unit' testing at all (Should one test internal implementation, or only test public behaviour?), and that any time spent writing unit tests would be better spent writing system tests.
While more tests is usually better (I have yet to be on a project that actually had too many tests), there's a point at which the ROI bottoms out, and you should move on. I'm assuming you have finite time to work on this project, by the way. ;)
Adding unit tests has some amount of diminishing returns -- after a certain point (Code Complete has some theories), you're better off spending your finite amount of time on something else. That may be more testing/quality activities like refactoring and code review, usability testing with real human users, etc., or it could be spent on other things like new features, or user experience polish.
As EJD said, you can't verify the absence of errors.
This means there are always more tests you could write. Any of these could be useful.
What you need to understand is that unit-testing (and other types of automated testing you use for development purposes) can help with development, but should never be viewed as a replacement for formal QA.
Some tests are much more valuable than others.
There are parts of your code that change a lot more frequently, are more prone to break, etc. These are the most economical tests.
You need to balance out the amount of testing you agree to take on as a developer. You can easily overburden yourself with unmaintainable tests. IMO, unmaintainable tests are worse than no tests because they:
Turn others off from trying to maintain a test suite or write new tests.
Detract from you adding new, meaningful functionality. If automated testing is not a net-positive result, you should ditch it like other engineering practices.
What should I test?
Test the "Happy Path" - this ensures that you get interactions right, and that things are wired together properly. But you don't adequately test a bridge by driving down it on a sunny day with no traffic.
Pragmatic Unit Testing recommends you use Right-BICEP to figure out what to test. "Right" for the happy path, then Boundary conditions, check any Inverse relationships, use another method (if it exists) to Cross-check results, force Error conditions, and finally take into account any Performance considerations that should be verified. I'd say if you are thinking about tests to write in this way, you're most likely figure out how to get to an adequate level of testing. You'll be able to figure out which ones are more useful and when. See the book for much more info.
Test at the right level
As others have mentioned, unit tests are not the only way to write automated tests. Other types of frameworks may be built off of unit tests, but provide mechanisms to do package level, system or integration tests. The best bang for the buck may be at a higher level, and just using unit testing to verify a single component's happy path.
Don't be discouraged
I'm painting a more grim picture here than I expect most developers will find in reality. The bottom line is that you make a commitment to learn how to write tests and write them well. But don't let fear of the unknown scare you into not writing any tests. Unlike production code, tests can be ditched and rewritten without many adverse effects.
Unit test any code that you think might change.
You should only really write unit tests for any code which you have written yourself. There is no need to test the functionality inherently provided to you.
For example, If you've been given a library with an add function, you should not be testing that add(1,2) returns 3. Now if you've WRITTEN that code, then yes, you should be testing it.
Of course, whoever wrote the library may not have tested it and it may not work... in which case you should write it yourself or get a separate one with the same functionality.
Well, you certainly shouldn't unit test everything, but at least the complicated tasks or those that will most likely contain errors/cases you haven't thought of.
The point of unit testing is being able to run a quick set of tests to verify that your code is correct. This lets you verify that your code matches your specification and also lets you make changes and ensure that they don't break anything.
Use your judgement. You don't want to spend all of your time writing unit tests or you won't have any time to write actual code to test.
When you've unit tested your unit tests, thinking you have then provided 200% coverage.
There is a development approach called test-driven development which essentially says that there is no such thing as too much (non-redundant) unit testing. That approach, however, is not a testing approach, but rather a design approach which relies on working code and a more or less complete unit test suite with tests which drive every single decision made about the codebase.
In a non-TDD situation, automated tests should exercise every line of code you write (in particular Branch coverage is good), but even then there are exceptions - you shouldn't be testing vendor-supplied platform or framework code unless you know for certain that there are bugs which will affect you in that platform. You shouldn't be testing thin wrappers (or, equally, if you need to test it, the wrapper is not thin). You should be testing all core business logic, and it is certainly helpful to have some set of tests that exercise your database at some elemental level, although those tests will never work in the common situation where unit tests are run every time you compile.
Specifically with regard to database testing is intrinsically slow, and depending on how much logic is held in your database, quite difficult to get right. Typically things like dbs, HTML/XML documents & templating, and other document-ish aspects of a program are verified moreso than tested. The difference is usually that testing tries to exercise execution paths whereas verification tries to verify inputs and outputs directly.
To learn more about this I would suggest reading up on "Code Coverage". There is a lot of material available if you're curious about this.
I don't understand how an unit test could possibly benefit.
Isn't it sufficient for a tester to test the entire output as a whole rather than doing unit tests?
Thanks.
What you are describing is integration testing. What integration testing will not tell you is which piece of your massive application is not working correctly when your output is no longer correct.
The advantage to unit testing is that you can write a test for each business assumption or algorithm step that you need your program to perform. When someone adds or changes code to your application, you immediately know exactly which step, which piece, and maybe even which line of code is broken when a bug is introduced. The time savings on maintenence for that reason alone makes it worthwhile, but there is an even bigger advantage in that regression bugs cannot be introduced (assuming your tests are running automatically when you build your software). If you fix a bug, and then write a test specifically to catch that bug in the future, there is no way someone could accidentally introduce it again.
The combination of integration testing and unit testing can let you sleep much easier at night, especially when you've checked in a big piece of code that day.
The earlier you catch bugs, the cheaper they are to fix. A bug found during unit testing by the coder is pretty cheap (just fix the darn thing).
A bug found during system or integration testing costs more, since you have to fix it and restart the test cycle.
A bug found by your customer will cost a lot: recoding, retesting, repackaging and so forth. It may also leave a painful boot print on your derriere when you inform management that you didn't catch it during unit testing because you didn't do any, thinking that the system testers would find all the problems :-)
How much money would it cost GM to recall 10,000 cars because the catalytic converter didn't work properly?
Now think of how much it would cost them if they discovered that immediately after those converters were delivered to them, but before they were put into those 10,000 cars.
I think you'll find the latter option to be quite a bit cheaper.
That's one reason why test driven development and continuous integration are (sometimes) a good thing - testing is done all the time.
In addition, unit tests don't check that the program works as a whole, just that each little bit performs as expected. That's often quite a lot more than higher level tests would check.
From my experience:
Integration and functional testing tend to be more indicative of the overall quality of the system, than unit test suit is.
High level testing (functional, acceptance) is a QA tool.
Unit testing is a development tool. Especially in a TDD context, where unit test becomes more of a design implement, rather than that of a quality assurance.
As a result of better design, quality of the entire system improves (indirectly).
Passing unit test suite is meant to ensure that a single component conforms to the developer's intentions (correctness). Acceptance test is the level that covers validity of the system (i.e. system does what user want it to do).
Summary:
Unit test is meant as a development tool first, QA tool second.
Acceptance test is meant as a QA tool.
There is still a need for a certain level of manual testing to be performed but unit testing is used to decrease the number of defects that make it to that stage. Unit testing tests the smallest parts of the system and if they all work the chances of the application as a whole working correctly are increased significantly.
It also assists when adding new features since regression testing can be performed quickly and automatically.
For a complex enough application, testing the entire output as a whole may not cover enough different possibilities. For example, any given application has a huge number of different code paths that can be followed depending on input. In typical testing, there may be many parts of your code that are simply never encountered, because they are only used in certain circumstances, so you can't be sure that any code that isn't run in your test situation, actually works. Also, errors in one section of code may be masked a majority of the time by something else in another section of code, so you may never discover some errors.
It is better to test each function or class separately. That way, the test is easier to write, because you are only testing a certain small section of the code. It's also easier to cover every possible code path when testing, and if you test each small part separately then you can detect errors even when those errors would often be masked by other parts of your code when run in your application.
Do yourself a favor and try out unit testing first. I was quite the skeptic myself until I realized just how darned helpful/powerful unit-tests can be. If you think about it, they aren't really there to add to your workload. They are there to provide you with peace of mind and allow you to continue extending your application while ensuring that your code is solid. You get immediate feedback as to when you may have broke something and this is something of extraordinary value.
To your question regarding why to test small sections of code consider this: Suppose your giant app uses a cool XOR encryption scheme that you wrote and eventually product management changes the requirements of how you generate these encrypted strings. So you say: "Heck, I wrote the the encryption routine so I'll go ahead and make the change. It'll take me 15 minutes and we'll all go home and have a party." Well, perhaps you introduced a bug during this process. But wait!!! Your handy dandy TestXOREncryption() test method immediately tells you that the expected output did not match the input. Bingo, this is why you broke down your unit tests ahead of time into small "units" to test for because in your big giant application you would not have figured this out nearly as fast.
Also, once you get into the frame of mind of regularly writing unit tests you'll realize that although you pay an upfront cost in the beginning in terms of time, you'll get that back 10 fold later in the development cycle when you can quickly identify areas in your code that have introduced problems.
There is no magic bullet with unit tests because your ability to identify problems is only as good as the tests you write. It boils down to delivering a better product and relieving yourself of stress and headaches. =)
Agree with most of the answers. Let's drill down on the topic of speed. Here are some real numbers:
Unit test results in 1 or 2 minutes from a
fresh compile. As true unit tests
(no interaction with external
systems like dbs) they can cover a
lot of logic really fast.
Automated functional test results in 1 or 2 hours. These run on a simplified platform, but sometimes cover multiple systems and the database - which really kills the speed.
Automated integration test results once a day. These exercise the full meal deal, but are so heavy and slow, we can only execute them once a day and it takes a few hours.
Manual regression results come in after a few weeks. We get stuff over to testers a few times a day, but your change isn't realistically regressed for week or two at best.
I want to find out what I broke in 1 or 2 minutes, not a few weeks, not even a few hours. That's where the 10fold ROI on unit tests that people talk about comes from.
This is a tough question to approach because it questions something of such enormous breadth. Here's my short answer, however:
Test Driven Development (or TDD) seeks to prove that every logical unit of an application (or block of code) functions exactly as it should. By making tests as automated as possible for productivity's sake, how could this really be harmful?
By testing every logical piece of code, you can trust the usage of the code up some hierarchy. Say I build an application that relies on a thread-safe stack implementation. Shouldn't the stack be guaranteed to work up at every stage before I build on it?
The key is that if something in the whole application breaks, meaning just looking at the total output/outcome, how do you know where it came from? Well, debugging, of course! Which puts you back where you started. TDD allows you to -hopefully- bypass this most painful stage in development.
Testers generally test end to end functionality. Obviously this is geared for going at user scenarios and has incredible value.
Unit Tests serve a different functionality. The are the developers way of verifying the components they write work correctly in the absence of other features or in combination with other features. This offers a range of value including
Provides un-ignorable documentation
Ability to isolate bugs to specific components
Verify invariants in the code
Provide quick, immediate feedback to changes in the code base.
One place to start is regression testing. Once you find a bug, write a small test that demonstrates the bug, fix it, then make sure the test now passes. In future you can run that test before each release to ensure that the bug has not been reintroduced.
Why do that at a unit level instead of a whole-program level? Speed. In good code it's much faster to isolate a small unit and write a tiny test than to drive a complex program through to the bug point. Then when testing a unit test will generally run significantly faster than an integration test.
Very simply: Unit tests are easier to write, since you're only testing a single method's functionality. And bugs are easier to fix, since you know exactly what method is broken.
But like the other answerers have pointed out, unit tests aren't the end-all-be-all of testing. They're just the smallest piece of the equation.
Probably the single biggest difficulty with software is the sheer number of interacting things, and the most useful technique is to reduce the number of things that have to be considered.
For example, using higher-level languages rather than lower-level improves productivity, because one line is a separate thing, and being able to write a program in fewer lines reduces the number of things.
Procedural programming came about as an attempt to reduce complexity by making it possible to treat a function as a thing. In order to do that, though, we have to be able to think about what the function does in a coherent manner, and with confidence that we're right. (Object-oriented programming does a similar thing, on a larger scale.)
There are several ways to do this. Design-by-contract is a way of exactly specifying what the function does. Using function parameters rather than global variables to call the function and get results reduces the complexity of the function.
Unit testing is one way to verify that the function does what it is supposed to. It's usually possible to test all the code in a function, and sometimes all the execution paths. It is a way to tell if the function works as it should or not. If the function works, we can think about it as a single thing, rather than as multiple things we have to keep track of.
It serves other purposes. Unit tests are usually quick to run, and so can catch bugs quickly, when they're easy to fix. If developers make sure a function passes the tests before being checked in, then the tests are a form of documenting what the function does that is guaranteed correct. The act of creating the tests forces the test writer to think about what the function should be doing. After that, whoever wanted the change can look at the tests to see if he or she was properly understood.
By way of contrast, larger tests are not exhaustive, and so can easily miss lots of bugs. They're bad at localizing bugs. They are usually performed at fairly long intervals, so they may detect a bug some time after it's made. They define parts of the total user experience, but provide no basis to reason about any part of the system. They should not be neglected, but they are not a substitute for unit tests.
As others have stated, the length of the feedback loop and isolation of the problem to a specific component are key benefits of Unit Tests.
Another way that they are complementary to functional tests is how coverage is tracked in some organizations:
Unit tests on code coverage
Functional tests on requirements coverage
Functional tests might miss features that were implemented but are not in the spec.
Being based on the code, Unit tests might miss that a certain feature wasn't implemented, which is where requirements based coverage analysis of Functional testing comes in.
A final point : there are some things that are easier/faster to test at the unit level, especially around error scenarios.
Unit testing will help you identify the source of your bug more clearly and let you know that you have a problem earlier. Both are good to have, but they are different, and unit testing does have benefits.
The software you test is a system. When you are testing it as a whole you are black box testing since you primarily deal with inputs and outputs. Black box testing is great when you have no means of getting inside of the system.
But since you usually do, you create a lot of unit tests that actually test your system as a white box. You can slice system open in many ways and organize your tests depending on system internal structure. White box testing provides you with many more ways of testing and analyzing systems. It's clearly complimentary to Black box testing and should not be considered as an alternative or competing methodology.