Related
When integrating submodule-releases into parent projects I regularly encounter bugs which are only visible and triggered when integrated. This is normal. Bugs are normal. We debug them, fix them and submit them into the submodule.
Now it happened several times that such fixes are overwritten by a submodule-developer later on and reoccur in the project which integrated this submodule.
Over time and due to virtually no intelligent tracking of their behavior and their symptoms it happens that it was forgotten and it is fixed again. This is extremely time-consuming especially if it is tricky one.
Hence my question: how can I technically store the 'behavior' of a bug to be "reminded" when I see its symptoms again?
Is there a tool which addresses my problem. Anything I can used to categorize fixed bugs by symptoms?
One idea I had was to extend a static-analyser (such as coverity or clang-analyzer) with custom-patterns. This would not address the behavior/symptoms-approach but I could analyze the code during compilation with patterns created during the first fixing: "if this code is written in a certain way it is not good". In my experience I could address quite an amount of bugs this way, but not all of them.
I added C and C++ tags as these are the languages I'm using.
UPDATE: As there were questions in the comments: We are using git. and the bugs which are re-appearing are committed months or even years after their first correction.
UPDATE: we are using Mantis for bug-tracking.
This is not a tool, but rather a process. I have used BFV (Bug Fix Verification) tests to do this. Everytime you fix a bug, you add an automated BFV test for it. The name of the test would be BFV#bugno - this is the bug number in the bug database. As part of all builds which are given to testing, run all BFV tests as part of regression testing. If BFV2042 fails, you can look up Bug#2042 in your bug database to be reminded about it. If you are running regression tests before accepting merges from submodule branches, then it's even better.
What defect rate can I expect in a C++ codebase that is written for an embedded processor (DSP), given that there have been no unit tests, no code reviews, no static code analysis, and that compiling the project generates about 1500 warnings. Is 5 defects/100 lines of code a reasonable estimate?
Your question is "Is 5 defects/100 lines of code a reasonable estimate?" That question is extremely difficult to answer, and it's highly dependent on the codebase & code complexity.
You also mentioned in a comment "to show the management that there are probably lots of bugs in the codebase" -- that's great, kudos, right on.
In order to open management's figurative eyes, I'd suggest at least a 3-pronged approach:
take specific compiler warnings, and show how some of them can cause undefined / disastrous behavior. Not all warnings will be as weighty. For example, if you have someone using an uninitialized pointer, that's pure gold. If you have someone stuffing an unsigned 16-bit value into an unsigned 8-bit value, and it can be shown that the 16-bit value will always be <= 255, that one isn't gonna help make your case as strongly.
run a static analysis tool. PC-Lint (or Flexelint) is cheap & provides good "bang for the buck". It will almost certainly catch stuff the compiler won't, and it can also run across translation units (lint everything together, even with 2 or more passes) and find more subtle bugs. Again, use some of these as indications.
run a tool that will give other metrics on code complexity, another source of bugs. I'd recommend M Squared's Resource Standard Metrics (RSM) which will give you more information and metrics (including code complexity) than you could hope for. When you tell management that a complexity score over 50 is "basically untestable" and you have a score of 200 in one routine, that should open some eyes.
One other point: I require clean compiles in my groups, and clean Lint output too. Usually this can accomplished solely by writing good code, but occasionally the compiler / lint warnings need to be tweaked to quiet the tool for things that aren't problems (use judiciously).
But the important point I want to make is this: be very careful when going in & fixing compiler & lint warnings. It's an admirable goal, but you can also inadvertantly break working code, and/or uncover undefined behavior that accidentally worked in the "broken" code. Yes, this really does happen. So tread carefully.
Lastly, if you have a solid set of tests already in place, that will help you determine if you accidentally break something while refactoring.
Good luck!
Despite my scepticism of the validity of any estimate in this case, I have found some statistics that may be relevant.
In this article, the author cites figures from a "a large body of empirical studies", published in Software Assessments, Benchmarks, and Best Practices (Jones, 2000). At SIE CMM Level 1, which sounds like the level of this code, one can expect a defect rate of 0.75 per function point. I'll leave it to you to determine how function points and LOC may relate in your code - you'll probably need a metrics tool to perform that analysis.
Steve McConnell in Code Complete cites a study of 11 projects developed by the same team, 5 without code reviews, 6 with code reviews. The defect rate for the non-reviewed code was 4.5 per 100 LOC, and for the reviewed it was 0.82. So on that basis, your estimate seems fair in the absence of any other information. However I have to assume a level of professionalism amongst this team (just from the fact that they felt the need to perform the study), and that they would have at least attended to the warnings; your defect rate could be much higher.
The point about warnings is that some are benign, and some are errors (i.e. will result in undesired behaviour of the software), if you ignore them on the assumption that they are all benign, you will introduce errors. Moreover some will become errors under maintenance when other conditions change, but if you have already chosen to accept a warning, you have no defence against introduction of such errors.
Take a look at the code quality. It would quickly give you a indication of the amount of problems hiding in the source. If the source is ugly and take a long time to understand there will be a lot of bugs in the code.
Well structured code with consistent style and that is easy to understand are going to contain less problems. Code shows how much effort and thought went into it.
My guess is if the source contains that many warnings there is going to be a lot of bugs hiding out in the code.
That also depends on who wrote the code (level of experience), and how big the code base is.
I would treat all warnings as errors.
How many errors do you get when you run a static analysis tool on the code?
EDIT
Run cccc, and check the mccabe's cyclic complexity. It should tell how complex the code it.
http://sourceforge.net/projects/cccc/
Run other static analysis tools.
If you want to get an estimate of the number of defects, the usual way of statistical estimatation is to subsample the data. I would pick three medium-sized subroutines at random, and check them carefully for bugs (eliminate compiler warnings, run static analysis tool, etc). If you find three bugs in 100 total lines of code selected at random, it seems reasonable that a similar density of bugs are in the rest of the code.
The problem mentioned here of introducing new bugs is an important issue, but you don't need to check the modified code back into the production branch to run this test. I would suggest a thorough set of unit tests before modifying any subroutines, and cleaning up all the code followed by very thorough system testing before releasing new code to production.
If you want to demonstrate the benefits of unit tests, code reviews, static analysis tools, I suggest doing a pilot study.
Do some unit tests, code reviews, and run static analysis tools on a portion of the code. Show management how many bugs you find using those methods. Hopefully, the results speak for themselves.
The following article has some numbers based on real-life projects to which static analysis has been applied to: http://www.stsc.hill.af.mil/crosstalk/2003/11/0311German.html
Of course the criteria by which an anomaly is counted can affect the results dramatically, leading to the large variation in the figures shown in Table 1. In this table, the number of anomalies per thousand lines of code for C ranges from 500 (!) to about 10 (auto generated).
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
What do software engineers encounter after another stressfull release? Well, the first thing we encounter in our group are the bugs we have released out in the open. The biggest problem that we as software engineers encounter after a stressfull release is spaghetti-code, also called the big ball of mud.
The time and money to chase perfection are seldom available, nor should they be. To survive, we must do what it takes to get our software working and out the door on time. Indeed, if a team completes a project with time to spare, today’s managers are likely to take that as a sign to provide less time and money or fewer people the next time around.
You need to deliver quality software on time, and under budget
Cost: Architecture is a long-term investment. It is easy for the people who are paying the bills to dismiss it, unless there is some tangible immediate benefit, such a tax write-off, or unless surplus money and time happens to be available. Such is seldom the case. More often, the customer needs something working by tomorrow. Often, the people who control and manage the development process simply do not regard architecture as a pressing concern. If programmers know that workmanship is invisible, and managers don't want to pay for it anyway, a vicious circle is born.
But if this was really the case than each longterm software project would eventually always lead to a big ball of mud.
We know that does not, always, happen. How come? Because the statement that managers do not regard architecture as a pressing concern is false. At least nowadays. Managers in the IT field very well know that maintainability is key to the business.
business becomes dependent upon the data driving it. Businesses have become critically dependent on their software and computing infrastructures. There are numerous mission critical systems that must be on-the-air twenty-four hours a day/seven days per week. If these systems go down, inventories can not be checked, employees can not be paid, aircraft cannot be routed, and so on. [..]
Therefore it is at the heart of the business to seek ways to keep systems far away from the big ball of mud. That the system is still maintainable. That the system actually works and that you, as programmer can prove it does. Does your manager ask you if you have finished your coding today, does she ask you if the release that has fixes A, B and C can be done today or does she ask if the software that will be released actually works? And have you proved it works? With what?
Now for my question:
What ways do we have to prove our managers and/or stakeholders that our software works? Are those green lights of our software-unit tests good enough? If yes, won't that only prove our big-ball of mud is still doing what we expect it to do? That the software is maintainable? How can you prove your design is right?
[Added later]
Chris Pebble his answer below is putting my team on the right track. The Quality Assurance is definitely the thing we are looking for. Thanks Chris. Having a QA policy agreed with the stakeholders is than the logical result of what my team is looking for.
The follow-up question is what should all be in that QA policy?
Having that buildserver running visible for my stakeholders
Having the buildserver not only 'just build' but adding tests that were part of the QA policy
Having an agreement from my stakeholders on our development process (where developers review each others code is part of)
more..
Some more information: The team I'm leading is building webservices that are consumed by other software teams. That is why a breaking webservice is immediately costing money. When the developers of the presentationlayer team, or the actual testers can't move forward we are in immediate stress and have to fix bugs ASAP, which in turn lead to quick hacks..
[ added later ]
Thanks for all the answers. It is indeed about 'trust'. We cannot do a release if the software is not trusted by the stakeholders, who are actively testing our software themselves using the website that is consuming our webservice.
When issues arise, the first question of our testers is: Is it a servicelayer problem or a presentationlayer problem? Which directs me to have a QA policy that ensures that our software is ok for the tests they are doing.
So, the only way I can (now) envision enabling trust with testers is to:
- Talk with the current test-team, go over the tests that they are able to manually execute (from their test-script and scenario's) and make sure that our team has those tests as unit-tests already checked against our webservice. That would be a good starting point for a 'sign-off' before we do a release that the presentationlayerteam has to integrate. It will take some effort to clarify that creating automatic tests for all those scenario's will take some time. But it will definately be usefull to ensure what we build is actually working.
I'm part of a team working on a large project for a governmental client.
The first module of phase 1 was a huge disaster, the team was un-managed, there were'nt a QA team and the developers were not motivated to work better. Instead, the manager kept shouting and deducting wages for people who didnt work overtime!!
The client, huh, dont ask about that, they were really pissed off, but they stayed with our company because they know that no one understand the business as we do.
So what was the solution:
First thing separated the management from programmers, and put a friendly team leader.
Second, get a qualified QA team. In the first few weeks, bugs were in 100s.
Third, put 2-3 developers as support team, there responsibility is not to do any new task, just fix bugs, work directly with the QA.
Fourth, motivate the guys, sometimes its not about the money or extra vacations, sometimes a good word will be perfect. Small example, after working 3 days in row for almost 15 hours a day, the team leader made a note to the manager. Two days later i received a letter from the CEO thanking me on my efforts and giving me 2 vacation days.
We will soon deliver the 4th module of the system, and as one of the support team i could say its at least 95% bug free. Which is a huge jump from our first modules.
Today we have a powerful development team, qualified QA and expert bug fixers.
Sorry for the long story, but thats how our team (during 4 months) proved to the manager and client that we are reliable, and just need the right environment.
You cant prove it beyond the scope of the tests, and unless there is a bulletproof specification (which there never is) then the tests never prove anything beyond the obvious.
What you can do as a team is approach your software design in a responsible manner and not give in to the temptations of writing bad code to please managers, demanding the necessary resources and time constraints, and treating the whole process as much as a craft as a job. The finest renaissance sculptors knew no-one would see the backs statues to be placed in the corners of cathedrals but still took the effort to make sure they weren't selling themselves short.
As a team the only way to prove your software is reliable is to build up a track record: do things correctly from the start, fix bugs before implementing new features, never give in to the quick hack fix, and make sure everyone shares the same enthusiasm and respect for the code.
In all but trivial cases, you cannot 'prove' that your software is correct.
That's the role of User Acceptance Testing: to show that that an acceptable level of usefulness has been reached.
I think this is putting the cart before the horse. It's tantamount to a foot soldier trying to explain to the general what battle maneuvers are and why protecting your flanks is important. If management can't tell the difference between quality code and a big ball of mud, you're always going to end up delivering a big ball of mud.
Unfortunately it is completely impossible to "prove" that your software is working bug-free (the windows xp commercials always annoyed me by announcing "the most secure version of windows ever", that's impossible to prove at release). It's up to management to setup and enforce a QA process and establish metrics as to what a deliverable product actually looks like and what level of bugs or unexpected behaviors is acceptable in the final release.
That said, if you're a small team and set your own QA policies with little input from management I think it would be advantageous to write out a basic QA process and have management sign off on it. For our web apps we currently support 4 browsers -- and management knows this -- so when the application breaks in some obscure handheld browser everyone clearly understands that this is not something we designed the application to support. It also provides good leverage for hiring additional development or test resources when management decides it wants to start testing for x.
As Billy Joel once say, "It's always been a matter of trust."
You need to understand that software development is "black magic" to all except those who write software. It's not obvious (actually, quite counter intuitive) to to the rest of your company that many of your initiatives lead to increasing quality and reducing risk of running over time and/or budget.
The key is to build a trusting, respectful relationship between development and other parts of the business. How do you build this trust? Well that's one of those touchy-feely people problems... you will need to experiment a little. I use the following tools often:
Process visibility - make sure everyone knows what you are doing and how things are progressing. Also, make sure everyone can see the impact of and changes that occur during development.
Point out the little victories - to build up trust, point out when things went exactly as you planned. Try to find situations where you had to make a judgement call and use the term "mitigated the risk" with your management.
Do not say, "I told you so." - let's say that you told management that you needed 2 months to undertake some task and they say, "well you only have three weeks." The outcome is not likely to be good (assuming your estimate is accurate). Make management aware of the problem and document all the things you had to do to try to meet the deadline. When quality is shown to be poor, you can work through the issues you faced (and recorded) in a professional manner rather than pointing the finger and saying, "I told you so."
If you have a good relationship with you manager, you can suggest that they read some books specific to software development so they can understand industry best practices.
Also, I would point out to your boss that not allowing you to act as a professional software developers is hurting your career. You really want to work somewhere that lets you grow professionally rather than somewhere that turns you into a hacker.
In most software projects, defects originate from requirements, design, coding and defect corrections. From my experience the majority of defects originate from the coding phase.
I am interested in finding out what practical approaches software developers use to reduce defect injection rates.
I have seen the following appraoches used with varying levels of success and associated cost
code inspections
unit tests
static code analysis tools
use of programming style
peer programming
In my experience it has been the fault of the process, not developers, that permit defects. See They Write the Right Stuff on how the process affects bugs.
Competitive Testing
Software developers should aspire to prevent testers from finding issues with the software they have written. Testers should be rewarded (does not have to be financial) for finding issues with software.
Sign Off
Put a person in charge of the software who has a vested interest in making sure the software is devoid of issues. The software is not shipped until that person is satisfied.
Requirements
Avoid changing requirements. Get time estimates from developers for how long it will take to implement the requirements. If the time does not match the required delivery schedule, do not hire more developers. Instead, eliminate some features.
Task Switching
Allow developers to complete the task they are working on before assigning them to another. After coming back to a new task, much time is spent getting familiar with where the task was abandoned and what remaining items are required to complete the it. Along the way, certain technical details can be missed.
Metrics
Gather as many possible metrics you can. Lines of code per method, per class, dependency relationships, and others.
Standards
Ensure everyone is adhering to corporate standards, including:
Source code formatting. This can be automated, and is not a discussion.
Naming conventions (variables, database entities, URLs, and such). Use tools when possible, and weekly code reviews to enforce.
Code must compile without warnings. Note and review all exceptions.
Consistent (re)use of APIs, both internally and externally developed.
Independent Review
Hire a third-party to perform code reviews.
Competent Programmers
Hire the best programmers you can afford. Let go of the programmers who shirk corporate standards.
Disseminate Information
Hold review sessions where developers can share (with the entire team) their latest changes to the framework(s). Allow them freedom to deprecate old portions of the code in favour of superior methods.
Task Tracking
Have developers log how long (within brackets of 15 minutes) each task has taken them. This is not to be used to measure performance, and must be stressed that it has no relation to review or salary. It is simply a measure of how long it takes certain technical tasks to be implemented. From there you can see, generally, how much time is being spent on different aspects of the system. This will allow you to change focus, if necessary.
Evaluate the Process
If many issues are still finding their way into the software, consider reevaluating the process with which the software is being developed. Metrics will help pinpoint the areas that need to be addressed.
First, bugs injected at requirements time are far, far more costly than coding bugs. A zero-value requirement, correctly implemented is a piece of zero-value, unused (or unusable) functionality.
Two things reduce the incidence of bugs
Agility. You are less likely to inject bugs at every step (requirements, design, etc.) if you aren't doing as much in each step. If you try to write all the requirements, you will make terrible mistakes. If you try to write requirements for the next sprint, odds are better that you will get those few requirements correct.
TDD. You are less likely to struggle with bad requirements or bad design if you have to write a test first. If you can't figure out what you're testing, you have a requirements bug. Stop coding. Step away from the keyboard.
If you can't figure out how to test it, you have a design bug. Again, stop coding. Fix the design so it's testable. Then move forward.
I think the main problem of injection rates can become from a lot of sources, and it vary from environment to environment.
You can use a lot of best practices like TDD, DDD, pair programming, continuous integration, etc. But you will never be free from bugs, because what creates bugs are human people, and not exactly the processes.
But IMO, using a bug tracker tool could bring you hints of which problem is more recurrent. From there, you can start attacking your main problem.
The majority of defects may occur during coding, but the impact of coding defects is generally much lower than the impact of errors made during the process of understanding requirements and while developing a resilient architecture. Thus the use of short executable-producing iterations focused on
identifying and correcting ambiguous, imprecise, or just plain incorrect requirements
exposing a suboptimal and/or brittle architecture
can save enormous amounts of time and collective stomach lining in a project of significant scope.
Unit testing, scenario testing, and static analysis tools can detect defects after they are created, but to reduce the number of defects created in the first place, reduce the number of interruptions that developers must endure:
reduce, eliminate, and/or consolidate meetings
provide an interrupt-free working environment
allow developers to correct their defects when they find them (while the responsible code is still fresh in their mind) rather than defer them to a later time when context must be re-established
Step 1 - Understand where your defects are being injected.
Use a technique such as Orthogonal Defect Classification (ODC) to measure when in the software lifecycle defects are injected and when they are detected. Once you know when the defects are injected and have identified when they were discovered you can start to understand the gaps in your process with respect to defect injection and removal.
Step 2 - Develop defect "filters" and adapt your process
Once you know when defects are being injected you can devise strategies to prevent them from entering the system. Different strategies are effective at different points in the software lifecycle. For example, static analysis tools don't help with defects that originated in the requirements, instead you should be looking into some kind of peer review or inspection, maybe even changing the way requirements are specified so you use automated analysis or achieve a more meaning sign-off, etc.
Generally I use a combination of inspection, static analysis, and testing (many different kinds) to filter as many bugs as I can, as soon after they are injected as I am able.
In addition:
Project knowledge base. It says how we do activity X (like 'form validation') in this project. This allows unification and re-use of tested solution, preventing bugs injected when re-inventing-the-wheel.
Production bug monitoring. When a production bug occurs it is investigated. Why this bug was not caught? How we can ensure that this won't happen again? Then we change the process accordingly.
Our management has recently been talking to some people selling C++ static analysis tools. Of course the sales people say they will find tons of bugs, but I'm skeptical.
How do such tools work in the real world? Do they find real bugs? Do they help more junior programmers learn?
Are they worth the trouble?
Static code analysis is almost always worth it. The issue with an existing code base is that it will probably report far too many errors to make it useful out of the box.
I once worked on a project that had 100,000+ warnings from the compiler... no point in running Lint tools on that code base.
Using Lint tools "right" means buying into a better process (which is a good thing). One of the best jobs I had was working at a research lab where we were not allowed to check in code with warnings.
So, yes the tools are worth it... in the long term. In the short term turn your compiler warnings up to the max and see what it reports. If the code is "clean" then the time to look at lint tools is now. If the code has many warnings... prioritize and fix them. Once the code has none (or at least very few) warnings then look at Lint tools.
So, Lint tools are not going to help a poor code base, but once you have a good codebase it can help you keep it good.
Edit:
In the case of the 100,000+ warning product, it was broken down into about 60 Visual Studio projects. As each project had all of the warnings removed it was changed so that the warnings were errors, that prevented new warnings from being added to projects that had been cleaned up (or rather it let my co-worker righteously yell at any developer that checked in code without compiling it first :-)
In my experience with a couple of employers, Coverity Prevent for C/C++ was decidedly worth it, finding some bugs even in good developers’ code, and a lot of bugs in the worst developers’ code. Others have already covered technical aspects, so I’ll focus on the political difficulties.
First, the developers whose code need static analysis the most, are the least likely to use it voluntarily. So I’m afraid you’ll need strong management backing, in practice as well as in theory; otherwise it might end up as just a checklist item, to produce impressive metrics without actually getting bugs fixed. Any static analysis tool is going to produce false positives; you’re probably going to need to dedicate somebody to minimizing the annoyance from them, e.g., by triaging defects, prioritizing the checkers, and tweaking the settings. (A commercial tool should be extremely good at never showing a false positive more than once; that alone may be worth the price.) Even the genuine defects are likely to generate annoyance; my advice on this is not to worry about, e.g., check-in comments grumbling that obviously destructive bugs are “minor.”
My biggest piece of advice is a corollary to my first law, above: Take the cheap shots first, and look at the painfully obvious bugs from your worst developers. Some of these might even have been found by compiler warnings, but a lot of bugs can slip through those cracks, e.g., when they’re suppressed by command-line options. Really blatant bugs can be politically useful, e.g., with a Top Ten List of the funniest defects, which can concentrate minds wonderfully, if used carefully.
As a couple people remarked, if you run a static analysis tool full bore on most applications, you will get a lot of warnings, some of them may be false positives or may not lead to an exploitable defect. It is that experience that leads to a perception that these types of tools are noisy and perhaps a waste of time. However, there are warnings that will highlight a real and potentially dangerous defects that can lead to security, reliability, or correctness issues and for many teams, those issues are important to fix and may be nearly impossible to discover via testing.
That said, static analysis tools can be profoundly helpful, but applying them to an existing codebase requires a little strategy. Here are a couple of tips that might help you..
1) Don't turn everything on at once, decide on an initial set of defects, turn those analyses on and fix them across your code base.
2) When you are addressing a class of defects, help your entire development team to understand what the defect is, why it's important and how to code to defend against that defect.
3) Work to clear the codebase completely of that class of defects.
4) Once this class of issues have been fixed, introduce a mechanism to stay in that zero issue state. Luckily, it is much easier make sure you are not re-introducing an error if you are at a baseline has no errors.
It does help. I'd suggest taking a trial version and running it through a part of your codebase which you think is neglected. These tools generate a lot of false positives. Once you've waded through these, you're likely to find a buffer overrun or two that can save a lot of grief in near future. Also, try at least two/three varieties (and also some of the OpenSource stuff).
I've used them - PC-Lint, for example, and they did find some things. Typically they are configurable and you can tell them 'stop bothering me about xyz', if you determine that xyz really isn't an issue.
I don't know that they help junior programmers learn a lot, but they can be used as a mechanism to help tighten up the code.
I've found that a second set of (skeptical, probing for bugs) eyes and unit testing is typically where I've seen more bug catching take place.
Those tools do help. lint has been a great tool for C developers.
But one objection that I have is that they're batch processes that run after you've written a fair amount of code and potentially generate a lot of messages.
I think a better approach is to build such a thing into your IDE and have it point out the problem while you're writing it so you can correct it right away. Don't let those problems get into the code base in the first place.
That's the difference between the FindBugs static analysis tool for Java and IntelliJ's Inspector. I greatly prefer the latter.
You are probably going to have to deal with a good amount of false positives, particularly if your code base is large.
Most static analysis tools work using "intra-procedural analysis", which means that they consider each procedure in isolation, as opposed to "whole-program analysis" which considers the entire program.
They typically use "intra-procedural" analysis because "whole-program analysis" has to consider many paths through a program that won't actually ever happen in practice, and thus can often generate false positive results.
Intra-procedural analysis eliminates those problems by just focusing on a single procedure. In order to work, however, they usually need to introduce an "annotation language" that you use to describe meta-data for procedure arguments, return types, and object fields. For C++ those things are usually implemented via macros that you decorate things with. The annotations then describe things like "this field is never null", "this string buffer is guarded by this integer value", "this field can only be accessed by the thread labeled 'background'", etc.
The analysis tool will then take the annotations you supply and verify that the code you wrote actually conforms to the annotations. For example, if you could potentially pass a null off to something that is marked as not null, it will flag an error.
In the absence of annotations, the tool needs to assume the worst, and so will report a lot of errors that aren't really errors.
Since it appears you are not using such a tool already, you should assume you are going to have to spend a considerably amount of time annotating your code to get rid of all the false positives that will initially be reported. I would run the tool initially, and count the number of errors. That should give you an estimate of how much time you will need to adopt it in your code base.
Wether or not the tool is worth it depends on your organization. What are the kinds of bugs you are bit by the most? Are they buffer overrun bugs? Are they null-dereference or memory-leak bugs? Are they threading issues? Are they "oops we didn't consider that scenario", or "we didn't test a Chineese version of our product running on a Lithuanian version of Windows 98?".
Once you figure out what the issues are, then you should know if it's worth the effort.
The tool will probably help with buffer overflow, null dereference, and memory leak bugs. There's a chance that it may help with threading bugs if it has support for "thread coloring", "effects", or "permissions" analysis. However, those types of analysis are pretty cutting-edge, and have HUGE notational burdens, so they do come with some expense. The tool probably won't help with any other type of bugs.
So, it really depends on what kind of software you write, and what kind of bugs you run into most frequently.
I think static code analysis is well worth, if you are using the right tool. Recently, we tried the Coverity Tool ( bit expensive). Its awesome, it brought out many critical defects,which were not detected by lint or purify.
Also we found that, we could have avoided 35% of the customer Field defects, if we had used coverity earlier.
Now, Coverity is rolled out in my company and when ever we get a customer TR in old software version, we are running coverity against it to bring out the possible canditates for the fault before we start the analysis in a susbsytem.
Paying for most static analysis tools is probably unnecessary when there's some very good-quality free ones (unless you need some very special or specific feature provided by a commercial version). For example, see this answer I gave on another question about cppcheck.
I guess it depends quite a bit on your programming style. If you are mostly writing C code (with the occasional C++ feature) then these tools will likely be able to help (e.g. memory management, buffer overruns, ...). But if you are using more sophisticated C++ features, then the tools might get confused when trying to parse your source code (or just won't find many issues because C++ facilities are usually safer to use).
As with everything the answer depends ... if you are the sole developer working on a knitting-pattern-pretty-printer for you grandma you'll probably do not want to buy any static analysis tools. If you are having a medium sized project for software that will go into something important and maybe on top of that you have a tight schedule, you might want to invest a little bit now that saves you much more later on.
I recently wrote a general rant on this: http://www.redlizards.com/blog/?p=29
I should write part 2 as soon as time permits, but in general do some rough calculations whether it is worth it for you:
how much time spent on debugging?
how many resources bound?
what percentage could have been found by static analysis?
costs for tool setup?
purchase price?
peace of mind? :-)
My personal take is also:
get static analysis in early
early in the project
early in the development cycle
early as in really early (before nightly build and subsequent testing)
provide the developer with the ability to use static analysis himself
nobody likes to be told by test engineers or some anonymous tool
what they did wrong yesterday
less debugging makes a developer happy :-)
provides a good way of learning about (subtle) pitfalls without embarrassment
This rather amazing result was accomplished using Elsa and Oink.
http://www.cs.berkeley.edu/~daw/papers/fmtstr-plas07.pdf
"Large-Scale Analysis of Format String Vulnerabilities in Debian Linux"
by Karl Chen, David Wagner,
UC Berkeley,
{quarl, daw}#cs.berkeley.edu
Abstract:
Format-string bugs are a relatively common security vulnerability, and can lead to arbitrary code execution. In collaboration with others, we designed and implemented a system to eliminate format string vulnerabilities from an entire Linux distribution, using typequalifier inference, a static analysis technique that can find taint violations. We successfully analyze 66% of C/C++ source packages in the Debian 3.1 Linux distribution. Our system finds 1,533 format string taint warnings. We estimate that 85% of these are true positives, i.e., real bugs; ignoring duplicates from libraries, about 75% are real bugs. We suggest that the technology exists to render format string vulnerabilities extinct in the near future.
Categories and Subject Descriptors D.4.6 [Operating Systems]: Security and Protection—Invasive Software;
General Terms: Security, Languages;
Keywords: Format string vulnerability, Large-scale analysis, Typequalifier inference
Static analysis that finds real bugs is worth it regardless of whether it's C++ or not. Some tend to be quite noisy, but if they can catch subtle bugs like signed/unsigned comparisons causing optimizations that break your code or out of bounds array accesses, they are definitely worth the effort.
At a former employer we had Insure++.
It helped to pinpoint random behaviour (use of uninitialized stuff) which Valgrind could not find. But most important: it helpd to remove mistakes which were not known as errors yet.
Insure++ is good, but pricey, that's why we bought one user license only.