Tools for Isolating a Stack smashing bug - c++

To put it mildly I have a small memory issue and am running out of tools and ideas to isolate the cause.
I have a highly multi-threaded (pthreads) C/C++ program that has developed a stack smashing issue under optimized compiles with GCC after 4.4.4 and prior to 4.7.1.
The symptom is that during the creation of one of the threads, I get a full stack smash, not just %RIP, but all parent frames and most of the registers are 0x00 or other non-sense address.
Which thread causes the issue is seemingly random, however judging by log messages it seems to be isolated to the same Hunk of code, and seems to come at a semi repeatable point in the creation of the new thread.
This has made it very hard to trap and isolate the offending code more narrowly than to a single compilation unit of may thousand lines, since print()'s with in the offending file have so far proved unreliable in trying to narrow down the active section.
The thread creation that leads off the thread that eventually smashes the stack is:
extern "C"
{
static ThreadReturnVal ThreadAPI WriterThread(void *act)
{
Recorder *rec = reinterpret_cast (act);
xuint64 writebytes;
LoggerHandle m_logger = XXGetLogger("WriterThread");
if (SetThreadAffinity(rec->m_cpu_mask))
{ ... }
SetThreadPrio((xint32)rec->m_thread_priority);
while (true)
{
... poll a ring buffer ... Hard Spin 100% use on a single core, this is that sort of crazy code.
}
}
I have tried a debug build, but the symptom is only present in optimized builds, -O2 or better.
I have tried Valgrind/memcheck and DRD but both fail to find any issue before the stack is blown away ( and takes about 12hr's to reach the failure )
A compile with -O2 -Wstack-protector sees nothing wrong,
however a build with -fstack-protector-all does protect me from the bug, but emits no errors.
Electric-Fence also traps, but only after the stack is gone.
Question: What other tools or techniques would be useful in narrowing down the offending section ?
Many thanks,
--Bill

A couple of options for approaching this sort of problem:
You could try setting a hardware breakpoint on a stack address before the corruption occurs and hope the debugger breaks early enough in the corruption to provide a vaguely useful debugging state. The tricky part here is choosing the right stack address; depending on how random the 'choice' of offending thread is, this might not be practical. But from one of your comments it sounds like it is often the newly created thread that gets smashed, so this might be doable. Try to break during thread creation, grab the thread's stack location, offset by some wild guess, set the hardware BP, and continue. Based on whether you break too early, too late, or not at all, adjust your offset, rinse, and repeat. This is basically advanced guess and check, and can be heavily hindered or outright unpractical if the corruption pattern is too random, but it is surprising how often this can lead to a semi-legible stack and successful debugging efforts.
Another option would be to start collecting crash dumps. Try to look for patterns between the crash dumps that might help bring you closer to the source of the corruption. Perhaps you'll get lucky and one of the crash dumps will crash 'faster'/'closer to the source'.
Unfortunately, both of these techniques are more art that science; they're non-deterministic, rely on a healthy dose of luck, etc. (at least in my experience.. that being said, there are people out there who can do amazing things with crash dumps, but it takes a lot of time to get to that level of skill).
One more side note: as others have pointed out, uninitialized memory is a very typical source of debug vs release differences, and could easily be your problem here. However, another possibility to keep in mind is timing differences. The order that threads get scheduled in, and for how long, is often dramatically different in debug vs release, and can easily lead to synchronization bugs being masked in one but not the other. These differences can be just due to execution speed differences, but I think some runtimes intentionally mess with thread scheduling in a debug environment.

You can use a static analysis tool to check for some sutble errors, maybe one of the found errors will be the cause of your bug. You can find some information on these tools here.

Related

MSVC Address Sanitizer - Any reason to use it in Release builds?

Microsoft recently brought Address Sanitizer (ASan) to Microsoft Visual Studio 2019, and I've been experimenting with it. This question is specific to C and C++. My question is this: Is there any reason to have ASan enabled for Release builds, as opposed to having it enabled only for Debug builds? Having ASan turned on is devastating to the performance and memory usage of my program. (CPU performance worse than halved, memory usage tripled.) Therefore my hope is that if I enable ASan just to check for potential issues, and it does not detect any problems in a Debug build, then I can safely assume that there would not be any problems in the Release build that would have otherwise been caught by ASan? Certainly we are not meant to leave ASan enabled in Release/Production builds?
Thanks for any insight.
Therefore my hope is that if I enable ASan just to check for potential issues, and it does not detect any problems in a Debug build, then I can safely assume that there would not be any problems in the Release build that would have otherwise been caught by ASan?
That's not likely (especially for larger complex programs). Consider a function like:
int array[250];
int test1(uint16_t a) {
return array[ (1016 / (a+1)) & 0xFF];
}
You can test it with lots of values of a (e.g. 0, 2, 3, 4, ...) and it will be fine, and it will pass the address sanitizer's test during debug. Then, after release, it might crash due to a different (untested) value of a (e.g. 1) that triggers the "array index out of range" bug.
To guarantee that something like that can't happen you could test every possible combination of parameters for every function (which would be ludicrously expensive); but (in the presence of global variables/state) that might still not be enough, and (in the presence of multiple threads) it won't find hidden race conditions (e.g. code that almost always works because threadA finishes before threadB, that crashes when subtle differences in timing cause threadB to finish before threadA).
Certainly we are not meant to leave ASan enabled in Release/Production builds?
Correct - it's too expensive to leave enabled.
The general idea is to minimize the risk of bugs (because you can't guarantee that all bugs were found and eliminated). Without much testing you might be able to say that you're "80% sure" there's no bugs in the released version; and with some rigorous testing with Asan (and optimizations) enabled you might be able to say that you're "95% sure" there's no bugs in the released version.
What this really comes down to is economics (money). It's a "cost of software bugs (lost sales, support costs, etc)" vs. "cost of finding bugs" compromise; where something like a computer game might be fine with less testing (because users just assume that games crash occasionally anyway) and something like a life support machine (where bugs cause deaths) you might refuse to use C or C++ in the first place.
The other thing worth considering is that users don't care much why your code failed - if it failed because of a bug then the user will be annoyed; and if it failed because Asan was enabled and detected a bug then the user will be equally annoyed. Leaving Asan enabled in release builds would help the user submit better information in a bug report, but most people don't submit bug reports (they just sit there annoyed; wondering if it was a temporary hardware glitch because they're too cheap to buy ECC RAM, or if it was a bug in the OS or a driver; and then restart the software).
Hard facts up front: A program's shape - broadly speaking - falls into one of two categories.
Its logical representation expressed as its source code.
Its binary representation, translated by a compiler (and linker) into something a machine can execute.
1. is what the programmer intended the program to do, and 2. is what a program translated that intent into.
Ideally, 1. is perfectly within a programming language's specification, and either representation has identical observable behavior. In reality, 1. is wrong, and 2. does whatever.
The exercise is thus: Evaluate how badly 2. is broken. To do that, you absolutely want a 2. that's as close to what your clients will run as ever possible. Fuzzing a 2. compiled as a Release configuration with ASan enabled gets you there. If you don't feel like reading along, fuzz this target as hard as humanly imaginable, and go your merry way.
That covers the unquestionable facts. What follows is strong opinions on what you (we, us) should really be doing. Stop reading, if you don't like opinions, strong opinions, or ethics.
<Ramblings, not quite done rambling, yet>

Software crash after steep rise of process' working set memory

I ask this question, because we're really stuck at finding the cause of a software crash. I know that questions like "Why does the software crash" are not appreciated, but we really don't know how to find the problem.
We currently do a longterm test of our software. To find potential memory leaks, we used the windows tool Performance monitor to track several memory metrics, such as Private bytes, Working set and Virtual bytes.
The software ran quite a long time (about 30 hours) without any problems. It does the same all the time, reading in an image from the harddrive, doing some inspection and showing some results.
Then suddenly it crashes. Inspecting the memory metrics in the performance monitor, we saw that strange steep rising of the working set bytes graph at 10.17AM. We encountered this several times and according to the dumpfiles, the exception code is always 0xc0000005 : "the thread tried to read from or write to a virtual address for which it does not have the appropriate access", but it appears at different positions, where no pointers are used.
Does someone know, what could be the cause of such a steep rise of the working set and why this could cause a software crash? How could we find out, if our software has a bug, when every time, the crash occurs the position of the crash is at another position?
The application is written in C++ and it runs on a windows 7 32bit pc.
It's actually impossible to know from the information that you have provided, but I would suggest that you have some memory corruption (hence the access violation). It could be a buffer-overflow issue... for example there is a missing null character from a string and so something is being appended indefinitely?
Recommended next step is to download the Debugging Tools for Windows suite. Setup WinDbg with your correct symbol files, and analyse the stack trace, to find the general area of the crash. Depending on the cause of the memory corruption this will be more or less useful. You could have corrupted the memory a long time before your crash occurs.
Ideally also run a static analysis tool on the code.
Given information you have now, there is little chance to get an answer. You need more information, more specifically:
Get more intelligence (is there anything specific about that files which cause crash? What about last-but-one file?)
Insert more tracing and logging (as much as you can without making it 2x slower). At least you'll see where it crashes, and then will be able to insert more tracing/logging around that place
As you're on Windows - consider handling c0000005 via _set_se_translator, converting it into C++ exception, and even more logging on the way this exception is unwinded.
There is no silver bullet for this kind of problems, only gathering more information and figuring it out.
P.S. As an unlikely shot - I've seen similar things to be caused by a bug in MS heap; if you're not using LFH yet (not sure, it might be default now) - there is an 1% chance changing your default heap to LFH will help.

How can I find a rare bug that seems to only occur in release builds?

I have a fairly large solution that occasionally crashes. Sadly, these crashes appear to only occur in release build. When I attach the debugger upon crashing, I get the message:
"No symbols are loaded for any call
stack frame. The source code cannot be
displayed"
This makes it quite hard to find the cause of the crashes. I am using the default release build settings of visual studio 2008, in which 'debug information format' is set to 'Program Database (/Zi)'.
Do you have any tips that might help me find the bug? For example, could I change some settings in my projects so that the crashes might still occur but get more meaningful information in the debugger?
Update: The problem was a very rarely occurring logic error that in itself should not cause a crash, but apparently caused a crash elsewhere. Solving the logic error solved the crashing behavior.
To anyone that came here looking for a resolution of a similar problem: best of luck, you're in for a rough ride. What eventually helped me locate the problem was adding a lot of bounds checks in the code (that I could enable/disable with preprocessor directives) and compiling for linux and running with gdb/valgrind.
First ensure you are building symbols (debug info) for release build, and that the debugger can find them (this may require configuring the symbol path—a symbol server would be better).
Second use the Modules view while debugging to confirm you have the symbols loaded.
The simplest way to get the symbols is to put the .pdb files in the same location as the assemblies.
Check out John Robbins blog for many many more details of doing this.
If the code crashes after optimisation is applied (as in the default release), it is most likely that your code is in some way flawed and relies on undefined behaviour which changes between the release and debug build.
Try switching off optimisation in the release build to see if the problem goes away (or switch it on in the debug build to see if it occurs). If it does, you should still aim to find and fix the bug, but you will at least know to be looking for undefined behaviour.
Set the compiler warning level to maximum (/W4) and warnings as errors (/Wx) and fix all warnings (and not simply by casting everything in sight - think about it!). When optimisation is applied, you may well get warnings that did not occur in the debug build because of the more extensive code analysis that is performed - this is useful static analysis.
You can if you wish switch debugging on in an optimised build, but it is unlikely that you will be able to follow what is going on since the optimiser may re-order code, and remove code and variables.
Sounds to me like that stack frame was blown. Trivial to do with a buffer overflow, just copy a large string in a small char[] for example. That wipes out the return address. The code just keeps running until the return, then bombs when it pops a bad address off the stack. Or worse, if the address happens to be valid.
The debugger cannot display anything meaningful since it cannot walk the stack to show you how the code got to the crash location. The actual crash location doesn't tell you anything.
Tuff as nails to debug. You have to get it reproducible and you need either stepping or tracing to find the last known-good function. The one that produces the crash after stepping out of it is the one with the bug. You can actually see the statement that does the damage, the debugger call stack suddenly goes catatonic. If you can't get a consistent repro then a thorough code review is all that's left. You can justify the time by calling it a "security review". Good luck with it.
A few reasons why a debug build might not allow a defect to express itself:
Some debug configurations initialize all variables.
Debug memory allocations and deallocations might be more forgiving of pointer abuse.
The debug build might execute at a different speed thus masking a race condition.
Since you are using C++ you might consider using a static analysis tool like valgrind to point out possible uninitialized data and pointer mishandling.
Race conditions may be tracked down by adding log output with time stamps. You first have to narrow down where in your "large solution" the problem occurs by observing what happened just prior to the crash. Be sure to use a deferred logging mechanism -- one that does the string processing later or in another thread so it doesn't itself impact the timing too much.
Did you know you can still debug release builds? Just hit F5 (rather than CTRL+F5) to run in debug.
Is it repeatable i.e. are you doing something specific when it goes bang?
If so, set a breakpoint in your code before the crash and hit F5 to run in debug (make sure you're using your release build though). Then step through until your app crashes. I generally find this faster than adding logging and debug print statements.
If not, just running in debug mode will sometimes catch the error and halt on the offending line.
Failing that, Richard and Amar's answers are good :-)
An uninitalized variable (pointer perhaps) could also be causing the problem. Perhaps you should run a static analysis program over your code - CppCheck isn't bad.

C++: Where to start when my application crashes at random places?

I'm developing a game and when I do a specific action in the game, it crashes.
So I went debugging and I saw my application crashed at simple C++ statements like if, return, ... Each time when I re-run, it crashes randomly at one of 3 lines and it never succeeds.
line 1:
if (dynamic) { ... } // dynamic is a bool member of my class
line 2:
return m_Fixture; // a line of the Box2D physical engine. m_Fixture is a pointer.
line 3:
return m_Density; // The body of a simple getter for an integer.
I get no errors from the app nor the OS...
Are there hints, tips or tricks to debug more efficient and get known what is going on?
That's why I love Java...
Thanks
Random crashes like this are usually caused by stack corruption, since these are branching instructions and thus are sensitive to the condition of the stack. These are somewhat hard to track down, but you should run valgrind and examine the call stack on each crash to try and identify common functions that might be the root cause of the error.
Are there hints, tips or tricks to debug more efficient and get known what is going on?
Run game in debugger, on the point of crash, check values of all arguments. Either using visual studio watch window or using gdb. Using "call stack" check parent routines, try to think what could go wrong.
In suspicious(potentially related to crash) routines, consider dumping all arguments to stderr (if you're using libsdl or on *nixlike systems), or write a logfile, or send dupilcates of all error messages using (on Windows) OutputDebugString. This will make them visible in "output" window in visual studio or debugger. You can also write "traces" (log("function %s was called", __FUNCTION__))
If you can't debug immediately, produce core dumps on crash. On windows it can be done using MiniDumpWriteDump, on linux it is set somewhere in configuration variables. core dumps can be handled by debugger. I'm not sure if VS express can deal with them on Windows, but you still can debug them using WinDBG.
if crash happens within class, check *this argument. It could be invalid or zero.
If the bug is truly evil (elusive stack corruption in multithreaded app that leads to delayed crash), write custom memory manager, that will override new/delete, provide alternative to malloc(if your app for some reason uses it, which may be possible), AND that locks all unused memory memory using VirtualProtect (windows) or OS-specific alternative. In this case all potentially dangerous operation will crash app instantly, which will allow you to debug the problem (if you have Just-In-Time debugger) and instantly find dangerous routine. I prefer such "custom memory manager" to boundschecker and such - since in my experience it was more useful. As an alternative you could try to use valgrind, which is available on linux only. Note, that if your app very frequently allocates memory, you'll need a large amount of RAM in order to be able to lock every unused memory block (because in order to be locked, block should be PAGE_SIZE bytes big).
In areas where you need sanity check either use ASSERT, or (IMO better solution) write a routine that will crash the application (by throwing an std::exception with a meaningful message) if some condition isn't met.
If you've identified a problematic routine, walk through it using debugger's step into/step over. Watch the arguments.
If you've identified a problematic routine, but can't directly debug it for whatever reason, after every statement within that routine, dump all variables into stderr or logfile (fprintf or iostreams - your choice). Then analyze outputs and think how it could have happened. Make sure to flush logfile after every write, or you might miss the data right before the crash.
In general you should be happy that app crashes somewhere. Crash means a bug you can quickly find using debugger and exterminate. Bugs that don't crash the program are much more difficult (example of truly complex bug: given 100000 values of input, after few hundreds of manipulations with values, among thousands of outputs, app produces 1 absolutely incorrect result, which shouldn't have happened at all)
That's why I love Java...
Excuse me, if you can't deal with language, it is entirely your fault. If you can't handle the tool, either pick another one or improve your skill. It is possible to make game in java, by the way.
These are mostly due to stack corruption, but heap corruption can also affect programs in this way.
stack corruption occurs most of the time because of "off by one errors".
heap corruption occurs because of new/delete not being handled carefully, like double delete.
Basically what happens is that the overflow/corruption overwrites an important instruction, then much much later on, when you try to execute the instruction, it will crash.
I generally like to take a second to step back and think through the code, trying to catch any logic errors.
You might try commenting out different parts of the code and seeing if it affects how the program is compiled.
Besides those two things you could try using a debugger like Visual Studio or Eclipse etc...
Lastly you could try to post your code and the error you are getting on a website with a community that knows programming and could help you work through the error (read: stackoverflow)
Crashes / Seg faults usually happen when you access a memory location that it is not allowed to access, or you attempt to access a memory location in a way that is not allowed (for example, attempting to write to a read-only location).
There are many memory analyzer tools, for example I use Valgrind which is really great in telling what the issue is (not only the line number, but also what's causing the crash).
There are no simple C++ statements. An if is only as simple as the condition you evaluate. A return is only as simple as the expression you return.
You should use a debugger and/or post some of the crashing code. Can't be of much use with "my app crashed" as information.
I had problems like this before. I was trying to refresh the GUI from different threads.
If the if statements involve dereferencing pointers, you're almost certainly corrupting the stack (this explains why an innocent return 0 would crash...)
This can happen, for instance, by going out of bounds in an array (you should be using std::vector!), trying to strcpy a char[]-based string missing the ending '\0' (you should be using std::string!), passing a bad size to memcpy (you should be using copy-constructors!), etc.
Try to figure out a way to reproduce it reliably, then place a watch on the corrupted pointer. Run through the code line-by-line until you find the very line that corrupts the pointer.
Look at the disassembly. Almost any C/C++ debugger will be happy to show you the machine code and the registers where the program crashed. The registers include the Instruction Pointer (EIP or RIP on x86/x64) which is where the program was when it stopped. The other registers usually have memory addresses or data. If the memory address is 0 or a bad pointer, there is your problem.
Then you just have to work backward to find out how it got that way. Hardware breakpoints on memory changes are very helpful here.
On a Linux/BSD/Mac, using GDB's scripting features can help a lot here. You can script things so that after the breakpoint is hit 20 times it enables a hardware watch on the address of array element 17. Etc.
You can also write debugging into your program. Use the assert() function. Everywhere!
Use assert to check the arguments to every function. Use assert to check the state of every object before you exit the function. In a game, assert that the player is on the map, that the player has health between 0 and 100, assert everything that you can think of. For complicated objects write verify() or validate() functions into the object itself that checks everything about it and then call those from an assert().
Another way to write in debugging is to have the program use signal() in Linux or asm int 3 in Windows to break into the debugger from the program. Then you can write temporary code into the program to check if it is on iteration 1117321 of the main loop. That can be useful if the bug always happens at 1117322. The program will execute much faster this way than to use a debugger breakpoint.
some tips :
- run your application under a debugger, with the symbol files (PDB) together.
- How to set Visual Studio as the default post-mortem debugger?
- set default debugger for WinDbg Just-in-time Debugging
- check memory allocations Overriding new and delete, and Overriding malloc and free
One other trick: turn off code optimization and see if the crash points make more sense. Optimization is allowed to float little bits of your code to surprising places; mapping that back to source code lines can be less than perfect.
Check pointers. At a guess, you're dereferencing a null pointer.
I've found 'random' crashes when there are some reference to a deleted object. As the memory is not necessarily overwritten, in many cases you don't notice it and the program works correctly, and than crashes after the memory was updated and is not valid anymore.
JUST FOR DEBUGGING PURPOSES, try commenting out some suspicious 'deletes'. Then, if it doesn't crash anymore, there you are.
use the GNU Debugger
Refactoring.
Scan all the code, make it clearer if not clear at first read, try to understand what you wrote and immediately fix what seems incorrect.
You'll certainly discover the problem(s) this way and fix a lot of other problems too.

Heap corruption under Win32; how to locate?

I'm working on a multithreaded C++ application that is corrupting the heap. The usual tools to locate this corruption seem to be inapplicable. Old builds (18 months old) of the source code exhibit the same behaviour as the most recent release, so this has been around for a long time and just wasn't noticed; on the downside, source deltas can't be used to identify when the bug was introduced - there are a lot of code changes in the repository.
The prompt for crashing behaviuor is to generate throughput in this system - socket transfer of data which is munged into an internal representation. I have a set of test data that will periodically cause the app to exception (various places, various causes - including heap alloc failing, thus: heap corruption).
The behaviour seems related to CPU power or memory bandwidth; the more of each the machine has, the easier it is to crash. Disabling a hyper-threading core or a dual-core core reduces the rate of (but does not eliminate) corruption. This suggests a timing related issue.
Now here's the rub:
When it's run under a lightweight debug environment (say Visual Studio 98 / AKA MSVC6) the heap corruption is reasonably easy to reproduce - ten or fifteen minutes pass before something fails horrendously and exceptions, like an alloc; when running under a sophisticated debug environment (Rational Purify, VS2008/MSVC9 or even Microsoft Application Verifier) the system becomes memory-speed bound and doesn't crash (Memory-bound: CPU is not getting above 50%, disk light is not on, the program's going as fast it can, box consuming 1.3G of 2G of RAM). So, I've got a choice between being able to reproduce the problem (but not identify the cause) or being able to idenify the cause or a problem I can't reproduce.
My current best guesses as to where to next is:
Get an insanely grunty box (to replace the current dev box: 2Gb RAM in an E6550 Core2 Duo); this will make it possible to repro the crash causing mis-behaviour when running under a powerful debug environment; or
Rewrite operators new and delete to use VirtualAlloc and VirtualProtect to mark memory as read-only as soon as it's done with. Run under MSVC6 and have the OS catch the bad-guy who's writing to freed memory. Yes, this is a sign of desperation: who the hell rewrites new and delete?! I wonder if this is going to make it as slow as under Purify et al.
And, no: Shipping with Purify instrumentation built in is not an option.
A colleague just walked past and asked "Stack Overflow? Are we getting stack overflows now?!?"
And now, the question: How do I locate the heap corruptor?
Update: balancing new[] and delete[] seems to have gotten a long way towards solving the problem. Instead of 15mins, the app now goes about two hours before crashing. Not there yet. Any further suggestions? The heap corruption persists.
Update: a release build under Visual Studio 2008 seems dramatically better; current suspicion rests on the STL implementation that ships with VS98.
Reproduce the problem. Dr Watson will produce a dump that might be helpful in further analysis.
I'll take a note of that, but I'm concerned that Dr Watson will only be tripped up after the fact, not when the heap is getting stomped on.
Another try might be using WinDebug as a debugging tool which is quite powerful being at the same time also lightweight.
Got that going at the moment, again: not much help until something goes wrong. I want to catch the vandal in the act.
Maybe these tools will allow you at least to narrow the problem to certain component.
I don't hold much hope, but desperate times call for...
And are you sure that all the components of the project have correct runtime library settings (C/C++ tab, Code Generation category in VS 6.0 project settings)?
No I'm not, and I'll spend a couple of hours tomorrow going through the workspace (58 projects in it) and checking they're all compiling and linking with the appropriate flags.
Update: This took 30 seconds. Select all projects in the Settings dialog, unselect until you find the project(s) that don't have the right settings (they all had the right settings).
My first choice would be a dedicated heap tool such as pageheap.exe.
Rewriting new and delete might be useful, but that doesn't catch the allocs committed by lower-level code. If this is what you want, better to Detour the low-level alloc APIs using Microsoft Detours.
Also sanity checks such as: verify your run-time libraries match (release vs. debug, multi-threaded vs. single-threaded, dll vs. static lib), look for bad deletes (eg, delete where delete [] should have been used), make sure you're not mixing and matching your allocs.
Also try selectively turning off threads and see when/if the problem goes away.
What does the call stack etc look like at the time of the first exception?
I have same problems in my work (we also use VC6 sometimes). And there is no easy solution for it. I have only some hints:
Try with automatic crash dumps on production machine (see Process Dumper). My experience says Dr. Watson is not perfect for dumping.
Remove all catch(...) from your code. They often hide serious memory exceptions.
Check Advanced Windows Debugging - there are lots of great tips for problems like yours. I recomend this with all my heart.
If you use STL try STLPort and checked builds. Invalid iterator are hell.
Good luck. Problems like yours take us months to solve. Be ready for this...
We've had pretty good luck by writing our own malloc and free functions. In production, they just call the standard malloc and free, but in debug, they can do whatever you want. We also have a simple base class that does nothing but override the new and delete operators to use these functions, then any class you write can simply inherit from that class. If you have a ton of code, it may be a big job to replace calls to malloc and free to the new malloc and free (don't forget realloc!), but in the long run it's very helpful.
In Steve Maguire's book Writing Solid Code (highly recommended), there are examples of debug stuff that you can do in these routines, like:
Keep track of allocations to find leaks
Allocate more memory than necessary and put markers at the beginning and end of memory -- during the free routine, you can ensure these markers are still there
memset the memory with a marker on allocation (to find usage of uninitialized memory) and on free (to find usage of free'd memory)
Another good idea is to never use things like strcpy, strcat, or sprintf -- always use strncpy, strncat, and snprintf. We've written our own versions of these as well, to make sure we don't write off the end of a buffer, and these have caught lots of problems too.
Run the original application with ADplus -crash -pn appnename.exe
When the memory issue pops-up you will get a nice big dump.
You can analyze the dump to figure what memory location was corrupted.
If you are lucky the overwrite memory is a unique string you can figure out where it came from. If you are not lucky, you will need to dig into win32 heap and figure what was the orignal memory characteristics. (heap -x might help)
After you know what was messed-up, you can narrow appverifier usage with special heap settings. i.e. you can specify what DLL you monitor, or what allocation size to monitor.
Hopefully this will speedup the monitoring enough to catch the culprit.
In my experience, I never needed full heap verifier mode, but I spent a lot of time analyzing the crash dump(s) and browsing sources.
P.S:
You can use DebugDiag to analyze the dumps.
It can point out the DLL owning the corrupted heap, and give you other usefull details.
You should attack this problem with both runtime and static analysis.
For static analysis consider compiling with PREfast (cl.exe /analyze). It detects mismatched delete and delete[], buffer overruns and a host of other problems. Be prepared, though, to wade through many kilobytes of L6 warning, especially if your project still has L4 not fixed.
PREfast is available with Visual Studio Team System and, apparently, as part of Windows SDK.
Is this in low memory conditions? If so it might be that new is returning NULL rather than throwing std::bad_alloc. Older VC++ compilers didn't properly implement this. There is an article about Legacy memory allocation failures crashing STL apps built with VC6.
The apparent randomness of the memory corruption sounds very much like a thread synchronization issue - a bug is reproduced depending on machine speed. If objects (chuncks of memory) are shared among threads and synchronization (critical section, mutex, semaphore, other) primitives are not on per-class (per-object, per-class) basis, then it is possible to come to a situation where class (chunk of memory) is deleted / freed while in use, or used after deleted / freed.
As a test for that, you could add synchronization primitives to each class and method. This will make your code slower because many objects will have to wait for each other, but if this eliminates the heap corruption, your heap-corruption problem will become a code optimization one.
You tried old builds, but is there a reason you can't keep going further back in the repository history and seeing exactly when the bug was introduced?
Otherwise, I would suggest adding simple logging of some kind to help track down the problem, though I am at a loss of what specifically you might want to log.
If you can find out what exactly CAN cause this problem, via google and documentation of the exceptions you are getting, maybe that will give further insight on what to look for in the code.
My first action would be as follows:
Build the binaries in "Release" version but creating debug info file (you will find this possibility in project settings).
Use Dr Watson as a defualt debugger (DrWtsn32 -I) on a machine on which you want to reproduce the problem.
Repdroduce the problem. Dr Watson will produce a dump that might be helpful in further analysis.
Another try might be using WinDebug as a debugging tool which is quite powerful being at the same time also lightweight.
Maybe these tools will allow you at least to narrow the problem to certain component.
And are you sure that all the components of the project have correct runtime library settings (C/C++ tab, Code Generation category in VS 6.0 project settings)?
So from the limited information you have, this can be a combination of one or more things:
Bad heap usage, i.e., double frees, read after free, write after free, setting the HEAP_NO_SERIALIZE flag with allocs and frees from multiple threads on the same heap
Out of memory
Bad code (i.e., buffer overflows, buffer underflows, etc.)
"Timing" issues
If it's at all the first two but not the last, you should have caught it by now with either pageheap.exe.
Which most likely means it is due to how the code is accessing shared memory. Unfortunately, tracking that down is going to be rather painful. Unsynchronized access to shared memory often manifests as weird "timing" issues. Things like not using acquire/release semantics for synchronizing access to shared memory with a flag, not using locks appropriately, etc.
At the very least, it would help to be able to track allocations somehow, as was suggested earlier. At least then you can view what actually happened up until the heap corruption and attempt to diagnose from that.
Also, if you can easily redirect allocations to multiple heaps, you might want to try that to see if that either fixes the problem or results in more reproduceable buggy behavior.
When you were testing with VS2008, did you run with HeapVerifier with Conserve Memory set to Yes? That might reduce the performance impact of the heap allocator. (Plus, you have to run with it Debug->Start with Application Verifier, but you may already know that.)
You can also try debugging with Windbg and various uses of the !heap command.
MSN
Graeme's suggestion of custom malloc/free is a good idea. See if you can characterize some pattern about the corruption to give you a handle to leverage.
For example, if it is always in a block of the same size (say 64 bytes) then change your malloc/free pair to always allocate 64 byte chunks in their own page. When you free a 64 byte chunk then set the memory protection bits on that page to prevent reads and wites (using VirtualQuery). Then anyone attempting to access this memory will generate an exception rather than corrupting the heap.
This does assume that the number of outstanding 64 byte chunks is only moderate or you have a lot of memory to burn in the box!
If you choose to rewrite new/delete, I have done this and have simple source code at:
http://gandolf.homelinux.org/~smhanov/blog/?id=10
This catches memory leaks and also inserts guard data before and after the memory block to capture heap corruption. You can just integrate with it by putting #include "debug.h" at the top of every CPP file, and defining DEBUG and DEBUG_MEM.
The little time I had to solve a similar problem.
If the problem still exists I suggest you do this :
Monitor all calls to new/delete and malloc/calloc/realloc/free.
I make single DLL exporting a function for register all calls. This function receive parameter for identifying your code source, pointer to allocated area and type of call saving this information in a table.
All allocated/freed pair is eliminated. At the end or after you need you make a call to an other function for create report for left data.
With this you can identify wrong calls (new/free or malloc/delete) or missing.
If have any case of buffer overwritten in your code the information saved can be wrong but each test may detect/discover/include a solution of failure identified. Many runs to help identify the errors.
Good luck.
Do you think this is a race condition? Are multiple threads sharing one heap? Can you give each thread a private heap with HeapCreate, then they can run fast with HEAP_NO_SERIALIZE. Otherwise, a heap should be thread safe, if you're using the multi-threaded version of the system libraries.
A couple of suggestions. You mention the copious warnings at W4 - I would suggest taking the time to fix your code to compile cleanly at warning level 4 - this will go a long way to preventing subtle hard to find bugs.
Second - for the /analyze switch - it does indeed generate copious warnings. To use this switch in my own project, what I did was to create a new header file that used #pragma warning to turn off all the additional warnings generated by /analyze. Then further down in the file, I turn on only those warnings I care about. Then use the /FI compiler switch to force this header file to be included first in all your compilation units. This should allow you to use the /analyze switch while controling the output