I am working on the largest project I've ever worked on, and I've never debugged something like this, so I don't know where to get started.
Some info on the crash:
I am using Visual Studio, and the debugger is completely useless. The only information it gives me is that it appears to be happening during a call to "memcpy". The call stack is completely empty except for the memcpy function, and the local variables are listed but it does not have values for any of them.
It happens occasionally on any computer.
It does not ALWAYS happen under any (known) condition, but it only ever happens under a few conditions. In particular it only happens when a particular type of object is destroyed, although that's not necessarily the direct cause, and investigating the destruction process has not been helpful.
A little more about the project:
It is a game using SFML 2.0, linked statically.
I am not calling memcpy anywhere in my own code.
Some questions:
Where could the call to memcpy be coming from? Is it in SFML or elsewhere?
How do I (using visual studio) get more information on a crash when the debugger isn't working?
This is an answer to "Where could the call to memcpy be coming from?"
In most cases this is the result of a call to the copy constructor of std::string with a this pointer of NULL, or a string operation on an already destructed string. This string can be a member of a class of you, of course.
This in itself won't help you to find the problem when the project is really large. However, you can almost certainly assume that you are using a reference or pointer (or iterator) to a custom object that is already destructed. A most straightforward way to find this access would be by running your program, compiled without optimization and with debug info, in valgrind. Unfortunately that isn't available for windows (see Is there a good Valgrind substitute for Windows?).
The main problem here seems to be that you aren't even getting a backtrace, because that would give a strong hint to where to look into, at least. I'm not familiar with windows though, so I can only guess what is the cause of that. Are you sure you have everything compiled with debug info?
First of all, thank you for taking the time to view my question and help. I noticed that a lot of questioners here show little or no appreciation, but I'm sincerely appreciative for the help and the community here :)
I wrote a C++ plugin (compromised of hundreds of source files) for an application I do not have the source code for (it's a video game). In other words, I only have the source code for my plugin, but not the game. Now, somewhere in those thousands of lines in my plugin, something causes the game engine to throw (probably an access violation) and I don't know where. By the time the debugger breaks, the stack is corrupted and all I get are hex addresses for DLLs I do not have the source for (but the exception occurs in my DLL for sure). I tried everything... I just can't seem to find where the exception occurs. Sometimes the debugger points to a "memory relocation" function (which I never used in my plugin), sometimes it points to the engine's GameFrame(), and other times it points to a damage callback (all these are just different member functions of a class).
I tried practically everything... I googled for hours trying to find out how to use other debuggers like WinDbg and Microsoft Application Verifier. I tried to comment out one or the other, or both, where the debugger points, but it still crashes. I even inserted OUTPUT("The name of the last executed function is: %s", __FUNCTION__) into EVERY function in my application hoping to painstakingly catch the last function but it seems any kind of I/O prevents the exception from occurring for some reason... And 10 minutes of debugging and the crash happens at some random last executed function.
I can't find out where this access violation is happening or where some temporary object is removed to cause these bad pointers (I check every pointer before using it), but damn, I'm reaching my limit's end here.
So, how does one debug the impossible... a random crash with a crappy debugger call stack? Thanks in advance for your patient and kind help!
My suggestion: try different debuggers (non MS), they catch different things.
My experience: a program I have source code and full debugging symbols corrupt the stack, VS nor WinDbg can help but Ollydbg comments a non-string var with the value "r for pattern.", so I had overwrote some string buffer onto this var. Also Ollydbg have option to walk the stack the hard way (not using dbghelp.dll)
From my experience, the old adage "Prevention is better than cure" is very relevant. It is best to prevent the bugs from creeping in, by following good software development practices (unit tests, regressions, code review, etc.) than to work it out later once the bugs show up.
Of course, real world is not perfect, and bugs do show up. To debug memory corruption, you have some nice tools like valgrind, which at least narrow down the problem sections for you to take a closer look at. Debugging a complex program is not easy, and if your debugger throws up, it requires a lot of persistence on your part. One technique I find useful is to selectively enable or disable certain modules, to narrow down the module has the problem.
Sometimes you need to use "referential transparency" to unload some modules. To give you a stripped down example, consider:
int foo = factorial(3);
If I suspect there's a problem in this code (and the debugger crashes before I can see the call stack), I have to try by removing this code, and see if the problem persists. However, foo may be used later, so I cannot just remove it. Instead I can replace it with int foo = 6; and continue.
Another important point is to always maintain a trace file, where your code keeps logging what it is doing. When a program crashes, the trace file can often help narrow down the problem. Of course, you disable the tracing by default, so that it doesn't cause a performance bottleneck.
Alright, I need a sanity check.
I was working on my project by starting to add a new class yesterday and decided to compile my progress so far. Upon executing a release build of the application I immediately noticed a bug. (Which in this case is an asteroid field not showing up for my game) This immediately confused me since I hadn't touched any code that could of created this bug. And everything else is rendered fine as usual.
Fast forward a day later and I think I've narrowed down the cause however it doesn't make any sense to me at all.
It only happens intermittently with release builds. (About 1 in 5 executions) But I can prevent it happening altogether by simply not including my new class in the project. Even though the class isn't being used or included by anything else in the project yet. I even went a step further and found just commenting out the class definitions and leaving the header is fine but the second I uncomment them it comes back. And for further testing I included it in an older build with the same results.
Would unused code affect the compilation of a release build? Does this sound like a Visual Studio bug? Or does it not make any sense at all? (In other words, did I just find a really convincing red herring?) Is there anything I can do to help figure out the source of this bug?
Would unused code affect the compilation of a release build?
Yes. As this SO answer explains, Visual C++ keeps unused functions that aren't marked as inline.
Does this sound like a Visual Studio bug?
It's unlikely, as I'll explain in answer to your next question. Eric Lippert explains that, because the compiler is used so often that the easy and commonly seen bugs have already been found and fixed, it is far more likely that the bug is in your code than that the bug is in the C++ compiler.
Did I just find a really convincing red herring?
Yes, I believe you did. Without seeing the code for drawing the asteroid field, I can't be sure, but it's likely that you are dereferencing a bad pointer somewhere in there. Now it starts to matter which data the pointer points to.
When the unused class is included in the compiled assembly, the memory devoted to your code is larger, which means that your heap has to start higher in memory, and a bunch of things move to higher addresses. This means that a bad pointer can point at different data depending on whether the unused class is compiled into the executable or not. When you dereference that pointer, then, you get different results. That's why removing the code appears to "fix" the bug - whenever you take out the unused class, the bad pointer points at the data you want, rather than at other data. As #awoodland rightly points out in the comments, you're really unlucky without the unused class and lucky with it, because you fail to find a bug that could manifest itself in all kinds of weird ways once you start distributing the code to your friends (or customers, if this is a commercial product). These kinds of "bad pointer that happens to work" bugs can easily cause your code to appear to work correctly on some machines and fail dramatically on other machines, which is very difficult to debug. Better that you found the bug now than later.
I know that E&C is a controversial subject and some say that it encourages a wrong approach to debugging, but still - I think we can agree that there are numerous cases when it is clearly useful - experimenting with different values of some constants, redesigning GUI parameters on-the-fly to find a good look... You name it.
My question is: Are we ever going to have E&C on GDB? I understand that it is a platform-specific feature and needs some serious cooperation with the compiler, the debugger and the OS (MSVC has this one easy as the compiler and debugger always come in one package), but... It still should be doable. I've even heard something about Apple having it implemented in their version of GCC [citation needed]. And I'd say it is indeed feasible.
Knowing all the hype about MSVC's E&C (my experience says it's the first thing MSVC users mention when asked "why not switch to Eclipse and gcc/gdb"), I'm seriously surprised that after quite some years GCC/GDB still doesn't have such feature. Are there any good reasons for that? Is someone working on it as we speak?
It is a surprisingly non-trivial amount of work, encompassing many design decisions and feature tradeoffs. Consider: you are debugging. The debugee is suspended. Its image in memory contains the object code of the source, and the binary layout of objects, the heap, the stacks. The debugger is inspecting its memory image. It has loaded debug information about the symbols, types, address mappings, pc (ip) to source correspondences. It displays the call stack, data values.
Now you want to allow a particular set of possible edits to the code and/or data, without stopping the debuggee and restarting. The simplest might be to change one line of code to another. Perhaps you recompile that file or just that function or just that line. Now you have to patch the debuggee image to execute that new line of code the next time you step over it or otherwise run through it. How does that work under the hood? What happens if the code is larger than the line of code it replaced? How does it interact with compiler optimizations? Perhaps you can only do this on a specially compiled for EnC debugging target. Perhaps you will constrain possible sites it is legal to EnC. Consider: what happens if you edit a line of code in a function suspended down in the call stack. When the code returns there does it run the original version of the function or the version with your line changed? If the original version, where does that source come from?
Can you add or remove locals? What does that do to the call stack of suspended frames? Of the current function?
Can you change function signatures? Add fields to / remove fields from objects? What about existing instances? What about pending destructors or finalizers? Etc.
There are many, many functionality details to attend to to make any kind of usuable EnC work. Then there are many cross-tools integration issues necessary to provide the infrastructure to power EnC. In particular, it helps to have some kind of repository of debug information that can make available the before- and after-edit debug information and object code to the debugger. For C++, the incrementally updatable debug information in PDBs helps. Incremental linking may help too.
Looking from the MS ecosystem over into the GCC ecosystem, it is easy to imagine the complexity and integration issues across GDB/GCC/binutils, the myriad of targets, some needed EnC specific target abstractions, and the "nice to have but inessential" nature of EnC, are why it has not appeared yet in GDB/GCC.
Happy hacking!
(p.s. It is instructive and inspiring to look at what the Smalltalk-80 interactive programming environment could do. In St80 there was no concept of "restart" -- the image and its object memory were always live, if you edited any aspect of a class you still had to keep running. In such environments object versioning was not a hypothetical.)
I'm not familiar with MSVC's E&C, but GDB has some of the things you've mentioned:
http://sourceware.org/gdb/current/onlinedocs/gdb/Altering.html#Altering
17. Altering Execution
Once you think you have found an error in your program, you might want to find out for certain whether correcting the apparent error would lead to correct results in the rest of the run. You can find the answer by experiment, using the gdb features for altering execution of the program.
For example, you can store new values into variables or memory locations, give your program a signal, restart it at a different address, or even return prematurely from a function.
Assignment: Assignment to variables
Jumping: Continuing at a different address
Signaling: Giving your program a signal
Returning: Returning from a function
Calling: Calling your program's functions
Patching: Patching your program
Compiling and Injecting Code: Compiling and injecting code in GDB
This is a pretty good reference to the old Apple implementation of "fix and continue". It also references other working implementations.
http://sources.redhat.com/ml/gdb/2003-06/msg00500.html
Here is a snippet:
Fix and continue is a feature implemented by many other debuggers,
which we added to our gdb for this release. Sun Workshop, SGI ProDev
WorkShop, Microsoft's Visual Studio, HP's wdb, and Sun's Hotspot Java
VM all provide this feature in one way or another. I based our
implementation on the HP wdb Fix and Continue feature, which they
added a few years back. Although my final implementation follows the
general outlines of the approach they took, there is almost no shared
code between them. Some of this is because of the architectual
differences (both the processor and the ABI), but even more of it is
due to implementation design differences.
Note that this capability may have been removed in a later version of their toolchain.
UPDATE: Dec-21-2012
There is a GDB Roadmap PDF presentation that includes a slide describing "Fix and Continue" among other bullet points. The presentation is dated July-9-2012 so maybe there is hope to have this added at some point. The presentation was part of the GNU Tools Cauldron 2012.
Also, I get it that adding E&C to GDB or anywhere in Linux land is a tough chore with all the different components.
But I don't see E&C as controversial. I remember using it in VB5 and VB6 and it was probably there before that. Also it's been in Office VBA since way back. And it's been in Visual Studio since VS2005. VS2003 was the only one that didn't have it and I remember devs howling about it. They intended to add it back anyway and they did with VS2005 and it's been there since. It works with C#, VB, and also C and C++. It's been in MS core tools for 20+ years, almost continuous (counting VB when it was standalone), and subtracting VS2003. But you could still say they had it in Office VBA during the VS2003 period ;)
And Jetbrains recently added it too their C# tool Rider. They bragged about it (rightly so imo) in their Rider blog.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
In unmanaged C/C++ code, what are the best practices to detect memory leaks? And coding guidelines to avoid? (As if it's that simple ;)
We have used a bit of a silly way in the past: having a counter increment for every memory allocation call and decrement while freeing. At the end of the program, the counter value should be zero.
I know this is not a great way and there are a few catches. (For instance, if you are freeing memory which was allocated by a platform API call, your allocation count will not exactly match your freeing count. Of course, then we incremented the counter when calling API calls that allocated memory.)
I am expecting your experiences, suggestions and maybe some references to tools which simplify this.
If your C/C++ code is portable to *nix, few things are better than Valgrind.
If you are using Visual Studio, Microsoft provides some useful functions for detecting and debugging memory leaks.
I would start with this article:
https://msdn.microsoft.com/en-us/library/x98tx3cf(v=vs.140).aspx
Here is the quick summary of those articles. First, include these headers:
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
Then you need to call this when your program exits:
_CrtDumpMemoryLeaks();
Alternatively, if your program does not exit in the same place every time, you can call this at the start of your program:
_CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
Now when the program exits all the allocations that were not free'd will be printed in the Output Window along with the file they were allocated in and the allocation occurrence.
This strategy works for most programs. However, it becomes difficult or impossible in certain cases. Using third party libraries that do some initialization on startup may cause other objects to appear in the memory dump and can make tracking down your leaks difficult. Also, if any of your classes have members with the same name as any of the memory allocation routines( such as malloc ), the CRT debug macros will cause problems.
There are other techniques explained in the MSDN link referenced above that could be used as well.
In C++: use RAII. Smart pointers like std::unique_ptr, std::shared_ptr, std::weak_ptr are your friends.
As a C++ Developer here's some simply guidelines:
Use pointers only when absolutely necessary
If you need a pointer, doublecheck if a SmartPointer is a possibility
Use the GRASP Creator pattern.
As for the detection of memory leaks personally I've always used Visual Leak Detector and find it to be very useful.
I've been using DevStudio for far too many years now and it always amazes me just how many programmers don't know about the memory analysis tools that are available in the debug run time libraries. Here's a few links to get started with:
Tracking Heap Allocation Requests - specifically the section on Unique Allocation Request Numbers
_CrtSetDbgFlag
_CrtSetBreakAlloc
Of course, if you're not using DevStudio then this won't be particularly helpful.
I’m amazed no one mentioned DebugDiag for Windows OS.
It works on release builds, and even at the customer site.
(You just need to keep your release version PDBs, and configure DebugDiag to use Microsoft public symbol server)
Visual Leak Detector is a very good tool, altough it does not supports the calls on VC9 runtimes (MSVCR90D.DLL for example).
Microsoft VC++ in debug mode shows memory leaks, although it doesn't show where your leaks are.
If you are using C++ you can always avoid using new explicitly: you have vector, string, auto_ptr (pre C++11; replaced by unique_ptr in C++11), unique_ptr (C++11) and shared_ptr (C++11) in your arsenal.
When new is unavoidable, try to hide it in a constructor (and hide delete in a destructor); the same works for 3rd party APIs.
There are various replacement "malloc" libraries out there that will allow you to call a function at the end and it will tell you about all the unfreed memory, and in many cases, who malloced (or new'ed) it in the first place.
If you're using MS VC++, I can highly recommend this free tool from the codeproject:
leakfinder by Jochen Kalmbach.
You simply add the class to your project, and call
InitAllocCheck(ACOutput_XML)
DeInitAllocCheck()
before and after the code you want to check for leaks.
Once you've build and run the code, Jochen provides a neat GUI tool where you can load the resulting .xmlleaks file, and navigate through the call stack where each leak was generated to hunt down the offending line of code.
Rational's (now owned by IBM) PurifyPlus illustrates leaks in a similar fashion, but I find the leakfinder tool actually easier to use, with the bonus of it not costing several thousand dollars!
Never used it myself, but my C friends tell me Purify.
If you're using Visual Studio it might be worth looking at Bounds Checker. It's not free, but it's been incredibly helpful in finding leaks in my code. It doesn't just do memory leaks either, but also GDI resource leaks, WinAPI usage errors, and other stuff. It'll even show you where the leaked memory was initialized, making it much easier to track down the leak.
I think that there is no easy answer to this question. How you might really approach this solution depends on your requirements. Do you need a cross platform solution? Are you using new/delete or malloc/free (or both)? Are you really looking for just "leaks" or do you want better protection, such as detecting buffer overruns (or underruns)?
If you are working on the windows side, the MS debug runtime libraries have some basic debug detection functionality, and as another has already pointed out, there are several wrappers that can be included in your source to help with leak detection. Finding a package that can work with both new/delete and malloc/free obviously gives you more flexibility.
I don't know enough about the unix side to provide help, although again, others have.
But beyond just leak detection, there is the notion of detecting memory corruption via buffer overruns (or underruns). This type of debug functionality is I think more difficult than plain leak detection. This type of system is also further complicated if you are working with C++ objects because polymorhpic classes can be deleted in varying ways causing trickiness in determining the true base pointer that is being deleted. I know of no good "free" system that does decent protection for overruns. we have written a system (cross platform) and found it to be pretty challenging.
I'd like to offer something I've used at times in the past: a rudimentary leak checker which is source level and fairly automatic.
I'm giving this away for three reasons:
You might find it useful.
Though it's a bit krufty, I don't let that embarass me.
Even though it's tied to some win32 hooks, that should be easy to alleviate.
There are things of which you must be careful when using it: don't do anything that needs to lean on new in the underlying code, beware of the warnings about cases it might miss at the top of leakcheck.cpp, realize that if you turn on (and fix any issues with) the code that does image dumps, you may generate a huge file.
The design is meant to allow you to turn the checker on and off without recompiling everything that includes its header. Include leakcheck.h where you want to track checking and rebuild once. Thereafter, compile leakcheck.cpp with or without LEAKCHECK #define'd and then relink to turn it on and off. Including unleakcheck.h will turn it off locally in a file. Two macros are provided: CLEARALLOCINFO() will avoid reporting the same file and line inappropriately when you traverse allocating code that didn't include leakcheck.h. ALLOCFENCE() just drops a line in the generated report without doing any allocation.
Again, please realize that I haven't used this in a while and you may have to work with it a bit. I'm dropping it in to illustrate the idea. If there turns out to be sufficient interest, I'd be willing to work up an example, updating the code in the process, and replace the contents of the following URL with something nicer that includes a decently syntax-colored listing.
You can find it here: http://www.cse.ucsd.edu/~tkammeye/leakcheck.html
For Linux:
Try Google Perftools
There are a lot of tools that do similar alloc/free counting, the pros of Goolge Perftools:
Quite fast (in comparison to valgrind: very fast)
Comes with nice graphical display of results
Has other useful capabilities: cpu-profiling, memory-usage profiling...
The best defense against leaks is a program structure which minimizes the use of malloc. This is not only good from a programming perspective, but also improves performance and maintainability. I'm not talking about using other things in place of malloc, but in terms of re-using objects and keeping very explicit tabs on all objects being passed around rather than allocating willy-nilly like one often gets used to in languages with garbage collectors like Java.
For example, a program I work on has a bunch of frame objects representing image data. Each frame object has sub-data, which the frame's destructor frees. The program keeps a list of all frames that are allocated, and when it needs a new one, checks a list of unused frame objects to see if it can re-use an existing one rather than allocate a new one. On shutdown, it just iterates through the list, freeing everything.
I would recommend using Memory Validator from software verify.
This tool proved itself to be of invaluable help to help me track down memory leaks and to improve the memory management of the applications i am working on.
A very complete and fast tool.
Are you counting the allocs and frees by interpolating your own syscall functions which record the calls and then pass the call to the real function?
This is the only way you can keep track of calls originating from code that you haven't written.
Have a look at the man page for ld.so. Or ld.so.1 on some systems.
Also do Google LD_PRELOAD and you'll find some interesting articles explaining the technique over on www.itworld.com.
At least for MS VC++, the C Runtime library has several functions that I've found helpful in the past. Check the MSDN help for the _Crt* functions.
Paul Nettle's mmgr is a long time favourite tool of mine. You include mmgr.h in your source files, define TEST_MEMORY, and it delivers a textfile full of memory problems that occurred during a run of your app.
General Coding Guideline:
Resources should be deallocated at the same "layer" (function/class/library) where they are allocated.
If this is not possible, try to use some automatic deallocation (boost shared pointer...)
Memory debugging tools are worth their weight in gold but over the years I've found that two simple ideas can be used to prevent most memory/resource leaks from being coded in the first place.
Write release code immediatly after writing the acquisition code for the resources you want to allocate. With this method its harder to "forget" and in some sense forces one to seriously think of the lifecycle of resources being used upfront instead of as an aside.
Use return as sparringly as possible. What is allocated should only be freed in one place if possible. The conditional path between acquisition of resource and release should be designed to be as simple and obvious as possible.
At the top of this list (when I read it) was valgrind. Valgrind is excellent if you are able to reproduce the leak on a test system. I've used it with great success.
What if you've just noticed that the production system is leaking right now and you have no idea how to reproduce it in test? Some evidence of what's wrong is captured in the state of that production system, and it might be enough to provide an insight on where the problem is so you can reproduce it.
That's where Monte Carlo sampling comes into the picture. Read Raymond Chen's blog article,
“The poor man's way of identifying memory leaks” and then check out my implementation (assumes Linux, tested only on x86 and x86-64)
http://github.com/tialaramex/leakdice/tree/master
Working on Motorola cell phones operating system, we hijacked memory allocation library to observe all memory allocations. It helped to find a lot of problems with memory allocations.
Since prevention is better then curing, I would recommend to use static analysis tool like Klockwork or PC-Lint
Valgrind is a nice option for Linux. Under MacOS X, you can enable the MallocDebug library which has several options for debugging memory allocation problems (see the malloc manpage, the "ENVIRONMENT" section has the relevant details). The OS X SDK also includes a tool called MallocDebug (usually installed in /Developer/Applications/Performance Tools/) that can help you to monitor usage and leaks.
Detect:
Debug CRT
Avoid:
Smart pointers, boehm GC
A nice malloc, calloc and reallloc replacement is rmdebug, it's pretty simple to use. It is much faster to then valgrind, so you can test your code extensively. Of course it has some downsides, once you found a leak you probably still need to use valgrind to find where the leak appears and you can only test mallocs that you do directly. If a lib leaks because you use it wrong, rmdebug won't find it.
http://www.hexco.de/rmdebug/
Most memory profilers slow my large complex Windows application to the point where the results are useless. There is one tool that works well for finding leaks in my application: UMDH - http://msdn.microsoft.com/en-us/library/ff560206%28VS.85%29.aspx
Mtrace appears to be the standard built-in one for linux. The steps are :
set up the environment variable MALLOC_TRACE in bash
MALLOC_TRACE=/tmp/mtrace.dat
export MALLOC_TRACE;
Add #include <mcheck.h> to the top of you main source file
Add mtrace(); at the start of main and muntrace(); at the bottom (before the return statement)
compile your program with the -g switch for debug information
run your program
display leak info with mtrace your_prog_exe_name /tmp/mtrace.dat
(I had to install the mtrace perl script first on my fedora system with yum install glibc_utils )