We have a big multi-threaded C++ application running on Linux. We see that occupied by the application memory grows fast and believe there are some leaks. We have tried every tool we have (valgrind, DynLeak, Purify) but did not find anything. Since this application can run on Windows, we have also tried Bounds Checker. Did not help, too.
We need a new tool that can help. I've looked at Google Perfomrance Tools, MMGR by Paul Nettle, MemCheck Deluxe. None of them impressed me.
Is there anywhere a good tool for this task?
The definition of a memory leak in C/C++ is very specific: it is memory that has been allocated and then the pointer was overwritten or otherwise lost. Valgrind generally detects such cases out of the box, but things are not always that simple.
Your application could very well be still using that memory. In that case you might have what a Java programmer would consider a leak, e.g. entering data in a structure and rarely (or never) removing entries.
You might be measuring the memory usage of your memory incorrectly. On Linux memory usage measurements are not as straight-forward as they seem. How have you measured your memory usage?
You should consider using the application hooks (Valgrind calls them client requests) of whatever memory analysis tool your are using, to avoid the issue with reports only being issued at program termination. Using those hooks might help you pin-point the location of your leak.
You should try using a heap profiler, such as massif from Valgrind, to look for memory allocation locations with inordinate amounts of allocated memory.
Make sure you are not using a custom allocator or garbage collector in your application. As far as I know, no memory analysis tool will work with a custom allocator without user interference.
If your memory leak is massive enough to be detectable within an acceptable amount of application run-time, you could try a binary search of old revisions through your version control system to identify the commit that introduced the problem. At least Mercurial
and Git offer built-in support for this task.
If by "did not help" you mean it did not report memory leaks, it is quite possible you don't have one and just use more and more memory that is still referenced by pointers and can be deleted.
To help you debug the problem, perhaps in your logging, you should also write memory size, number of objects (their type) and a few other stats which are useful to you. At least until you become more familiar with the tools you mentioned.
I'm looking for a real time memory tracker library (or considering writing one) that is built on the CRT Debug Heap functions. This would primarily be for performance/memory usage analysis, although it should also detect leaks.
I'm looking to get similar data to what Memory Validator provides except integrated into a program (not an external tool).
If there is a library that already suits my needs that would be great. If not, could you point me towards some resources for writing one and possibly list some pros and cons of building it on top of the CRT debug heap functions as opposed to overloading new and delete and writing a memory tracker singleton (or suggest a different better route to take).
Obvious you are on the windows platform, I suggest the Visual leak detector lib.
In most scenario memory tracking is use to detect memory leak, but only print the memory allocation file name and line number is not enough to find out the real problem, you need the backtrace for the leaked memory.
You can't get the details MemoryValidator provides from just the CRT debug heap. It doesn't supply a callstack or even file/line without #defining new (evil! and breaks placement new).
I believe I have a double delete and some memory corruption happening somehow in my complex c++ application on OpenBSD. I would like to track down the first location my object is deleted, and any points at which deallocated memory is accessed.
I would usually look into valgrind, but it is linux only. Failing that, I would instrument my new and delete operators with some kind of tracking code, but I've been finding it difficult to determine the correct google search for this.
Is there any package for openbsd which will give me information on memory errors? Is there any kind of standard way to redefine new and delete to detect overflows, invalid accesses, double frees?
This helps a lot:
man malloc
Debugging options can be enabled system-wide, environment-wide, or program-specific.
I don't know if you are willing to use a third party tool, but C++ memory validator is very good.
http://www.softwareverify.com/cpp-memory.php
It isolates the memory/ handle leaks, tells you how much memory is leaked and shows you the position in the code. If only it could then fix the leak for you : ) 30 day free trial is available too.
I've used it to find leaks in my legacy C++ MFC application where the previous developer didn't seem to think there was a need to relase memory ever!
When someone mention the memory management the c++ is capable of doing, how can i see this thing? is this done in theory like guessing?
i took a logical design intro course and it covered the systems of numbers and boolean algebra and combinational logic,will this help?
so say in Visual Studio , is there some kind of a tool to visualize the memory,i hope im not being ridiculous here ?
thank you.
C++ has a variety of memory areas:
space used for global and static variables, which is pre-allocated by the compiler
"stack" memory that's used for preserving caller context during function calls, passing some function arguments (others may fit in CPU registers), and local variables
"heap" memory that's allocated using new or new[](C++'s preferred approach) or malloc (a lower-level function inherited from C), and released with delete, delete[] or free respectively.
The heap is important in that it supports run-time requests for arbitrary amounts of memory, and the usage persists until delete or free is explicitly used, rather than being tied to the lifetime of particular function calls as per stack memory.
I'm not aware of any useful tools for visualising and categorising the overall memory usage of a running C++ program, less still for relating that back to which pointers in the source code currently have how much memory associated with them. As a very general guideline, it's encouraged to write code in such a way that pointers are only introduced when the program is ready to point them at something, and they go out of scope when they will no longer pointer at something. When that's impractical, it can be useful to set them to NULL (0), so that if you're monitoring the executing program in a debugger you can tell the pointer isn't meant to point to legitimate data at that point.
Memory management is not something that you can easily visualize while you are programming. Instead, it refers to how your program is allocating and freeing memory while it is running. Many debuggers will provide a way to halt a program while it is running and view information about the dynamic memory that it has allocated. You can plan your classes and interfaces with proper memory management techniques, but it's not as simple as "hit this button for a chart of your memory usage".
You can also implement something like this to keep track of your memory allocations and warn you about anything that your program didn't free. A garbage collector can free you from some of the hassles associated with memory management.
In Visual Studio, there is a Memory Window (Alt+6) that will let you read/write memory manually provided it is a valid memory location for the operation you are trying to do, during debugging.
On Windows platform, you can get a first initial feel of memory management using tools like perfmon.exe, taskmgr.exe and many other tools from sysinternals
I am debugging a (native) multi-threaded C++ application under Visual Studio 2008. On seemingly random occasions, I get a "Windows has triggered a break point..." error with a note that this might be due to a corruption in the heap. These errors won't always crash the application right away, although it is likely to crash short after.
The big problem with these errors is that they pop up only after the corruption has actually taken place, which makes them very hard to track and debug, especially on a multi-threaded application.
What sort of things can cause these errors?
How do I debug them?
Tips, tools, methods, enlightments... are welcome.
Application Verifier combined with Debugging Tools for Windows is an amazing setup. You can get both as a part of the Windows Driver Kit or the lighter Windows SDK. (Found out about Application Verifier when researching an earlier question about a heap corruption issue.) I've used BoundsChecker and Insure++ (mentioned in other answers) in the past too, although I was surprised how much functionality was in Application Verifier.
Electric Fence (aka "efence"), dmalloc, valgrind, and so forth are all worth mentioning, but most of these are much easier to get running under *nix than Windows. Valgrind is ridiculously flexible: I've debugged large server software with many heap issues using it.
When all else fails, you can provide your own global operator new/delete and malloc/calloc/realloc overloads -- how to do so will vary a bit depending on compiler and platform -- and this will be a bit of an investment -- but it may pay off over the long run. The desirable feature list should look familiar from dmalloc and electricfence, and the surprisingly excellent book Writing Solid Code:
sentry values: allow a little more space before and after each alloc, respecting maximum alignment requirement; fill with magic numbers (helps catch buffer overflows and underflows, and the occasional "wild" pointer)
alloc fill: fill new allocations with a magic non-0 value -- Visual C++ will already do this for you in Debug builds (helps catch use of uninitialized vars)
free fill: fill in freed memory with a magic non-0 value, designed to trigger a segfault if it's dereferenced in most cases (helps catch dangling pointers)
delayed free: don't return freed memory to the heap for a while, keep it free filled but not available (helps catch more dangling pointers, catches proximate double-frees)
tracking: being able to record where an allocation was made can sometimes be useful
Note that in our local homebrew system (for an embedded target) we keep the tracking separate from most of the other stuff, because the run-time overhead is much higher.
If you're interested in more reasons to overload these allocation functions/operators, take a look at my answer to "Any reason to overload global operator new and delete?"; shameless self-promotion aside, it lists other techniques that are helpful in tracking heap corruption errors, as well as other applicable tools.
Because I keep finding my own answer here when searching for alloc/free/fence values MS uses, here's another answer that covers Microsoft dbgheap fill values.
You can detect a lot of heap corruption problems by enabling Page Heap for your application . To do this you need to use gflags.exe that comes as a part of Debugging Tools For Windows
Run Gflags.exe and in the Image file options for your executable, check "Enable Page Heap" option.
Now restart your exe and attach to a debugger. With Page Heap enabled, the application will break into debugger whenever any heap corruption occurs.
To really slow things down and perform a lot of runtime checking, try adding the following at the top of your main() or equivalent in Microsoft Visual Studio C++
_CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF | _CRTDBG_CHECK_ALWAYS_DF );
A very relevant article is Debugging Heap corruption with Application Verifier and Debugdiag.
What sort of things can cause these errors?
Doing naughty things with memory, e.g. writing after the end of a buffer, or writing to a buffer after it's been freed back to the heap.
How do I debug them?
Use an instrument which adds automated bounds-checking to your executable: i.e. valgrind on Unix, or a tool like BoundsChecker (Wikipedia suggests also Purify and Insure++) on Windows.
Beware that these will slow your application, so they may be unusable if yours is a soft-real-time application.
Another possible debugging aid/tool might be MicroQuill's HeapAgent.
One quick tip, that I got from Detecting access to freed memory is this:
If you want to locate the error
quickly, without checking every
statement that accesses the memory
block, you can set the memory pointer
to an invalid value after freeing the
block:
#ifdef _DEBUG // detect the access to freed memory
#undef free
#define free(p) _free_dbg(p, _NORMAL_BLOCK); *(int*)&p = 0x666;
#endif
The best tool I found useful and worked every time is code review (with good code reviewers).
Other than code review, I'd first try Page Heap. Page Heap takes a few seconds to set up and with luck it might pinpoint your problem.
If no luck with Page Heap, download Debugging Tools for Windows from Microsoft and learn to use the WinDbg. Sorry couldn't give you more specific help, but debuging multi-threaded heap corruption is more an art than science. Google for "WinDbg heap corruption" and you should find many articles on the subject.
What type of allocation functions are you using? I recently hit a similar error using the Heap* style allocation functions.
It turned out that I was mistakenly creating the heap with the HEAP_NO_SERIALIZE option. This essentially makes the Heap functions run without thread safety. It's a performance improvement if used properly but shouldn't ever be used if you are using HeapAlloc in a multi-threaded program [1]. I only mention this because your post mentions you have a multi-threaded app. If you are using HEAP_NO_SERIALIZE anywhere, delete that and it will likely fix your problem.
[1] There are certain situations where this is legal, but it requires you to serialize calls to Heap* and is typically not the case for multi-threaded programs.
If these errors occur randomly, there is high probability that you encountered data-races. Please, check: do you modify shared memory pointers from different threads? Intel Thread Checker may help to detect such issues in multithreaded program.
You may also want to check to see whether you're linking against the dynamic or static C runtime library. If your DLL files are linking against the static C runtime library, then the DLL files have separate heaps.
Hence, if you were to create an object in one DLL and try to free it in another DLL, you would get the same message you're seeing above. This problem is referenced in another Stack Overflow question, Freeing memory allocated in a different DLL.
In addition to looking for tools, consider looking for a likely culprit. Is there any component you're using, perhaps not written by you, which may not have been designed and tested to run in a multithreaded environment? Or simply one which you do not know has run in such an environment.
The last time it happened to me, it was a native package which had been successfully used from batch jobs for years. But it was the first time at this company that it had been used from a .NET web service (which is multithreaded). That was it - they had lied about the code being thread safe.
You can use VC CRT Heap-Check macros for _CrtSetDbgFlag: _CRTDBG_CHECK_ALWAYS_DF or _CRTDBG_CHECK_EVERY_16_DF.._CRTDBG_CHECK_EVERY_1024_DF.
I'd like to add my experience. In the last few days, I solved an instance of this error in my application. In my particular case, the errors in the code were:
Removing elements from an STL collection while iterating over it (I believe there are debug flags in Visual Studio to catch these things; I caught it during code review)
This one is more complex, I'll divide it in steps:
From a native C++ thread, call back into managed code
In managed land, call Control.Invoke and dispose a managed object which wraps the native object to which the callback belongs.
Since the object is still alive inside the native thread (it will remain blocked in the callback call until Control.Invoke ends). I should clarify that I use boost::thread, so I use a member function as the thread function.
Solution: Use Control.BeginInvoke (my GUI is made with Winforms) instead so that the native thread can end before the object is destroyed (the callback's purpose is precisely notifying that the thread ended and the object can be destroyed).
I had a similar problem - and it popped up quite randomly. Perhaps something was corrupt in the build files, but I ended up fixing it by cleaning the project first then rebuilding.
So in addition to the other responses given:
What sort of things can cause these errors?
Something corrupt in the build file.
How do I debug them?
Cleaning the project and rebuilding. If it's fixed, this was likely the problem.
I have also faced this issue. In my case, I allocated for x size memory and appended the data for x+n size. So, when freeing it shown heap overflow. Just make sure your allocated memory sufficient and check for how many bytes added in the memory.