I have a C++ application that has some minimal leaks, and I would like to fix them. I am using AppVerifier to dump the leaked objects, and I can get the addresses and first few bytes of the allocated memory.
Unfortunately, those first bytes and raw address is not enough to pinpoint the allocation stack trace, is there a method to get complete allocation data dump, and find the stack that's allocating the memory?
I could put _CrtSetBreakAlloc via the leak number, but unfortunatelly it's a threaded application and those numbers float up and down.
Does anyone have a suggestion what I could try?
With the gflags utility you can enable storing call stack information (gflags +ust). However, your applications will now run slower and take more memory.
Side-remark: To be honest, I never got all those Microsoft utilities (leak-tracing in the C-RunTime, Gflags, UMDH, AppVerifier, LeakDiag) to do exactly what I wanted. In the end, I simply wrote my own memory allocator in which I can add whatever tracing I want (call stack, red zone marking, delayed freeing, consistency checking, ...).
You could try using UMDH to track memory leaks. You first have to use GFlags to turn on storing call stack tracing whenever memory is allocated. The docs on UMDH state how to use it.
But recently I've finally tried out visual leak detector, and it works fabulous on my monstrous, big app.
http://vld.codeplex.com
Related
In C++ programs, I have sometimes had problems with "weak" memory leaks. By that, I mean that some objects accumulate resources, but, eventually, these objects are destroyed properly and their memory is released, so that these leaks do not show up using the traditional memory debugging tools like valgrind or address sanitizers.
A typical example would be a poorly-written cache that keeps all the cached results from the beginning of the program. It grows forever, but its memory is reclaimed at the end of the program, when the cache is destroyed.
How can one debug this ? Are there tools available to see where are the largest objects allocated by the program ? To dump the current state of allocated memory (including call stack) ? To see which objects are growing ? I'm using Linux, but I am interested in other platforms as well.
If other platforms are an option, I would recommend Visual Studio on Windows.
It has powerful profiling options, including one for memory usage.
https://learn.microsoft.com/en-us/visualstudio/profiling/memory-usage
While debugging you can take a snapshot to see where memory is being used.
You can also take memory usage snapshots at different times and compare them.
You can use a profiler like e.g. Intel VTune (this is available for Linux as well) to trace memory consumption of your application. In VTune, you can see the memory consumption over time, and select a time window to see where memory was allocated during that window.
It still will be difficult to detect such problems if your application allocates and deallocates a lot of memory correctly, and only a small fraction is deallocated too late. In that case you need to check a lot of allocations/deallocations before you can find the bad one(s).
I am trying valgrind for detecting memory leak. It works well on heap leak(i.e. memory allocation from malloc or new). however, does it support check mmap leaking in linux?
Thanks
Chang
Not directly, it's very hard to debug, take a look to valgrind.h
VALGRIND_MALLOCLIKE_BLOCK should be put immediately after the point where a
heap block -- that will be used by the client program -- is allocated.
It's best to put it at the outermost level of the allocator if possible;
for example, if you have a function my_alloc() which calls
internal_alloc(), and the client request is put inside internal_alloc(),
stack traces relating to the heap block will contain entries for both
my_alloc() and internal_alloc(), which is probably not what you want.
For Memcheck users: if you use VALGRIND_MALLOCLIKE_BLOCK to carve out
custom blocks from within a heap block, B, that has been allocated with
malloc/calloc/new/etc, then block B will be *ignored* during leak-checking
-- the custom blocks will take precedence.
VALGRIND_FREELIKE_BLOCK is the partner to VALGRIND_MALLOCLIKE_BLOCK. For
Memcheck, it does two things:
- It records that the block has been deallocated. This assumes that the
block was annotated as having been allocated via
VALGRIND_MALLOCLIKE_BLOCK. Otherwise, an error will be issued.
- It marks the block as being unaddressable.
VALGRIND_FREELIKE_BLOCK should be put immediately after the point where a
heap block is deallocated.
Sadly, valgrind's memcheck doesn't support mmap tracking (at least not out of the box), but there's hope.
I've recently came across valgrind-mmt, a valgrind fork for tracing mmap memory accesses and allocations:
https://nouveau.freedesktop.org/wiki/Valgrind-mmt
It is developed by envytools, and seems to be used mostly for graphic drivers developement.
The mmap tracing tool, mmt, does deep tracing on all access to mmapped memory, including loads and stores. This may be too much for the job of finding leaking mmap memory, and the output of the tool needs be processed and analyzed, but with some careful work it may be useful for detecting mmap leak scenarios. Personally, I've had only partial success using it, but perhaps others will be more lucky.
Note that, it may not be suitable for anonymous mmap allocations.
For getting started:
Clone and build the envytools/envytools repository
Clone and build the envytools/valgrind repository
Run valgrind with the following parameters:
/usr/local/bin/valgrind --tool=mmt --mmt-trace-file=[mmapped-file-to-be-traced] --log-file=mmt-log.bin
Decode mmt output: demmt -l mmt-log.bin > mmt-log.txt
I want to verify the memory stability of a C++ application I wrote and compiled for Linux.
It is a network application that responds to remote clients connectings in a rate of 10-20 connections per second.
On long run, memory was rising to 50MB, eventhough the app was making calls to delete...
Investigation shows that Linux does not immediately free memory. So here are my questions :
How can force Linux to free memory I actually freed? At least I want to do this once to verify memory stability.
Otherwise, is there any reliable memory indicator that can report memory my app is actually holding?
What you are seeing is most likely not a memory leak at all. Operating systems and malloc/new heaps both do very complex accounting of memory these days. This is, in general, a very good thing. Chances are any attempt on your part to force the OS to free the memory will only hurt both your application performance and overall system performance.
To illustrate:
The Heap reserves several areas of virtual memory for use. None of it is actually committed (backed by physical memory) until malloc'd.
You allocate memory. The Heap grows accordingly. You see this in task manager.
You allocate more memory on the Heap. It grows more.
You free memory allocated in Step 2. The Heap cannot shrink, however, because the memory in #3 is still allocated, and Heaps are unable to compact memory (it would invalidate your pointers).
You malloc/new more stuff. This may get tacked on after memory allocated in step #3, because it cannot fit in the area left open by free'ing #2, or because it would be inefficient for the Heap manager to scour the heap for the block left open by #2. (depends on the Heap implementation and the chunk size of memory being allocated/free'd)
So is that memory at step #2 now dead to the world? Not necessarily. For one thing, it will probably get reused eventually, once it becomes efficient to do so. In cases where it isn't reused, the Operating System itself may be able to use the CPU's Virtual Memory features (the TLB) to "remap" the unused memory right out from under your application, and assign it to another application -- on the fly. The Heap is aware of this and usually manages things in a way to help improve the OS's ability to remap pages.
These are valuable memory management techniques that have the unmitigated side effect of rendering fine-grained memory-leak detection via Process Explorer mostly useless. If you want to detect small memory leaks in the heap, then you'll need to use runtime heap leak-detection tools. Since you mentioned that you're able to build on Windows as well, I will note that Microsoft's CRT has adequate leak-checking tools built-in. Instructions for use found here:
http://msdn.microsoft.com/en-us/library/974tc9t1(v=vs.100).aspx
There are also open-source replacements for malloc available for use with GCC/Clang toolchains, though I have no direct experience with them. I think on Linux Valgrind is the preferred and more reliable method for leak-detection anyway. (and in my experience easier to use than MSVCRT Debug).
I would suggest using valgrind with memcheck tool or any other profiling tool for memory leaks
from Valgrind's page:
Memcheck
detects memory-management problems, and is aimed primarily at
C and C++ programs. When a program is run under Memcheck's
supervision, all reads and writes of memory are checked, and calls to
malloc/new/free/delete are intercepted. As a result, Memcheck can
detect if your program:
Accesses memory it shouldn't (areas not yet allocated, areas that have been freed, areas past the end of heap blocks, inaccessible areas
of the stack).
Uses uninitialised values in dangerous ways.
Leaks memory.
Does bad frees of heap blocks (double frees, mismatched frees).
Passes overlapping source and destination memory blocks to memcpy() and related functions.
Memcheck reports these errors as soon as they occur, giving the source
line number at which it occurred, and also a stack trace of the
functions called to reach that line. Memcheck tracks addressability at
the byte-level, and initialisation of values at the bit-level. As a
result, it can detect the use of single uninitialised bits, and does
not report spurious errors on bitfield operations. Memcheck runs
programs about 10--30x slower than normal. Cachegrind
Massif
Massif is a heap profiler. It performs detailed heap profiling by
taking regular snapshots of a program's heap. It produces a graph
showing heap usage over time, including information about which parts
of the program are responsible for the most memory allocations. The
graph is supplemented by a text or HTML file that includes more
information for determining where the most memory is being allocated.
Massif runs programs about 20x slower than normal.
Using valgrind is as simple as running application with desired switches and give it as an input of valgrind:
valgrind --tool=memcheck ./myapplication -f foo -b bar
I very much doubt that anything beyond wrapping malloc and free [or new and delete ] with another function can actually get you anything other than very rough estimates.
One of the problems is that the memory that is freed can only be released if there is a long contiguous chunk of memory. What typically happens is that there are "little bits" of memory that are used all over the heap, and you can't find a large chunk that can be freed.
It's highly unlikely that you will be able to fix this in any simple way.
And by the way, your application is probably going to need those 50MB later on when you have more load again, so it's just wasted effort to free it.
(If the memory that you are not using is needed for something else, it will get swapped out, and pages that aren't touched for a long time are prime candidates, so if the system runs low on memory for some other tasks, it will still reuse the RAM in your machine for that space, so it's not sitting there wasted - it's just you can't use 'ps' or some such to figure out how much ram your program uses!)
As suggested in a comment: You can also write your own memory allocator, using mmap() to create a "chunk" to dole out portions from. If you have a section of code that does a lot of memory allocations, and then ALL of those will definitely be freed later, to allocate all those from a separate lump of memory, and when it's all been freed, you can put the mmap'd region back into a "free mmap list", and when the list is sufficiently large, free up some of the mmap allocations [this is in an attempt to avoid calling mmap LOTS of times, and then munmap again a few millisconds later]. However, if you EVER let one of those memory allocations "escape" out of your fenced in area, your application will probably crash (or worse, not crash, but use memory belonging to some other part of the application, and you get a very strange result somewhere, such as one user gets to see the network content supposed to be for another user!)
Use valgrind to find memory leaks : valgrind ./your_application
It will list where you allocated memory and did not free it.
I don't think it's a linux problem, but in your application. If you monitor the memory usage with « top » you won't get very precise usages. Try using massif (a tool of valgrind) : valgrind --tool=massif ./your_application to know the real memory usage.
As a more general rule to avoid leaks in C++ : use smart pointers instead of normal pointers.
Also in many situations, you can use RAII (http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization) instead of allocating memory with "new".
It is not typical for an OS to release memory when you call free or delete. This memory goes back to the heap manager in the runtime library.
If you want to actually release memory, you can use brk. But that opens up a very large can of memory-management worms. If you directly call brk, you had better not call malloc. For C++, you can override new to use brk directly.
Not an easy task.
The latest dlmalloc() has a concept called an mspace (others call it a region). You can call malloc() and free() against an mspace. Or you can delete the mspace to free all memory allocated from the mspace at once. Deleting an mspace will free memory from the process.
If you create an mspace with a connection, allocate all memory for the connection from that mspace, and delete the mspace when the connection closes, you would have no process growth.
If you have a pointer in one mspace pointing to memory in another mspace, and you delete the second mspace, then as the language lawyers say "the results are undefined".
I got the Problem, that my application has an infinite growing Memory Leak, which is not detected. What I do very simplified, is to create an object, run a method on it and then delete the object. Each time I do this, the memory usage in the TaskManager grows by around 50-100MB. This exhausts my whole memory after some runs. I do this by multithreading, but there are no static variables, so there is no collision between the different objects in my threads. They only use static methods of other objects, that don't modify any other memory than passed in the parameters - so it's thread-safe.
What I tried to find out the reason:
Using crtdbg.h (CRT-Memeory-Leak-Detection), but there are only leaks which exist since the start of my application - they will get deleted at shutdown and they are not that big.
I was looking for the virtual destructors in all the objects that I inherit from, but they are all OK
What else can I try to find out where my application leaks? I can't find any leaks in the HEAP and I don't know any other reasons than the destructor problem that can cause Leaks in the STACK (by this I mean that an object doesn't destroy a local std::string objects which has allocated space in heap). I don't know if there are other reasons for "STACK-Leaks", but I know that in the parts of my method, where the memory grows the most, there are no HEAP allocation.
You probably want to use a nicer, more robust leak detector. You may also need to use a leak detector that can output a heap report at different times while your program is running. Finally, you should consider that your problem might be due to heap fragmentation rather than just a leak.
You can try Visual Leak Detector which is free from Google.
This question contains a list of other memory check products, from the basic to the quite advanced/expensive. CRTDBG is the lowest-common-denominator solution; I've had good luck with BoundsChecker, although it is not free.
Not sure how you have used CRTDBG library but it provides lots of goodies:
http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx
You can use _CrtMemCheckpoint in divide and conquer manner. It allows you to measure difference in memory use between two points in your code. With multithreading this can be difficult.
Another is _CrtDumpMemoryLeaks (which i suppose is executed anyway on app end) with _CRTDBG_MAP_ALLOC enabled, this should show exact location of memory allocations.
Another hint is that maybe you have your CRTDBG overconfigured, with lots of small allocations it can create huge internal memory structures.
Try switching off parts of your code, and check if problem persists.
If you build your app on daily basis, try running previous versions to spot where problem appeared, then compare changes in source code repository.
...
I am currently looking into an issue where an application is using alot of private memory for a C++ app. It looks like alot is in commited and reserve memory based on the dump analysis.
I use Windbg. Is there any way I can see what excactly is in commited and reserve memory? I have narrowed it down to one specific heap.
My theory is that it's not being released. I cannot live debug, I only have dumps to go on.
I have already used the command !address -filter and get a pretty little out put, but how do i move forward?
Any suggestions would help.
You can use the !heap -s command to get memory usage info in WinDbg. There is a tutorial on Leak Detection with windbg here.
There are few ways how you can diagnoze memory leaks:
Using heap stack trace, and then examining process dump in WinDbg
Taking snapshots of process memory state using UMDH tool.
The later option (UMDH tool) is part of WinDbg package and is usually the easiest option to investigate memory leaks. Both options are actually based on the same feature, which is an ability of NT heap to keep call stacks at the allocation along with the allocated entry.
Note that besides leaking memory in heaps, you might have other types of leaks that would result in increase of commited memory space. For example you might have called VirtualAlloc directly and forgot to VirtualFree it.