I research an embedded device that use GLIBC 2.25.
When I look at /proc/PID/maps I see under the heap section some anonymous sections ,I understand that sections create when the process use new
I dump those sections with dd and there is there interesting value that I want to understand is that buffer allocated or free, and what is the size of this buffer.
How can I do that please?
You can use the gdb (GNU Debugger) tool to inspect the memory of a running process. You can attach to the process using its PID and use the x command to examine memory at a specific address. You can also use the info proc mapping command to view the memory maps of the process, including the size of the heap. Additionally, you can use the heap command to list heap blocks and the malloc_info command to show detailed information about heap blocks.
You can also use the malloc_stats function to display information about the heap usage such as the number of bytes allocated, the number of bytes free and the number of bytes in use.
You can also use the pmap command to display the memory map of a process, including the heap size. This command is available on some systems and may not be present on others.
It's also worth noting that the /proc/PID/maps file can also give you an idea about the heap section of a process.
Please keep in mind that you need to have the right permission to access the process you want to inspect.
Instead of analyzing the memory from proc, you may want to try following options, limited to your env.
use tools like valgrind if you suspect any kind of leaks or invalid read/writes.
rather than looking at output of dd, attach to running process and inspect memory within process, gives you context to make sense of memory usage.
use logging to dump addresses of allocation/free/read/write. This allows you to build better understanding of memory usage.
You may have to use all of the above options depending upon the complexity of your task.
Related
I'm trying to iterate all the malloc_chunk in all arenas. (debugging based on core file, for memory leak and memory corruption investigation)
As i know each arena have top_chunk which point to the top chunk inside of one arena, based on top_chunk, inside of it, there's prev_size and size, based on the code (glibc/malloc/malloc.c):
I can get the previous continuous chunks, and then loop all the chunks in one arena. (i can statistic the chunks with the size and the number, which like WinDBG: !heap -stat -h) and also based on prev_size and size, i can check the chunk is corrupt or not.
In arena(malloc_state), there's a member variable: next which point to next arena. Then i can loop all the arena's chunks.
But i met a problem is if the chunk is not allocated, the prev_size is invalid, how to get the previous malloc_chunk?? Or this way is not correct.
Question Background:
The memory leak bug we have is memory leak reported in several online data node(our project is distributed storage cluster).
What we did and result:
We use valrgind to reproduce the bug in test cluster, but unfortunately we get nothing.
I tried to investigate more about the heap, tried to analyze the heap chunk and follow the way which i did before in WinDBG(which have very interesting heap commands to digger the memory leak and memory corruption), but i was blocked by the Question which i asked.
We use valgrind-massif to analyze the allocation(which i think it's very detail and interesting, could show which allocation takes how much memory). Massif show several clues, we follow this and check code, finally found the leak(a map is very huge, and in-proper usage of it, but i would erase in holder-class's destructor, that's why valgrind not report this).
I'll digger more about the gdb-heap source code to know more about glic malloc structure.
The free open source program https://github.com/vmware/chap does what you want here for glibc malloc. Just grab a core (either because the core crashed or grab a lib core by using gcore or using the generate command from within gdb). Then just open the core by doing:
chap yourCoreFileName
Once you get to the chap prompt, if you want to iterate through all the chunks, both free and not, you can do any of the following, depending on the verbosity you want, but keeping in mind that an "allocation" in chap does not contain the chunk header, but rather starts at the address returned by malloc.
Try any of the following:
count allocations
summarize allocations
describe allocations
show allocations
If you only care about allocations that are currently in use try any of the following:
count used
summarize used
describe used
show used
If you only care about allocations that are leaked try any of the following:
count leaked
summarize leaked
describe leaked
show leaked
More details are available in documentation available from the github URL mentioned above.
In terms of corruption, chap does some checking at startup and reports many kinds of corruption, although the output may be a bit cryptic at times.
First, before digging into the implementation details of malloc, your time may be better spent with a tool like valgrind or even run under the MALLOC_CHECK_ environment variable to let the internal heap consistency checking do the work for you.
But, since you asked....
glibc's malloc.c has some helpful comments about looking at the previous chunk.
Some particularly interesting ones are:
/* Note that we cannot even look at prev unless it is not inuse */
And:
If prev_inuse is set for any given chunk, then you CANNOT determine the size of the previous chunk, and might even get a memory addressing fault when trying to do so.
This is just a limitation of the malloc implimentation. When a previous chunk is in use, the footer that would store the size is used by the user-data of the allocation instead.
While it doesn't help your case, you can check whether a previous chunk is in use by following what the prev_inuse macro does.
#define PREV_INUSE 0x1
#define prev_inuse(p) ((p)->size & PREV_INUSE)
It checks the low-order bit of the current chunk's size. (All chunk sizes are divisible by 4 so the lower 2 bits can be used for status.) That would help you stop your iteration before going off into no-man's land.
Unfortunately, you'd still be terminating your loop early, before visiting every chunk.
If you really want to iterate over all chunks, I'd recommend that you start at malloc_state::top and follow the next_chunk until next_chunk points to the top.
Try pmap <PID> -XX command to trace down the memory usage from different aspects.
How to get the total memory, used memory, free memory from C++ code on Linux system?
Run your program through valgrind. For a program called foo, for example:
valgrind foo
It'll run the program in a harness that keeps track of memory use and print out that information after the program terminates.
If you don't have valgrind installed for some reason, you should be able to find it in your distro's package repository.
As commented by Chris Stratton, you can -on Linux- query a lot of system information in /proc/ so read proc(5); which contain textual pseudo-files (a bit like pipes) to be read sequentially. These are not real disk files so are read very quickly. You'll need to open and close them at every measurement.
From inside a process, you can query its address space in virtual memory using /proc/self/maps -and /proc/self/smaps; outside of that process, for another process of pid 1234, use /proc/1234/maps & /proc/1234/smaps; you can get system wide memory information thru /proc/meminfo
So try the following commands in a terminal:
cat /proc/meminfo
cat /proc/$$/maps
cat /proc/$$/smaps
cat /proc/self/maps
to understand more what they could give you.
Be aware that malloc and free (used by new and delete) are often requesting space (from the kernel) using syscalls like mmap(2) but are managing previously free-d memory to reuse it, so often don't release memory back to the kernel with munmap. Read wikipage on C memory management. In other words, the used heap is not accessible outside the process (since some unused, but reusable for future malloc-s, memory remains mmap-ed). See also mallinfo(3) and malloc_stats(3).
As Justin Lardinois answered, use valgrind to detect memory leaks.
Advanced Linux Programming is a good book to read. It has several related chapters.
I am trying valgrind for detecting memory leak. It works well on heap leak(i.e. memory allocation from malloc or new). however, does it support check mmap leaking in linux?
Thanks
Chang
Not directly, it's very hard to debug, take a look to valgrind.h
VALGRIND_MALLOCLIKE_BLOCK should be put immediately after the point where a
heap block -- that will be used by the client program -- is allocated.
It's best to put it at the outermost level of the allocator if possible;
for example, if you have a function my_alloc() which calls
internal_alloc(), and the client request is put inside internal_alloc(),
stack traces relating to the heap block will contain entries for both
my_alloc() and internal_alloc(), which is probably not what you want.
For Memcheck users: if you use VALGRIND_MALLOCLIKE_BLOCK to carve out
custom blocks from within a heap block, B, that has been allocated with
malloc/calloc/new/etc, then block B will be *ignored* during leak-checking
-- the custom blocks will take precedence.
VALGRIND_FREELIKE_BLOCK is the partner to VALGRIND_MALLOCLIKE_BLOCK. For
Memcheck, it does two things:
- It records that the block has been deallocated. This assumes that the
block was annotated as having been allocated via
VALGRIND_MALLOCLIKE_BLOCK. Otherwise, an error will be issued.
- It marks the block as being unaddressable.
VALGRIND_FREELIKE_BLOCK should be put immediately after the point where a
heap block is deallocated.
Sadly, valgrind's memcheck doesn't support mmap tracking (at least not out of the box), but there's hope.
I've recently came across valgrind-mmt, a valgrind fork for tracing mmap memory accesses and allocations:
https://nouveau.freedesktop.org/wiki/Valgrind-mmt
It is developed by envytools, and seems to be used mostly for graphic drivers developement.
The mmap tracing tool, mmt, does deep tracing on all access to mmapped memory, including loads and stores. This may be too much for the job of finding leaking mmap memory, and the output of the tool needs be processed and analyzed, but with some careful work it may be useful for detecting mmap leak scenarios. Personally, I've had only partial success using it, but perhaps others will be more lucky.
Note that, it may not be suitable for anonymous mmap allocations.
For getting started:
Clone and build the envytools/envytools repository
Clone and build the envytools/valgrind repository
Run valgrind with the following parameters:
/usr/local/bin/valgrind --tool=mmt --mmt-trace-file=[mmapped-file-to-be-traced] --log-file=mmt-log.bin
Decode mmt output: demmt -l mmt-log.bin > mmt-log.txt
I am currently looking into an issue where an application is using alot of private memory for a C++ app. It looks like alot is in commited and reserve memory based on the dump analysis.
I use Windbg. Is there any way I can see what excactly is in commited and reserve memory? I have narrowed it down to one specific heap.
My theory is that it's not being released. I cannot live debug, I only have dumps to go on.
I have already used the command !address -filter and get a pretty little out put, but how do i move forward?
Any suggestions would help.
You can use the !heap -s command to get memory usage info in WinDbg. There is a tutorial on Leak Detection with windbg here.
There are few ways how you can diagnoze memory leaks:
Using heap stack trace, and then examining process dump in WinDbg
Taking snapshots of process memory state using UMDH tool.
The later option (UMDH tool) is part of WinDbg package and is usually the easiest option to investigate memory leaks. Both options are actually based on the same feature, which is an ability of NT heap to keep call stacks at the allocation along with the allocated entry.
Note that besides leaking memory in heaps, you might have other types of leaks that would result in increase of commited memory space. For example you might have called VirtualAlloc directly and forgot to VirtualFree it.
I need to get the following to analyze a memory leak issue. How to do that?
Orphan Block Addresses Orphan Call
Stack
Are there any good resources/tools to know about/fix memory leaks.
Thanks
If you're on linux, use valgrind. It's your new best friend. I'm not sure what tools are available for Windows.
valgrind --leak-check=full
The Microsoft Application Verifier performs memory analysis similar to valgrind if you are on a Windows platform.
In Windows, you can use the MiniDumpWriteDump function in dbghelp.dll.
How to create minidump for my process when it crashes?
This can be very helpful in tracking down errors in deployed applications because you can use your debug symbols to inspect a minidump made in the field with no debug info. It's not very useful for tracking memory leaks, however.
For memory leaks under Windows (aside from commercial tools like Purify, BoundsChecker and GlowCode, of course) you can use WinDbg from the free Debugging Tools for Windows package, along with Win32 heap tags to track down the source of memory leaks.
http://www.codeproject.com/KB/cpp/MemoryLeak.aspx
http://blogs.msdn.com/alikl/archive/2009/02/15/identifying-memory-leak-with-process-explorer-and-windbg.aspx
Yes, as J. Paulett commented, at least on the Linux platform Valgrind is an excellent starting point.
On Windows I was able to get the necessary details using UIforETW, which is handling the necessary command line arguments for xperf.
This blog post explains everything in great detail: https://randomascii.wordpress.com/2015/04/27/etw-heap-tracingevery-allocation-recorded/
Recording
Step 1: A TracingFlags registry entry is created and set to ‘1’ in the Image File Execution Options for each process name that will be trace to tell the Windows heap to configure itself for tracing when a process with that name is launched. As is always the case with Image File Execution Options the options don’t affect already running processes – only processes launched when the registry key is set are affected.
Step 2: An extra ETW session is created using the “-heap -Pids 0” incantation. This session will record information from processes that had a TracingFlags registry entry of ‘1’ when they started.
The details are a bit messy but now that UIforETW is written I don’t have to bother explaining the details, and you don’t have to pretend to listen. If you want to record a heap trace use UIforETW, and if you want to know how it works then look at the code, or click the Show commands button to see most of the dirty laundry.
Analysis
The recording can be inspected with WPA (Windows Performance Analyzer) which can be conveniently launched from UIforETW.
The recommended columns are: Process, Handle, Type, Stack.
The allocation types are:
AIFO – Allocated Inside Freed Outside (hint, hint)
AOFI – Allocated Outside Freed Inside
AOFO – Allocated Outside Freed Outside
AIFI – Allocated Inside Freed Inside