Suppose I have a program written in C/C++ and I'd like to find out how much of memory was used for data (heap, stack) and how much of memory was used for code (libs, executable files, etc).
I have once measured the dynamic memory space used using 'valgrind' but I don't think is has a feature to profile memory footprint for data and code.
Platform : Mac (possibly Linux)
Your development environment should have some sort of linker options. Generally in such you can instruct it to create a link map. The information you are looking for is likely to be in the link map, or calculable based on the information in the link map.
Related
I am authoring a C++ program and find it consumes too much memory. I would like to know which part of the program consumes the most number of memory, ideally, I would like to know how much percentage of memory are consumed by what kind of C++ objects the program is using at a particular moment.
In Java, I know tools like Eclipse Memory Analyzer (https://www.eclipse.org/mat/) which could take a heap dump and show/visualize such memory usage, and I wonder if this can be done for a C++ program. For example, I expect to use a tool/approach letting me know a particular vector<shared_ptr<MyObject>> is holding 30% of the memory.
Note:
I develop the program mainly on macOS (compile using Apple Clang), so it will be better if the approach works on macOS. But I do deploy to Linux as well (compile using gcc) so approaches/tools on Linux is okay.
I tried using Apple's Intruments for such purpose, but so far I can only use it to find memory allocation issue. I have no idea how to figure out the memory consumption of the program at a particular moment (the memory consumption should be related with C++ objects in the program so that I can do some action to reduce it accordingly).
I don't find an easy way to visualize/summarize each part of my program's memory yet. So far, the best tool/approach that I found is Apple's Instruments (if you are on macOS).
By using Instruments, you can use Allocations profiling template. When using this profiling template, you can choose File ==> Recording Options ==> Check Discard events for freed memory option
And you will be able to figure out the un-free memory (aka. the data that are still in the memory) during allocation recording. If you have your program's debug symbol loaded, you can see which function leads to this result.
Although this doesn't address all the issues, it does help to identify part of the problem.
How to find heap memory size of a c++ program under linux platform ?I need heap memory space before the usage of new or malloc and also after that.can anyone help?
#include <malloc.h>
#include <iostream>
int main()
{
//here need heap memory space
unsigned char* I2C_Read_Data= new unsigned char[250];
//get heap memory space After the usage of new
return 0;
}
You can also add heap tracking to your own programs by overloading the new and delete operators. In a game engine I am working on, I have all memory allocation going through special functions, which attach each allocation to a particular heap tracker object. This way, at any given moment, I can pull up a report and see how much memory is being taken up by entities, actors, Lua scripts, etc.
It's not as thorough as using an external profiler (particularly when outside libraries handle their own memory management), but it is very nice for seeing exactly what memory you were responsible for.
Use valgrind's heap profiler: Massif
On Linux you can read /proc/[pid]/statm to get memory usage information.
Provides information about memory usage, measured in pages. The
columns are:
size total program size
(same as VmSize in /proc/[pid]/status)
resident resident set size
(same as VmRSS in /proc/[pid]/status)
share shared pages (from shared mappings)
text text (code)
lib library (unused in Linux 2.6)
data data + stack
dt dirty pages (unused in Linux 2.6)
See the man page for more details.
Answer by Adam Zalcman to this question describes some interesting details of the heap allocation
You can use the getrlimit function call and pass the RLIMIT_DATA for the resource. That should give you the size of the data segment for your program.
Apart from external inspection, you can also instrument your implementation of malloc to let you inspect those statistics. jemalloc and tcmalloc are implementations that, on top of performing better for multithreaded code that typical libc implementations, add some utility functions of that sort.
To dig deeper, you should learn a bit more how heap allocation works. Ultimately, the OS is the one assigning memory to processes as they ask for it, however requests to the OS (syscalls) are slower than regular calls, so in general an implementation of malloc will request large chunks to the OS (4KB or 8KB blocks are common) and the subdivise them to serve them to its callers.
You need to identify whether you are interested in the total memory consumed by the process (which includes the code itself), the memory the process requested from the OS within a particular procedure call, the memory actually in use by the malloc implementation (which adds its own book-keeping overhead, however small) or the memory you requested.
Also, fragmentation can be a pain for the latter two, and may somewhat blurs the differences between really used and assigned to.
You can try "mallinfo" and "malloc_info". They might work. mallinfo has issues when you allocate more than 2GB. malloc_info is o/s specific and notably very weird. I agree - very often it's nice to do this stuff without 3rd party tools.
I need to shave off as much memory as possible. I am using standard C++ with STL. The program does not do much (yet) and it still takes 960Kb [according to top]! The executabe size is only 64KB.
The code is 3000 lines long, I am not going to post obviously. I believe the problem is not with my code, but with the system libraries.
A single main() function (includes all my code but doesn't use it) uses 732Kb of RAM!
Simple Code:
int main() {
sleep(1000); //do nothing
return 0;
}
//Uses 732kb of RAM
My code has no global variables (apart from ones in libraries that are hidden from me).
I am using standard libraries: libstdc++ (STL), GNU libc. Also a single BSD socket and libev and the non-standard STL rope class.
Is there some memory-profiler I can run?
Platform: Linux 2.6.18-32, 32-bit processor, 16MB total system RAM, no swap available
Compiler: GCC 4
Standard Library: GCC's libstdc++
Compiler Options: -Os (no debugging symbols)
I am not making heavy use of templates: containers and iterators that's all. However I am making heavy use of the SGI STL rope class.
The test environment is a basic server running Linux with 128MB RAM, Pentium III 667 Mhz, CentOS 5.5, no emulation.
UPDATE: I am wondering if the libraries themselves (code size) are causing the problem. Doesn't shared libraries require to be loaded into RAM?
Start stripping out functionality until the memory usage goes down. Go extreme first -- if you can replace main with sleep(1000); and your memory use is still high, look to code and static data -- anything initialized at global scope or static inside a class or function, along with template instantiations of different types and debug symbols.
UPDATE: Removed incorrect commentary about STL allocators. It may apply to other compiler/STL versions (check the history if you want to see it) but it's not applicable to this question.
Be aware that malloc/operator new will often be stingy about giving free memory back to the OS, which will cause your program as a whole not to shrink its apparent usage over time; that memory will get reused throughout your program by future allocations, so it's usually not a huge issue aside from keeping your "memory use" numbers at or near their high-water mark indefinitely.
UPDATE: I am wondering if the
libraries themselves (code size) are
causing the problem. Doesn't shared
libraries require to be loaded into
RAM?
Bingo. On Mac OS X at least, Top includes the size of shared libraries in the physical memory usage. Only one copy of each library is resident in memory, of course.
Check the documentation for top for a workaround, or just chuck it and use malloc_info(). Be careful to find a way to account for code, stack, and global usage, though.
Get the linker to emit a link map file; you can use that to determine exactly how much statically linked code and static data space your code requires.
Stack, heap space, and shared libraries are additional to that, and are allocated at run-time.
If you have 16Mb of RAM does it really matter? It is likley that there is a relatively large but fixed overhead, and that your overall memory footprint will not grow linearly with lines of code added.
Since the target is a linux, I would think you could learn something about the details of memory usage, particularly shared library components by looking in the maps and smaps files in /proc/{pid_number}
I am aware of Valgrind, but it just detects memory management issues. What I am searching is a tool that gives me an overview, which parts of my program do consume how much memory. A graphical representation with e.g. a tree map (as KCachegrind does for Callgrind) would be cool.
I am working on a Linux machine, so windows tools will not help me very much.
Use massif, which is part of the Valgrind tools. massif-visualizer can help you graph the data or you can just use the ms_print command.
Try out the heap profiler delivered with gperftools, by Google. I've always built it from sources, but it's available as a precompiled package under several Linux distros.
It's as simple to use as linking a dynamic library to your executables and running the program. It collects information about every dynamic memory allocation (as far as I've seen) and save to disk a memory dump every time one of the following happens:
HEAP_PROFILE_ALLOCATION_INTERVAL bytes have been allocated by the program (default: 1Gb)
the high-water memory usage mark increases by HEAP_PROFILE_INUSE_INTERVAL bytes (default: 100Mb)
HEAP_PROFILE_TIME_INTERVAL seconds have elapsed (default: inactive)
You explicitly call HeapProfilerDump() from your code
The last one, in my experience, is the most useful because you can control exactly when to have a snapshot of the heap usage and then compare two different snapshots and see what's wrong.
Eventually, there are several possible output formats, like textual or graphical (in the form of a directed graph):
Using this tool I've been able to spot incorrect memory usages that I couldn't find using Massif.
A "newer" option is HeapTrack. Contrary to massif, it is an instrumented version of malloc/free that stores all the calls and dumps a log.
The GUI is nice (but requires Qt5 IIRC) and the results timings (because you may want to track time as well) are less biased than valgrind (as they are not emulated).
Use callgrind option with valgrind
I'm trying to take a snapshot of the memory used by a large application running on both Unix / Windows. My ultimate aim would be to have a kind of chart breaking down the memory used by which area of code.
The program is split into about 30 different projects most of which are either static libraries or dynamic dlls. Some of these are written in C, some C++ and others a mixture of the two. In total the code across all projects is about 600,000 lines.
With the heap I could try and overload every 'malloc/free' and 'new/delete' across all projects and track it that way, but that is quite daunting with an application this size.
Also, that wouldn't pick up all the static global data littered around the projects too.
Thanks for any help.
You could give valgrind a try.
Here is a quote about one of the tools:
Massif
Massif is a heap profiler. It performs detailed heap profiling by taking regular snapshots of a program's heap. It produces a graph showing heap usage over time, including information about which parts of the program are responsible for the most memory allocations. The graph is supplemented by a text or HTML file that includes more information for determining where the most memory is being allocated. Massif runs programs about 20x slower than normal.
It supports on Linux now, but if doing the analysis on Linux and applying the results to the Windows version works for you this might help you.
If you are working with ELF binaries you could check the object files "*.o" before linking with an elf analyzer and see how big are the static memory sections and the size the bss(non-initialized static data) will have once loaded.