I am trying to improve the performance of my C++ program and I found that converting memory allocations (mallocs) into object pool is giving great results.
The problem is detecting the places from which malloc is called, since the code base is quite large. I can't use simple gdb with break points because there are many timers and signal handlers running in parallel.
Is there a way in gdb using which I can print the entire stack trace whenever malloc is called without having to do it manually each time.
OR
Can you suggest any other tool which will help me do the same.
You can script gdb using Python.
You can also implement your own malloc function and link with that. The return address will be on the stack, which will give you the caller.
The valgrind suite of tools contains massif which you can use for precisely this purpose:
valgrind --tool=massif ./mybinary
This collects details of all allocations including stack traces that you can examine after the program has finished executing. Please see the massif documentation for more details on the output: http://valgrind.org/docs/manual/ms-manual.html. Hope that helps.
P.S. Also checkout the TCMalloc library - it already possibly already does what you want, although you can do better depending on your specific application. The best thing is that no source code changes are needed - you simply replace the malloc function from glibc using a linker directive.
Related
I am authoring a C++ program and find it consumes too much memory. I would like to know which part of the program consumes the most number of memory, ideally, I would like to know how much percentage of memory are consumed by what kind of C++ objects the program is using at a particular moment.
In Java, I know tools like Eclipse Memory Analyzer (https://www.eclipse.org/mat/) which could take a heap dump and show/visualize such memory usage, and I wonder if this can be done for a C++ program. For example, I expect to use a tool/approach letting me know a particular vector<shared_ptr<MyObject>> is holding 30% of the memory.
Note:
I develop the program mainly on macOS (compile using Apple Clang), so it will be better if the approach works on macOS. But I do deploy to Linux as well (compile using gcc) so approaches/tools on Linux is okay.
I tried using Apple's Intruments for such purpose, but so far I can only use it to find memory allocation issue. I have no idea how to figure out the memory consumption of the program at a particular moment (the memory consumption should be related with C++ objects in the program so that I can do some action to reduce it accordingly).
I don't find an easy way to visualize/summarize each part of my program's memory yet. So far, the best tool/approach that I found is Apple's Instruments (if you are on macOS).
By using Instruments, you can use Allocations profiling template. When using this profiling template, you can choose File ==> Recording Options ==> Check Discard events for freed memory option
And you will be able to figure out the un-free memory (aka. the data that are still in the memory) during allocation recording. If you have your program's debug symbol loaded, you can see which function leads to this result.
Although this doesn't address all the issues, it does help to identify part of the problem.
I'm using CentOS 7 and I'm running a C++ Application. Recently I switched to a newer version of a library which the application was using for various MySQL C API functions. But after integrating the new library, I saw a tremendous increase in memory usage of the program i.e. the application crashes if left running for more than a day or two. Precisely, what happens is the memory usage for the application starts increasing upto a point where the application alone is using 74.9% of total memory of the system and then it is forcefully shut down by the system.
Is there any way of how to track memory usage of the whole application including the static variables as well. I've already tried valgrind's tool Massif.
Can anyone tell me what could be the possible reasons for the increased memory usage or any tools that can give me a deep insight of how the memory is being allocated (both static and dynamic). Is there any tool which can tell us about Memory Allocation for a C++ Application running in a linux environment?
Thanks in advance!
Static memory is allocate when the program starts. Are you seeing memory growth or a startup increase?
Since it takes 'a day or two to crash', the trouble is likely a memory leak or unbounded growth of a data structure. Valgrind should be able to help with both. If valgrind shows a big leak with the --leak-check-full option then you will likely have found the issue.
To check for unbounded growth, put a preemptive _exit() in the program at a point where you suspect the heap has grown. For example, put a timer on the main loop and have the program _exit after 10 minutes. If the valgrind shows a big 'in use at exit' then you likely have unbounded growth of a data structure but not a leak. Massif can help track this down. The ms_print gives details of allocations with function stack.
If you find an issue, try switching back to the older version of your library. If the problem goes away, check and make sure you are using the API properly in the new version. If you don't have the source code then you are a bit stuck in terms of a fix.
If you want to go the extra mile, you can write a shared library interposer for malloc/free to see what is happening. Here is a good start. Linux has the backtrace functionality that can help with determining the exact stack.
Finally, if you must use the 3rd party library and find the heap growing without bound or leaking then you can use the shared library interposer to directly call free/delete. This is a risky last-ditch unrecommended strategy but I've used in production to limp a process along.
I have a c++ program that dies because of out of memory error.
Do you know a tool for c++ program that shows which program line allocated how much heap?
I would like to figure which part of the program consumes most of the heap.
Thanks.
Platform: Microsoft C++...Windows
By the way, can heap corruption cause excessive memory usage?
Solution:
It was not heap corruption. Some uninitialised variables were used to allocated the amount of memory.
I used valgrind which showed the problem immediately (both the presence of huge allocations and uninitilased variables).
You could use something like valgrind on *nix platform or crtdbg checks on windows platform.
It sounds like you want a profiling tool of some sort. A quick Google search brought up this Stack Overflow question:
What's your favorite profiling tool (for C++)
(It says "c" in the URL, but the question is about C++.)
In visual studio, if you get an exception like that while debugging, the code will break at the point of the exception. Then you can navigate up the call stack to see exactly where you are in every function up to that point. I assume something similar would hold for other platforms.
Paul Nettle MMGR You find a "mmgr" there, which I have used some time ago. Its very useful IMHO.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have a small doubt regarding profiling applications which never exit until we manually reboot the machine.
I used tools like valgrind which talks about memory leaks or bloating of any application which exits after sometime.
But is there any tool which can be used to tell about memory consumption, bloating, overhead created by the application at various stages if possible?
NOTE: I am more intrested to know about apps which dont exit ... If an app exits I can use tools like valgrind ..
I'd consider adding a graceful exit from the program.
dtrosset's point is well put but apparently misunderstood. Add a means to terminate the program so you can perform a clean analysis. This can be something as simple as adding a signal handler for SIGUSR1, for example, that terminates the program at a point in time you decide. There are a variety of methods at your disposal depending on your OS.
There's a big difference between an application which never exits (embedded, daemons, etc) and one that cannot be exited. The prior is normal, the latter is bad design.
If anything, that application can be forcibly aborted (SIGKILL on *nix, terminate on win32) and you'd get your analysis. That method doesn't give your application the opportunity to clean up before it's destroyed so there will be very likely be retained memory reported.
Profiling is intrusive, so you don't want to deploy the app with the profiler attached, anyway. Therefore, include some #ifdef PROFILE_MODE-code that exits the app after an appropriate amount of time. Compile with -DPROFLILE_MODE, profile. Deploy without PROFILE_MODE.
Modify your program slightly so that you can request a Valgrind leak check at any point - when the command to do that is recieved, your program should use VALGRIND_DO_LEAK_CHECK from memcheck.h (this will have no effect if the program isn't running under Valgrind).
You can use GNU gprof, but it has also the problem that it requires an exit of the program.
You can overcom this by calling internal functions of gprof. (see below) It may be a real "dirty" hack, depending on the version of gcc and, and, and,... but it works.
#include "sys/gmon.h"
extern "C" //the internal functions and vars of gprof
{
void moncontrol (int mode);
void monstartup (unsigned long lowpc, unsigned long highpc);
void _mcleanup (void);
extern void _start(void), etext(void);
extern int __libc_enable_secure;
}
// call this whenever you want to write profiling information to file
void WriteProfilingInformation(char* Name)
{
setenv("GMON_OUT_PREFIX",Name,1); // set file name
int old = __libc_enable_secure; // save old value
__libc_enable_secure = 0; // has to be zero to change profile file name
_mcleanup();
__libc_enable_secure = old; // reset to old value
monstartup(lowpc, highpc); // restart profiler
moncontrol(1); // enable profiler
}
Rational Purify can do that, at least on windows. There seem to be a linux version but I don't know if it can do the same.
Some tools allow you to force a memory analysis at any point during the program's execution. This method is not as reliable as checking on exit, but it gives you a starting point.
Here's a Windows example using LeakDiag.
Have you tried GNU Gprof?
Note that in this document, "cc" and "gcc" are interchangeable. ("cc" is assumed as an alias for "gcc.")
http://www.cs.utah.edu/dept/old/texinfo/as/gprof_toc.html
Your question reads as if you were looking for top. It nicely displays (among other things) the current memory consumption of all running processes. (Limited to one page in the terminal.) On Linux, hit “M” to sort by memory usage. The man page shows more options for sorting and filtering.
I have used rational purify API's to check incremental leaks. Haven't used the API's in linux. I found the VALGRIND_DO_LEAK_CHECK option in Valgrind User Manual, I think this would suffice your requirement.
For windows, DebugDiag does that.
Generates a report in the end with probable memory leaks.
Also has memory pressure analysis.
And it's available for free # microsoft. Download it from here
You need stackshots. Either use pstack or lsstack, or just run it under a debugger or IDE and pause it (Ctrl-C) at random. It will not tell you about memory leaks, but it will give you a good idea of how the time is being used and why.
If time is being used because of memory leaks, you will see a good percent of samples ending in memory management routines. If they are in malloc or new, higher up the stack you will see what objects are being allocated and why, and you can consider how to do that less often.
Work of program that profiling memory leaks is based on detecting memory that was freed by OS not by program.
I am debugging a (native) multi-threaded C++ application under Visual Studio 2008. On seemingly random occasions, I get a "Windows has triggered a break point..." error with a note that this might be due to a corruption in the heap. These errors won't always crash the application right away, although it is likely to crash short after.
The big problem with these errors is that they pop up only after the corruption has actually taken place, which makes them very hard to track and debug, especially on a multi-threaded application.
What sort of things can cause these errors?
How do I debug them?
Tips, tools, methods, enlightments... are welcome.
Application Verifier combined with Debugging Tools for Windows is an amazing setup. You can get both as a part of the Windows Driver Kit or the lighter Windows SDK. (Found out about Application Verifier when researching an earlier question about a heap corruption issue.) I've used BoundsChecker and Insure++ (mentioned in other answers) in the past too, although I was surprised how much functionality was in Application Verifier.
Electric Fence (aka "efence"), dmalloc, valgrind, and so forth are all worth mentioning, but most of these are much easier to get running under *nix than Windows. Valgrind is ridiculously flexible: I've debugged large server software with many heap issues using it.
When all else fails, you can provide your own global operator new/delete and malloc/calloc/realloc overloads -- how to do so will vary a bit depending on compiler and platform -- and this will be a bit of an investment -- but it may pay off over the long run. The desirable feature list should look familiar from dmalloc and electricfence, and the surprisingly excellent book Writing Solid Code:
sentry values: allow a little more space before and after each alloc, respecting maximum alignment requirement; fill with magic numbers (helps catch buffer overflows and underflows, and the occasional "wild" pointer)
alloc fill: fill new allocations with a magic non-0 value -- Visual C++ will already do this for you in Debug builds (helps catch use of uninitialized vars)
free fill: fill in freed memory with a magic non-0 value, designed to trigger a segfault if it's dereferenced in most cases (helps catch dangling pointers)
delayed free: don't return freed memory to the heap for a while, keep it free filled but not available (helps catch more dangling pointers, catches proximate double-frees)
tracking: being able to record where an allocation was made can sometimes be useful
Note that in our local homebrew system (for an embedded target) we keep the tracking separate from most of the other stuff, because the run-time overhead is much higher.
If you're interested in more reasons to overload these allocation functions/operators, take a look at my answer to "Any reason to overload global operator new and delete?"; shameless self-promotion aside, it lists other techniques that are helpful in tracking heap corruption errors, as well as other applicable tools.
Because I keep finding my own answer here when searching for alloc/free/fence values MS uses, here's another answer that covers Microsoft dbgheap fill values.
You can detect a lot of heap corruption problems by enabling Page Heap for your application . To do this you need to use gflags.exe that comes as a part of Debugging Tools For Windows
Run Gflags.exe and in the Image file options for your executable, check "Enable Page Heap" option.
Now restart your exe and attach to a debugger. With Page Heap enabled, the application will break into debugger whenever any heap corruption occurs.
To really slow things down and perform a lot of runtime checking, try adding the following at the top of your main() or equivalent in Microsoft Visual Studio C++
_CrtSetDbgFlag(_CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF | _CRTDBG_CHECK_ALWAYS_DF );
A very relevant article is Debugging Heap corruption with Application Verifier and Debugdiag.
What sort of things can cause these errors?
Doing naughty things with memory, e.g. writing after the end of a buffer, or writing to a buffer after it's been freed back to the heap.
How do I debug them?
Use an instrument which adds automated bounds-checking to your executable: i.e. valgrind on Unix, or a tool like BoundsChecker (Wikipedia suggests also Purify and Insure++) on Windows.
Beware that these will slow your application, so they may be unusable if yours is a soft-real-time application.
Another possible debugging aid/tool might be MicroQuill's HeapAgent.
One quick tip, that I got from Detecting access to freed memory is this:
If you want to locate the error
quickly, without checking every
statement that accesses the memory
block, you can set the memory pointer
to an invalid value after freeing the
block:
#ifdef _DEBUG // detect the access to freed memory
#undef free
#define free(p) _free_dbg(p, _NORMAL_BLOCK); *(int*)&p = 0x666;
#endif
The best tool I found useful and worked every time is code review (with good code reviewers).
Other than code review, I'd first try Page Heap. Page Heap takes a few seconds to set up and with luck it might pinpoint your problem.
If no luck with Page Heap, download Debugging Tools for Windows from Microsoft and learn to use the WinDbg. Sorry couldn't give you more specific help, but debuging multi-threaded heap corruption is more an art than science. Google for "WinDbg heap corruption" and you should find many articles on the subject.
What type of allocation functions are you using? I recently hit a similar error using the Heap* style allocation functions.
It turned out that I was mistakenly creating the heap with the HEAP_NO_SERIALIZE option. This essentially makes the Heap functions run without thread safety. It's a performance improvement if used properly but shouldn't ever be used if you are using HeapAlloc in a multi-threaded program [1]. I only mention this because your post mentions you have a multi-threaded app. If you are using HEAP_NO_SERIALIZE anywhere, delete that and it will likely fix your problem.
[1] There are certain situations where this is legal, but it requires you to serialize calls to Heap* and is typically not the case for multi-threaded programs.
If these errors occur randomly, there is high probability that you encountered data-races. Please, check: do you modify shared memory pointers from different threads? Intel Thread Checker may help to detect such issues in multithreaded program.
You may also want to check to see whether you're linking against the dynamic or static C runtime library. If your DLL files are linking against the static C runtime library, then the DLL files have separate heaps.
Hence, if you were to create an object in one DLL and try to free it in another DLL, you would get the same message you're seeing above. This problem is referenced in another Stack Overflow question, Freeing memory allocated in a different DLL.
In addition to looking for tools, consider looking for a likely culprit. Is there any component you're using, perhaps not written by you, which may not have been designed and tested to run in a multithreaded environment? Or simply one which you do not know has run in such an environment.
The last time it happened to me, it was a native package which had been successfully used from batch jobs for years. But it was the first time at this company that it had been used from a .NET web service (which is multithreaded). That was it - they had lied about the code being thread safe.
You can use VC CRT Heap-Check macros for _CrtSetDbgFlag: _CRTDBG_CHECK_ALWAYS_DF or _CRTDBG_CHECK_EVERY_16_DF.._CRTDBG_CHECK_EVERY_1024_DF.
I'd like to add my experience. In the last few days, I solved an instance of this error in my application. In my particular case, the errors in the code were:
Removing elements from an STL collection while iterating over it (I believe there are debug flags in Visual Studio to catch these things; I caught it during code review)
This one is more complex, I'll divide it in steps:
From a native C++ thread, call back into managed code
In managed land, call Control.Invoke and dispose a managed object which wraps the native object to which the callback belongs.
Since the object is still alive inside the native thread (it will remain blocked in the callback call until Control.Invoke ends). I should clarify that I use boost::thread, so I use a member function as the thread function.
Solution: Use Control.BeginInvoke (my GUI is made with Winforms) instead so that the native thread can end before the object is destroyed (the callback's purpose is precisely notifying that the thread ended and the object can be destroyed).
I had a similar problem - and it popped up quite randomly. Perhaps something was corrupt in the build files, but I ended up fixing it by cleaning the project first then rebuilding.
So in addition to the other responses given:
What sort of things can cause these errors?
Something corrupt in the build file.
How do I debug them?
Cleaning the project and rebuilding. If it's fixed, this was likely the problem.
I have also faced this issue. In my case, I allocated for x size memory and appended the data for x+n size. So, when freeing it shown heap overflow. Just make sure your allocated memory sufficient and check for how many bytes added in the memory.