I am authoring a C++ program and find it consumes too much memory. I would like to know which part of the program consumes the most number of memory, ideally, I would like to know how much percentage of memory are consumed by what kind of C++ objects the program is using at a particular moment.
In Java, I know tools like Eclipse Memory Analyzer (https://www.eclipse.org/mat/) which could take a heap dump and show/visualize such memory usage, and I wonder if this can be done for a C++ program. For example, I expect to use a tool/approach letting me know a particular vector<shared_ptr<MyObject>> is holding 30% of the memory.
Note:
I develop the program mainly on macOS (compile using Apple Clang), so it will be better if the approach works on macOS. But I do deploy to Linux as well (compile using gcc) so approaches/tools on Linux is okay.
I tried using Apple's Intruments for such purpose, but so far I can only use it to find memory allocation issue. I have no idea how to figure out the memory consumption of the program at a particular moment (the memory consumption should be related with C++ objects in the program so that I can do some action to reduce it accordingly).
I don't find an easy way to visualize/summarize each part of my program's memory yet. So far, the best tool/approach that I found is Apple's Instruments (if you are on macOS).
By using Instruments, you can use Allocations profiling template. When using this profiling template, you can choose File ==> Recording Options ==> Check Discard events for freed memory option
And you will be able to figure out the un-free memory (aka. the data that are still in the memory) during allocation recording. If you have your program's debug symbol loaded, you can see which function leads to this result.
Although this doesn't address all the issues, it does help to identify part of the problem.
I have a program on XCode's written with C++.
Some builds my memory consumption skyrockets out of nowhere, while other builds the memory stays constant and works as expected.
Are there any built in debugging features with XCodes that will allow me to view memory consumption on a per variable or per object basis?
Thanks in advance.
You should first look into using Instruments (wiki). You can run Instruments in Xcode using ⌘+I. It is built on top of DTrace.
DTrace background
Why is that important? DTrace lets you ask really interesting, specific questions about the runtime behaviors of your application, whether during debugging or in production.
The beauty of DTrace is in the providers, as well as the scripts (written in the D scripting language - not the same thing as dlang). Many languages have DTrace providers that allow you to probe into their standard libraries. There is also a syscall provider that lets you see when applications are hitting lower level system calls. Some third party applications have their own application-specific DTrace providers, often along with handy scripts you can use. Or you can create your own (see below).
Objc/C++ refresher
Keep in mind that both Objective-C and C++ essentially coexist with and extend C. So your new and delete in C++ are really just malloc and free with some size calculations done for you based on the class you're working with. In Objective-C, it's more or less the same story under the hood - things must be malloc'd onto the heap. So whether you're writing C, C++, or Objective-C code (or even Swift code!) there is some common ground we can rely on.
Instruments
Getting back to the issue at hand, Instruments simplifies using DTrace by offering a GUI with shortcuts to handy, pre-made scripts. You don't even have to see the scripts. For example, you can launch Instruments and tell it to monitor memory allocations or leaks, and suddenly you get a rolling graph of memory usage in your application.
Once you launch Instruments and start recording a session with the Memory Allocation tool, you should be looking at the "persistent bytes" column (used to be "live bytes"). This is a good starting point. This number will increase and decrease. You'll typically see a net increase over time, but you should see it decrease whenever you're expecting a deallocation to occur - the larger the deallocation, the easier it is to notice. If you are not seeing decreases at those moments, then you'll want to explore your code for bad memory management.
Taking It Further
You can even create custom probes.
Here's another in-depth article that describes what you're talking about, with tracking malloc/free usage from specific lines of your code.
my program is running and creating variables, I need to know what's the total of Bytes these variables take.
I don't want to know how much is the physical memory space that the system gives my program to be executed, I know I can open the processes manager and find out.
I neither want to write into my code some sizeof and agregations so I can know the total size of the variable pool (let say the code is too complex to be modify like that).
Finally I'm using Microsoft VC++ 2010 Express, I just want to know if there is a workspace which monitor that kind of information.
Thanks in advance.
Check this out: Memory Performance Information . There are few metrics of a running process you might be interested in, you will primarily want private bytes, and this data is available both programmatically or through tools like Performance Monitor. You can also enumerate heaps of the process with GetProcessHeaps (and even HeapWalk if you need details) and check heap allocation sizes directly.
Valgrind Massif profiler is a great tool (see here ) but only for Unix/Linux I think. In your case, on Windows I think Insure++ or softwareverify are good choices (they are commercial tools).
A free alternative is Google's tcmalloc which provides a heap profiler here
I've just restarted my firefox web browser again because it started stuttering and slowing down. This happens every other day due to (my understanding) of excessive memory usage.
I've noticed it takes 40M when it starts and then, by the time I notice slow down, it goes to
1G and my machine has nothing more to offer unless I close other applications.
I'm trying to understand the technical reasons behind why its such a difficult problem to sol
ve.
Mozilla have a page about high memory usage:
http://support.mozilla.com/en-US/kb/High+memory+usage
But I'm looking for a slightly more in depth and satisfying explanation. Not super technical but enough to give the issue more respect and please the crowd here.
Some questions I'm already pondering (they could be silly so take it easy):
When I close all tabs, why doesn't the memory usage go all the way down?
Why is there no limits on extensions/themes/plugins memory usage?
Why does the memory usage increase if it's left open for long periods of time?
Why are memory leaks so difficult to find and fix?
App and language agnostic answers also much appreciated.
Browsers are like people - they get old, they get bloated, and they get ditched for younger and leaner models.
Firefox is not just a browser, it's an ecosystem.
While I feel that recent versions are quite bloated, the core product is generally stable.
However, firefox is an ecosystem/platform for:
1) Badly written plug-ins
2) Badly written JavaScript code that executes within it.
3) Adobe flash as a platform for heavyweight video and for poorly written ad scripts such as 'hit Osama bin Laden with a duck to reduce your mortgage rate and receive a free iPod* (participation required).
4) Quicktime and other media player.
5) Some embedded Java code.
The description of a memory leak suggests a script running amok or a third-party tool requesting more memory. If you ever run Flash on a Mac, that's almost a given along with 90% CPU utilization.
The goal of most programming languages is not to save you but to give you tools to save yourself. You can write bad and bloated code with memory leaks in any language, including ones with garbage collection. Third party tools are usually not as well tested as the platform itself. Web pages that try to do too much are also not uncommon.
If you want to do an experiment to demonstrate this, get a mac with Firefox and go to a well-written site like Stack Overflow and spend an hour. Your memory usage shouldn't grow much. Then spend 5 minutes visiting random pages on Myspace.
Now let me try and answer your questions based on my guesses since I'm not familiar with the source code
When I close all tabs, why
doesn't the memory usage go all the
way down?
Whereas each browser instance is an independent process with its own memory, the tabs in a single window are all within the same process. Firefox used to have some sort of in-memory caching and merely closing a tab doesn't clear the relevant information immediately from the in-memory cache. If you reopened a tab to the same site, you might get better performance. There was some advanced option to allow you to disable it, something like browser.cache.memory.enable. Or just search for disabling the memory cache.
* Why is there no limits on extensions/themes/plugins memory usage?
For the same reason that Windows or Linux doesn't have a vetting process on applications you can run on them. It's an open environment and you assume the risk. If you want an environment where applications and extensions are 'validated', Apple might be the way to go :)
* Why does the memory usage increase if it's left open for long periods of time?
Not all calculations and actions in a script have visual manifestations. A script could be doing some stuff in the background (like requesting extra materials, pre-fetching stuff, just bugs) even if you don't see it.
* Why are memory leaks so difficult to find and fix?
It's about bookkeeping. Think about every item you ever borrowed (even a pen) or that someone borrowed from you in your entire life. Are they all accounted for? Memory leaks are the same way (you borrow memory from the system), except that you pass items around. Then look at the stuff on your desk, did you leave anything lying around because 'you might need it soon' even though you probably won't? same story.
Why are memory leaks so difficult to find and fix?
Because some developers refuse to use tools like Electric Fence.
Memory leaks are present in the first place because you want to keep things in memory and not on disk. For example, suppose you have a web page, which have images, CSS, JavaSript, text. If to display the page you will go to the hard disk every time you want to use the JavaScript interpreter or a CSS parser, or a font rendering engine to display a text with, then the browser will be very slow and sometimes won't work at all (because one JavaScript piece might need variables present which are left by another JavaScript piece, for example). Therefore, a browser tries to keep all things necessary for its work in memory, and those things get cross-referenced easily (JavaScript calling into Adobe Flash, Adobe Flash calling into JavaScript and so on). And you have to be very careful with such resource references, because cleaning them prematurely and out of order will break the code (better to keep a resource around then to die suddenly because it isn't there).
P.S. See also this article for some gory details.
I'm working on a multithreaded C++ application that is corrupting the heap. The usual tools to locate this corruption seem to be inapplicable. Old builds (18 months old) of the source code exhibit the same behaviour as the most recent release, so this has been around for a long time and just wasn't noticed; on the downside, source deltas can't be used to identify when the bug was introduced - there are a lot of code changes in the repository.
The prompt for crashing behaviuor is to generate throughput in this system - socket transfer of data which is munged into an internal representation. I have a set of test data that will periodically cause the app to exception (various places, various causes - including heap alloc failing, thus: heap corruption).
The behaviour seems related to CPU power or memory bandwidth; the more of each the machine has, the easier it is to crash. Disabling a hyper-threading core or a dual-core core reduces the rate of (but does not eliminate) corruption. This suggests a timing related issue.
Now here's the rub:
When it's run under a lightweight debug environment (say Visual Studio 98 / AKA MSVC6) the heap corruption is reasonably easy to reproduce - ten or fifteen minutes pass before something fails horrendously and exceptions, like an alloc; when running under a sophisticated debug environment (Rational Purify, VS2008/MSVC9 or even Microsoft Application Verifier) the system becomes memory-speed bound and doesn't crash (Memory-bound: CPU is not getting above 50%, disk light is not on, the program's going as fast it can, box consuming 1.3G of 2G of RAM). So, I've got a choice between being able to reproduce the problem (but not identify the cause) or being able to idenify the cause or a problem I can't reproduce.
My current best guesses as to where to next is:
Get an insanely grunty box (to replace the current dev box: 2Gb RAM in an E6550 Core2 Duo); this will make it possible to repro the crash causing mis-behaviour when running under a powerful debug environment; or
Rewrite operators new and delete to use VirtualAlloc and VirtualProtect to mark memory as read-only as soon as it's done with. Run under MSVC6 and have the OS catch the bad-guy who's writing to freed memory. Yes, this is a sign of desperation: who the hell rewrites new and delete?! I wonder if this is going to make it as slow as under Purify et al.
And, no: Shipping with Purify instrumentation built in is not an option.
A colleague just walked past and asked "Stack Overflow? Are we getting stack overflows now?!?"
And now, the question: How do I locate the heap corruptor?
Update: balancing new[] and delete[] seems to have gotten a long way towards solving the problem. Instead of 15mins, the app now goes about two hours before crashing. Not there yet. Any further suggestions? The heap corruption persists.
Update: a release build under Visual Studio 2008 seems dramatically better; current suspicion rests on the STL implementation that ships with VS98.
Reproduce the problem. Dr Watson will produce a dump that might be helpful in further analysis.
I'll take a note of that, but I'm concerned that Dr Watson will only be tripped up after the fact, not when the heap is getting stomped on.
Another try might be using WinDebug as a debugging tool which is quite powerful being at the same time also lightweight.
Got that going at the moment, again: not much help until something goes wrong. I want to catch the vandal in the act.
Maybe these tools will allow you at least to narrow the problem to certain component.
I don't hold much hope, but desperate times call for...
And are you sure that all the components of the project have correct runtime library settings (C/C++ tab, Code Generation category in VS 6.0 project settings)?
No I'm not, and I'll spend a couple of hours tomorrow going through the workspace (58 projects in it) and checking they're all compiling and linking with the appropriate flags.
Update: This took 30 seconds. Select all projects in the Settings dialog, unselect until you find the project(s) that don't have the right settings (they all had the right settings).
My first choice would be a dedicated heap tool such as pageheap.exe.
Rewriting new and delete might be useful, but that doesn't catch the allocs committed by lower-level code. If this is what you want, better to Detour the low-level alloc APIs using Microsoft Detours.
Also sanity checks such as: verify your run-time libraries match (release vs. debug, multi-threaded vs. single-threaded, dll vs. static lib), look for bad deletes (eg, delete where delete [] should have been used), make sure you're not mixing and matching your allocs.
Also try selectively turning off threads and see when/if the problem goes away.
What does the call stack etc look like at the time of the first exception?
I have same problems in my work (we also use VC6 sometimes). And there is no easy solution for it. I have only some hints:
Try with automatic crash dumps on production machine (see Process Dumper). My experience says Dr. Watson is not perfect for dumping.
Remove all catch(...) from your code. They often hide serious memory exceptions.
Check Advanced Windows Debugging - there are lots of great tips for problems like yours. I recomend this with all my heart.
If you use STL try STLPort and checked builds. Invalid iterator are hell.
Good luck. Problems like yours take us months to solve. Be ready for this...
We've had pretty good luck by writing our own malloc and free functions. In production, they just call the standard malloc and free, but in debug, they can do whatever you want. We also have a simple base class that does nothing but override the new and delete operators to use these functions, then any class you write can simply inherit from that class. If you have a ton of code, it may be a big job to replace calls to malloc and free to the new malloc and free (don't forget realloc!), but in the long run it's very helpful.
In Steve Maguire's book Writing Solid Code (highly recommended), there are examples of debug stuff that you can do in these routines, like:
Keep track of allocations to find leaks
Allocate more memory than necessary and put markers at the beginning and end of memory -- during the free routine, you can ensure these markers are still there
memset the memory with a marker on allocation (to find usage of uninitialized memory) and on free (to find usage of free'd memory)
Another good idea is to never use things like strcpy, strcat, or sprintf -- always use strncpy, strncat, and snprintf. We've written our own versions of these as well, to make sure we don't write off the end of a buffer, and these have caught lots of problems too.
Run the original application with ADplus -crash -pn appnename.exe
When the memory issue pops-up you will get a nice big dump.
You can analyze the dump to figure what memory location was corrupted.
If you are lucky the overwrite memory is a unique string you can figure out where it came from. If you are not lucky, you will need to dig into win32 heap and figure what was the orignal memory characteristics. (heap -x might help)
After you know what was messed-up, you can narrow appverifier usage with special heap settings. i.e. you can specify what DLL you monitor, or what allocation size to monitor.
Hopefully this will speedup the monitoring enough to catch the culprit.
In my experience, I never needed full heap verifier mode, but I spent a lot of time analyzing the crash dump(s) and browsing sources.
P.S:
You can use DebugDiag to analyze the dumps.
It can point out the DLL owning the corrupted heap, and give you other usefull details.
You should attack this problem with both runtime and static analysis.
For static analysis consider compiling with PREfast (cl.exe /analyze). It detects mismatched delete and delete[], buffer overruns and a host of other problems. Be prepared, though, to wade through many kilobytes of L6 warning, especially if your project still has L4 not fixed.
PREfast is available with Visual Studio Team System and, apparently, as part of Windows SDK.
Is this in low memory conditions? If so it might be that new is returning NULL rather than throwing std::bad_alloc. Older VC++ compilers didn't properly implement this. There is an article about Legacy memory allocation failures crashing STL apps built with VC6.
The apparent randomness of the memory corruption sounds very much like a thread synchronization issue - a bug is reproduced depending on machine speed. If objects (chuncks of memory) are shared among threads and synchronization (critical section, mutex, semaphore, other) primitives are not on per-class (per-object, per-class) basis, then it is possible to come to a situation where class (chunk of memory) is deleted / freed while in use, or used after deleted / freed.
As a test for that, you could add synchronization primitives to each class and method. This will make your code slower because many objects will have to wait for each other, but if this eliminates the heap corruption, your heap-corruption problem will become a code optimization one.
You tried old builds, but is there a reason you can't keep going further back in the repository history and seeing exactly when the bug was introduced?
Otherwise, I would suggest adding simple logging of some kind to help track down the problem, though I am at a loss of what specifically you might want to log.
If you can find out what exactly CAN cause this problem, via google and documentation of the exceptions you are getting, maybe that will give further insight on what to look for in the code.
My first action would be as follows:
Build the binaries in "Release" version but creating debug info file (you will find this possibility in project settings).
Use Dr Watson as a defualt debugger (DrWtsn32 -I) on a machine on which you want to reproduce the problem.
Repdroduce the problem. Dr Watson will produce a dump that might be helpful in further analysis.
Another try might be using WinDebug as a debugging tool which is quite powerful being at the same time also lightweight.
Maybe these tools will allow you at least to narrow the problem to certain component.
And are you sure that all the components of the project have correct runtime library settings (C/C++ tab, Code Generation category in VS 6.0 project settings)?
So from the limited information you have, this can be a combination of one or more things:
Bad heap usage, i.e., double frees, read after free, write after free, setting the HEAP_NO_SERIALIZE flag with allocs and frees from multiple threads on the same heap
Out of memory
Bad code (i.e., buffer overflows, buffer underflows, etc.)
"Timing" issues
If it's at all the first two but not the last, you should have caught it by now with either pageheap.exe.
Which most likely means it is due to how the code is accessing shared memory. Unfortunately, tracking that down is going to be rather painful. Unsynchronized access to shared memory often manifests as weird "timing" issues. Things like not using acquire/release semantics for synchronizing access to shared memory with a flag, not using locks appropriately, etc.
At the very least, it would help to be able to track allocations somehow, as was suggested earlier. At least then you can view what actually happened up until the heap corruption and attempt to diagnose from that.
Also, if you can easily redirect allocations to multiple heaps, you might want to try that to see if that either fixes the problem or results in more reproduceable buggy behavior.
When you were testing with VS2008, did you run with HeapVerifier with Conserve Memory set to Yes? That might reduce the performance impact of the heap allocator. (Plus, you have to run with it Debug->Start with Application Verifier, but you may already know that.)
You can also try debugging with Windbg and various uses of the !heap command.
MSN
Graeme's suggestion of custom malloc/free is a good idea. See if you can characterize some pattern about the corruption to give you a handle to leverage.
For example, if it is always in a block of the same size (say 64 bytes) then change your malloc/free pair to always allocate 64 byte chunks in their own page. When you free a 64 byte chunk then set the memory protection bits on that page to prevent reads and wites (using VirtualQuery). Then anyone attempting to access this memory will generate an exception rather than corrupting the heap.
This does assume that the number of outstanding 64 byte chunks is only moderate or you have a lot of memory to burn in the box!
If you choose to rewrite new/delete, I have done this and have simple source code at:
http://gandolf.homelinux.org/~smhanov/blog/?id=10
This catches memory leaks and also inserts guard data before and after the memory block to capture heap corruption. You can just integrate with it by putting #include "debug.h" at the top of every CPP file, and defining DEBUG and DEBUG_MEM.
The little time I had to solve a similar problem.
If the problem still exists I suggest you do this :
Monitor all calls to new/delete and malloc/calloc/realloc/free.
I make single DLL exporting a function for register all calls. This function receive parameter for identifying your code source, pointer to allocated area and type of call saving this information in a table.
All allocated/freed pair is eliminated. At the end or after you need you make a call to an other function for create report for left data.
With this you can identify wrong calls (new/free or malloc/delete) or missing.
If have any case of buffer overwritten in your code the information saved can be wrong but each test may detect/discover/include a solution of failure identified. Many runs to help identify the errors.
Good luck.
Do you think this is a race condition? Are multiple threads sharing one heap? Can you give each thread a private heap with HeapCreate, then they can run fast with HEAP_NO_SERIALIZE. Otherwise, a heap should be thread safe, if you're using the multi-threaded version of the system libraries.
A couple of suggestions. You mention the copious warnings at W4 - I would suggest taking the time to fix your code to compile cleanly at warning level 4 - this will go a long way to preventing subtle hard to find bugs.
Second - for the /analyze switch - it does indeed generate copious warnings. To use this switch in my own project, what I did was to create a new header file that used #pragma warning to turn off all the additional warnings generated by /analyze. Then further down in the file, I turn on only those warnings I care about. Then use the /FI compiler switch to force this header file to be included first in all your compilation units. This should allow you to use the /analyze switch while controling the output