I have a daemon process on which I want to perform a memory profile. So I took valgrind as a choice and ran it using massif tool, but since the process never dies, massif never returns the output file. Even I try to send a TERM signal to the process, I am not receiving any output from massif.
So now I tried installing a plugin of valgrind in my eclipse and started trying to run the profile on an already created binary of my daemon process, but when I start the profiler, it says 2 kinds of errors:
failing saying not able to load a library. I didn't find any way to set the library path in the profile configuration.
failing bad permissions to read a memory address.
So I am not even able to run the profiler in eclipse.
I tried gdb, I tried getting the memory info, but that is what "/proc//maps" would give. So of no use.
Finally here is my use case:
I have a daemon process that never quits and I want to perform memory profiling on it.
I want to get snapshots of no of memory allocations happened, max memory allocations, which instruction is trying to allocate the most number of allocations etc etc.
Better if I could get a visual interface for the memory profiling so that I can even share it with my manager.
So please suggest me is there any such profiler that helps and any pointers to where to get the documentation etc.
Thanks in Advance!
Vinay.
When running your program under valgrind, various commands
(depending on the tool) can be executed from the shell, using
vgdb in standalone mode.
When running with --tool=massif, you can do on demand snapshot, while
your program is running.
See http://www.valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.valgrind-monitor-commands for more information.
Related
I'm working on a C++ program and I want to find the total memory used by it in its execution. My operating system is Ubuntu 19.10. I found this as a related question, but it seems to address a much different problem. Any help/directions would be great. Thanks!
You can use the command line tool top to monitor the memory usage of a process. Simply run:
top -p PID
where PID is the process ID of the C++ executable that you want to monitor the memory usage of.
I am currently developing a distributed software in C++ using linux which is executed in more than 20 nodes simultaneously. So one of the most challenging issue that I found is how to debug it.
I heard that is possible to manage in a single gdb session multiple remote sessions (e.g. in the master node I create the gdb session and in every other node I launch the program using gdbserver), is it possible? If so can you give an example? Do you know any other way to do it?
Thanks
You can try to do it like this:
First start nodes with gdbserver on remote hosts. It is even possible to start it without a program to debug, if you start it with --multi flag. When server is in multi mode, you can control it from your local session, I mean that you can make it start a program you want to debug.
Then, start multiple inferiors in your gdb session
gdb> add-inferior -copies <number of servers>
switch them to a remote target and connect them to remote servers
gdb> inferior 1
gdb> target extended-remote host:port // use extended to switch gdbserver to multi mode
// start a program if gdbserver was started in multi mode
gdb> inferior 2
...
Now you have them all attached to one gdb session. The problem is that, AFAIK, it is not much better than to start multiple gdb's from different console tabs. On the other hand you can write some scripts or auto tests this way. See the gdb tutorial: server and inferiors.
I don't believe there is one, simple, answer to debugging "many remote applications". Yes, you can attach to a process on another machine, and step through it in GDB. But it's quite awkward to debug a large number of interdependent processes, especially when the problem is complicated.
I believe a good set of logging capabilities in the code, supplemented with additional logs for specific debugging as needed, is more likely to give you a good/fast result.
Another option might be to run the processes on one machine, rather than on multiple machines. Perhaps even use threads within one process, to simulate the behaviour of multiple machines, simplifying the debugging process. Of course, this doesn't prevent bugs that appear ONLY when you run 20 processes on 20 different machines. But the basic idea is to reduce the number of those bugs to a minimum, and debug most things in a "simpler environment".
Aggressive use of defensive programming paradigms, such as liberal use of assert is clearly a good idea (perhaps with a macro to turn it off for the production runs, but make sure that you don't just leave error paths completely unchecked - it is MUCH harder to detect that the reason something crashes is that a memory allocation failed than to track down where that NULL pointer came from some 20 function calls away from a failed allocation.
I have a C++ DLL that is used for authentication that gets loaded by a Windows service for every login.
In that DLL I use the Windows ADSI function ADsOpenObject() to get a user object from Active Directory.
HRESULT hr = ADsOpenObject(L"LDAP://rootDSE",
L"username",
L"password",
m_dwADSFlags,
IID_IDirectorySearch,
(void**)&m_DSSearch);
Generally this works since years. But currently I get the error code
-2147024882 (0x8007000E)
which is OUT_OF_MEMORY. When I restart the service that is using my DLL then it runs fine for weeks but then the errors start occuring.
Now I can't find what is out of memory. The task scheduler looks fine and free memory is plenty.
What can I do to fix that?
which is OUT_OF_MEMORY.
It is E_OUTOFMEMORY, a COM error code. The description isn't very helpful, this error code tends to be returned for any "out of resources" errors by Microsoft code, not just memory. Could by an internal limit being reached, could be a winapi call that fails.
And it is not necessarily limited to the direct software involved. A mishaving device driver that leaks kernel pool memory can be the indirect source of the mishap for example.
You'll be lucky if you can find something back in the Application event log, look both in the machine that reports the error as well as the domain server. Task Manager might give a clue, add the Handles, GDI Objects, USER Object, Commit size, Page pool and NP Pool columns. Pretty hard to give specific advice beyond this. It is no doubt a leak, quacks loudly like one when you have to restart a machine periodically to recover. Good luck hunting it down.
Are you calling Release on m_DSSearch? Also if you are searching you need to call CloseSearchHandle or AbandonSearch. If you are not doing either of those, you could be slowly leaking memory.
Your process could fragment the heap to a point so that ADsOpenObject can't find a consecutive piece of memory that is large enough.
You can use VMMap to profile your memory usage:
http://technet.microsoft.com/en-us/sysinternals/dd535533
You could try enabling low fragmentation heap:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750(v=vs.85).aspx
If it is now starting to appear and it wasn't earlier, I would suppose one of two possibilities: it has to do with the time (more specifically the year, a milleniumbug of some kind). Another option would be a problem with 32/64 bit architecture.
Now mind that I don't program C++. But I do know a bit about MS errors (I have worked on a MS helpdesk...)
I have a 64bit application that runs as a service under Server 2003.
When I attach the VS Profiler or windbg I see lots of callstacks like the one below. I understand that processes spawned in the debugger (or profiler) use the debug heap etc... But this is not the case since this service is started by the OS and I am only attaching to it.
I do not understand why it is unwinding the stack. And the profiler shows that a measurable amount of time is spent doing that. Some more info:
• These are release bits built with vc9, running on Server 2003.
• System environment variable _NO_DEBUG_HEAP is set to 1.
• I am using Microsoft symbol servers.
Why is it capturing the stack trace? It seems it’s logging it.. but I can’t find where.
My objective is to verify that the app is really unwinding the stack and, if that is true, try to avoid it.
Any ideas?
Callstack
ntdll!RtlVirtualUnwind
ntdll!RtlpWalkFrameChain
ntdll!RtlCaptureStackBackTrace
ntdll!RtlpCaptureStackTraceForLogging
ntdll!RtlLogStackBackTrace
ntdll!RtlDebugAllocateHeap
ntdll!RtlAllocateHeapSlowly
ntdll!RtlAllocateHeap
MSVCR90!malloc
MSVCR90!operator new
It's possible that somebody has used gflags.exe to enable user stack trace capturing for this process or systemwide, or some other flag that requires tracking of CRT allocation operations.
You should be able to check this possibility in the registry using the info here.
Steve's answer is great, and eventually led to a resolution for our own server after a looong investigation. Our situation was slightly different, and the cause for the stack captures was due to a script periodically running a UMDH capture. UMDH will enable the stack collection when executed against that process for the first time. It gives the following warning:
UMDH has enabled allocation stack collection for the current running process.
Looking at Coldfusion's Server Monitor, I see many requests with averages from 100KB to 700KB average memory usage. This seems high to me, is it?
If it is high, it would explain why the server seems to love to eat up RAM and never let it go. What would be a good way of finding out why the memory usage is so high. There is nothing that I can think of that would be using this much memory. Are there any good ways of debugging Coldfusion requests?
We've had good luck with SeeFusion and simply using Jconsole on the servers to monitor memory and threads.
One thing we did find is that during the time in which the data is being rendered to be returned to the user the memory usage was highest. Once the form was displayed memory dropped off significantly. Make sure garbage collection is set properly, as well.
Are you actually getting 'out of memory' exceptions? Or 'out of stack space' errors? Check the logs carefully as I had experience with CF restarted spontaneaouly before - check the server.log files.
If the answer is yes, then - as mentioned - you can use a tool like JConsole to monitor memory usage. If you see that memory goes up and up prior to these exceptions then you need to find out where your memory leak is. The best way to do this is to get a dump of the JVM at that time and use a meomry analysis tool. I documented our experience with this.
If there are no exceptions etc. then be aware than JRun does use a lot of memory. Typically between 500MB and 1.5GB on our Win32 servers even if the CF Admin setting for the JVM heap size is 1024MB. JRun and is running on top of Java, running an application server so this will be high enough. What you need to watch out for is an ever-increasing use of memory using JConsole.
As for GC tuning - only mess with these settings if you really have a problem. It's complicated and will result in server instability if done incorrectly or without understanding of how Java GC works. You can get the basics here, and then you can pass the settings (e.g. -XX:MaxPermSize=192m) as JVM argument in CF admin.
Getting to the bottom of memory issues can be challenging and time consuming so best of luck!
Sounds like a memory leak. You want to generate a JVM heap dump - a snapshot of JVM memory. Easiest way to do so is with jvisualvm. Then analyze that snapshot using the Leaks Report in Eclipse Memory Analyzer. Marc Escher has a blog entry on this topic.
Steps to Setup a JVM JMX Port for getting a ColdFusion Heap Dump
Edit jvm.config and add to java.args: -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=true
Copy \ColdFusion9\runtime\jre\lib\management\jmxremote.access.template -> jxmremote.access
Edit jxmremote.access add "controlRole changeIt" to the end of the file replacing "changeIt" with password of your choice
In Windows Explorer > jxmremote.access > Right Click > Properties > Security > Advanced
Owner > Administrators
Permissions > Uncheck "Allow inheritable...." > Dialog Click "Copy"
Add "Administrators" with Full Control
Remove all other Permission entries
Click "OK"
Restart ColdFusion
Generating a Heap Dump
on ColdFusion box install the JDK
run jvisualvm from jdk-install-dir/bin/jvisualvm
open a JMX connection (e.g. localhost:3333)
on the "monitor" tab click the Heap Dump button and save the result by right clicking its node
take this file and open in Eclipse Memory Analyzer and run the Leaks Report
dig through it, use the Eclipse help to understand how to evaluate
I've used this approach to find a number of memory leaks in ColdFusion code.