I have a 64bit application that runs as a service under Server 2003.
When I attach the VS Profiler or windbg I see lots of callstacks like the one below. I understand that processes spawned in the debugger (or profiler) use the debug heap etc... But this is not the case since this service is started by the OS and I am only attaching to it.
I do not understand why it is unwinding the stack. And the profiler shows that a measurable amount of time is spent doing that. Some more info:
• These are release bits built with vc9, running on Server 2003.
• System environment variable _NO_DEBUG_HEAP is set to 1.
• I am using Microsoft symbol servers.
Why is it capturing the stack trace? It seems it’s logging it.. but I can’t find where.
My objective is to verify that the app is really unwinding the stack and, if that is true, try to avoid it.
Any ideas?
Callstack
ntdll!RtlVirtualUnwind
ntdll!RtlpWalkFrameChain
ntdll!RtlCaptureStackBackTrace
ntdll!RtlpCaptureStackTraceForLogging
ntdll!RtlLogStackBackTrace
ntdll!RtlDebugAllocateHeap
ntdll!RtlAllocateHeapSlowly
ntdll!RtlAllocateHeap
MSVCR90!malloc
MSVCR90!operator new
It's possible that somebody has used gflags.exe to enable user stack trace capturing for this process or systemwide, or some other flag that requires tracking of CRT allocation operations.
You should be able to check this possibility in the registry using the info here.
Steve's answer is great, and eventually led to a resolution for our own server after a looong investigation. Our situation was slightly different, and the cause for the stack captures was due to a script periodically running a UMDH capture. UMDH will enable the stack collection when executed against that process for the first time. It gives the following warning:
UMDH has enabled allocation stack collection for the current running process.
Related
I have a daemon process on which I want to perform a memory profile. So I took valgrind as a choice and ran it using massif tool, but since the process never dies, massif never returns the output file. Even I try to send a TERM signal to the process, I am not receiving any output from massif.
So now I tried installing a plugin of valgrind in my eclipse and started trying to run the profile on an already created binary of my daemon process, but when I start the profiler, it says 2 kinds of errors:
failing saying not able to load a library. I didn't find any way to set the library path in the profile configuration.
failing bad permissions to read a memory address.
So I am not even able to run the profiler in eclipse.
I tried gdb, I tried getting the memory info, but that is what "/proc//maps" would give. So of no use.
Finally here is my use case:
I have a daemon process that never quits and I want to perform memory profiling on it.
I want to get snapshots of no of memory allocations happened, max memory allocations, which instruction is trying to allocate the most number of allocations etc etc.
Better if I could get a visual interface for the memory profiling so that I can even share it with my manager.
So please suggest me is there any such profiler that helps and any pointers to where to get the documentation etc.
Thanks in Advance!
Vinay.
When running your program under valgrind, various commands
(depending on the tool) can be executed from the shell, using
vgdb in standalone mode.
When running with --tool=massif, you can do on demand snapshot, while
your program is running.
See http://www.valgrind.org/docs/manual/manual-core-adv.html#manual-core-adv.valgrind-monitor-commands for more information.
I have a C++ DLL that is used for authentication that gets loaded by a Windows service for every login.
In that DLL I use the Windows ADSI function ADsOpenObject() to get a user object from Active Directory.
HRESULT hr = ADsOpenObject(L"LDAP://rootDSE",
L"username",
L"password",
m_dwADSFlags,
IID_IDirectorySearch,
(void**)&m_DSSearch);
Generally this works since years. But currently I get the error code
-2147024882 (0x8007000E)
which is OUT_OF_MEMORY. When I restart the service that is using my DLL then it runs fine for weeks but then the errors start occuring.
Now I can't find what is out of memory. The task scheduler looks fine and free memory is plenty.
What can I do to fix that?
which is OUT_OF_MEMORY.
It is E_OUTOFMEMORY, a COM error code. The description isn't very helpful, this error code tends to be returned for any "out of resources" errors by Microsoft code, not just memory. Could by an internal limit being reached, could be a winapi call that fails.
And it is not necessarily limited to the direct software involved. A mishaving device driver that leaks kernel pool memory can be the indirect source of the mishap for example.
You'll be lucky if you can find something back in the Application event log, look both in the machine that reports the error as well as the domain server. Task Manager might give a clue, add the Handles, GDI Objects, USER Object, Commit size, Page pool and NP Pool columns. Pretty hard to give specific advice beyond this. It is no doubt a leak, quacks loudly like one when you have to restart a machine periodically to recover. Good luck hunting it down.
Are you calling Release on m_DSSearch? Also if you are searching you need to call CloseSearchHandle or AbandonSearch. If you are not doing either of those, you could be slowly leaking memory.
Your process could fragment the heap to a point so that ADsOpenObject can't find a consecutive piece of memory that is large enough.
You can use VMMap to profile your memory usage:
http://technet.microsoft.com/en-us/sysinternals/dd535533
You could try enabling low fragmentation heap:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366750(v=vs.85).aspx
If it is now starting to appear and it wasn't earlier, I would suppose one of two possibilities: it has to do with the time (more specifically the year, a milleniumbug of some kind). Another option would be a problem with 32/64 bit architecture.
Now mind that I don't program C++. But I do know a bit about MS errors (I have worked on a MS helpdesk...)
The project my team has been working on has reached a point where we need to deploy it to computers without the development environment (Visual Studio 2005) installed on them. We fixed the dependency issues we had at first, but we're still having issues.
Now, once the installer is finished, our project gets stuck somewhere before entering WinMain. It only takes up 13MB of RAM, but takes up 50% of the cpu cycles.
Are there any suggestions as to how debug this problem?
Edit: Clarification - this is a C++ project.
Is it possible the hang occurs while some global variable is initialized? That happens before WinMain, and from a global variable's constructor any code could be run. Also, take a look at the busy thread's stack using Process Explorer (make sure you deploy the PBD in order to get a meaningful stack trace). The stack trace should make it obvious where is that thread hanging.
You might have to resort to old-time debugging - outputting print statements to a console that refer to what part of the application has been run successfully. Without the IDE installed on the target machine, there really aren't many options for debugging.
If your running vista or windows 7 you can create a memory dump from task manager (right click and select create dump file) and then transfer that to your dev computer, load the symbols and it will show you where the program was at that time.
If my C++ app crashes on Windows I want to send useful debugging information to our server.
On Linux I would use the GNU backtrace() function - is there an equivalent for Windows?
Is there a way to extract useful debugging information after a program has crashed? Or only from within the process?
(Advice along the lines of "test you app so it doesn't crash" is not helpful! - all non-trivial programs will have bugs)
The function Stackwalk64 can be used to snap a stack trace on Windows.
If you intend to use this function, you should be sure to compile your code with FPO disabled - without symbols, StackWalk64 won't be able to properly walk FPO'd frames.
You can get some code running in process at the time of the crash via a top-level __try/__except block by calling SetUnhandledExceptionFilter. This is a bit unreliable since it requires you to have code running inside a crashed process.
Alternatively, you can just the built-in Windows Error Reporting to collect crash data. This is more reliable, since it doesn't require you to add code running inside the compromised, crashed process. The only cost is to get a code-signing certificate, since you must submit a signed binary to the service. https://sysdev.microsoft.com/en-US/Hardware/signup/ has more details.
You can use the Windows API call MiniDumpWriteDump if you wish to roll your own code. Both Windows XP and Vist automate this process and you can sign up at https://winqual.microsoft.com to gain access to the error reports.
Also check out http://kb.mozillazine.org/Breakpad and http://www.codeproject.com/KB/debug/crash_report.aspx for other solutions.
This website provides quite a detailed overview of stack retrieval on Win32 after a C++ exception:
http://www.eptacom.net/pubblicazioni/pub_eng/except.html
Of course, this will only work from within the process, so if the process gets terminated or crashes to the point where it terminates before that code is run, it won't work.
Generate a minidump file. You can then load it up in windbg or Visual Studio and inspect the entire stack where the crash occurred.
Here's a good place to start reading.
Its quite simple to dump the current stackframe addresses into a log file. All you have to do is get such a function called on program faults (i.e. a interrupt handler in Windows) or asserts. This can be done at released versions as well. The log file then can be matched with a map file resulting in a call stack with function names.
I published a article about this some years ago.
See http://www.ddj.com/architect/185300443
Let me describe how I handle crashes in my C++/WTL application.
First, in the main function, I call _set_se_translator, and pass in a function that will throw a C++ exception instead of using structured windows exceptions. This function gets an error code, for which you can get a Windows error message via FormatMessage, and a PEXCEPTION_POINTERS argument, which you can use to write a minidump (code here). You can also check the exception code for certain "meltdown" errors that you should just bail from, like EXCEPTION_NONCONTINUABLE_EXCEPTION or EXCEPTION_STACK_OVERFLOW :) (If it's recoverable, I prompt the user to email me this minidump file.)
The minidump file itself can be opened in Visual Studio like a normal project, and providing you've created a .pdb file for your executable, you can run the project and it'll jump to the exact location of the crash, together with the call stack and registers, which can be examined from the debugger.
If you want to grab a callstack (plus other good info) for a runtime crash, on a release build even on site, then you need to set up Dr Watson (run DrWtsn32.exe). If you check the 'generate crash dumps' option, when an app crashes, it'll write a mini dump file to the path specified (called user.dmp).
You can take this, combine it with the symbols you created when you built your server (set this in your compiler/linker to generate pdb files - keep these safe at home, you use them to match the dump so they can work out the source where the crash occurred)
Get yourself windbg, open it and use the menu option to 'load crash dump'. Once it's loaded everything you can type '~#kp' to get a callstack for every thread (or click the button at the top for the current thread).
There's good articles to know how to do this all over the web, This one is my favourite, and you'll want to read this to get an understanding of how to helpyourself manage the symbols really easily.
You will have to set up a dump generation framework in your application, here is how you may do it.
You may then upload the dump file to the server for further analysis using dump analyzers like windbg.
You may want to use adplus to capture the crash callstack.
You can download and install Debugging tools for Windows.
Usage of adplus is mentioned here:
Adplus usage
This creates the complete crash or hang dump. Once you have the dump, Windbg comes to the rescue. Map the correct pdbs and symbols and you are all set to analyze the dump. To start with use the command "!analyze -v"
we have a problem with an application we're developing. Very seldom, like once in a hundred, the application crashes at start up. When the crash happens it brings down the whole system, the computer starts to beep and freezes up completely, the only way to recover is to turn off the power (we're using Windows XP). The rarity of the crash combined with the fact that we can't break into the debugger or even generate a stackdump when it occurs makes it extremely hard to debug.
I'm looking for something that logs all function calls to a file. Does such a tool exist? It shouldn't be impossible to implement, profilers like VTune does something very similar.
We're using visual studio 2008 (C++).
Thanks
A.B.
Logging function entries/exits is a low-level approach to your problem. I would suggest using automatic debugger instrumentation (using Debugger key under Image File Execution Options with regedit or using gflags from the package I provide a link to below) and trying to repro the problem until it crashes. Additionally, you can have the debugger log function call history of suspected module(s) using a script or have collect any other information.
But not knowing the details of your application it is very hard to suggest a solution. Is it a user app, service or a driver? What does "crashes at startup" mean - at windows startup or app's startup?
Use this debugger package to troubleshoot.
The only problem with the logging idea is that when the system crashes, the latest log entries might still be in the cache and have no chance to be written to disk...
If it was me I would try running the program on a different PC - it might be flaky hardware or drivers causing the problem. An application program "shouldn't" be able to bring down the system.
A few Ideas-
There is a good chance that just prior to your crash there is some sort of exception in the application. if you set you handler for all unhandled exceptions using SetUnhandledExceptionFilter() and write a stack trace to your log file, you might have a chance to catch the crash in action.
Just remember to flush the file after every write.
Another option is to use a tool such as strace which logs all of the system calls into the kernel (there are multiple flavors and implementations for that so pick your favorite). if you look at the log just before the crash you might find the culprit
Have you considered using a second machine as a remote debugger (via the network)? When the application (and system) crashes, the second machine should still show some useful information, if not the actual point of the problem. I believe VC++ has that ability, at least in some versions.
For Visual C++ _penter() and _pexit() can be used to instrument your code.
See also Method Call Interception in C++.
GCC (including the version MingGW for Windows development) has a code generation switch called -finstrument-functions that tells the compiler to emit special calls to functions called __cyg_profile_func_enter and __cyg_profile_func_exit around every function call. For Visual C++, there are similar options called /GH and /Gh. These cause the compiler to emit calls to __penter and __pexit around function calls.
These instrumentation modes can be used to implement a logging system, with you implementing the calls that the compiler generates to output to your local filesystem or to another computer on your network.
If possible, I'd also try running your system using valgrind or a similar checking tool. This might catch your problem before it gets out-of-hand.