I tried to catch the stack trace of an address, but it always show me nothing,
my issue is analyse memory leak, so i have queried all heap statics and then queried highest allocated memory, it returned me lot of allocation,
and i tried to search many of allocation address by “!heap -p –a ####“, #### is adress.
but it never return me any call stack,
but if I search this address/any other adresses in memory, it shows me similar allocation.
also if i try to run this command “dt ntdll!_DPH_HEAP_BLOCK StackTrace ####,” , it return me NULL stack trace.
is it because of page heap for application is not enabled????
For native memory issues, you cannot easily use dumps to determine the root cause.
Microsoft has a tool DebugDiag for 32 bit processes,
http://www.microsoft.com/downloads/en/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
You can use it to track native memory leaks in some typical scenarios,
http://support.microsoft.com/kb/919790
Besides, involving Microsoft support team can speed up the root cause analysis,
http://support.microsoft.com
There are a number of things here, firstly you will not get a complete stack trace for symbols you do not have access to, for this you need the pdbs without the private symbols stripped.
For the microsoft symbols entering '.symfix;.reload' in WinDbg should fix that problem.
Secondly although you will not get a stack trace for 3rd party dlls you will for your own application dlls, you will need to make sure your pdbs with public symbols match the version used to generate the dump.
You could ask the customer to generate the user stack trace in gflags and reproduce the error or to send you the pdbs with full symbol information.
It is also possible to open the dump file in Visual studio:
File->Open->Project/Solution navigate to dump file and click ‘OK’
In Solution Explorer right click project->Debug->Start new instance
You may find this approach more familiar but you will not have access to WinDbg tools but as this is a post mortem analysis you can open the dump in visual studio and WinDbg as you are not invasively debugging anyway.
Related
I have an ARM Cortex-M4 embedded system running FreeRTOS. I have implemented a crash log dump mechanism that writes a file to a storage device in the event of a fatal error, such as divide-by-zero, null-pointer, address errors, invalid instruction, or assertions. In this file, I write, among other things, the contents of the stack at the time of the exception.
This system captures crashes that happen in the field, so the idea is to analyze the crashes that are returned to me and determine the source of the problem as best I can. I can easily connect the log to the .elf that was generated when that version of the code was compiled. I just need a way to parse it.
I assume there are tools that can do this already (I can't be the first to do this), but I'm having trouble finding something on The Series of Tubes(tm) that fits the bill.
Is there a good starting point to create a tool that can parse the .elf from compilation and follow a stack dump to create such a report?
For the benefit of anyone else with this problem, here's what I am doing:
Google has a tool called "breakpad" that can parse .elf and crashlog files in the "minidump" format, which was originally created by Microsoft and adapted by Google for Chrome.
I am writing a tool to convert my stack traces to minidump format in the hopes of using the breakpad tools to analyze my crash logs.
Here's a link to breakpad: https://github.com/google/breakpad/blob/master/docs/getting_started_with_breakpad.md
One of our application (windows form application C++, MSVS 2010 )crashes after few minutes of usage. Task manager tells that the memory usage grows to 60% of total system RAM in just few seconds of the run.
I used Intel inspector to get any idea of memory leaks. I was expecting I will get a a list of functions that are creating problem. But what I got is only the dlls as can be seen in the following screenshot.
The application is using a couple of third party libraries such as those starting with Pv, OpenCv cdio, CAIO etc. As you can see the last one is an opencv library, and is occupying close to 400MB. (How is this possible ? )
Also the right panel shows different types of leaks which have occurred.
I want to pin point the memory leak code and correct it. What should be my strategy, what functions should I start looking into? Why the inspector is not giving me correct source code and just giving me dlls? I am sure dlls are perfect as these are used by millions of people.
Please advice,
Thanks
Update
I think I have done something wrong in various compiler setting while generating the exe. .
As can be seen above, no symbol information is loaded. That was the reason I was unable to get the source code where memory leaks were happening. Pressing F1 reaches me to the following instructions:
Troubleshooting No Symbolic Information Symptoms
In the Sources window, the Intel Inspector displays no source code for any code locations in the problem set.
Details Intel Inspector cannot display source code for viewing and editing unless the application has symbols available.
If the Intel Inspector detects no symbols for a location, it sets the call stack and code pane to the first location where it can find symbols.
If the Intel Inspector cannot find any location in the call stack with symbols, it provides the module name and relative virtual address (RVA) for the location.
Possible Correction Strategies
1- When you compile and link an application on Windows* systems:
a) Enable the debug information compiler option.
b) Set the linker option to generate debug information.
2- Configure the project to search non-standard directories.
3- Copy the symbol files, such as *.pdb files, to a location where the Intel Inspector can find them.
So now I am focusing on the above correction strategies. My latest question are:
1- how do I set the above three strategies in MSVS 2010.
2- Do I need to use debgug exe or a release exe while using Intel inspector ?
If this is your source code, and you are sure your code is causing the leaks, you can use Visual Leak Detector.
You just need very minimal changes in project - I would say just #include<vld.h> (which you can make conditional). It will report all memory leaks on Debug Output window. This differs from VC++ standard leaking staticitics - it shows where memory was allocated.
Probally it couldn't load symbols for some module / modules and thus the information is a bit incorrect. Is symbol file (like opencv_core240.pdb) opencv_core240.dll available? Check it!
Also I would suggest to try another memory leak detectors to compare their results to each other.
In general when using Inspector the recommendation is to use a debug build of the code. Release builds may optimize away some important pieces of code.
You can also enable just debugging symbols in a release build, which is important when using Amplifier and Advisor. You can do this by going to Project -> [project name] properties... -> Linker -> Debugging -> Generate Debug Info -> Yes and Project->[project name] properties...->C/C++->General->Debug Information Format -> Program Database. Even if you are in debug configuration make sure these settings are set correctly, because they may have been accidentally modified.
WRT what you are seeing in the report:
OpenCV (and others) isn't occupying 350MB, rather there has been a leak of that size (which means the last pointer to dynamically allocated memory was overwritten without releasing that memory). Is it possible that you're misusing the library APIs?
Also, you may find it useful to look at the call stack at the location of the leak. You will possibly find the API leading to the memory leak which can help you pinpoint the issue.
Ran into some access violation in visual studio 2010 and here's the callstack:
Most of the call stack are assembly code in the dll(almost illegible to me). I want to trace back to the line in my code which caused the violation, but it seems there's no user function in the call stack.
How can I find the line in my function causing the violation ? Do I need to adjust some settings ?
Getting a reliable stack trace out of optimized C or C++ code is difficult. The optimizer chooses speed over diagnosability. The debugger needs PDB files for such code to know how to interpret the stack frames correctly and find the return address to the calling method.
Clearly you don't have these PDBs, you are getting the raw addresses from the operating system DLLs instead of their function names. Getting those PDBs is pretty simple, Microsoft has a public server that does nothing but deliver those PDBs for any released version of Windows, including service packs and security updates.
Telling the debugger about that server is required, the feature is off by default. It is particularly easy for VS2010, the server name is preprogrammed in the dialog, you only have to turn it on. Tools + Options, Debugging, Symbols, tick the checkbox in front of "Microsoft Symbol Servers". Set the cache directory, any writable directory will do.
Start debugging again, it will take a while at first to cache the PDBs. When it is done, you'll see a greatly improved stack trace. Accurate and with function names for the Windows DLLs.
If my C++ app crashes on Windows I want to send useful debugging information to our server.
On Linux I would use the GNU backtrace() function - is there an equivalent for Windows?
Is there a way to extract useful debugging information after a program has crashed? Or only from within the process?
(Advice along the lines of "test you app so it doesn't crash" is not helpful! - all non-trivial programs will have bugs)
The function Stackwalk64 can be used to snap a stack trace on Windows.
If you intend to use this function, you should be sure to compile your code with FPO disabled - without symbols, StackWalk64 won't be able to properly walk FPO'd frames.
You can get some code running in process at the time of the crash via a top-level __try/__except block by calling SetUnhandledExceptionFilter. This is a bit unreliable since it requires you to have code running inside a crashed process.
Alternatively, you can just the built-in Windows Error Reporting to collect crash data. This is more reliable, since it doesn't require you to add code running inside the compromised, crashed process. The only cost is to get a code-signing certificate, since you must submit a signed binary to the service. https://sysdev.microsoft.com/en-US/Hardware/signup/ has more details.
You can use the Windows API call MiniDumpWriteDump if you wish to roll your own code. Both Windows XP and Vist automate this process and you can sign up at https://winqual.microsoft.com to gain access to the error reports.
Also check out http://kb.mozillazine.org/Breakpad and http://www.codeproject.com/KB/debug/crash_report.aspx for other solutions.
This website provides quite a detailed overview of stack retrieval on Win32 after a C++ exception:
http://www.eptacom.net/pubblicazioni/pub_eng/except.html
Of course, this will only work from within the process, so if the process gets terminated or crashes to the point where it terminates before that code is run, it won't work.
Generate a minidump file. You can then load it up in windbg or Visual Studio and inspect the entire stack where the crash occurred.
Here's a good place to start reading.
Its quite simple to dump the current stackframe addresses into a log file. All you have to do is get such a function called on program faults (i.e. a interrupt handler in Windows) or asserts. This can be done at released versions as well. The log file then can be matched with a map file resulting in a call stack with function names.
I published a article about this some years ago.
See http://www.ddj.com/architect/185300443
Let me describe how I handle crashes in my C++/WTL application.
First, in the main function, I call _set_se_translator, and pass in a function that will throw a C++ exception instead of using structured windows exceptions. This function gets an error code, for which you can get a Windows error message via FormatMessage, and a PEXCEPTION_POINTERS argument, which you can use to write a minidump (code here). You can also check the exception code for certain "meltdown" errors that you should just bail from, like EXCEPTION_NONCONTINUABLE_EXCEPTION or EXCEPTION_STACK_OVERFLOW :) (If it's recoverable, I prompt the user to email me this minidump file.)
The minidump file itself can be opened in Visual Studio like a normal project, and providing you've created a .pdb file for your executable, you can run the project and it'll jump to the exact location of the crash, together with the call stack and registers, which can be examined from the debugger.
If you want to grab a callstack (plus other good info) for a runtime crash, on a release build even on site, then you need to set up Dr Watson (run DrWtsn32.exe). If you check the 'generate crash dumps' option, when an app crashes, it'll write a mini dump file to the path specified (called user.dmp).
You can take this, combine it with the symbols you created when you built your server (set this in your compiler/linker to generate pdb files - keep these safe at home, you use them to match the dump so they can work out the source where the crash occurred)
Get yourself windbg, open it and use the menu option to 'load crash dump'. Once it's loaded everything you can type '~#kp' to get a callstack for every thread (or click the button at the top for the current thread).
There's good articles to know how to do this all over the web, This one is my favourite, and you'll want to read this to get an understanding of how to helpyourself manage the symbols really easily.
You will have to set up a dump generation framework in your application, here is how you may do it.
You may then upload the dump file to the server for further analysis using dump analyzers like windbg.
You may want to use adplus to capture the crash callstack.
You can download and install Debugging tools for Windows.
Usage of adplus is mentioned here:
Adplus usage
This creates the complete crash or hang dump. Once you have the dump, Windbg comes to the rescue. Map the correct pdbs and symbols and you are all set to analyze the dump. To start with use the command "!analyze -v"
Sometimes my c++ program crashes in debug mode, and what I got is a message box saying that an assertion failed in some of the internal memory management routines (accessing unallocated memory etc.). But I don't know where that was called from, because I didn't get any stack trace. How do I get a stack trace or at least see where it fails in my code (instead of library/ built-in routines)?
If you have a crash, you can get information about where the crash happened whether you have a debug or a release build. And you can see the call stack even if you are on a computer that does not have the source code.
To do this you need to use the PDB file that was built with your EXE. Put the PDB file inside the same directory as the EXE that crashed. Note: Even if you have the same source code, building twice and using the first EXE and the second PDB won't work. You need to use the exact PDB that was built with your EXE.
Then attach a debugger to the process that crashed. Example: windbg or VS.
Then simply checkout your call stack, while also having your threads window open. You will have to select the thread that crashed and check on the callstack for that thread. Each thread has a different call stack.
If you already have your VS debugger attached, it will automatically go to the source code that is causing the crash for you.
If the crash is happening inside a library you are using that you don't have the PDB for. There is nothing you can do.
If you run the debug version on a machine with VS, it should offer to bring it up and let you see the stack trace.
The problem is that the real problem is not on the call stack any more. If you free a pointer twice, that can result in this problem somewhere else unrelated to the program (the next time anything accesses the heap datastructures)
I wrote this blog on some tips for getting the problem to show up in the call stack so you can figure out what is going on.
http://www.atalasoft.com/cs/blogs/loufranco/archive/2007/02/06/6-_2200_Pointers_2200_-on-Debugging-Unmanaged-Code.aspx
The best tip is to use the gflags utility to make pointer issues cause immediate problems.
You can trigger a mini-dump by setting a handler for uncaught exceptions. Here's an article that explains all about minidumps
Google actually implemented their own open source crash handler called BreakPad, which also mozilla use I think (that's if you want something more serious - a rich and robust crash handler).
If I remember correctly that message box should have a button which says 'retry'. This should then break the program (in the debugger) at the point where the assertion happened.
CrashFinder can help you locate the place of the exception given the DLL and the address of the exception reported.
You can take this code and integrate it into your application to have a stack trage automatically generated when there is an uncaught exception. This is generally performed using __try{} __except{} or with a call to SetUnhandledExceptionFilter which allows you to specify a callback to all unhandled exceptions.
You can also have a post-mortem debugger installed on the client system. This is a decent, general way to get information when you do not have dump creation built into your application (maybe for an older version for which you must still get information).
Dr. Watson on Windows can be installed by running: drwtsn32 -i Running drwtsn32 (without any options) will bring up the configuration dialog. This will allow the creation of crash dump files, which you can later analyze with WinDbg or something similar.
You can use Poppy for this. You just sprinkle some macros across your code and it will gather the stack trace, together with the actual parameter values, local variables, loop counters, etc. It is very lightweight so it can be left in the release build to gather this information from crashes on end-user machines