I've developed a program by a customer who's experiencing when he do a certain operation. This isn't happening always on the same place and on the same data and, moreover, it is not happening nor in my local developing machine nor in my test Virtual Machine (which is free of all developing equipment).
Given these conditions, I've decided to compile with MAP (enabled in Configuring Properties-> Linker->Debugger with option /MAP) to see which function is causing crash.
If I've correctly understood, when the program crash I've to check down the offset error and then, search in my MAP under the column RVA+BASE:
Address Publics by Value Rva+Base Lib:Object
0001:00037af0 ?PersonalizzaPlancia#CDlgGestioneDatiProgetto#MosaicoDialogs##IAEXXZ 00438af0 f DlgGestioneDatiProgetto.obj
0001:00038000 ?SalvaTemporanei#CDlgGestioneDatiProgetto#MosaicoDialogs##IAEXXZ 00439000 f DlgGestioneDatiProgetto.obj
Actually, my crash happens at offset: 00038C90 So I should think that it's somewhere in the method:
MosaicoDialogs::CDlgGestioneDatiProgetto::PersonalizzaPlancia
but this is not absolutely possible, so assuming that the computer can't be wrong, I'm the one who's doing it bad.
Can someone explain me how to read MAP in correct way?
don't bother - instead, build the project with symbols enabled and strip them into a pdb file.
Modify the program a little, to write a minidump when it crashes using a unhandled exception handler
Give the newly compiled program to the customer, and when it crashes call MiniDumpWriteDump.
Ask the customer to send this .dmp file to you, and you then simply load it up in Visual Studio (or WinDbg) and it will match up the symbols to the program, and will also match up the code. You should be able to see the exact line of code and some of the variables involved. (if using VS, when you load the .dmp file, top right corner will be an option to "start debugging" click that as it will 'start debugging' at the point of the crash)
Try it first locally - put a div by zero error somewhere in your program and see if you can debug the dump after its been run. Note that you must keep the exact same symbol file for each build of your program - they match exactly. You cannot expect a symbol file for one build to match another build, even if nothing changed.
There are tutorials for this kind of thing, such as this one from CodeProject that looks like it describes what you need.
Reading of MAP files to find out crash location is explained nicely in this code project article.
http://www.codeproject.com/Articles/3472/Finding-crash-information-using-the-MAP-file
Hope helps.
For postmortem debugging, there's an alternative that would not required the use of a map file. Rather, it would require you to create a simple registry script to enable some WER (Windows Error Reporting) flags to trap the crash dump file. First, build your application with debug symbols. Then, follow the instructions for Collecting User-Mode Dumps. Basically, you create a sub key under the "LocalDumps" key. This sub key must be the name of your application, for example, "myapplication.exe". Then, create the "DumpCount", "DumpType", and "DumpFolder" keys/values. Have the user run the registry script. This will enable trapping the dump locally. Then, have the user force the crash to collect the dump file. The user can then send the dump file to you to debug using the symbols you created earlier. Lastly, you'll need to create a registry script that removes the keys/values you added to the registry.
Related
I'm new in debugging with symbols (when no access to the testing machine is possible).
I already provided the client with Debug build with .pdb file but for some reason the dump file I get contains no entries specific to my .dll (although the customer insists the problem occurs there, in particular, the app hangs). The debug build was made with VC++ 2008 x86 (I also tried older VC++ 6.0 with no difference).
The stack trace customer provides looks like
ChildEBP RetAddr
01ece854 773f8e44 ntdll!NtWaitForSingleObject+0x15
01ece8b8 773f8d28 ntdll!RtlpWaitOnCriticalSection+0x13e
01ece8e0 02a92003 ntdll!RtlEnterCriticalSection+0x150
WARNING: Stack unwind information not available. Following frames may be wrong.
01ece8f0 02a8b4fa MyDllName!DllRegisterServer+0xbd2dc
01ece920 02a8b49e MyDllName!DllRegisterServer+0xb67d3
01ece930 02a8746c MyDllName!DllRegisterServer+0xb6777
01ece93c 029dc5ca MyDllName!DllRegisterServer+0xb2745
01ece99c 02a819e4 MyDllName!DllRegisterServer+0x78a3
01ecea80 02a09776 MyDllName!DllRegisterServer+0xaccbd
01eceb00 02a32506 MyDllName!DllRegisterServer+0x34a4f
01eceb58 029f44bf MyDllName!DllRegisterServer+0x5d7df
01ececdc 029f5e20 MyDllName!DllRegisterServer+0x1f798
01eceda0 029f76da MyDllName!DllRegisterServer+0x210f9
01ecedf4 291fe0ce MyDllName!DllRegisterServer+0x229b3
01ecee98 29365243 ClientAppName!Class.Method2+0x262
01eceeb8 293378d9 ClientAppName!Class.Method1+0x37
But I'm not sure what all of this exactly means. Does DllRegisterServer+0x229b3 mean "function which has address +229b3 to the address of DllRegisterServer in map file"?
In map file, I have something like
0002:0006d720 _DllRegisterServer#0 10137720 f DllName.obj
But when I sum 229b3 and 6d720, I don't have any match in the map file for the resulting value.
And why the stack trace shows DllRegisterServer as an address base? It's not the first address in the map file. There are many function before it, should they have negative offset then (seems meaningless)?
I guess I understand reading debugging things wrong, but can't figure out what exactly is wrong..
If I could find out the function names, this would let me move further.
Things get even more complicated as I don't think my .DLL has no critical sections but the customer insists it's my dll which causes entering a critical section and never getting out. For now, I don't yet know how to prove him wrong (or maybe find out that it's indeed my lib which somehow, indirectly, does this, maybe, Windows sockets or DNS resolve name to an IP address somewhere behind the scenes are using critical sections).
This recent blog post by Raymond Chen is exactly the answer you're looking for: Restoring symbols to a stack trace originally generated without symbols.
For some reason, the debugger (or whatever is producing that stack trace) is failing to find the debug symbols for your module, so it's doing the best it can with only the DLL's export table. To paraphrase Raymond:
Ugh. A stack trace taken without working symbols. (There's no way
that DllRegisterServer is a deeply recursive 750 KB function. Just by
casual inspection, you know that the symbols are wrong.)
To see how to fix this, you just have to understand what the debugger
does when it has no symbols to work from: It uses the symbols from the
exported function table. For every address it wants to resolve, it
looks for the nearest exported function whose address is less than or
equal to the target value.
Assuming you have the correct, matching symbols file (.pdb) for the version of the DLL that generated the stack trace, you can trick the debugger into loading the DLL as if it were a process dump, and then you can load the symbols for it:
C:> ntsd -z MyDllName.dll
NTSD is the Microsoft NT Symbolic Debugger, which is installed by default on all modern Windows versions. You can also use WinDbg, but I'm not sure if there's a way to use Visual Studio with this technique.
Once you've got the DLL loaded into the debugger with symbols, you can then let the debugger do the heavy lifting to decode the stack trace. See the blog post for more detailed examples of that.
I'd guess that your DLL is badly behaved and you're deadlocking on the loader lock.
See "Another reason not to do anything scary in your DllMain: Inadvertent deadlock" here
Is there any api's which load a crash dump, symbols from the symbol store and then allow programmatic access to the information?
I want to create a tool to auto process crash dumps and produce a report based on them.
Use a command file for Windbg which sets up your configs and then runs !analyze. Set windbg to output to a file (as described here: windbg: Command output to text file)
Then you can enjoy trying to sort, categorize, etc. the output from that. !analyzewill do a decent job of analyzing most crashes. If it can't, I doubt you'd be doing a better job with your own code, unless you have a lot of experience in analyzing crash logs by program (I haven't ever tried, although I have a decent idea of what to look for, I wouldn't necessarily want to write code to actually do it - I have written code to write logs when the system has crashed, so I know what you usually need).
The Windows dbghelp API loads symbols and line number information to support making stack back traces. Do not know API for reading crash dump files.
Debug diagnostic tool has a com interface that can load a dump, process it using symbols and give you the information back.
http://www.microsoft.com/en-au/download/details.aspx?id=26798
Tutorial: http://codenasarre.wordpress.com/2011/06/14/how-to-control-a-debugger-engine/
We have a problem for which I am looking for a direction. We have an old MFC based application, and due to various reasons, the application is crashing sometimes intermittently in some weird scenarios. And even the customers who are using our application and getting these crashes are finding difficulty in identifying the pattern of crash. So, I had a thought that, if we can have a mechanism by which we can generate a log whenever the application crashes. Like for example, the call stack or any other information in that log. I know,, we can use the crash dump in this case, but then I feel like having a log is a better option. So any help or information in this regard would be really helpful.
Thank you.
You can find a good implementation of crash reporter in the link here.
When you compile your release build, make sure that both DEBUG and /MAP are enabled. Store your binaries together with your .map files and let your customer run this version until a crash is produced. In the Event Viewer you will then find a crash log with a crash offset. Then debug step into your code (F10) and use the crash offset together with some nifty tricks and tricks to jump (set the EIP register to... well, you have to google this a bit) to the location where the crash occurred. You should now be able to find the error!
If my C++ app crashes on Windows I want to send useful debugging information to our server.
On Linux I would use the GNU backtrace() function - is there an equivalent for Windows?
Is there a way to extract useful debugging information after a program has crashed? Or only from within the process?
(Advice along the lines of "test you app so it doesn't crash" is not helpful! - all non-trivial programs will have bugs)
The function Stackwalk64 can be used to snap a stack trace on Windows.
If you intend to use this function, you should be sure to compile your code with FPO disabled - without symbols, StackWalk64 won't be able to properly walk FPO'd frames.
You can get some code running in process at the time of the crash via a top-level __try/__except block by calling SetUnhandledExceptionFilter. This is a bit unreliable since it requires you to have code running inside a crashed process.
Alternatively, you can just the built-in Windows Error Reporting to collect crash data. This is more reliable, since it doesn't require you to add code running inside the compromised, crashed process. The only cost is to get a code-signing certificate, since you must submit a signed binary to the service. https://sysdev.microsoft.com/en-US/Hardware/signup/ has more details.
You can use the Windows API call MiniDumpWriteDump if you wish to roll your own code. Both Windows XP and Vist automate this process and you can sign up at https://winqual.microsoft.com to gain access to the error reports.
Also check out http://kb.mozillazine.org/Breakpad and http://www.codeproject.com/KB/debug/crash_report.aspx for other solutions.
This website provides quite a detailed overview of stack retrieval on Win32 after a C++ exception:
http://www.eptacom.net/pubblicazioni/pub_eng/except.html
Of course, this will only work from within the process, so if the process gets terminated or crashes to the point where it terminates before that code is run, it won't work.
Generate a minidump file. You can then load it up in windbg or Visual Studio and inspect the entire stack where the crash occurred.
Here's a good place to start reading.
Its quite simple to dump the current stackframe addresses into a log file. All you have to do is get such a function called on program faults (i.e. a interrupt handler in Windows) or asserts. This can be done at released versions as well. The log file then can be matched with a map file resulting in a call stack with function names.
I published a article about this some years ago.
See http://www.ddj.com/architect/185300443
Let me describe how I handle crashes in my C++/WTL application.
First, in the main function, I call _set_se_translator, and pass in a function that will throw a C++ exception instead of using structured windows exceptions. This function gets an error code, for which you can get a Windows error message via FormatMessage, and a PEXCEPTION_POINTERS argument, which you can use to write a minidump (code here). You can also check the exception code for certain "meltdown" errors that you should just bail from, like EXCEPTION_NONCONTINUABLE_EXCEPTION or EXCEPTION_STACK_OVERFLOW :) (If it's recoverable, I prompt the user to email me this minidump file.)
The minidump file itself can be opened in Visual Studio like a normal project, and providing you've created a .pdb file for your executable, you can run the project and it'll jump to the exact location of the crash, together with the call stack and registers, which can be examined from the debugger.
If you want to grab a callstack (plus other good info) for a runtime crash, on a release build even on site, then you need to set up Dr Watson (run DrWtsn32.exe). If you check the 'generate crash dumps' option, when an app crashes, it'll write a mini dump file to the path specified (called user.dmp).
You can take this, combine it with the symbols you created when you built your server (set this in your compiler/linker to generate pdb files - keep these safe at home, you use them to match the dump so they can work out the source where the crash occurred)
Get yourself windbg, open it and use the menu option to 'load crash dump'. Once it's loaded everything you can type '~#kp' to get a callstack for every thread (or click the button at the top for the current thread).
There's good articles to know how to do this all over the web, This one is my favourite, and you'll want to read this to get an understanding of how to helpyourself manage the symbols really easily.
You will have to set up a dump generation framework in your application, here is how you may do it.
You may then upload the dump file to the server for further analysis using dump analyzers like windbg.
You may want to use adplus to capture the crash callstack.
You can download and install Debugging tools for Windows.
Usage of adplus is mentioned here:
Adplus usage
This creates the complete crash or hang dump. Once you have the dump, Windbg comes to the rescue. Map the correct pdbs and symbols and you are all set to analyze the dump. To start with use the command "!analyze -v"
Sometimes my c++ program crashes in debug mode, and what I got is a message box saying that an assertion failed in some of the internal memory management routines (accessing unallocated memory etc.). But I don't know where that was called from, because I didn't get any stack trace. How do I get a stack trace or at least see where it fails in my code (instead of library/ built-in routines)?
If you have a crash, you can get information about where the crash happened whether you have a debug or a release build. And you can see the call stack even if you are on a computer that does not have the source code.
To do this you need to use the PDB file that was built with your EXE. Put the PDB file inside the same directory as the EXE that crashed. Note: Even if you have the same source code, building twice and using the first EXE and the second PDB won't work. You need to use the exact PDB that was built with your EXE.
Then attach a debugger to the process that crashed. Example: windbg or VS.
Then simply checkout your call stack, while also having your threads window open. You will have to select the thread that crashed and check on the callstack for that thread. Each thread has a different call stack.
If you already have your VS debugger attached, it will automatically go to the source code that is causing the crash for you.
If the crash is happening inside a library you are using that you don't have the PDB for. There is nothing you can do.
If you run the debug version on a machine with VS, it should offer to bring it up and let you see the stack trace.
The problem is that the real problem is not on the call stack any more. If you free a pointer twice, that can result in this problem somewhere else unrelated to the program (the next time anything accesses the heap datastructures)
I wrote this blog on some tips for getting the problem to show up in the call stack so you can figure out what is going on.
http://www.atalasoft.com/cs/blogs/loufranco/archive/2007/02/06/6-_2200_Pointers_2200_-on-Debugging-Unmanaged-Code.aspx
The best tip is to use the gflags utility to make pointer issues cause immediate problems.
You can trigger a mini-dump by setting a handler for uncaught exceptions. Here's an article that explains all about minidumps
Google actually implemented their own open source crash handler called BreakPad, which also mozilla use I think (that's if you want something more serious - a rich and robust crash handler).
If I remember correctly that message box should have a button which says 'retry'. This should then break the program (in the debugger) at the point where the assertion happened.
CrashFinder can help you locate the place of the exception given the DLL and the address of the exception reported.
You can take this code and integrate it into your application to have a stack trage automatically generated when there is an uncaught exception. This is generally performed using __try{} __except{} or with a call to SetUnhandledExceptionFilter which allows you to specify a callback to all unhandled exceptions.
You can also have a post-mortem debugger installed on the client system. This is a decent, general way to get information when you do not have dump creation built into your application (maybe for an older version for which you must still get information).
Dr. Watson on Windows can be installed by running: drwtsn32 -i Running drwtsn32 (without any options) will bring up the configuration dialog. This will allow the creation of crash dump files, which you can later analyze with WinDbg or something similar.
You can use Poppy for this. You just sprinkle some macros across your code and it will gather the stack trace, together with the actual parameter values, local variables, loop counters, etc. It is very lightweight so it can be left in the release build to gather this information from crashes on end-user machines