I have a crash dump file from a customer which I have to analyze. I am new to the world of crash dump analysis. (The source code is C++).
Here is what I have tried:-
I opened the .dmp file with MS Visual Studio which indicated the following error - You cannot debug a 64-bit dump of a 32-bit process. So, I thought of giving WinDbg a try.
When I opened the file in WinDbg after setting the symbol search path, I started the getting the following - Debuggee not connected.
Can anyone point me out in the right direction? Should I be asking the customer to provide a 32-bit dump from his point or can this dump file be debugged.
Also, provide the necessary documentation to get started.
To some extent, you can debug a 64-bit dump of a 32-bit process with Windbg,
by use of the wow64exts. However if possible I think it’s best to have a 32 bits dump.
If the customer can provide a 32-bit dump , get it.
Here is a sample of the wow64exts:
0:008> k
Child-SP RetAddr Call Site
00000000`0291f128 00000000`779d263a wow64cpu!CpupSyscallStub+0x2
00000000`0291f130 00000000`7792c4f6 wow64cpu!WaitForMultipleObjects32+0x1d
00000000`0291f1e0 00000000`7792b8f5 wow64!RunCpuSimulation+0xa
00000000`0291f230 000007fe`e51fd6af wow64!Wow64LdrpInitialize+0x435
00000000`0291f770 000007fe`e519c1ae ntdll!_LdrpInitialize+0xde
00000000`0291f7e0 00000000`00000000 ntdll!LdrInitializeThunk+0xe
0:008> .load wow64exts
0:008> !sw
Switched to 32bit mode
0:008:x86> k
ChildEBP RetAddr
02a1f2dc 7783c752 ntdll_779e0000!NtWaitForMultipleObjects+0xc
02a1f460 75b956c0 KERNELBASE!WaitForMultipleObjectsEx+0x10b
02a1f4d4 75b9586a kernel32!WerpReportFaultInternal+0x1c4
02a1f4e8 75b67828 kernel32!WerpReportFault+0x6d
02a1f4f4 778c07c4 kernel32!BasepReportFault+0x19
The most useful tool for crash dump analysis is to load it into Windbg (File -> Open crash dump) and then use the
!analyze -v
command. This applies a number of heuristics to rewind slightly from the actual crash site to work out where the cause of the crash is likely to be, eg to where a null pointer dereference occurred. There's a good tutorial here. A really good site to bookmark is John Robbins' blog which has lots of great articles about Windbg.
Related
In my application I have setup signal handler to catch Segfaults, and print bactraces.
My application loads some plugins libraries, when process starts.
If my application crashes with a segfault, due to an error in the main executable binary, I can analyze the backtrace with:
addr2line -Cif -e ./myapplication 0x4...
It accurately displays the function and the source_file:line_no
However how do analyze if the crash occurs due to an error in the plugin as in the backtrace below?
/opt/myapplication(_Z7sigsegvv+0x15)[0x504245]
/lib64/libpthread.so.0[0x3f1c40f500]
/opt/myapplication/modules/myplugin.so(_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi+0x6af)[0x7f5588fe4bbf]
/opt/myapplication/modules/myplugin.so(_Z11myplugin_reqmodP12CONNECTION_TP7Filebuf+0x68)[0x7f5588fe51e8]
/opt/myapplication(_ZN10Processors7ExecuteEiP12CONNECTION_TP7Filebuf+0x5b)[0x4e584b]
/opt/myapplication(_Z15process_requestP12CONNECTION_TP7Filebuf+0x462)[0x4efa92]
/opt/myapplication(_Z14handle_requestP12CONNECTION_T+0x1c6d)[0x4d4ded]
/opt/myapplication(_Z13process_entryP12CONNECTION_T+0x240)[0x4d79c0]
/lib64/libpthread.so.0[0x3f1c407851]
/lib64/libc.so.6(clone+0x6d)[0x3f1bce890d]
Both my application and plugin libraries have been compiled with gcc and are unstripped.
My application when executed, loads the plugin.so with dlopen
Unfortunately, the crash is occurring at a site where I cannot run the application under gdb.
Googled around frantically for an answer but all sites discussing backtrace and addr2line exclude scenarios where analysis of faulty plugins may be required.
I hope some kind-hearted hack knows solution to this dilemma, and can share some insights. It would be so invaluable for fellow programmers.
Tons of thanks in advance.
Here are some hints that may help you debug this:
The address in your backtrace is an address in the address space of the process at the time it crashed. That means that, if you want to translate it into a 'physical' address relative to the start of the .text section of your library, you have to subtract the start address of the relevant section of pmap from the address in your backtrace.
Unfortunately, this means that you need a pmap of the process before it crashed. I admittedly have no idea whether loading addresses for libraries on a single system are constant if you close and rerun it (imaginably there are security features which randomize this), but it certainly isn't portable across systems, as you have noticed.
In your position, I would try:
demangling the symbol names with c++filt -n or manually. I don't have a shell right now, so here is my manual attempt: _ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi is ICAPSection::process(CONNECTION_T *, Filebuf *, int). This may already be helpful. If not:
use objdump or nm (I'm pretty sure they can do that) to find the address corresponding to the mangled name, then add the offset (+0x6af as per your stacktrace) to this, then look up the resulting address with addr2line.
us2012's answer was quite the trick required to solve the problem. I am just trying to restate it here just to help any other newbie struggling with the same problem, or if somebody wishes to offer improvements.
In the backtrace it is clearly visible that the flaw exists in the code for myplugin.so. And the backtrace indicates that it exists at:
/opt/myapplication/modules/myplugin.so(_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi+0x6af)[0x7f5588fe4bbf]
The problem of locating the line corresponding to this fault cannot be determined as simplistically as:
addr2line -Cif -e /opt/myapplication/modules/myplugin.so 0x7f5588fe4bbf
The correct procedure here would be to use nm or objdump to determine the address pointing to the mangled name. (Demangling as done by us2012 is not really necessary at this point). So using:
nm -Dlan /opt/myapplication/modules/myplugin.so | grep "_ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi"
I get:
0000000000008510 T _ZN11ICAPSection7processEP12CONNECTION_TP7Filebufi /usr/local/src/unstable/myapplication/sources/modules/myplugin/myplugin.cpp:518
Interesting to note here is that myplugin.cpp:518 actually points to the line where the opening "{" of the function ICAPSection::process(CONNECTION_T *, Filebuf *, int)
Next we add 0x6af to the address (revealed by the nm output above) 0000000000008510 using linux shell command
printf '0x%x\n' $(( 0x0000000000008510 + 0x6af ))
And that results in 0x8bbf
And this is the actual source_file:line_no of the faulty code, and can be precisely determined with addr2line as:
addr2line -Cif -e /opt/myapplication/modules/myplugin.so 0x8bbf
Which displays:
std::char_traits<char>::length(char const*)
/usr/include/c++/4.4/bits/char_traits.h:263
std::string::assign(char const*)
/usr/include/c++/4.4/bits/basic_string.h:970
std::string::operator=(char const*)
/usr/include/c++/4.4/bits/basic_string.h:514
??
/usr/local/src/unstable/myapplication/sources/modules/myplugin/myplugin.cpp:622
I am not too sure why the function name was not displayed here, but myplugin.cpp:622 was quite precisely where the fault was.
I develop a win32 application using i686-w64-mingw32-gcc 4.7.2 crosscompiler under Ubuntu 12.10.
When I run my application compiled in the release mode and it crashes, I may receive an output like bellow.
I am wondering how to interpret it? Is there some useful information that I could get from that? For example would it be possible to get lines in my source code from the Stack dump?
wine: Unhandled page fault on read access to 0x00006c11 at address 0x401cb1 (thread 0009), starting debugger...
Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
Unhandled exception: page fault on read access to 0x00006c11 in 32-bit code (0x00401cb1).
Register dump:
CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b
EIP:00401cb1 ESP:007af750 EBP:007af9c8 EFLAGS:00010206( R- -- I - -P- )
EAX:00406a20 EBX:00000004 ECX:00006c11 EDX:00006c11
ESI:00000068 EDI:00110440
Stack dump:
0x007af750: 007af7df 00000030 00000043 00000004
0x007af760: 00142fc0 3ff54e5b 00406a20 3ff57208
0x007af770: 0207251c 3ff4cb7d 02072574 3ff51f7f
0x007af780: ea9e6eeb 3ff49b90 e09fe868 3ff4c562
0x007af790: 00006c11 00000045 0209c5b4 007afa0c
0x007af7a0: 00000004 019b0000 007af7f8 00110000
000c: sel=0067 base=00000000 limit=00000000 16-bit --x
Backtrace:
=>0 0x00401cb1 in analyser (+0x1cb1) (0x007af9c8)
1 0x00536147 in analyser (+0x136146) (0x007afd98)
2 0x004013fa __tmainCRTStartup+0x279() [/home/ruben/mingw-w64/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:313] in analyser (0x007afe70)
3 0x7b859ddc call_process_entry+0xb() in kernel32 (0x007afe88)
4 0x7b85b04f in kernel32 (+0x4b04e) (0x007afec8)
5 0x7bc71d90 call_thread_func_wrapper+0xb() in ntdll (0x007afed8)
6 0x7bc7486d call_thread_func+0x7c() in ntdll (0x007affa8)
7 0x7bc71d6e RtlRaiseException+0x21() in ntdll (0x007affc8)
8 0x7bc49f4e call_dll_entry_point+0x61d() in ntdll (0x007affe8)
0x00401cb1: movl 0x0(%edx),%eax
Modules:
Module Address Debug info Name (89 modules)
PE 240000- 273000 Deferred ssleay32
PE 280000- 37e000 Deferred libeay32
...
I am wondering how to interpret it?
It appears pretty clear already: your code, more precisely: instruction at address 0x401cb1, attempted to dereference memory at address 0x00006c11. That memory access was not valid, and triggered a page fault.
Is there some useful information that I could get from that?
Yes, as above.
For example would it be possible to get lines in my source code from the Stack dump?
Sure: when you build your application in release mode, add the -g flag. Then
addr2line -fe analyser 0x401cb1
should tell you which file/line the address 0x401cb1 corresponds to.
I have a dump file which I managed to create, from my DLL which is created for any unhandled exception.
When I did something like int* tt = new int[4]; return int[n]; with n = 4, I would get the dump file, and could open it, and see at what line the error is caused. This was possible for both directly from a release exe, and a release DLL.
Now this was an easy error, and I only entered it to test my memory dump creation.
I now have a 900kb dump file, and the event log says the error comes from my .DLL, yet if I open the file, it does not display any source code.
The call stack is
KERNELBASE.dll!RaiseException() + 0x3d bytes
clr.dll!RaiseTheExceptionInternalOnly() + 0x18f bytes
clr.dll!IL_Throw() + 0xe2 bytes
000007fe81f65fd7()
00000000034d1610()
000000002d06ecb8()
436f93ce00050011()
436f93cf00110012()
000000002d06ec50()
00006d930c4f7680()
clr.dll!InlinedCallFrame::`vftable'()
000000002d06f3d8()
which does not help me at all to figure out where in the DLL my error is coming from.
Another issue with debugging this is, it only happens on a live-PC, but never on my debugging system. Can anyone help me finding a way how to debug this? It seems to happen on the calling of the DLL, but: not every time, only like every 2nd time (sometimes on 1st try, sometimes on 5th). I am completely lost on what is happening here.
Edit:
Updated the call stack with the Microsoft symbols loaded, but I still do not know where this may be coming from.
You need to load the symbols for kernelbase.dll. And possibly clr.dll.
Presumably you are using visual studio?
Set it up to access symbols from the microsoft symbol server: http://msdn.microsoft.com/en-us/library/b8ttk8zy(v=vs.80).aspx
You may need to right click on items in the callstack and tell it to load symbols.
Additionally be sure to keep a copy of the pdb file for any releases of software you make.
We're using Fogbugz for tracking issues and I am in the middle of writing a C++ wrapper around the XML API for Fogbugz.
The best practice seems to be to use the "scout" field so that similar/same crashes are just counted but not reported again. To do that we need a unique string for a particular cause of a crash.
In Win32 - after getting a dmp file or other crash handler what is a good way to make a unique string for a crash? (we're going to create a dmp file and send it to the fogbugz server)
In previous postings/articles/etc Joel has made various suggestions but much of those counted on a language like C# that use reflection and have a lot of information that is either harder to get or not possible to get.
Have any other people gotten things like stack traces or other things to make scout entries in fogbugz?
EDIT
To clarify - we don;t want a unique id for every incident - there are likely crashes that have the same code path. We want to capture that. I was thinking that we would get the last few stack calls that are in our code (not ones from win32 DLLs) - but not sure how to go about doing this.
Reporting every crash as unique is not right. Reporting all crashes under the same case is not right. Different users repeating a scenario that causes a crash should map to the same incident.
EDIT
What I think we want is a general "signature" of a crash - based on what is on the stack. Similar stacks should have the same signature. For example - take the top 5 methods that are in our app and then the first call (if any) we make into an MS DLL. This would probably be sufficient for a signature and would likely correlate the crashes that are "the same".
So how does one get the list of methods on the stack? And how can you tell if they are from your own app or in another DLL?
EDIT - NOTE
We want to create a "bucket id"/signature while in the exception handler so that we can create the minidump and send it to fogbugz as a scout description. Alternatively we can load up the dump on t he next start of the app and send it then with a signature we generate.
Here in my project I use the Address Memory of the Crash as a "Unique" ID.
IMO the best thing you can use will be bucket id from dump analysis. Use properly configured Debugging Tools for Windows (windbg), one can do !analyze -v and classify your dumps into different buckets based on bucket id. Bucket id guaranteed that if two dumps are the same, their bucket id will be the same. That solves part of the puzzle.
Many times two dumps rooted from same problem will create different bucket id's (maybe version difference, say your 1.0 and 1.1 both crash at same point). You can use faulting module and stack signature to correlate bugs from the same point of fault.
There will be certain things that causes very random dumps (e.g. heap corruption, the faulting module is typically the victim). Therefore dump analysis should be considered best-effort. When you can't, you can't.
I used something like this to generate exceptions in my last app (MSVC), so every error would get logged with the sourcefile and line it occured on:
class Error {
//...
public: Error(string file, string line, string error) ;
};
#define ERROR(err) Error(__FILE__, __LINE__, err)
It's probably a little bit late, but I will add my solution here, too, in case it can help other people.
You can do this using fools from "Debugging Tools for Windows", for example windbg.exe or better kd.exe.
Running the command "kd.exe -z "path_to_dump.dmp" -c "kd;q" >> dumpstack.txt, you might get the following result:
Microsoft (R) Windows Debugger Version 10.0.15063.400 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [d:\work\bugs\14122\myexe.exe.2624.dmp]
User Mini Dump File with Full Memory: Only application data is available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*C:\Symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*C:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Version 15063 MP (4 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
15063.0.x86fre.rs2_release.170317-1834
Machine Name:
Debug session time: Fri Oct 13 00:09:01.000 2017 (UTC + 1:00)
System Uptime: 0 days 0:18:33.797
Process Uptime: 0 days 0:03:40.000
................................................................
.....................................................
Loading unloaded module list
..............................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(a40.2580): Security check failure or stack buffer overrun - code c0000409 (first/second chance not available)
eax=00000001 ebx=00000000 ecx=00000007 edx=77cc4350 esi=00000000 edi=00000000
eip=62ae7666 esp=0b75e17c ebp=0b75e1a8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
msvcr120!abort+0x28:
62ae7666 cd29 int 29h
0:068> kd: Reading initial command 'kb;q'
ChildEBP RetAddr Args to Child
0b75e178 62addc5f 935dda1f 00000000 00000000 msvcr120!abort+0x28
0b75e1a8 0b75e7d4 62a9b436 0b75e1dc 62a52aa5 msvcr120!terminate+0x33
WARNING: Frame IP not in any known module. Following frames may be wrong.
0b75e1ac 62a9b436 0b75e1dc 62a52aa5 00000000 0xb75e7d4
0b75e1b4 62a52aa5 00000000 62a59740 0b75e7d4 msvcr120!__FrameUnwindToState+0x89
0b75e1c8 62a52b33 00000000 00000000 00000000 msvcr120!_EH4_CallFilterFunc+0x12
0b75e1f4 62a5a0f3 62b1f7b8 62a4f7c6 0b75e324 msvcr120!_except_handler4_common+0x8e
0b75e214 77cd6152 0b75e324 0b75e7c4 0b75e344 msvcr120!_except_handler4+0x1e
0b75e238 77cd6124 0b75e324 0b75e7c4 0b75e344 ntdll!ExecuteHandler2+0x26
0b75e30c 77cc4266 0b75e324 0b75e344 0b75e324 ntdll!ExecuteHandler+0x24
0b75e30c 74cf28f2 0b75e324 0b75e344 0b75e324 ntdll!KiUserExceptionDispatcher+0x26
0b75e684 62a59339 e06d7363 00000001 00000003 KERNELBASE!RaiseException+0x62
0b75e6c4 6001821c 0b75e6e4 6004e1bc 946a8f2a msvcr120!_CxxThrowException+0x5b
0b75e6f8 60018042 0b75e720 946a8efa ffffffff mymodule!FunctionC+0x7c
0b75e730 60016544 946a8ece ffffffff 092889d8 mymodule!FunctionB+0x32
0b75e754 600166b8 00842338 6000588d 00000001 myothermodule!FunctionB+0x44
From this stack, you can create a unique bucket if you take for example only your methods from the stack and concatenate them in a string: "mymodule!FunctionC+0x7c;mymodule!FunctionB+0x32;myothermodule!FunctionB+0x44". In order for this to work, you need to have access to you personal symbols server, either using the environment variable _NT_SYMBOL_PATH or with the -y command line switch.
You can alternatively create a string from the return addresses only (second column): "62addc5f,0b75e7d4,62a9b436,62a52aa5,62a52b33,62a5a0f3,77cd6152,77cd6124,77cc4266,74cf28f2,62a59339,6001821c,60018042,60016544,600166b8"
Just use an MD5 string generated from the dump file and you will likely to get a unique string for every crash.
I would start with collecting the data on how often every function in your code has been "flashed" in a crash report stack trace. Every report would have to be added to some kind of database, and every function would have to be indexed so that you could later query, which functions seem to crash more often than others. (And of course, functions like main() will be in every report, but that's understandable).
Or, you think that only crash reports seem to be the problem, you could just remove all those entries from crash stack traces, and then hash the rest (your functions). That way you could see if any particular call chain of your own functions causes a crash repeatedly, no matter what external functions have been called in between.
Then of course, some of the more complicated problems will not be captured this way anyway, as the stack trace will be completely different. To help that, you could record other data from your application along with the stack trace in every report, like sizes of buffers, counters, states of different parts of the application and so on... And then do some statistics on that.
How do I restore the stack trace function name instead of <UNKNOWN>?
Event Type: Error
Event Source: abcd
Event Category: None
Event ID: 16
Date: 1/3/2010
Time: 10:24:10 PM
User: N/A
Computer: CMS01
Description:
Server.exe caused a in module at 2CA001B:77E4BEF7
Build 6.0.0.334
WorkingSetSize: 1291071488 bytes
EAX=02CAF420 EBX=00402C88 ECX=00000000 EDX=7C82860C ESI=02CAF4A8
EDI=02CAFB68 EBP=02CAF470 ESP=02CAF41C EIP=77E4BEF7 FLG=00000206
CS=2CA001B DS=2CA0023 SS=7C820023 ES=2CA0023 FS=7C82003B GS=2CA0000
2CA001B:77E4BEF7 (0xE06D7363 0x00000001 0x00000003 0x02CAF49C)
2CA001B:006DFAC7 (0x02CAF4B8 0x00807F50 0x00760D50 0x007D951C)
2CA001B:006DFC87 (0x00003561 0x7F6A0F38 0x008E7290 0x00021A6F)
2CA001B:0067E4C3 (0x00003561 0x00000000 0x02CAFBB8 0x02CAFB74)
2CA001B:00674CB2 (0x00003561 0x006EBAC7 0x02CAFB68 0x02CAFA64)
2CA001B:00402CA4 (0x00003560 0x00000000 0x00000000 0x02CAFBB8)
2CA001B:00402B29 (0x00003560 0x00000001 0x02CAFBB8 0x00000000)
2CA001B:00683096 (0x00003560 0x563DDDB6 0x00000000 0x02CAFC18)
2CA001B:00688E32 (0x02CAFC58 0x7C7BE590 0x00000000 0x00000000)
2CA001B:00689F0C (0x02CAFC58 0x7C7BE590 0x00000000 0x00650930)
2CA001B:0042E8EA (0x7F677648 0x7F677648 0x00CAFD6C 0x02CAFD6C)
2CA001B:004100CA (0x563DDB3E 0x00000000 0x00000000 0x008E7290)
2CA001B:0063AC39 (0x7F677648 0x02CAFD94 0x02CAFD88 0x77B5B540)
2CA001B:0064CB51 (0x7F660288 0x563DD9FE 0x00000000 0x00000000)
2CA001B:0063A648 (0x00000000 0x02CAFFEC 0x77E6482F 0x008E7290)
2CA001B:0063A74D (0x008E7290 0x00000000 0x00000000 0x008E7290)
2CA001B:77E6482F (0x0063A740 0x008E7290 0x00000000 0x00000000)
Get the exact same build of the program, objdump it, and have a look at what function is at the address. If symbol names have been stripped from the executable, this might be a little bit difficult.
If the program code is in any way dynamic, you may have to run it into a debugger to find the addresses of functions.
If the program is deliberately obfuscated and nasty, and it randomises its function addresses in some way at runtime (evasive things like viruses, or copy-protection code sometimes do this) then all bets are off.
In general:
The easiest way to find out what caused the crash is to follow the steps necessary to reproduce it in an instance of the application running in a debugger. All other approaches are much more difficult. This is why developers will often not spend time trying to tackle bugs which there are no known methods to reproduce.
You need to have the debug info file (.pdb) next to the exe when the crash occurs.
Then hopefully, your crash dumping code can load it and use the information in it.
Try to run the same built from your IDE, pause it and jump to the adresses in assembly. Then you can switch to source code and see the functions there.
At least that's how it works in Delphi. And I know of this possibility in Visual Studio as well.
If you don't have the built in your IDE, you have to use a Debugger like OllyDbg. Run the exe that caused the errors and pause the application in OllyDbg. Go to the adresses from the stack trace.
Open a similar application project in your IDE, run it and pause it. Search for the same binary pattern you see in OllyDbg and then switch to source code.
The last possibility I know is analysing the map file if you built it during your built.
I use StackWalker - just compile it into the source of your program, and if it crashes you can generate a stack trace at that time, including function names.
The most common cause is that you in fact don't have a module at the specified address. This can happen e.g. when you dereference an uninitialized function pointer, or call a virtual function using an invalid this pointer.
This is probably not the case here: "77E4BEF7" is with very high probability a Windows DLL, and 006DFAC7 an address in one of your modules. You don't need PDB files for this: Windows should always know the module name (.EXE or .DLL) - in fact, it needs first that module name to even find the proper PDB file.
Now, the remaining question is why you don't have the module information. This I can't tell from the information above. You need information about the actual system. For instance, does if have DEP? Is DEP enabled for your process?