I have a stack where one of the addresses is not in a module (or at least it can't be resolved to any module). My basic question is where exactly does this code exist? Is it a lump of code that has been generated in a newly created memory segment for example?
Callstack:
ntdll.dll!00000000779914da()
kernel32.dll!0000000077741928()
kernel32.dll!00000000777755f1()
kernel32.dll!000000007778a7a2()
avcuf64.dll!000000007502a2f2()
avcuf64.dll!0000000074fdbd94()
-> 0000000075fd0be2()
kernel32.dll!0000000077757d54()
msvcr120d.dll!000007fee58b1188()
msvcr120d.dll!000007fee58b052c()
msvcr120d.dll!000007fee57c348e()
msvcr120d.dll!000007fee57c11f9()
msvcr120d.dll!000007fee57c123d()
msvcr120d.dll!000007fee57c1259()
Performance.exe!main()
Modules:
Performance.exe 0000000001000000-0000000001034000
ntdll.dll 0000000077940000-0000000077AE9000
kernel32.dll 0000000077720000-000000007783F000
KernelBase.dll 000007FEFDA00000-000007FEFDA6C000
avcuf64.dll 0000000074FD0000-0000000075064000
msvcr120d.dll 000007FEE57A0000-000007FEE59B1000
This callstack starts in main() and I'm calling 'getchar()' out of the C++ stdlib. Also avcuf64.dll is Bitdefender my virus software, so I presume that's some kind of key logger protection going on.
Thanks
Related
I develop a win32 application using i686-w64-mingw32-gcc 4.7.2 crosscompiler under Ubuntu 12.10.
When I run my application compiled in the release mode and it crashes, I may receive an output like bellow.
I am wondering how to interpret it? Is there some useful information that I could get from that? For example would it be possible to get lines in my source code from the Stack dump?
wine: Unhandled page fault on read access to 0x00006c11 at address 0x401cb1 (thread 0009), starting debugger...
Application tried to create a window, but no driver could be loaded.
Make sure that your X server is running and that $DISPLAY is set correctly.
Unhandled exception: page fault on read access to 0x00006c11 in 32-bit code (0x00401cb1).
Register dump:
CS:0023 SS:002b DS:002b ES:002b FS:0063 GS:006b
EIP:00401cb1 ESP:007af750 EBP:007af9c8 EFLAGS:00010206( R- -- I - -P- )
EAX:00406a20 EBX:00000004 ECX:00006c11 EDX:00006c11
ESI:00000068 EDI:00110440
Stack dump:
0x007af750: 007af7df 00000030 00000043 00000004
0x007af760: 00142fc0 3ff54e5b 00406a20 3ff57208
0x007af770: 0207251c 3ff4cb7d 02072574 3ff51f7f
0x007af780: ea9e6eeb 3ff49b90 e09fe868 3ff4c562
0x007af790: 00006c11 00000045 0209c5b4 007afa0c
0x007af7a0: 00000004 019b0000 007af7f8 00110000
000c: sel=0067 base=00000000 limit=00000000 16-bit --x
Backtrace:
=>0 0x00401cb1 in analyser (+0x1cb1) (0x007af9c8)
1 0x00536147 in analyser (+0x136146) (0x007afd98)
2 0x004013fa __tmainCRTStartup+0x279() [/home/ruben/mingw-w64/src/mingw-w64/mingw-w64-crt/crt/crtexe.c:313] in analyser (0x007afe70)
3 0x7b859ddc call_process_entry+0xb() in kernel32 (0x007afe88)
4 0x7b85b04f in kernel32 (+0x4b04e) (0x007afec8)
5 0x7bc71d90 call_thread_func_wrapper+0xb() in ntdll (0x007afed8)
6 0x7bc7486d call_thread_func+0x7c() in ntdll (0x007affa8)
7 0x7bc71d6e RtlRaiseException+0x21() in ntdll (0x007affc8)
8 0x7bc49f4e call_dll_entry_point+0x61d() in ntdll (0x007affe8)
0x00401cb1: movl 0x0(%edx),%eax
Modules:
Module Address Debug info Name (89 modules)
PE 240000- 273000 Deferred ssleay32
PE 280000- 37e000 Deferred libeay32
...
I am wondering how to interpret it?
It appears pretty clear already: your code, more precisely: instruction at address 0x401cb1, attempted to dereference memory at address 0x00006c11. That memory access was not valid, and triggered a page fault.
Is there some useful information that I could get from that?
Yes, as above.
For example would it be possible to get lines in my source code from the Stack dump?
Sure: when you build your application in release mode, add the -g flag. Then
addr2line -fe analyser 0x401cb1
should tell you which file/line the address 0x401cb1 corresponds to.
I was reading up on various things on CodeProject and I found this article: http://www.codeproject.com/Articles/29527/Reverse-Engineering-and-Function-Calling-by-Addres
So what I did was I created an injector and a DLL, and grabbed the sample executable file. It basically outputs this when you press F11:
http://i.stack.imgur.com/YIygV.jpg
So I followed the entire tutorial, but the thing is that the address used in the DLL is always changing. This one to be specific:
pFunctionAddress pMySecretFunction = (pFunctionAddress)(0x004113C0);
In his tutorial the address for the function is 0x004113C0. In mine it is something else, and I take the one I have and use it. It works perfectly, but when I close the executable and open it, it won't work anymore, and OllyDbg shows that the address is a totally new one.
So I researched a bit and I started adding breakpoints with OllyDbg. I found out that the address is always going to be:
main + 4C
Where I guess "main" is the main module of these executable. How can I find this address to the function always? Because it changes all the time and I am clueless at this point. In this article I read it doesn't go through what happens when the executable is re-opened, and I've spent 5 hours trying to find a solution.
Thanks in advance!
EDIT:
Huge thanks to everyone. Thanks to mfc especially, I have finally figured it out! What I ended up doing was whenever I hit DLL_PROCESS_ATTACH, I set a global HMODULE to the address of the executable, like this:
HMODULE g_hExeModule;
g_hExeModule = GetModuleHandle(L"TutExample.exe");
And after a few tests it seems like the function address is always the address of the executable + 0x11014, so in the call I just do:
pFunctionAddress pMySecretFunction = (pFunctionAddress)((DWORD)g_hExeModule + 0x11014);
so if I find a way to get the address of "main" I can add a 4C offset and the function will always be there, I think
Again, functions do have an address:
void *(funcPtr)() = (void (*)())((char *)&main + 0x4C);
// If you were right, and you also substituted the appropriate
// function signature above, then this should work:
funcPtr();
The function that you are trying to call is inside of the exe file, so the reference offset should be relative to the memory address that the exe is loaded.
The offset to the target function should be a constant, changed only after each compilation of the source code.
To find out more about your exe, add these two line into your exe:
printf(_T("Exe loaded at: %08X"), GetModuleHandle(_T("TutExample.exe")));
printf(_T("Target function at: %08X"), mySecretFunction);
I cannot edit my post nor add any remark, so I have to post this as a new answer.
Your result:
Exe loaded at: 00000000 (wrong, probably: 00BE0000 and offset is: 00001005)
Target function at: 00BE1005
Exe loaded at: 00000000 (wrong, probably: 01230000 and offset is: 00001005)
Target function at: 01231005
Exe loaded at: 00000000 (wrong, probably: 012A0000 and offset is: 00001005)
Target function at: 012A1005
Please check the name of your compiled exe, is it "TutExample.exe" ? if not, change it to the exact name in the call to GetModuleHandle.
The value of "00000000" indicates that the GetModuleHandle fails because the name "TutExample.exe" is not found in the current memory space.
The address of target function seems ok. Just minus this address with the address of loaded exe and you will get the offset inside the exe memory layout.
You can do this same math inside your injected dll to alway tracks the target function address correctly no matter how the os loads the exe.
Newer OS's have a feature called ASLR (Address space layout randomization). Criminals use some of the trick you are using. So to make bad guys lives more difficult, EXE and DLL's get assigned a different address every time you run a program.
If you compiled the DLL, there is an option to disable ASLR for your DLL.
I have a dump file which I managed to create, from my DLL which is created for any unhandled exception.
When I did something like int* tt = new int[4]; return int[n]; with n = 4, I would get the dump file, and could open it, and see at what line the error is caused. This was possible for both directly from a release exe, and a release DLL.
Now this was an easy error, and I only entered it to test my memory dump creation.
I now have a 900kb dump file, and the event log says the error comes from my .DLL, yet if I open the file, it does not display any source code.
The call stack is
KERNELBASE.dll!RaiseException() + 0x3d bytes
clr.dll!RaiseTheExceptionInternalOnly() + 0x18f bytes
clr.dll!IL_Throw() + 0xe2 bytes
000007fe81f65fd7()
00000000034d1610()
000000002d06ecb8()
436f93ce00050011()
436f93cf00110012()
000000002d06ec50()
00006d930c4f7680()
clr.dll!InlinedCallFrame::`vftable'()
000000002d06f3d8()
which does not help me at all to figure out where in the DLL my error is coming from.
Another issue with debugging this is, it only happens on a live-PC, but never on my debugging system. Can anyone help me finding a way how to debug this? It seems to happen on the calling of the DLL, but: not every time, only like every 2nd time (sometimes on 1st try, sometimes on 5th). I am completely lost on what is happening here.
Edit:
Updated the call stack with the Microsoft symbols loaded, but I still do not know where this may be coming from.
You need to load the symbols for kernelbase.dll. And possibly clr.dll.
Presumably you are using visual studio?
Set it up to access symbols from the microsoft symbol server: http://msdn.microsoft.com/en-us/library/b8ttk8zy(v=vs.80).aspx
You may need to right click on items in the callstack and tell it to load symbols.
Additionally be sure to keep a copy of the pdb file for any releases of software you make.
We're using Fogbugz for tracking issues and I am in the middle of writing a C++ wrapper around the XML API for Fogbugz.
The best practice seems to be to use the "scout" field so that similar/same crashes are just counted but not reported again. To do that we need a unique string for a particular cause of a crash.
In Win32 - after getting a dmp file or other crash handler what is a good way to make a unique string for a crash? (we're going to create a dmp file and send it to the fogbugz server)
In previous postings/articles/etc Joel has made various suggestions but much of those counted on a language like C# that use reflection and have a lot of information that is either harder to get or not possible to get.
Have any other people gotten things like stack traces or other things to make scout entries in fogbugz?
EDIT
To clarify - we don;t want a unique id for every incident - there are likely crashes that have the same code path. We want to capture that. I was thinking that we would get the last few stack calls that are in our code (not ones from win32 DLLs) - but not sure how to go about doing this.
Reporting every crash as unique is not right. Reporting all crashes under the same case is not right. Different users repeating a scenario that causes a crash should map to the same incident.
EDIT
What I think we want is a general "signature" of a crash - based on what is on the stack. Similar stacks should have the same signature. For example - take the top 5 methods that are in our app and then the first call (if any) we make into an MS DLL. This would probably be sufficient for a signature and would likely correlate the crashes that are "the same".
So how does one get the list of methods on the stack? And how can you tell if they are from your own app or in another DLL?
EDIT - NOTE
We want to create a "bucket id"/signature while in the exception handler so that we can create the minidump and send it to fogbugz as a scout description. Alternatively we can load up the dump on t he next start of the app and send it then with a signature we generate.
Here in my project I use the Address Memory of the Crash as a "Unique" ID.
IMO the best thing you can use will be bucket id from dump analysis. Use properly configured Debugging Tools for Windows (windbg), one can do !analyze -v and classify your dumps into different buckets based on bucket id. Bucket id guaranteed that if two dumps are the same, their bucket id will be the same. That solves part of the puzzle.
Many times two dumps rooted from same problem will create different bucket id's (maybe version difference, say your 1.0 and 1.1 both crash at same point). You can use faulting module and stack signature to correlate bugs from the same point of fault.
There will be certain things that causes very random dumps (e.g. heap corruption, the faulting module is typically the victim). Therefore dump analysis should be considered best-effort. When you can't, you can't.
I used something like this to generate exceptions in my last app (MSVC), so every error would get logged with the sourcefile and line it occured on:
class Error {
//...
public: Error(string file, string line, string error) ;
};
#define ERROR(err) Error(__FILE__, __LINE__, err)
It's probably a little bit late, but I will add my solution here, too, in case it can help other people.
You can do this using fools from "Debugging Tools for Windows", for example windbg.exe or better kd.exe.
Running the command "kd.exe -z "path_to_dump.dmp" -c "kd;q" >> dumpstack.txt, you might get the following result:
Microsoft (R) Windows Debugger Version 10.0.15063.400 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Loading Dump File [d:\work\bugs\14122\myexe.exe.2624.dmp]
User Mini Dump File with Full Memory: Only application data is available
************* Symbol Path validation summary **************
Response Time (ms) Location
Deferred srv*C:\Symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*C:\Symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
Windows 10 Version 15063 MP (4 procs) Free x86 compatible
Product: WinNt, suite: SingleUserTS
15063.0.x86fre.rs2_release.170317-1834
Machine Name:
Debug session time: Fri Oct 13 00:09:01.000 2017 (UTC + 1:00)
System Uptime: 0 days 0:18:33.797
Process Uptime: 0 days 0:03:40.000
................................................................
.....................................................
Loading unloaded module list
..............................
This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(a40.2580): Security check failure or stack buffer overrun - code c0000409 (first/second chance not available)
eax=00000001 ebx=00000000 ecx=00000007 edx=77cc4350 esi=00000000 edi=00000000
eip=62ae7666 esp=0b75e17c ebp=0b75e1a8 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
msvcr120!abort+0x28:
62ae7666 cd29 int 29h
0:068> kd: Reading initial command 'kb;q'
ChildEBP RetAddr Args to Child
0b75e178 62addc5f 935dda1f 00000000 00000000 msvcr120!abort+0x28
0b75e1a8 0b75e7d4 62a9b436 0b75e1dc 62a52aa5 msvcr120!terminate+0x33
WARNING: Frame IP not in any known module. Following frames may be wrong.
0b75e1ac 62a9b436 0b75e1dc 62a52aa5 00000000 0xb75e7d4
0b75e1b4 62a52aa5 00000000 62a59740 0b75e7d4 msvcr120!__FrameUnwindToState+0x89
0b75e1c8 62a52b33 00000000 00000000 00000000 msvcr120!_EH4_CallFilterFunc+0x12
0b75e1f4 62a5a0f3 62b1f7b8 62a4f7c6 0b75e324 msvcr120!_except_handler4_common+0x8e
0b75e214 77cd6152 0b75e324 0b75e7c4 0b75e344 msvcr120!_except_handler4+0x1e
0b75e238 77cd6124 0b75e324 0b75e7c4 0b75e344 ntdll!ExecuteHandler2+0x26
0b75e30c 77cc4266 0b75e324 0b75e344 0b75e324 ntdll!ExecuteHandler+0x24
0b75e30c 74cf28f2 0b75e324 0b75e344 0b75e324 ntdll!KiUserExceptionDispatcher+0x26
0b75e684 62a59339 e06d7363 00000001 00000003 KERNELBASE!RaiseException+0x62
0b75e6c4 6001821c 0b75e6e4 6004e1bc 946a8f2a msvcr120!_CxxThrowException+0x5b
0b75e6f8 60018042 0b75e720 946a8efa ffffffff mymodule!FunctionC+0x7c
0b75e730 60016544 946a8ece ffffffff 092889d8 mymodule!FunctionB+0x32
0b75e754 600166b8 00842338 6000588d 00000001 myothermodule!FunctionB+0x44
From this stack, you can create a unique bucket if you take for example only your methods from the stack and concatenate them in a string: "mymodule!FunctionC+0x7c;mymodule!FunctionB+0x32;myothermodule!FunctionB+0x44". In order for this to work, you need to have access to you personal symbols server, either using the environment variable _NT_SYMBOL_PATH or with the -y command line switch.
You can alternatively create a string from the return addresses only (second column): "62addc5f,0b75e7d4,62a9b436,62a52aa5,62a52b33,62a5a0f3,77cd6152,77cd6124,77cc4266,74cf28f2,62a59339,6001821c,60018042,60016544,600166b8"
Just use an MD5 string generated from the dump file and you will likely to get a unique string for every crash.
I would start with collecting the data on how often every function in your code has been "flashed" in a crash report stack trace. Every report would have to be added to some kind of database, and every function would have to be indexed so that you could later query, which functions seem to crash more often than others. (And of course, functions like main() will be in every report, but that's understandable).
Or, you think that only crash reports seem to be the problem, you could just remove all those entries from crash stack traces, and then hash the rest (your functions). That way you could see if any particular call chain of your own functions causes a crash repeatedly, no matter what external functions have been called in between.
Then of course, some of the more complicated problems will not be captured this way anyway, as the stack trace will be completely different. To help that, you could record other data from your application along with the stack trace in every report, like sizes of buffers, counters, states of different parts of the application and so on... And then do some statistics on that.
How do I restore the stack trace function name instead of <UNKNOWN>?
Event Type: Error
Event Source: abcd
Event Category: None
Event ID: 16
Date: 1/3/2010
Time: 10:24:10 PM
User: N/A
Computer: CMS01
Description:
Server.exe caused a in module at 2CA001B:77E4BEF7
Build 6.0.0.334
WorkingSetSize: 1291071488 bytes
EAX=02CAF420 EBX=00402C88 ECX=00000000 EDX=7C82860C ESI=02CAF4A8
EDI=02CAFB68 EBP=02CAF470 ESP=02CAF41C EIP=77E4BEF7 FLG=00000206
CS=2CA001B DS=2CA0023 SS=7C820023 ES=2CA0023 FS=7C82003B GS=2CA0000
2CA001B:77E4BEF7 (0xE06D7363 0x00000001 0x00000003 0x02CAF49C)
2CA001B:006DFAC7 (0x02CAF4B8 0x00807F50 0x00760D50 0x007D951C)
2CA001B:006DFC87 (0x00003561 0x7F6A0F38 0x008E7290 0x00021A6F)
2CA001B:0067E4C3 (0x00003561 0x00000000 0x02CAFBB8 0x02CAFB74)
2CA001B:00674CB2 (0x00003561 0x006EBAC7 0x02CAFB68 0x02CAFA64)
2CA001B:00402CA4 (0x00003560 0x00000000 0x00000000 0x02CAFBB8)
2CA001B:00402B29 (0x00003560 0x00000001 0x02CAFBB8 0x00000000)
2CA001B:00683096 (0x00003560 0x563DDDB6 0x00000000 0x02CAFC18)
2CA001B:00688E32 (0x02CAFC58 0x7C7BE590 0x00000000 0x00000000)
2CA001B:00689F0C (0x02CAFC58 0x7C7BE590 0x00000000 0x00650930)
2CA001B:0042E8EA (0x7F677648 0x7F677648 0x00CAFD6C 0x02CAFD6C)
2CA001B:004100CA (0x563DDB3E 0x00000000 0x00000000 0x008E7290)
2CA001B:0063AC39 (0x7F677648 0x02CAFD94 0x02CAFD88 0x77B5B540)
2CA001B:0064CB51 (0x7F660288 0x563DD9FE 0x00000000 0x00000000)
2CA001B:0063A648 (0x00000000 0x02CAFFEC 0x77E6482F 0x008E7290)
2CA001B:0063A74D (0x008E7290 0x00000000 0x00000000 0x008E7290)
2CA001B:77E6482F (0x0063A740 0x008E7290 0x00000000 0x00000000)
Get the exact same build of the program, objdump it, and have a look at what function is at the address. If symbol names have been stripped from the executable, this might be a little bit difficult.
If the program code is in any way dynamic, you may have to run it into a debugger to find the addresses of functions.
If the program is deliberately obfuscated and nasty, and it randomises its function addresses in some way at runtime (evasive things like viruses, or copy-protection code sometimes do this) then all bets are off.
In general:
The easiest way to find out what caused the crash is to follow the steps necessary to reproduce it in an instance of the application running in a debugger. All other approaches are much more difficult. This is why developers will often not spend time trying to tackle bugs which there are no known methods to reproduce.
You need to have the debug info file (.pdb) next to the exe when the crash occurs.
Then hopefully, your crash dumping code can load it and use the information in it.
Try to run the same built from your IDE, pause it and jump to the adresses in assembly. Then you can switch to source code and see the functions there.
At least that's how it works in Delphi. And I know of this possibility in Visual Studio as well.
If you don't have the built in your IDE, you have to use a Debugger like OllyDbg. Run the exe that caused the errors and pause the application in OllyDbg. Go to the adresses from the stack trace.
Open a similar application project in your IDE, run it and pause it. Search for the same binary pattern you see in OllyDbg and then switch to source code.
The last possibility I know is analysing the map file if you built it during your built.
I use StackWalker - just compile it into the source of your program, and if it crashes you can generate a stack trace at that time, including function names.
The most common cause is that you in fact don't have a module at the specified address. This can happen e.g. when you dereference an uninitialized function pointer, or call a virtual function using an invalid this pointer.
This is probably not the case here: "77E4BEF7" is with very high probability a Windows DLL, and 006DFAC7 an address in one of your modules. You don't need PDB files for this: Windows should always know the module name (.EXE or .DLL) - in fact, it needs first that module name to even find the proper PDB file.
Now, the remaining question is why you don't have the module information. This I can't tell from the information above. You need information about the actual system. For instance, does if have DEP? Is DEP enabled for your process?