I build my 32 bit application file in windows 2008 server machine using visual studio 2010. this 32 bit application will use IBM wepshere MQ Client ( 7.5.0.3).
while running in the debugging mode i am getting exception so i debugged using WinDbg software
dump info is
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Program Files (x86)\IBM\WebSphere MQ\bin\mqe.dll -
FAULTING_IP:
ntdll32!RtlpCreateSplitBlock+47b
77d23fcb 8b11 mov edx,dword ptr [ecx]
EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000077d23fcb (ntdll32!RtlpCreateSplitBlock+0x000000000000047b)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 000000003b9ac9ff
Attempt to read from address 000000003b9ac9ff
CONTEXT: 0000000000000000 -- (.cxr 0x0;r)
eax=0adaff08 ebx=02960000 ecx=3b9ac9ff edx=00100000 esi=0ae2bf90 edi=02960174
eip=77d23fcb esp=079bd490 ebp=079bd4b4 iopl=0 nv up ei pl zr na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
ntdll32!RtlpCreateSplitBlock+0x47b:
77d23fcb 8b11 mov edx,dword ptr [ecx] ds:002b:3b9ac9ff=????????
FAULTING_THREAD: 000000000000046c
PROCESS_NAME: MessageBroker.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 0000000000000000
EXCEPTION_PARAMETER2: 000000003b9ac9ff
READ_ADDRESS: 000000003b9ac9ff
FOLLOWUP_IP:
ntdll32!RtlpCreateSplitBlock+47b
77d23fcb 8b11 mov edx,dword ptr [ecx]
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
APP: messagebroker.exe
ANALYSIS_VERSION: 6.3.9600.17298 (debuggers(dbg).141024-1500) amd64fre
**BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ_ZEROED_STACK**
PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ
DEFAULT_BUCKET_ID: INVALID_POINTER_READ
LAST_CONTROL_TRANSFER: from 0000000077d23c5a to 0000000077d23fcb
**STACK_TEXT:**
079bd4b4 77d23c5a 0ae2bf90 0adaff08 02960150 ntdll32!RtlpCreateSplitBlock+0x47b
079bd5a0 77d23cfe 00000840 00000848 0ae2b74a ntdll32!RtlpAllocateHeap+0xb5d
079bd624 74e04d83 02960000 00000000 00000840 ntdll32!RtlAllocateHeap+0x23a
079bd644 008a8894 00000840 00000002 079bd800 MSVCR80!malloc+0x7a
**WARNING: Stack unwind information not available. Following frames may be wrong.
00000000 00000000 00000000 00000000 00000000 mqe+0x68894**
STACK_COMMAND: .cxr 0x0 ; kb
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll32!RtlpCreateSplitBlock+47b
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll32
IMAGE_NAME: ntdll32.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 51fb1072
FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_ntdll32.dll!RtlpCreateSplitBlock
**BUCKET_ID: X64_APPLICATION_FAULT_INVALID_POINTER_READ_ZEROED_STACK_ntdll32!RtlpCreateSplitBlock+47b**
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:invalid_pointer_read_c0000005_ntdll32.dll!rtlpcreatesplitblock
FAILURE_ID_HASH: {c09c2d45-666c-6fbd-b6fd-d9b79ba8c4fa}
Above dump show that Mqe.dll(IBM WebSphere MQ dll) is passing invalid pointer
Based on the above dump i am unable find the issue. The stack call very small so i am unable to find where my application calling MQ application ( Mqe.ddl).
Please suggest me how to find the issue and how to debugging Mqe.dll.
MQE.dll is quite possibly an innocent bystander here. It is definitely allocating memory via malloc, and then the crash happens inside of the memory manager. This is typically a result of memory corruption that happened at some point in the past. Buffer overflow, underflow, or use-after-free are the typical reasons for heap corruption.
Options to help track the issue down:
Use !heap -v in WinDbg to perform a heap analysis on the minidump.
Analyze the memory around where the memory manager is attempting to work, you may identify some know data (such as strings) that might give you hints as to who corrupted the memory.
I would walk backwards in the assembly and figure out where the read address (0x3b9ac9ff) came from.
The current registers may also contain pointers inside the heap where things are bad, you also use those as starting points for investigating.
Rerun the application with PageHeap to try and catch an overflow immediately when it happens.
Code review your allocations and uses of any heap memory.
Related
The process has crashed and grave(tombstone) been generated. It was triggered by signal 6(SIGABRT). And the backtrace is shallow:
signal 6 (SIGABRT), code 0 (?), fault addr --------
r0 00000000 r1 000008b5 r2 00000006 r3 000008b5
r4 00000006 r5 00000002 r6 2b094be4 r7 0000010c
r8 00000038 r9 00000002 sl 2c653680 fp 2c653c98
ip 2c655920 sp 2c653560 lr 2af9ecf4 pc 2af9d8d4 cpsr 20000010
backtrace:
#00 pc 0002c8d4 /lib/libc.so.6 (raise+60)
#01 pc ffffffff <'unknown'>
in the stack, find the trace about the suspected library and also in the thread, transfer the address via the maps. To get more info about the symbols with the tool address2line. While the symbol may be in the .data section. The output like this:
$d
crtstuff.c:?
Then I want to get the info via the address of near the accident point from the library(so) in the runtime by gdb. Whether it feasible? If yes, how to do it?
how to do it?
I think you are looking for (gdb) info symbol 0x123456
I am trying to find where in my code a specific interrupt happened. In this case it is on a stm32f4 microcontroller and the interrupt is the SysTick_Handler.
What i want is basically to figure out from where the systick interrupt happened. I am using arm-none-eabi-gdb to try to find the backtrace, but the only information i am getting from there is:
(gdb) bt
#0 SysTick_Handler () at modules/profiling.c:66
#1 <signal handler called>
#2 0x55555554 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?)
How can I get some information about where the program was before the interrupt fired?
Looking at the arm documentation here, it seems I should be able to read the stack pointer, and get the PC from there. But then this is exactly what the unwinder in GDB is doing isnt it?
You were on the right track at the end of your question. The ARM Cortex-M cores have two stack pointers, the main stack pointer (MSP, used for interrupts) and the process stack pointer (PSP, used for tasks).
When an interrupt with priority comes in, the current register values (for most of the registers) are pushed onto the current stack (PSP if interrupting the background application, or MSP if interrupting a lower priority interrupt), and then the stack is switched to the MSP (if not already there).
When you first enter an interrupt, the link register (LR, return address) will have a value that is mostly F's rather than an actual return address. This value tells the core how to exit when branched to. Typically, you'll see a value of 0xFFFFFFFD if the background task was interrupted, or 0xFFFFFFF1 if a lower priority interrupt was interrupted. These values will differ if you are using the floating point unit. The magic in this value, though, is that bit 2 (0x4) tells you whether your stack frame is on the PSP or MSP.
Once you determine which stack your frame is on, you can find the address you were executing from by looking at the appropriate stack pointer minus 24 (6 32-bit locations). See Figure 2.3 in your link. This will point you to the PC from which you were interrupted.
As many of you commented, the PC would be in two different stacks, the way I solved it was by actually finding a HardFault_Handling code in assembly and taking what i needed from there. To get the PC value correctly I am using the following code.
register int *r0 __asm("r0");
__asm( "TST lr, #4\n"
"ITE EQ\n"
"MRSEQ r0, MSP\n"
"MRSNE r0, PSP\n" // stack pointer now in r0
"ldr r0, [r0, #0x18]\n" // stored pc now in r0
//"add r0, r0, #6\n" // address to stored pc now in r0
);
The value of where the interrupt happended can now be accessed by
uint32_t PC = *r0;
and can now be used for whatever I want it. Unfortunately I did not manage to get GDB to unwind the stack automatically for me. But at least I found out where the interrupt was firing, which was the goal.
We keep seeing this question in various forms and folks keep saying there are two stacks. So I tried it myself with the systick.
The documentation says that we are in thread mode out of reset, and if you halt with openocd it says that
target halted due to debug-request, current mode: Thread
I have some code to dump registers:
20000000 APSR
00000000 IPSR
00000000 EPSR
00000000 CONTROL
00000000 SP_PROCESS
20000D00 SP_PROCESS after I modified it
20000FF0 SP_MAIN
20000FF0 mov r0,sp
then I dump the stack up to 0x20001000 which is where I know my stack started
20000FF0 00000000
20000FF4 00000000
20000FF8 00000000
20000FFC 0100005F
I setup and wait for a systick interrupt, the handler dumps registers and ram and then goes into an infinite loop. bad practice in general but just debugging/learning here. Before the interrupt I prep some registers:
.thumb_func
.globl iwait
iwait:
mov r0,#1
mov r1,#2
mov r2,#3
mov r3,#4
mov r4,#13
mov r12,r4
mov r4,#15
mov r14,r4
b .
and in the handler I see
20000000 APSR
0000000F IPSR
00000000 EPSR
00000000 CONTROL
20000D00 SP_PROCESS
20000FC0 SP_MAIN
20000FC0 mov r0,sp
20000FC0 0000000F
20000FC4 20000FFF
20000FC8 00000000
20000FCC FFFFFFF9 this is our special lr (not one rjp mentioned)
20000FD0 00000001 this is r0
20000FD4 00000002 this is r1
20000FD8 00000003 this is r2
20000FDC 00000004 this is r3
20000FE0 0000000D this is r12
20000FE4 0000000F this is r14/lr
20000FE8 01000074 and this is where we were interrupted from
20000FEC 21000000 this is probably the xpsr mentioned
20000FF0 00000000 stuff that was there before
20000FF4 00000000
20000FF8 00000000
20000FFC 0100005F
01000064 <iwait>:
1000064: 2001 movs r0, #1
1000066: 2102 movs r1, #2
1000068: 2203 movs r2, #3
100006a: 2304 movs r3, #4
100006c: 240d movs r4, #13
100006e: 46a4 mov ip, r4
1000070: 240f movs r4, #15
1000072: 46a6 mov lr, r4
1000074: e7fe b.n 1000074 <iwait+0x10>
1000076: bf00 nop
So in this case, straight out of the ARM documentation, it is not using the sp_process it is using sp_main. It is pushing the items the manual says it is pushing including the interrupted/return address which is 0x1000074.
Now, if I set the SPSEL bit (be careful to set the PSP first), it appears that a mov r0,sp in application/thread mode uses the PSP not MSP. But then the handler uses msp for a mov r0,sp but appears to put the
before in thread/foreground
20000000 APSR
00000000 IPSR
00000000 EPSR
00000000 SP_PROCESS
20000D00 SP_PROCESS modified
00000000 CONTROL
00000002 CONTROL modified
20000FF0 SP_MAIN
20000D00 mov r0,sp
now in the handler
20000000 APSR
0000000F IPSR
00000000 EPSR
00000000 CONTROL (interesting!)
20000CE0 SP_PROCESS
20000FE0 SP_MAIN
20000FE0 mov r0,sp
dump of that stack
20000FE0 0000000F
20000FE4 20000CFF
20000FE8 00000000
20000FEC FFFFFFFD
20000FF0 00000000
20000FF4 00000000
20000FF8 00000000
20000FFC 0100005F
dump of sp_process stack
20000CE0 00000001
20000CE4 00000002
20000CE8 00000003
20000CEC 00000004
20000CF0 0000000D
20000CF4 0000000F
20000CF8 01000074 our return value
20000CFC 21000000
So to be in this position of dealing with the alternate stack that folks keep mentioning, you have to put yourself in that position (or some code you rely on). Why you would want to do that for simple bare metal programs, who knows, the control register of all zeros is nice and easy, can share one stack just fine.
I dont use gdb, but you need to get it to dump all the registers sp_process and sp_main then depending on what you find, then dump a dozen or so words at each and in there you should see the 0xFFFFFFFx as a marker then count down from that to see the return address. You can have your handler read the two stack pointers as well then you can look at gprs. With gnu assembler mrs rX,psp; mrs rX,msp; For the process and main stack pointers.
This is called DEBUGGING. The easiest way to get started is to just stick a bunch of printf() calls here and there throughout the code. Run the program. If it prints out:
got to point A
got to point B
got to point C
and dies, then you know it died between "C" and "D." You can now refine that downwards by festooning the code between "C" and "D" with more closely spaced printf() calls.
This is the best way for a beginner to get started. Many seasoned experts also prefer printf() for debugging. Debuggers can get in the way.
Our application (written in C++, VS 2010 project) has been running fine on all operating systems prior to Windows 8 (and still does). On Windows 8, however, when orderly exiting the application, an access violation occurs:
mfc100.dll!_DllMain#12() <<< Crash here
mfc100.dll!__CRT_INIT#12()
mfc100.dll!__DllMainCRTStartup#12()
ntdll.dll!_LdrxCallInitRoutine#16()
ntdll.dll!LdrpCallInitRoutine()
ntdll.dll!LdrShutdownProcess()
ntdll.dll!RtlExitUserProcess()
kernel32.dll!_ExitProcessImplementation#4()
mscoreei.dll!RuntimeDesc::ShutdownAllActiveRuntimes(unsigned int,class RuntimeDesc *,enum RuntimeDesc::ShutdownCompatMode)
mscoreei.dll!_CorExitProcess#4()
mscoree.dll!_ShellShim_CorExitProcess#4()
msvcr100d.dll!__crtCorExitProcess(int status) line693 C
msvcr100d.dll!__crtExitProcess(int status) line 699 C
msvcr100d.dll!doexit(int code, int quick, int retcaller) line 621 C
msvcr100d.dll!exit(int code) Zeile 393 C
my.exe!__tmainCRTStartup() Zeile 568 C
my.exe!WinMainCRTStartup() Zeile 371 C
kernel32.dll!#BaseThreadInitThunk#12()
ntdll.dll!__RtlUserThreadStart()
ntdll.dll!__RtlUserThreadStart#8()
In an MSDN forum topic it has been suggested to run GC.Collect() before exit, but I couldn't make any difference with such a call shortly before exit.
I am a bit at a loss about how I should debug the problem. As far as I understand, CorExitProcess takes care of cleaning up the managed resources of the application. So could this be a fault in a managed component?
Or is it more likely that some function pointer in _DllMain has been overwritten/corrupted? If so, how would I set a data breakpoint at the address in question? There is a post explaning how to debug a similar issue, but he's having the issue in his own DLL so he can actually peak at the exact source of the problem which I can't.
Any suggestions?
Edit:
Additional information, windbg !analyze -v:
FAULTING_IP:
mfc100+258e6c
64298e6c 8b4654 mov eax,dword ptr [esi+54h]
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 64298e6c (mfc100+0x00258e6c)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 53f21f0c
Attempt to read from address 53f21f0c
CONTEXT: 00000000 -- (.cxr 0x0;r)
eax=53f21eb8 ebx=00000000 ecx=64187d2d edx=7fcde000 esi=53f21eb8 edi=00000001
eip=64298e6c esp=00c3f1b8 ebp=00c3f2ec iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00210206
mfc100+0x258e6c:
64298e6c 8b4654 mov eax,dword ptr [esi+54h] ds:0023:53f21f0c=????????
FAULTING_THREAD: 00000520
DEFAULT_BUCKET_ID: WRONG_SYMBOLS
PROCESS_NAME: ww.exe
ADDITIONAL_DEBUG_TEXT:
You can run '.symfix; .reload' to try to fix the symbol path and load symbols.
MODULE_NAME: mfc100
FAULTING_MODULE: 77bc0000 ntdll
DEBUG_FLR_IMAGE_TIMESTAMP: 4d5f29b8
ERROR_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.
EXCEPTION_PARAMETER1: 00000000
EXCEPTION_PARAMETER2: 53f21f0c
READ_ADDRESS: 53f21f0c
FOLLOWUP_IP:
mfc100+258e6c
64298e6c 8b4654 mov eax,dword ptr [esi+54h]
APP: ww.exe
ANALYSIS_VERSION: 6.3.9600.17029 (debuggers(dbg).140219-1702) x86fre
MANAGED_STACK: !dumpstack -EE
OS Thread Id: 0x520 (0)
Current frame:
ChildEBP RetAddr Caller, Callee
PRIMARY_PROBLEM_CLASS: WRONG_SYMBOLS
BUGCHECK_STR: APPLICATION_FAULT_WRONG_SYMBOLS
LAST_CONTROL_TRANSFER: from 6429da08 to 64298e6c
STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
00c3f2ec 6429da08 64040000 00000000 00000001 mfc100+0x258e6c
00c3f330 6429dac7 64040000 00c3f35c 77be077a mfc100+0x25da08
00c3f33c 77be077a 64040000 00000000 00000001 mfc100+0x25dac7
00c3f35c 77be07f0 6429daa9 64040000 00000000 ntdll!RtlAddMandatoryAce+0x14e
00c3f3a4 77bfa529 6429daa9 64040000 00000000 ntdll!RtlAddMandatoryAce+0x1c4
00c3f49c 77bfa40e 00000000 00000000 6f2d4890 ntdll!RtlExitUserProcess+0x1e7
00c3f4b0 76ff4231 00000000 77e8f3b0 ffffffff ntdll!RtlExitUserProcess+0xcc
00c3f4c4 6f8b3712 00000000 bd3cbe8b 01f1c054 KERNEL32!ExitProcess+0x15
00c3f74c 6f8c19a2 00000001 00c3f76c 6f1686ad mscoreei!GetFileVersion+0x1835
00c3f758 6f1686ad 00000000 77bdab85 6f8a0000 mscoreei!CorExitProcess+0x27
00c3f76c 70737954 00000000 00c3f784 7073798d mscoree!CorExitProcess+0x94
00c3f778 7073798d 00000000 00c3f7c8 70737ab0 MSVCR100!_query_new_mode+0x159
00c3f784 70737ab0 00000000 a2b843a9 00375f5c MSVCR100!_query_new_mode+0x192
00c3f7c8 70737b1d 00000000 00000000 00000000 MSVCR100!_query_new_mode+0x2b5
00c3f7dc 003274ab 00000000 d1ef1931 00000000 MSVCR100!exit+0x11
00c3f864 76ff173e 7fcdf000 00c3f8b4 77c16911 ww!_enc$textbss$begin+0x64ab
00c3f870 77c16911 7fcdf000 a613e810 00000000 KERNEL32!BaseThreadInitThunk+0x12
00c3f8b4 77c168bd ffffffff 77c8560a 00000000 ntdll!LdrInitializeThunk+0x1f0
00c3f8c4 00000000 003275da 7fcdf000 00000000 ntdll!LdrInitializeThunk+0x19c
STACK_COMMAND: .cxr 0x0 ; kb
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: mfc100+258e6c
FOLLOWUP_NAME: MachineOwner
IMAGE_NAME: mfc100.dll
BUCKET_ID: WRONG_SYMBOLS
FAILURE_BUCKET_ID: WRONG_SYMBOLS_c0000005_mfc100.dll!Unknown
ANALYSIS_SOURCE: UM
FAILURE_ID_HASH_STRING: um:wrong_symbols_c0000005_mfc100.dll!unknown
FAILURE_ID_HASH: {9e516b68-081f-78d6-cf23-b42f2b3cb573}
Followup: MachineOwner
---------
Screenshot of there the crash occurs:
As discussed in comments, our similar problem was where we had a native C++ application that communicated with a managed C# application running as a COM server. To allow the managed component to communicate events to the C++ app, an event sink was exposed as a simple ATL COM interface from the native side, which on the .NET side was automatically encapsulated in a Runtime Callable Wrapper.
The access violation on application close - which wasn't always visible except in the event logs - was due to the fact that the RCW didn't call Release() on our ATL COM interfaces until it was garbage collected. As this happened when the .NET runtime closed, which was after the native runtime had shut down, it tried to callback into dead code.
The solution for us was to expose a "shutdown" method on the .NET side that disposed of all the communicating objects, then called:
GC.Collect();
GC.WaitForPendingFinalizers();
Ok, I understand that this might not exactly mirror your problem, but the route in to finding out what was causing it was to use the Managed Debugging Assistants, particularly reportAvOnCOMRelease.
We activated the MDA by registry keys and ran the native app via a debugger to see the additional output that identified the COM interfaces that were being held too long. Probably as a first step, it would be wise to activate all of the MDA options to glean as much info as possible from the crash.
I tried debugging this using data breakpoints, but that didn't help a lot. I could see that at some point the data being accessed was overwritten, but that didn't happen in a call stack containing any of my own code.
So I resorted in a simpler method and started removing parts of the program until the error disappeared. In a large application it may be hard to remove some parts without breaking others, but I was able to narrow down the source of the issue.
Basically, the problem stopped occurring after removing a certain call to FreeLibrary. After further investigation it turned out that this call happens during DllMain, which is not allowed:
The entry-point function should perform only simple initialization or termination tasks. It must not call the LoadLibrary or LoadLibraryEx function (or a function that calls these functions), because this may create dependency loops in the DLL load order. This can result in a DLL being used before the system has executed its initialization code. Similarly, the entry-point function must not call the FreeLibrary function (or a function that calls FreeLibrary) during process termination, because this can result in a DLL being used after the system has executed its termination code.
In another SO question, one user apparently noticed a change since Windows 8 in this regard, which would explain why the error only happens on this version of Windows.
We'll now change our application so that FreeLibrary is called at a different point of time.
I have a third party kernel mode driver that is controlled by a windows service that I wrote in C++. The service initialization code appears to run perfectly, but when the service thread is run it encounters an exception that appears to indicate stack or heap corruption (it varies depending on the circumstances of debugging). The exception occurs in an address that is "not in any loaded module", making it almost impossible to tell what happened. Running in windbg with application verifier as an admin gives this after analyze -v:
Breakpoint 0 hit
eax=001ffb04 ebx=7ffda000 ecx=001ffb04 edx=002e1b34 esi=00000000 edi=001ff8a4
eip=00282051 esp=001ff76c ebp=001ff8b4 iopl=0 nv up ei ng nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000282
MyService!CServiceBase::Run+0x71:
00282051 8bf4 mov esi,esp
0:000> g
Detected memory leaks!
Dumping objects ->
{53} normal block at 0x058BFFF0, 8 bytes long.
Data: < ( > C0 15 28 00 04 FB 1F 00
Object dump complete.
(4e6c.4e9c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
eax=00000001 ebx=7ffda000 ecx=bffefce9 edx=002f8950 esi=00000000 edi=001ff8a4
eip=bfe1045d esp=001ff778 ebp=001ff8b4 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246
bfe1045d ?? ???
The _crtdbg "memory leaks" I believe are objects that I allocated during service initialization that will not be freed until service stop.
0:000> !analyze -v
*******************************************************************************
* *
* Exception Analysis *
* *
*******************************************************************************
Current verifier stop:
APPLICATION_VERIFIER_LUAPRIV_OK_OBJECT_DUMP (3317)
The application was able to access the object's security descriptor.
The application was granted the requested access to this object. A standard user should also be able to access this object.
Arguments:
Arg1: 6d101d5c, Object Name
Arg2: 0012019f, Access Requested
Arg3: 05156f60, Security Descriptor
Arg4: 05986fa0, String Security Descriptor
Previous verifier stop:
APPLICATION_VERIFIER_LUAPRIV_OK_OBJECT_GRANT (331e)
Safe Object.
The application opened an object (such as a file or registry key) and requested access that is granted to at least one non-privileged entity (listed). This suggests that the same operation will work when attempted by non-privileged/standard users.
Arguments:
Arg1: 6d101d5c, Object Type
Arg2: 05148fd8, Object Name
Arg3: 05156f8c, Access Control Entry
Arg4: 00000000, N/A
*** WARNING: Unable to verify checksum for ...nfapi.dll
FAULTING_IP:
+0
bfe1045d ?? ???
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: bfe1045d
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: bfe1045d
Attempt to read from address bfe1045d
FAULTING_THREAD: 00004e9c
DEFAULT_BUCKET_ID: BAD_INSTRUCTION_PTR
PROCESS_NAME: MyService.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced memory at 0x%08lx. The memory could not be %s.
EXCEPTION_PARAMETER1: 00000000
EXCEPTION_PARAMETER2: bfe1045d
READ_ADDRESS: bfe1045d
FOLLOWUP_IP:
MyService!main+71 [.....svc.cpp # 51]
00282d81 83c404 add esp,4
FAILED_INSTRUCTION_ADDRESS:
+9
bfe1045d ?? ???
NTGLOBALFLAG: 2000100
APPLICATION_VERIFIER_FLAGS: 80c43267
APP: myservice.exe
IP_ON_HEAP: 7ffda000
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.
PRIMARY_PROBLEM_CLASS: BAD_INSTRUCTION_PTR
BUGCHECK_STR: APPLICATION_FAULT_BAD_INSTRUCTION_PTR_INVALID_POINTER_READ
FRAME_ONE_INVALID: 1
LAST_CONTROL_TRANSFER: from 7ffda000 to bfe1045d
STACK_TEXT:
WARNING: Frame IP not in any known module. Following frames may be wrong.
001ff774 7ffda000 cccccccc cccccccc cccccccc 0xbfe1045d
001ff8b4 00282d81 001ffb04 bfe107a5 00000000 0x7ffda000
001ffb4c 002aac69 00000001 05868fa8 0586af60 MyService!main+0x71
001ffb98 002aab2f 001ffbac 75fbd2e9 7ffda000 MyService!__tmainCRTStartup+0x129
001ffba0 75fbd2e9 7ffda000 001ffbec 77691603 MyService!mainCRTStartup+0xf
001ffbac 77691603 7ffda000 762ef334 00000000 KERNEL32!BaseThreadInitThunk+0xe
001ffbec 776915d6 002aab20 7ffda000 00000000 ntdll!__RtlUserThreadStart+0x23
001ffc04 00000000 002aab20 7ffda000 00000000 ntdll!_RtlUserThreadStart+0x1b
FAULTING_SOURCE_LINE: .....svc.cpp
FAULTING_SOURCE_FILE: .....svc.cpp
FAULTING_SOURCE_LINE_NUMBER: 51
FAULTING_SOURCE_CODE:
47: {
48: try
49: {
50: CSampleService service(SVCNAME, TRUE, TRUE, TRUE);
> 51: if (!CServiceBase::Run(service))
52: {
53: printf("Service failed to run w/err 0x%08lx\n", GetLastError());
54: }
55: return 0;
56: } catch (char *str )
SYMBOL_STACK_INDEX: 2
SYMBOL_NAME: myservice!main+71
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: MyService
IMAGE_NAME: MyService.exe
DEBUG_FLR_IMAGE_TIMESTAMP: 530b8caa
STACK_COMMAND: ~0s ; kb
FAILURE_BUCKET_ID: BAD_INSTRUCTION_PTR_c0000005_MyService.exe!main
BUCKET_ID: APPLICATION_FAULT_BAD_INSTRUCTION_PTR_INVALID_POINTER_READ_BAD_IP_myservice!main+71
Followup: MachineOwner
---------
The difficulty here is that windows service startup code cannot be debugged directly, you have to skip the service start code and step into the initialization code. After the service starts you can attach to the process to debug, but this exception is thrown before the service starts successfully but after the initialization code is completed. If I step past the end of the initialization code the call stack is only hex addresses and the code is assembly in ntdll.dll. I can see that something is being corrupted in memory that later is used in service run but I can't detect what that is, all of the local and global variables appear to be correct completely through the initialization code. The third party code runs successfully as an exe, and the service starts successfully if I do not call the third party code.
When run from a command line the service start times out, which can occur if an exception occurs.
int main(int argc, char *argv[])
{
try
{
CSampleService service(SVCNAME, TRUE, TRUE, TRUE);
if (!CServiceBase::Run(service))
{
printf("Service failed to run w/err 0x%08lx\n", GetLastError());
}
return;
} catch (char *str )
{
char sMessage[MAX_PATH];
sprintf_s(sMessage,"Service failed to run, exception=%s", str);
SvcReportEvent(sMessage, 0);
}
catch (...)
{
char sMessage[MAX_PATH];
sprintf_s(sMessage,"Service failed to run, exception occured");
SvcReportEvent(sMessage, 0);
}
}
EDIT: I have now solved the problem. I used Windbg in postportem mode (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\servicename.exe
Debugger (string) set to "C:\Program Files\Debugging Tools for Windows (x86)\windbg.exe"), set my service to be interactive by ORing SERVICE_INTERACTIVE_PROCESS with the service type. I used __debugbreak() statements to break into the debugger in ServiceMain, which brought up an odd fullscreen mode window with only windbg running after an "interactive mode prompt" dialog. Once there, I was able to debug the running service itself, immediately after startup. It turned out the service did have a mistake in it that caused the stack overflow, which I was able to see and correct. Thank you, everyone!
UPDATE
Thanks to feedback below I was able to home in on ADPlus.vbs, which is part of the debugging tools for Windows.
Don't forget to set up _NT_SYMBOL_PATH before you run it.
Using this we've been able to see much more clearly in to the application with far greater clarity than we ever have using the regular dumps produced via Windows when the application crashes.
Many thanks to all for the responses.
ORIGINAL QUESTION
We have an server application written in Visual C++ that some times (relatively rarely) crashes on customer sites. We haven't been able to understand why this happens based on looking at our own log files so the next step is to start looking at crash dumps.
We've just purposefully put a bug in to our app (a null pointer) so that we can generate a crash dump and verify that the dumps produced are valuable, but thus far I can't make head or tail of what i'm seeing.
I think my first question is whether i've even got WinDbg set up correctly (the other developer here is loading the dump in to Visual Studio 2010 and seeing the same errors so i'm assuming it's fine, or we're both wrong :) ) - and then next question is, how do I understand what it's telling me.
The main confusion is that the dump seems to be telling me it has reached a break point, which seems odd to me since there was no debugger connected.
The app was running on a Windows Server 2003 system when it crashed. I believe I have pointed WinDbg at the PDB file for the DLL and EXE correctly.
FAULTING_IP:
ntdll!DbgBreakPoint+0
7c81a3e1 cc int 3
EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 7c81a3e1 (ntdll!DbgBreakPoint)
ExceptionCode: 80000003 (Break instruction exception)
ExceptionFlags: 00000000
NumberParameters: 3
Parameter[0]: 00000000
Parameter[1]: 8779fdb0
Parameter[2]: 00000003
DEFAULT_BUCKET_ID: STATUS_BREAKPOINT
PROCESS_NAME: CallPlusServerLauncher.exe
ERROR_CODE: (NTSTATUS) 0x80000003 - {EXCEPTION} Breakpoint A breakpoint has been reached.
EXCEPTION_CODE: (HRESULT) 0x80000003 (2147483651) - One or more arguments are invalid
EXCEPTION_PARAMETER1: 00000000
EXCEPTION_PARAMETER2: 8779fdb0
EXCEPTION_PARAMETER3: 00000003
NTGLOBALFLAG: 0
APPLICATION_VERIFIER_FLAGS: 0
ADDITIONAL_DEBUG_TEXT: Followup set based on attribute [Is_ChosenCrashFollowupThread] from Frame:[0] on thread:[ffffffff]
FAULTING_THREAD: ffffffff
PRIMARY_PROBLEM_CLASS: STATUS_BREAKPOINT
BUGCHECK_STR: APPLICATION_FAULT_STATUS_BREAKPOINT
STACK_TEXT:
1bd0ffc8 7c83fe08 00000005 00000004 00000001 ntdll!DbgBreakPoint
1bd0fff4 00000000 00000000 00000000 00000000 ntdll!DbgUiRemoteBreakin+0x36
FOLLOWUP_IP:
ntdll!DbgBreakPoint+0
7c81a3e1 cc int 3
SYMBOL_STACK_INDEX: 0
SYMBOL_NAME: ntdll!DbgBreakPoint+0
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: ntdll
IMAGE_NAME: ntdll.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 49900d60
STACK_COMMAND: ddS 1bd10000 1bd0c000 ; dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; ~439s; .ecxr ; kb
BUCKET_ID: MANUAL_BREAKIN
FAILURE_BUCKET_ID: STATUS_BREAKPOINT_80000003_ntdll.dll!DbgBreakPoint
WATSON_STAGEONE_URL: http://watson.microsoft.com/StageOne/CallPlusServerLauncher_exe/0_0_0_0/4df87414/ntdll_dll/5_2_3790_4455/49900d60/80000003/0001a3e1.htm?Retriage=1
Followup: MachineOwner
DbgBreakPoint -- Looks to me like you broke execution using a remote debugger.
If you didn't then I have seen DbgBreakPoint show up when you have code pages (Edit: I meant page heap) turned on (you should know if you did this) and there was a detection of invalid memory access.
Asserts can also trigger a breakpoint exception. For example I have (too often) seen them come out of the heap checking around a delete when the heap has got corrupted by double-delete or overflow. But only with the debug runtime I thought, is that what you have deployed?