Very strange segfault calling WinUsb_GetOverlappedResult - c++

Question

I have this code:
void GetResult(WINUSB_INTERFACE_HANDLE InterfaceHandle, LPOVERLAPPED lpOverlapped)
{
DWORD numBytes = 0;
WinUsb_GetOverlappedResult(
InterfaceHandle,
lpOverlapped,
&numBytes,
TRUE
);
return;
uint8_t stack[64];
}
WinUsb_GetOverlappedResult is a __stdcall function declared as follows:
WINBOOL WINAPI WinUsb_GetOverlappedResult (WINUSB_INTERFACE_HANDLE InterfaceHandle, LPOVERLAPPED lpOverlapped, LPDWORD lpNumberOfBytesTransferred, WINBOOL bWait);
Compiling in debug mode with GCC 5.3.0 (MinGW) it all works fine. (I can't compile with VC++ because I'm using GCC extensions.)
However if I change it to stack[80] then it segfaults!!
Here is the disassembly in each case. 64 (doesn't crash):
Dump of assembler code for function GetResult(void*, _OVERLAPPED*):
88 {
0x00408523 <+0>: push %ebp
0x00408524 <+1>: mov %esp,%ebp
0x00408526 <+3>: sub $0x68,%esp
89 DWORD numBytes = 0;
0x00408529 <+6>: movl $0x0,-0xc(%ebp)
90 WinUsb_GetOverlappedResult(
91 InterfaceHandle,
92 lpOverlapped,
93 &numBytes,
94 TRUE
95 );
=> 0x00408530 <+13>: movl $0x1,0xc(%esp)
0x00408538 <+21>: lea -0xc(%ebp),%eax
0x0040853b <+24>: mov %eax,0x8(%esp)
0x0040853f <+28>: mov 0xc(%ebp),%eax
0x00408542 <+31>: mov %eax,0x4(%esp)
0x00408546 <+35>: mov 0x8(%ebp),%eax
0x00408549 <+38>: mov %eax,(%esp)
0x0040854c <+41>: call 0x409d58 <WinUsb_GetOverlappedResult#16>
0x00408551 <+46>: sub $0x10,%esp
96 return;
0x00408554 <+49>: nop
97
98 uint8_t stack[64];
99 }
0x00408555 <+50>: leave
0x00408556 <+51>: ret
And 80 (does crash):
Dump of assembler code for function GetResult(void*, _OVERLAPPED*):
88 {
0x00408523 <+0>: push %ebp
0x00408524 <+1>: mov %esp,%ebp
0x00408526 <+3>: sub $0x78,%esp
89 DWORD numBytes = 0;
0x00408529 <+6>: movl $0x0,-0xc(%ebp)
90 WinUsb_GetOverlappedResult(
91 InterfaceHandle,
92 lpOverlapped,
93 &numBytes,
94 TRUE
95 );
=> 0x00408530 <+13>: movl $0x1,0xc(%esp)
0x00408538 <+21>: lea -0xc(%ebp),%eax
0x0040853b <+24>: mov %eax,0x8(%esp)
0x0040853f <+28>: mov 0xc(%ebp),%eax
0x00408542 <+31>: mov %eax,0x4(%esp)
0x00408546 <+35>: mov 0x8(%ebp),%eax
0x00408549 <+38>: mov %eax,(%esp)
0x0040854c <+41>: call 0x409d58 <WinUsb_GetOverlappedResult#16>
0x00408551 <+46>: sub $0x10,%esp
96 return;
0x00408554 <+49>: nop
97
98 uint8_t stack[80];
99 }
0x00408555 <+50>: leave
0x00408556 <+51>: ret
The effect of the __stdcall is to add the line sub $0x10,%esp which I guess is to cancel out ret $0x10 in the function.
In any case these seem very similar and I have no idea why it is crashing. I'm not even 100% sure where it is crashing (GDB is rather unhelpful) but it is somewhere around WinUsb function call.
It's quite hard to debug because if I run the debugger with any breakpoints set, it doesn't crash. I suspect it may be timing related -
I can also prevent the crash with a few extra Sleep(100)s. Once it seemed to crash in PerfIncrementULongLongCounterValue() but who knows...
Does anyone have any clue why this might be happening?
Edit
WinUsb_GetOverlappedResult() just calls straight through to GetOverlappedResult() according to its assembly, so I replace the call with that. Now you need stack[96] to cause the crash, but when it does it at least tells me where the real crash is (I think)!
Here is the disassembly of GetOverlappedResult(). It crashes where indicated because ebp is 0.
0x76feaba0 8b ff mov %edi,%edi
0x76feaba2 <+0x0002> 55 push %ebp
0x76feaba3 <+0x0003> 8b ec mov %esp,%ebp
0x76feaba5 <+0x0005> 83 ec 0c sub $0xc,%esp
0x76feaba8 <+0x0008> a1 94 4b 09 77 mov 0x77094b94,%eax
0x76feabad <+0x000d> 33 c5 xor %ebp,%eax
0x76feabaf <+0x000f> 89 45 fc mov %eax,-0x4(%ebp)
0x76feabb2 <+0x0012> 83 7d 14 00 cmpl $0x0,0x14(%ebp)
0x76feabb6 <+0x0016> 53 push %ebx
0x76feabb7 <+0x0017> 56 push %esi
0x76feabb8 <+0x0018> 57 push %edi
0x76feabb9 <+0x0019> 0f 84 b3 00 00 00 je 0x76feac72 <KERNELBASE!GetOverlappedResult+210>
0x76feabbf <+0x001f> 83 cf ff or $0xffffffff,%edi
0x76feabc2 <+0x0022> 8b 5d 08 mov 0x8(%ebp),%ebx
0x76feabc5 <+0x0025> 83 cb 01 or $0x1,%ebx
0x76feabc8 <+0x0028> 85 ff test %edi,%edi
0x76feabca <+0x002a> 0f 84 a9 00 00 00 je 0x76feac79 <KERNELBASE!GetOverlappedResult+217>
0x76feabd0 <+0x0030> b8 01 00 00 00 mov $0x1,%eax
0x76feabd5 <+0x0035> c7 45 f4 01 00 00 00 movl $0x1,-0xc(%ebp)
0x76feabdc <+0x003c> 89 45 f8 mov %eax,-0x8(%ebp)
0x76feabdf <+0x003f> 84 d8 test %bl,%al
0x76feabe1 <+0x0041> 0f 84 5e f3 03 00 je 0x77029f45 <KERNELBASE!GetCurrentProcess+43221>
0x76feabe7 <+0x0047> 6a 00 push $0x0
0x76feabe9 <+0x0049> 68 dc 10 f2 76 push $0x76f210dc
0x76feabee <+0x004e> 50 push %eax
0x76feabef <+0x004f> 68 ab ab ab ab push $0xabababab
0x76feabf4 <+0x0054> ff 15 68 80 09 77 call *0x77098068
0x76feabfa <+0x005a> 8b f0 mov %eax,%esi
0x76feabfc <+0x005c> 85 f6 test %esi,%esi
0x76feabfe <+0x005e> 74 0e je 0x76feac0e <KERNELBASE!GetOverlappedResult+110>
0x76feac00 <+0x0060> 8d 45 f4 lea -0xc(%ebp),%eax
0x76feac03 <+0x0063> 8b ce mov %esi,%ecx
0x76feac05 <+0x0065> 50 push %eax
0x76feac06 <+0x0066> ff 15 5c 8a 09 77 call *0x77098a5c
0x76feac0c <+0x006c> ff d6 call *%esi
0x76feac0e <+0x006e> 33 c0 xor %eax,%eax
0x76feac10 <+0x0070> 83 e3 fe and $0xfffffffe,%ebx
0x76feac13 <+0x0073> 89 45 f8 mov %eax,-0x8(%ebp)
0x76feac16 <+0x0076> 39 45 f4 cmp %eax,-0xc(%ebp)
0x76feac19 <+0x0079> 0f 85 26 f3 03 00 jne 0x77029f45 <KERNELBASE!GetCurrentProcess+43221>
0x76feac1f <+0x007f> 8b 75 0c mov 0xc(%ebp),%esi
0x76feac22 <+0x0082> 81 3e 03 01 00 00 cmpl $0x103,(%esi)
0x76feac28 <+0x0088> 74 26 je 0x76feac50 <KERNELBASE!GetOverlappedResult+176>
Crash:
0x76feac2a <+0x008a> 8b 45 10 mov 0x10(%ebp),%eax
0x76feac2d <+0x008d> 8b 4e 04 mov 0x4(%esi),%ecx
0x76feac30 <+0x0090> 89 08 mov %ecx,(%eax)
0x76feac32 <+0x0092> 8b 0e mov (%esi),%ecx
0x76feac34 <+0x0094> 85 c9 test %ecx,%ecx
0x76feac36 <+0x0096> 78 31 js 0x76feac69 <KERNELBASE!GetOverlappedResult+201>
0x76feac38 <+0x0098> b8 01 00 00 00 mov $0x1,%eax
0x76feac3d <+0x009d> 8b 4d fc mov -0x4(%ebp),%ecx
0x76feac40 <+0x00a0> 5f pop %edi
0x76feac41 <+0x00a1> 5e pop %esi
0x76feac42 <+0x00a2> 33 cd xor %ebp,%ecx
0x76feac44 <+0x00a4> 5b pop %ebx
0x76feac45 <+0x00a5> e8 0b f0 02 00 call 0x77019c55 <PerfIncrementULongLongCounterValue+197>
0x76feac4a <+0x00aa> 8b e5 mov %ebp,%esp
0x76feac4c <+0x00ac> 5d pop %ebp
0x76feac4d <+0x00ad> c2 10 00 ret $0x10
0x76feac50 <+0x00b0> 8b 46 10 mov 0x10(%esi),%eax
0x76feac53 <+0x00b3> 85 c0 test %eax,%eax
0x76feac55 <+0x00b5> 74 46 je 0x76feac9d <KERNELBASE!GetOverlappedResult+253>
0x76feac57 <+0x00b7> 6a 00 push $0x0
0x76feac59 <+0x00b9> 57 push %edi
0x76feac5a <+0x00ba> 50 push %eax
0x76feac5b <+0x00bb> e8 50 01 00 00 call 0x76feadb0 <WaitForSingleObjectEx>
0x76feac60 <+0x00c0> 85 c0 test %eax,%eax
0x76feac62 <+0x00c2> 74 c6 je 0x76feac2a <KERNELBASE!GetOverlappedResult+138>
0x76feac64 <+0x00c4> e9 fb f2 03 00 jmp 0x77029f64 <KERNELBASE!GetCurrentProcess+43252>
0x76feac69 <+0x00c9> e8 d2 f1 ff ff call 0x76fe9e40 <OpenThreadToken+64>
0x76feac6e <+0x00ce> 33 c0 xor %eax,%eax
0x76feac70 <+0x00d0> eb cb jmp 0x76feac3d <KERNELBASE!GetOverlappedResult+157>
0x76feac72 <+0x00d2> 33 ff xor %edi,%edi
0x76feac74 <+0x00d4> e9 49 ff ff ff jmp 0x76feabc2 <KERNELBASE!GetOverlappedResult+34>
0x76feac79 <+0x00d9> 8b 75 0c mov 0xc(%ebp),%esi
0x76feac7c <+0x00dc> 81 3e 03 01 00 00 cmpl $0x103,(%esi)
0x76feac82 <+0x00e2> 74 0a je 0x76feac8e <KERNELBASE!GetOverlappedResult+238>
0x76feac84 <+0x00e4> 33 c9 xor %ecx,%ecx
0x76feac86 <+0x00e6> 8d 45 f8 lea -0x8(%ebp),%eax
0x76feac89 <+0x00e9> f0 09 08 lock or %ecx,(%eax)
0x76feac8c <+0x00ec> eb 9c jmp 0x76feac2a <KERNELBASE!GetOverlappedResult+138>
0x76feac8e <+0x00ee> 68 e4 03 00 00 push $0x3e4
0x76feac93 <+0x00f3> ff 15 c4 80 09 77 call *0x770980c4
0x76feac99 <+0x00f9> 33 c0 xor %eax,%eax
0x76feac9b <+0x00fb> eb a0 jmp 0x76feac3d <KERNELBASE!GetOverlappedResult+157>
0x76feac9d <+0x00fd> 8b c3 mov %ebx,%eax
0x76feac9f <+0x00ff> eb b6 jmp 0x76feac57 <KERNELBASE!GetOverlappedResult+183>
0x76feaca1 <+0x0101> cc int3
0x76feaca2 <+0x0102> cc int3
0x76feaca3 <+0x0103> cc int3
0x76feaca4 <+0x0104> cc int3
0x76feaca5 <+0x0105> cc int3
0x76feaca6 <+0x0106> cc int3
0x76feaca7 <+0x0107> cc int3
0x76feaca8 <+0x0108> cc int3
0x76feaca9 <+0x0109> cc int3
0x76feacaa <+0x010a> cc int3
0x76feacab <+0x010b> cc int3
0x76feacac <+0x010c> cc int3
0x76feacad <+0x010d> cc int3
0x76feacae <+0x010e> cc int3
0x76feacaf <+0x010f> cc int3

Answer 1

Well I think I figured this out. Maybe. The thing I changed is that I don't move my OVERLAPPED structure any more. I can only assume that WinUsb retains a pointer to the OVERLAPPED you pass when you start the write. If it moves then presumably things break.
This isn't mentioned anywhere I can find int the documentation for OVERLAPPED but changing my code so that the OVERLAPPED is dynamically allocated once and never moved seems to stop the crashes.
Unfortunately I never found a good way to debug it. The best way would be a reversible debugger but they don't seem to exist for Windows.