How to trap stack overflow in a Windows x64 C++ application - c++

Question

I am trying to compile an application to x64 platform architecture in Windows. A couple of threads, handling the parsing of a scripting language, uses this code recommended by Microsoft to trap stack overflows and avoid access violation exceptions:
__try
{
DoSomethingThatMightUseALotOfStackMemory();
}
__except(EXCEPTION_EXECUTE_HANDLER)
{
LPBYTE lpPage;
static SYSTEM_INFO si;
static MEMORY_BASIC_INFORMATION mi;
static DWORD dwOldProtect;
// Get page size of system
GetSystemInfo(&si);
// Find SP address
_asm mov lpPage, esp;
// Get allocation base of stack
VirtualQuery(lpPage, &mi, sizeof(mi));
// Go to page beyond current page
lpPage = (LPBYTE)(mi.BaseAddress)-si.dwPageSize;
// Free portion of stack just abandoned
if (!VirtualFree(mi.AllocationBase,
(LPBYTE)lpPage - (LPBYTE)mi.AllocationBase,
MEM_DECOMMIT))
{
exit(1);
}
// Reintroduce the guard page
if (!VirtualProtect(lpPage, si.dwPageSize,
PAGE_GUARD | PAGE_READWRITE,
&dwOldProtect))
{
exit(1);
}
Sleep(2000);
}
Unfortunately it uses one line of inline assembler to get the stack pointer. Visual Studio does not support inline assembly for x64 mode and I can't find a compiler intrinsic for getting the stack pointer neither.
Is it possible to do this in a x64 friendly manner?

Answer 1

As pointed out in a comment to the question, the whole "hack" above can be replaced by the _resetstkoflw function. This works fine in both x86 and x64 mode.
The code snippet above then becomes:
// Filter for the stack overflow exception. This function traps
// the stack overflow exception, but passes all other exceptions through.
int stack_overflow_exception_filter(int exception_code)
{
if (exception_code == EXCEPTION_STACK_OVERFLOW)
{
// Do not call _resetstkoflw here, because at this point
// the stack is not yet unwound. Instead, signal that the
// handler (the __except block) is to be executed.
return EXCEPTION_EXECUTE_HANDLER;
}
else
return EXCEPTION_CONTINUE_SEARCH;
}
void example()
{
int result = 0;
__try
{
DoSomethingThatMightUseALotOfStackMemory();
}
__except(stack_overflow_exception_filter(GetExceptionCode()))
{
// Here, it is safe to reset the stack.
result = _resetstkoflw();
}
// Terminate if _resetstkoflw failed (returned 0)
if (!result)
return 3;
return 0;
}