COM method call accidentally corrupts the stack - c++

Question

I have a bit of code which is calling a method from a COM object (IDirect3D9), but every call causes a run-time check failure #0. The failure is caused by ESP not being properly preserved across the call, so some kind of stack issue (as COM methods are all __stdcall). The unusual part is the simplicity of the method signature and the circumstances.
The code is built in 32-bit mode only, with MSVC 10 (VS 2010 SP1), using the DirectX SDK (June 2010) headers and libs. I've reinstalled the SDK to make sure the headers weren't corrupt, without luck.
I've run the code with both VS' debugger and WinDBG attached, as well as multiple times after reboots/updated drivers. The problem occurs every time, and is identical. Enabling heap validation (and most other options) in gflags doesn't seem to provide any more information, nor does running with Application Verifier. Both simply report the same error as the popup, or the segfault caused shortly after.
Without the call (returning a constant value instead), the program runs as expected. I'm out of ideas on what could be going wrong here.
The function in question is IDirect3D9::GetAdapterModeCount, called from a D3D8-to-9 wrapper (part of a graphics upgrade project for old games). For more general info, the full file is here.
I've tried all the following forms of the call:
UINT r = m_Object->GetAdapterModeCount(D3DADAPTER_DEFAULT, D3DFMT_X8R8G8B8);
UINT r = m_Object->GetAdapterModeCount(0, (D3DFORMAT)22);
UINT adapter = D3DADAPTER_DEFAULT;
D3DFORMAT format = D3DFMT_X8R8G8B8; // and other values
UINT r = m_Object->GetAdapterModecount(adapter, format);
All of which cause the check failure. m_Object is a valid IDirect3D9, and is used previously for a variety of other calls, specifically:
201, 80194887, Voodoo3D8, CVoodoo3D8::GetAdapterCount() == 3
201, 80195309, Voodoo3D8, CVoodoo3D8::GetAdapterIdentifier(0, 2, 0939CBAC) == 0
201, 80195309, Voodoo3D8, CVoodoo3D8::GetAdapterDisplayMode(0, 0018F5B4) == 0
201, 80196541, Voodoo3D8, CVoodoo3D8::GetAdapterModeCount(0, D3DFMT_X8R8G8B8) == 80
The sequence is logged by debug trace code, and appears to be correct and returning the expected values (3 monitors and so forth). The first 3 calls, by the same object on my part (a single instance of CVoodoo3D8), all succeed with no stack warnings. The fourth does not.
If I reorder the calls, to cause GetAdapterModeCount to be called immediately before any of the others in the same object, the same run-time check failure appears. From testing, this seems to rule out an immediately-previous call breaking the stack; the 4 methods calling those 4 functions all occur at different places, and calling GetAdapterModeCount anywhere from within this file causes the issue.
Which brings us to the unusual part. A different class (CVoodoo3D9) also calls the same sequence of IDirect3D9 methods, with similar parameters, but does not fail (it is the equivalent wrapper class for D3D9). The objects are not used at the same time (the code picks on or the other depending on the render process I need), but both give the same behavior every time. The code for the other class is held in another file, which led me to suspect preprocessor issues (more on that shortly).
After that didn't provide any information, I examined the calling conventions of my code and parameters. Again, nothing came to light. The codebase compiles with /w4 /wX and has for some time, with SAL on most functions and all PREfast rules enabled (and passing).
In particular, the call fails when called within this class, whether the call to my method comes from my code or another program using the object. It fails regardless of where it is called, but only within this file.
The full method is:
UINT STDMETHODCALLTYPE CVoodoo3D8::GetAdapterModeCount(UINT Adapter)
{
UINT r = m_Object->GetAdapterModeCount(D3DADAPTER_DEFAULT, D3DFMT_X8R8G8B8);
gpVoodooLogger->LogMessage(LL_Debug, VOODOO_D3D_NAME, Format("CVoodoo3D8::GetAdapterModeCount(%d, D3DFMT_X8R8G8B8) == %d") << Adapter << r);
return r;
}
The check failure occurs immediately after the call to GetAdapterModeCount and again as my method returns, if allowed to execute to that point.
The preprocessor output, as given by the preprocess-to-file option, has the method declaration (from d3d9.h) correctly as:
virtual __declspec(nothrow) UINT __stdcall GetAdapterModeCount( UINT Adapter,D3DFORMAT Format) = 0;
The declaration of my method is essentially identical:
virtual __declspec(nothrow) UINT __stdcall GetAdapterModeCount(UINT Adapter);
My method hardly expands, becoming:
UINT __stdcall CVoodoo3D8::GetAdapterModeCount(UINT Adapter)
{
UINT r = m_Object->GetAdapterModeCount(D3DADAPTER_DEFAULT, D3DFMT_X8R8G8B8);
gpVoodooLogger->LogMessage(LL_Debug, L"Voodoo3D8", Format("CVoodoo3D8::GetAdapterModeCount(%d, D3DFMT_X8R8G8B8) == %d") << Adapter << r);
return r;
}
The preprocessor output seems correct for both methods, in the declaration and definition.
The assembly listing up to the point of failure is:
UINT STDMETHODCALLTYPE CVoodoo3D8::GetAdapterModeCount(UINT Adapter)
{
642385E0 push ebp
642385E1 mov ebp,esp
642385E3 sub esp,1Ch
642385E6 push ebx
642385E7 push esi
642385E8 push edi
642385E9 mov eax,0CCCCCCCCh
642385EE mov dword ptr [ebp-1Ch],eax
642385F1 mov dword ptr [ebp-18h],eax
642385F4 mov dword ptr [ebp-14h],eax
642385F7 mov dword ptr [ebp-10h],eax
642385FA mov dword ptr [ebp-0Ch],eax
642385FD mov dword ptr [ebp-8],eax
64238600 mov dword ptr [ebp-4],eax
UINT r = m_Object->GetAdapterModeCount(D3DADAPTER_DEFAULT, D3DFMT_X8R8G8B8);
64238603 mov esi,esp
64238605 push 16h
64238607 push 0
64238609 mov eax,dword ptr [this]
6423860C mov ecx,dword ptr [eax+8]
6423860F mov edx,dword ptr [this]
64238612 mov eax,dword ptr [edx+8]
64238615 mov ecx,dword ptr [ecx]
64238617 push eax
64238618 mov edx,dword ptr [ecx+18h]
6423861B call edx
6423861D cmp esi,esp
6423861F call _RTC_CheckEsp (6424B520h)
64238624 mov dword ptr [r],eax
For clarification, the error comes at 6423861F (the call to _RTC_CheckEsp), suggesting that the call or preparation broke the stack. I am working with the assumption that since the same call works in other places, it is not something within the call breaking things.
To my untrained eye, the only unusual part is the pair of mov register, dword ptr [register+8]. As it is a 32-bit system, I'm not sure if +8 could be incrementing too far, or how it could be getting into the build if so.
Shortly after my method returns, apparently due to the call breaking ESP, the program segfaults. If I don't call GetAdapterModeCount and simply return a value, the program executes as expected.
Additionally, a release build (no RTC) segfaults at a similar point, with the stack:
d3d8.dll!CEnum::EnumAdapterModes() + 0x13b bytes
Voodoo_DX89.dll!ClassCreate() + 0x963 bytes
Although I'm not sure of the implications of the address. It is not, so far as I can tell, the same place that segfaults in debug builds; those are within the program after my methods returns, this appears to be during one of my methods which retrieves data from D3D8. Edit: The segfault occurs in a later call, which I'm currently debugging.
At this point, I'm at a complete loss as to what is going wrong or how, and am out of things to check.

Answer 1

I don't see anything wrong with what you're doing or with your generated assembly code.
I can answer your one concern, though.
ecx,dword ptr [eax+8]
What this is doing is moving the address of m_Object into the ecx register. The +8 is the offset within your class to m_Object, which is probably correct.
Something to look at. Step through the assembly code until you reach this point:
6423861B call edx
6423861D cmp esi,esp
At that point check the esi and esp registers (in VS just hover your mouse over the register names).
Before the call is executed, ESI should be 12 higher than ESP. After the call, they should be equal. If they are not, post what they are.
Update:
So what is catching my eye is that of the 4 methods you are showing that you are calling, only GetAdapterModeCount has a different signature between D3D8 and D3D9, and that signature is different by 4 bytes, which is the difference in your stack.
How is m_Object obtained? Since this is some kind of adapter between D3D8 and D3D9, is it possible that your m_Object is actually an IDirect3D8 object that is being cast as IDirect3D9 at some point? That would explain the error, and why it works in another context, if you are obtaining the D3D object in a different way.