Hooking DirectX EndScene from an injected DLL - c++

Question

I want to detour EndScene from an arbitrary DirectX 9 application to create a small overlay. As an example, you could take the frame counter overlay of FRAPS, which is shown in games when activated.
I know the following methods to do this:
Creating a new d3d9.dll, which is then copied to the games path. Since the current folder is searched first, before going to system32 etc., my modified DLL gets loaded, executing my additional code.
Downside: You have to put it there before you start the game.
Same as the first method, but replacing the DLL in system32 directly.
Downside: You cannot add game specific code. You cannot exclude applications where you don't want your DLL to be loaded.
Getting the EndScene offset directly from the DLL using tools like IDA Pro 4.9 Free. Since the DLL gets loaded as is, you can just add this offset to the DLL starting address, when it is mapped to the game, to get the actual offset, and then hook it.
Downside: The offset is not the same on every system.
Hooking Direct3DCreate9 to get the D3D9, then hooking D3D9->CreateDevice to get the device pointer, and then hooking Device->EndScene through the virtual table.
Downside: The DLL cannot be injected, when the process is already running. You have to start the process with the CREATE_SUSPENDED flag to hook the initial Direct3DCreate9.
Creating a new Device in a new window, as soon as the DLL gets injected. Then, getting the EndScene offset from this device and hooking it, resulting in a hook for the device which is used by the game.
Downside: as of some information I have read, creating a second device may interfere with the existing device, and it may bug with windowed vs. fullscreen mode etc.
Same as the third method. However, you'll do a pattern scan to get EndScene.
Downside: doesn't look that reliable.
How can I hook EndScene from an injected DLL, which may be loaded when the game is already running, without having to deal with different d3d9.dll's on other systems, and with a method which is reliable? How does FRAPS for example perform it's DirectX hooks?
The DLL should not apply to all games, just to specific processes where I inject it via CreateRemoteThread.

Answer 1

You install a system wide hook. (SetWindowsHookEx) With this done, you get to be loaded into every process.
Now when the hook is called, you look for a loaded d3d9.dll.
If one is loaded, you create a temporary D3D9 object, and walk the vtable to get the address of the EndScene method.
Then you can patch the EndScene call, with your own method. (Replace the first instruction in EndScene by a call to your method.
When you are done, you have to patch the call back, to call the original EndScene method. And then reinstall your patch.
This is the way FRAPS does it. (Link)
You can find a function address from the vtable of an interface.
So you can do the following (Pseudo-Code):
IDirect3DDevice9* pTempDev = ...;
const int EndSceneIndex = 26 (?);
typedef HRESULT (IDirect3DDevice9::* EndSceneFunc)( void );
BYTE* pVtable = reinterpret_cast<void*>( pTempDev );
EndSceneFunc = pVtable + sizeof(void*) * EndSceneIndex;
EndSceneFunc does now contain a pointer to the function itself. We can now either patch all call-sites or we can patch the function itself.
Beware that this all depends on the knowledge of the implementation of COM-Interfaces in Windows. But this works on all windows versions (either 32 or 64, not both at the same time).

Answer 2

I know this question is old, but this should work for any program using DirectX9, You are creating your own instance basically, and then getting the pointer to the VTable, then you just hook it. You will need detours 3.X btw:
//Just some typedefs:
typedef HRESULT (WINAPI* oEndScene) (LPDIRECT3DDEVICE9 D3DDevice);
static oEndScene EndScene;
//Do this in a function or whatever
HMODULE hDLL=GetModuleHandleA("d3d9");
LPDIRECT3D9(__stdcall*pDirect3DCreate9)(UINT) = (LPDIRECT3D9(__stdcall*)(UINT))GetProcAddress( hDLL, "Direct3DCreate9");
LPDIRECT3D9 pD3D = pDirect3DCreate9(D3D_SDK_VERSION);
D3DDISPLAYMODE d3ddm;
HRESULT hRes = pD3D->GetAdapterDisplayMode(D3DADAPTER_DEFAULT, &d3ddm );
D3DPRESENT_PARAMETERS d3dpp;
ZeroMemory( &d3dpp, sizeof(d3dpp));
d3dpp.Windowed = true;
d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD;
d3dpp.BackBufferFormat = d3ddm.Format;
WNDCLASSEX wc = { sizeof(WNDCLASSEX),CS_CLASSDC,TempWndProc,0L,0L,GetModuleHandle(NULL),NULL,NULL,NULL,NULL,("1"),NULL};
RegisterClassEx(&wc);
HWND hWnd = CreateWindow(("1"),NULL,WS_OVERLAPPEDWINDOW,100,100,300,300,GetDesktopWindow(),NULL,wc.hInstance,NULL);
hRes = pD3D->CreateDevice(
D3DADAPTER_DEFAULT,
D3DDEVTYPE_HAL,
hWnd,
D3DCREATE_SOFTWARE_VERTEXPROCESSING | D3DCREATE_DISABLE_DRIVER_MANAGEMENT,
&d3dpp, &ppReturnedDeviceInterface);
pD3D->Release();
DestroyWindow(hWnd);
if(pD3D == NULL){
//printf ("WARNING: D3D FAILED");
return false;
}
pInterface = (unsigned long*)*((unsigned long*)ppReturnedDeviceInterface);
EndScene = (oEndScene) (DWORD) pInterface[42];
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)EndScene, newEndScene);
DetourTransactionCommit();
And then your function:
HRESULT WINAPI D3D9Hook::newEndScene(LPDIRECT3DDEVICE9 pDevice)
{
//Do your stuff here
//Call the original (if you want)
return EndScene(pDevice);
}

Answer 3

A slightly old question I know - but in case anyone is interested in doing this with C#, here is my example on hooking the Direct3D 9 API using C#. This utilizes EasyHook an open source .NET assembly that allows you to 'safely' install hooks from managed code into unmanaged functions. (Note: EasyHook takes care of all the issues surrounding DLL injection - e.g. CREATE_SUSPENDED, ACL's, 32 vs 64-bit and so on)
I use a similar VTable approach as mentioned by Christopher via a small C++ helper dll to dynamically determine the address of the IDirect3DDevice9 functions to hook. This is done by creating a temporary window handle, and creating a throw-away IDirect3Device9 within the injected assembly before then hooking the desired functions. This allows your application to hook a target that is already running (Update: note that this is possible entirely within C# also - see comments on the linked page).
Update: there is also an updated version for hooking Direct3D 9, 10 and 11 still using EasyHook and with SharpDX instead of SlimDX