How to hook functions that aren't in the Win32 API? - c++

Question

I made a very simple program that create a windows and display "Random number is: [random number from 0 to 9]" in the top left corner. The function that display the number looks like this:
void DisplayThings(HDC hdc, HWND hWnd, int randomNum)
{
std::stringstream text;
text << "Random number is: " << randomNum;
TextOut(hdc, 0, 0, text.str().c_str(), text.str().length());
}
In OllyDBG, I found it at the address 0x11211A0:
Next, I made a dll that tries to detour the DisplayThings function with the help of Microsoft Detour, here is what the dll looks like:
#pragma comment(lib, "detours.lib")
#include <Windows.h>
#include <detours.h>
#include <tchar.h>
#include <sstream>
typedef void (*pDisplayThingsFunc)(HDC hdc, HWND hWnd, int randomNum);
void DisplayThingsFunc(HDC hdc, HWND hWnd, int randomNum)
{
printf("function is being detoured\n");
TextOut(hdc, 0, 20, L"detoured", 8);
}
pDisplayThingsFunc DisplayThingsFuncToDetour = (pDisplayThingsFunc)(0x11211A0);
BOOL APIENTRY DllMain( HANDLE hModule, DWORD ul_reason_for_call, LPVOID lpReserved )
{
//DWORD *hiddenValueAdress = (DWORD*)(*(DWORD*)0x020FAB8);
switch (ul_reason_for_call)
{
case DLL_PROCESS_ATTACH:
AllocConsole();
freopen("CONOUT$", "w", stdout);
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)DisplayThingsFuncToDetour, DisplayThingsFunc);
DetourTransactionCommit();
break;
case DLL_PROCESS_DETACH:
FreeConsole();
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourDetach(&(PVOID&)DisplayThingsFuncToDetour, DisplayThingsFunc);
DetourTransactionCommit();
break;
}
return TRUE;
}
Once I inject the dll, the address get correctly replaced with a JMP to my new functions, but then something weird happens... The console keep printing "function is being detoured" but the TextOut function fails to display anything...
Any help would be appreciated!

Answer 1

The compiler is optimizing the code, because it deduced that it could safely do so for this function.
It is not exported.
It is not used with function pointers.
It is not used outside the scope that is analyzed. (Depends on compiler settings)
There is most likely only one caller (if there are more different callers with different settings it's less likely that arguments get optimized away).
You could change one of those things (for example by exporting the symbol, by assigning it to a function pointer etc..).
Or you could try to hook it as-is now.
First you need to deduce where the argument you are interested in is getting passed trough, which is most likely a register.
The easiest way to do this is to analyze the call to TextOut and trace the hdc argument back.
If you are lucky and it got optimized simply by making it fastcall, change the type of both your function and your function pointer to fastcall, and be done.
If you are not that lucky, you might have to grab it from an extra register with inline assembly.