Winapi hook via mhook causes program crash or hang - c++

Question

I am trying to hook StartDocW to intercept printing via mhook. I use AppInit_DLLs to load my library.
DLL code is simple:
#include <windows.h>
#include "mhook/mhook-lib/mhook.h"
using StartDocPtr = int(*)(HDC, const DOCINFO*);
StartDocPtr orig;
int HookedStartDocW(HDC hdc, const DOCINFO* lpdi) {
return orig(hdc, lpdi);
}
BOOL WINAPI DllMain(__in HINSTANCE, __in DWORD Reason, __in LPVOID) {
orig = (StartDocPtr)GetProcAddress(GetModuleHandle("gdi32"), "StartDocW");
switch (Reason)
{
case DLL_PROCESS_ATTACH:
Mhook_SetHook((PVOID*)&orig, &HookedStartDocW);
break;
case DLL_PROCESS_DETACH:
Mhook_Unhook((PVOID*)&orig);
break;
}
}
Hook is working and printing is done OK. But If I change HookStartDocW to following:
int HookedStartDocW(HDC hdc, const DOCINFO* lpdi) {
char buf[40];
GetModuleFileName(NULL, buf, 40);
return orig(hdc, lpdi);
}
Programs on printing will crash immediately. Even if I just leave char buf[40] and comment GetModuleHandle - program will hang. Why is this happening?
Moreover, if program crashes\hangs on printing (if I add anything besides return orig(hdc, lpdi)) - PC starts to behave very weirdly, refusing to run programs, etc. If I reboot it - Windows just endlessly spins on boot screen, the only way to bring it back to live - is to boot via liveCD and rename\delete my hook DLL.
Printing programs: Excel 2016, notepad.
Compiler - MSVC 2015, x64 release DLL compilation, using MBCS instead of unicode.

Answer 1

Your hook is declared wrong.
Look at the actual declaration of StartDocW() in Wingdi.h:
__gdi_entry WINGDIAPI int WINAPI StartDocW(__in HDC hdc, __in CONST DOCINFOW *lpdi);
You can ignore __gdi_entry. WINGDIAPI simply resolves to __declspec(dllimport). What is important in this declaration is the WINAPI.
Like almost all Win32 API functions, StartDocW() uses the __stdcall calling convention. The WINAPI macro resolves to __stdcall.
Your code does not specify any calling convention at all, so it uses your compiler's default, which is usually __cdecl instead. So you are mismanaging the call stack. That is why your code crashes.
You are also using DOCINFO when you should be using DOCINFOW instead. It is clear in your code that you are compiling for MBCS and not for UNICODE, so DOCINFO maps to DOCINFOA. You can't pass a DOCINFOA to StartDocW(), it expects a DOCINFOW instead.
You need to fix your declarations, eg:
#include <windows.h>
#include "mhook/mhook-lib/mhook.h"
using StartDocPtr = int (WINAPI *)(HDC, const DOCINFOW*);
StartDocPtr orig = nullptr;
int WINAPI HookedStartDocW(HDC hdc, const DOCINFOW* lpdi) {
//...
return orig(hdc, lpdi);
}
BOOL WINAPI DllMain(__in HINSTANCE, __in DWORD Reason, __in LPVOID) {
orig = (StartDocPtr) GetProcAddress(GetModuleHandle(TEXT("gdi32")), "StartDocW");
switch (Reason)
{
case DLL_PROCESS_ATTACH:
Mhook_SetHook((PVOID*)&orig, &HookedStartDocW);
break;
case DLL_PROCESS_DETACH:
Mhook_Unhook((PVOID*)&orig);
break;
}
}