Hooking with MS Detours crash when hook function is called C++ - c++

Question

I wrote a simple program that does 3 things:
It calls MessageBoxA (MBA) with the text "NOT HOOKED"
Then it loads a dll file that I've created that hooks the MBA function and recalls MBA with the text "HOOKED".
After that, it calls MBA again with the same text ("NOT HOOKED").
Of course, the second MBA call should be hooked and display a message with "HOOKED" text it.
Eventually it calls FreeLibrary and exits.
Here is the .cpp file:
#include <iostream>
#include <Windows.h>
using namespace std;
int main()
{
//Place the path of the dll file here, "DLLs\\HookDLL.dll" is the default path.
char dllPath[] = "HookDLL.dll";
//Display a pop-up message with the "NOT HOOKED" message and title.
MessageBoxA(NULL, "NOT HOOKED", "NOT HOOKED", MB_OK);
//Load the dll file
HMODULE hModule = LoadLibraryA((LPCSTR)dllPath);
//If hModule is null, then the dll wasn't loaded.
//An error message will be printed out to the console.
if (!hModule) {
cout << "Couldn't load the DLL file!" << endl;
return 1;
}
//This is the tricky part.
//This should display a pop-up message like before with the "NOT HOOKED" message and title,
//but the dll that was loaded should hook MessageBoxA function,
//and call a new one with a "HOOKED" message and title instead.
MessageBoxA(NULL, "NOT HOOKED", "NOT HOOKED", MB_OK);
FreeLibrary(hModule);
return 0;
}
here is the .dll file:
#include "pch.h"
#include "detours.h"
#include <iostream>
#include <Windows.h>
using namespace std;
typedef int(WINAPI* MBA)(HWND, LPCSTR, LPCSTR, UINT);
MBA originalMBA = NULL;
int HookedMessageBoxA(
HWND hWnd,
LPCSTR lpText,
LPCSTR lpCaption,
UINT uType
) {
return originalMBA(NULL, "HOOKED", "HOOKED", MB_OK);
}
BOOL APIENTRY DllMain(HMODULE hModule,
DWORD reason,
LPVOID lpReserved
)
{
if (reason == DLL_PROCESS_ATTACH) {
DetourRestoreAfterWith();
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
originalMBA = (MBA)DetourFindFunction("user32.dll", "MessageBoxA");//Pointer the the original MBA (MessageBoxA) function.
DetourAttach(&(PVOID&)originalMBA, (PVOID)HookedMessageBoxA);
DetourTransactionCommit();
}
return TRUE;
}
When I build and run on Debug mode, it crashes on the second MBA call (in the .cpp file of course):
It displays the hooked MBA with "HOOKED" in it like it should, then the it crashes, printing the error below and the program exits with code 3:
Run-Time Check Failure #0 - The value of ESP was not properly saved across a function call.
This is usually a result of calling a function declared with one calling convention with a function
pointer declared with a different calling convention.
I've checked this error and the exit code and found lots of informative posts and solutions, but I couldn't make any of them work for me (maybe I've done something wrong).
NOTE1:
If I'm on Release mode, I can call as many MBA calls as I want as long as they precedes the FreeLibrary call, and all of them will be hooked fine by the .dll file, and the program will exit correctly.
But if I try to call the MBA function after the FreeLibrary call - the program crashes with this error:
Exception thrown at 0x50011000 in ProgrammingTask.exe: 0xC0000005: Access violation executing location 0x50011000.
NOTE2:
I tried detaching the dll using DetourDetouch, but it didn't solve it, maybe I did it wrong.
Also, I tried reading about CreateRemoteThread, but was too messy for me.
Thanks in advance.

Answer 1

FOUND IT!
Forgot to add __stdcall to the HookedMessageBox function.
So, instead of int HookedMessageBoxA
I rewrote it to this:
int __stdcall HookedMessageBoxA
Thanks all for your help! <3