MessageBox() returning 0 when called from Win32 button press - c++

Question

I'm barely a week into Win32 GUI programming, so I'm confident/hopeful this is something simple. For your consideration, I've provided a sample of my code below.
As a quick brief, my application watches for another application to be open so that I can modify that application's memory. I've placed this initial check at the top of my UI looper as follows such that it will only run once (in which case the user can only cancel and the app closes, or execution resumes past the if statement with hWndApp having a value assigned):
if (!hWndApp) {
appCheck();
}
If I start my application and the other application isn't running, appCheck(); fires and the waterfall of execution therefrom acts accordingly: a message box appears (and will keep appearing if the user clicks "Retry") until the other application's window is found. If the user clicks "Cancel," the application exits and all is well. If they click "Retry" once the other application is running, then my application will finish painting to the screen and execution is normal.
The funny business starts the next time I call appCheck(); (which happens when one of either two buttons is clicked) if the other application has been opened, then closed.
While debugging (via Visual Studio 2017), the variable I assign the MessageBox() call to equals 0 when the other application has been opened, then closed, then the following:
Push button to call appCheck();
getProcessHandleAndPID(); is then called within the if statement because the window can't be found
AppCheckMessageBox(); is then called within the while loop because the window can't be found
The debugger then points to the first line inside of AppCheckMessageBox();, which is the entire int msgboxID = MessageBox(); bit
Pressing F11 (Step Into), no window shows and the application exists after the default choice in the switch statement is triggered since the value of msgboxID is 0 (indicating the MessageBox() call failed, so I tried calling GetLastError() before exit(EXIT_FAILURE), but to no avail).
I'm not sure where my error lies, but I've sought many solutions--none of which have panned out. Finally, if you see any variables below that look misnamed or undeclared, it's solely due to my modification of the code for this post to try to generalize/ shorten it. I may have left out some global variable declarations, etc.
Thank you for any help/guidance you can provide! I'm at my wit's end with this right now...lol.
//--------------------------------------------------
// Function to get process handle and PID of app /
//------------------------------------------------
void getProcessHandleAndPID()
{
hWndApp = FindWindow(NULL, _T("NameOfApplication"));
while (!hWndApp) {
AppCheckMessageBox();
}
//Much more code here
}
//---------------------------------
// Function to show message box /
//-------------------------------
int AppCheckMessageBox()
{
int msgboxID = MessageBox(
NULL, //I tried making this hWndApp as well, but no difference
(LPCWSTR)L"Cancel to exit or start the app and click Retry.",
(LPCWSTR)L"Application Not Found!",
MB_ICONSTOP | MB_RETRYCANCEL | MB_SYSTEMMODAL | MB_SETFOREGROUND
);
switch (msgboxID)
{
case IDCANCEL:
exit(EXIT_FAILURE);
case IDRETRY:
getProcessHandleAndPID();
break;
default:
//GetLastError();
exit(EXIT_FAILURE);
}
return msgboxID;
}
//------------------------------------------------
// Function to check if application is running /
//----------------------------------------------
void appCheck() {
if (!FindWindow(NULL, _T("NameOfApplication"))) {
getProcessHandleAndPID();
}
}
//--------------------------------------------------
// Function to get process handle and PID of app /
//------------------------------------------------
LRESULT CALLBACK WndProc(HWND hWnd, UINT message, WPARAM wParam, LPARAM lParam)
{
if (!hWndApp) {
appCheck();
}
switch (message)
{
case WM_PAINT:
//Code here
break;
case WM_CREATE:
//Code here
break;
case WM_COMMAND:
switch (HIWORD(wParam))
{
case BN_CLICKED:
if (LOWORD(wParam) == BTN_ENABLE) {
appCheck();
//Do stuff with button press
}
if (LOWORD(wParam) == BTN_DISABLE) {
appCheck();
//Do stuff with button press
}
break;
}
break;
case WM_DESTROY:
PostQuitMessage(0);
break;
default:
return DefWindowProc(hWnd, message, wParam, lParam);
break;
}
return 0;
}

Answer 1

Not really an answer, but too long for comments:
This isn't your real code. Many things are missing, there are mismatched braces which would prevent compilation, etc. Always show your real code if you want help.
As David says, calling appCheck() every time you receive a message is the wrong approach. If you want to call it once after your window opens, do that before entering your message loop.
Your code is needlessly recursive (getProcessHandleAndPID() calls AppCheckMessageBox() which calls getProcessHandleAndPID() which calls...) and will potentially lead to a stack overflow.