Weird bug with WndProc and DispatchMessage. Member Functions don't exist? - c++

Question

So a bit of a weird bug going on when getting a WM_KEYDOWN msg in my Window class.
I have a Global WndProc function that determines which window instance it is and sends the message to it's own local WndProc function.
//Every Windows Message will hit this function.
LRESULT CALLBACK GlobalWndProc(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam) {
//Get the Window Instance from the userdata
Window* targetWindow = (Window*)GetWindowLongPtr(hwnd, GWLP_USERDATA);
//If no window exists, then it must be the first time so we should set it
if (!targetWindow) {
//First let's try and extract the Window instance pointer from the lparam
CREATESTRUCT* createStruct = (CREATESTRUCT*)lparam;
if (createStruct) {
targetWindow = (Window*)createStruct->lpCreateParams;
SetWindowLongPtr(hwnd, GWLP_USERDATA, (LONG_PTR)targetWindow);
}
else {
//It was some other message which we just can't deal with right now
return DefWindowProc(hwnd, msg, wparam, lparam);
}
}
//Now we can pipe it to the Window's local wnd proc function
return targetWindow->LocalWndProc(hwnd, msg, wparam, lparam);
}
And my local wndproc looks like this:
LRESULT CALLBACK Window::LocalWndProc(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam) {
switch (msg) {
case WM_KEYDOWN:
ToggleFullScreen(!m_fullScreen);
return 0;
break;
case WM_DESTROY:
PostQuitMessage(0);
return 0;
break;
case WM_CLOSE:
PostQuitMessage(0);
return 0;
break;
default:
return DefWindowProc(hwnd, msg, wparam, lparam);
}
}
So it's pretty simple at this point. If any key is pressed, the window should call its member function ToggleFullScreen.
Now for some reason when i run this and I hit any key on the keyboard, I get an exception fault:
Unhandled exception at 0x77c015ee in Athena_Debug.exe: 0xC0000005: Access violation reading location 0x0000012d.
With CallStack:
ntdll.dll!77c015ee()
ntdll.dll!77bf015e()
user32.dll!7588788a()
Athena_Debug.exe!Athena::Window::HandleWindowsMessage() Line 195 + 0xc bytes C++
Athena_Debug.exe!Athena::AthenaCore::StartEngine() Line 96 + 0x12 bytes C++
Athena_Debug.exe!WinMain(HINSTANCE__ * hInstance, HINSTANCE__ * hPrevInstance, char * lpCmdLine, int nCmdShow) Line 44 C++
Athena_Debug.exe!__tmainCRTStartup() Line 547 + 0x2c bytes C
Athena_Debug.exe!WinMainCRTStartup() Line 371 C
kernel32.dll!7702339a()
The line it actually breaks on is the DispatchMessage(&msg) line located here:
MSG Window::HandleWindowsMessage() {
MSG msg;
ZeroMemory(&msg, sizeof(MSG));
if (m_realTime) {
PeekMessage(&msg, m_hwnd, 0, 0, PM_REMOVE);
}
else {
GetMessage(&msg, m_hwnd, 0, 0);
}
if (msg.message != WM_NULL) {
TranslateMessage(&msg);
DispatchMessage(&msg);
}
return msg;
}
So the interesting thing is that if I comment out ToggleFullScreen and instead put an OutputDebugTrace there it works just fine and outputs the debug trace. It appears like it can't resolve the function for an instance? If I comment everything out in ToggleFullScreen, it still crashes. If i create a brand new function that does nothing and call it in response to WM_KEYDOWN it still errors.
Anyone have any idea why this is happening or some ideas on how I can go about fixing it?
Thanks!

Answer 1

I would be guessing that you have wrong pointer obntained from GetWindowLongPtr, as it was not set by SetWindowLongPtr. As I wrote in my comment, you should not test if it's empty (I have no idea what is there by default but i wouldn't assume anything.), but set it when you recieve WM_CRERATE (which is guaranteed to be the first message sent to a window). Have you tested if SetWindowLongPtr is acctually called?
Other messages are not dependent on any contents of Window class, so they are working well with wrong this pointer - it is an error to call method for wrong pointer, but if method is not actually using it, it will still work well.
EDIT:
Another possbile cause: Why are you not using result value from PeekMessage? i see that you are testing for WM_NULL, which should be 0 at this point (you are zeroing memory), but PeekMessage does not guarantee that it's not touching MSG structure even if it's not retrieving anything. Using PeekMessage result should be used at all times. Plus zeroing memory is rather inefficient at this point.

Answer 2

The first message does not have to be WM_NCCREATE (or WM_CREATE), you have to be prepared to get WM_SIZE or WM_GETMINMAXINFO (and probably other messages) before any create message!
In your code you would change if (createStruct) { to if (WM_NCCREATE==msg && createStruct) {...
You can see this in Raymond Chen's scratch program.

Answer 3

Your WindowProc callback should look more like this:
//Every Windows Message will hit this function.
LRESULT CALLBACK GlobalWndProc(HWND hwnd, UINT msg, WPARAM wparam, LPARAM lparam)
{
Window* targetWindow;
if (msg == WM_CREATE)
{
//extract the Window instance pointer from the lparam
CREATESTRUCT* createStruct = (CREATESTRUCT*)lparam;
targetWindow = (Window*)createStruct->lpCreateParams;
SetWindowLongPtr(hwnd, GWLP_USERDATA, (LONG_PTR)targetWindow);
}
else
{
//Get the Window Instance from the userdata
targetWindow = (Window*)GetWindowLongPtr(hwnd, GWLP_USERDATA);
}
if (targetWindow)
{
//Now we can pipe it to the Window's local wnd proc function
return targetWindow->LocalWndProc(hwnd, msg, wparam, lparam);
}
//It was some other message which we just can't deal with right now
return DefWindowProc(hwnd, msg, wparam, lparam);
}