How to Unhook windows hooks - c++

Question

I have a program (not a DLL) that creates a thread, and in this thread I create a hook using SetWindowsHookExW() to get keyboard events. I also have a global variable that I can use to turn off this thread.
The thing is that I don't know how to release the hooks and continue the function/thread, because it also does some cleanup. When I use UnhookWindowsHookEx() and place breakpoints in the following lines, the breakpoints are never reached.
The example on MSDN is "broken", because it needs an app.h; Also, every tutorial on the Internet, either input systems, keylogger, etc, they talk about DLLs, that I'm not using, and some of them implement a wrapper Unhook function but they never use it.
What I've been trying:
#include <Windows.h>
bool bContinue;
wchar_t vkTecla;
LRESULT CALLBACK KeyboardCallBack(int nCode, WPARAM wParam, LPARAM lParam)
{
if (nCode >= 0)
{
vkTecla = ((KBDLLHOOKSTRUCT*)lParam)->vkCode;
if (wParam == WM_KEYDOWN)
{
// Do stuff here
}
// ...
}
if (bContinue)
{
UnhookWindowsHookEx(hHook);
return 0;
}
return CallNextHookEx(hHook, nCode, wParam, lParam);
}
void MainThread(HANDLE hfile)
{
hHook = SetWindowsHookExW(WH_KEYBOARD_LL, KeyboardCallBack, 0, 0);
MSG msg;
// I tested these 3 cases
/*************/
while(GetMessageW(&msg, 0, 0, 0));
// Or
GetMessageW(&msg, 0, 0, 0);
/*************/
while(GetMessageW(&msg, 0, 0, 0))
{
TranslateMessage(&msg);
DispatchMessage(&msg);
}
/*************/
while (bContinue)
{
// PeekMessage(...);
GetMessageW(&msg, 0, 0, 0);
TranslateMessage(&msg);
DispatchMessage(&msg);
}
// In this example I would remove the other Unhook function
UnhookWindowsHookEx(hHook);
/*************/
CloseHandle(hfile);
}
If I do the first example, the:
while(GetMessageW(&msg, 0, 0, 0));
and check bContinue inside of the callback function;
or use the third one:
while (bContinue)
{
GetMessageW(&msg, 0, 0, 0);
TranslateMessage(&msg);
DispatchMessage(&msg);
}
UnhookWindowsHookEx(hHook);
And place a breakpoint in CloseHandle(hArquivo);, when I make the bContinue = false, this line never gets executed.
So the question is:
Where do I place UnhookWindowsHookEx()? Is it inside the Callback function, or somewhere else?
When I release the hook, will the thread continue? Because at the moment, I don't get the program to stop at breakpoints past the unhooking that have some cleanup.

Answer 1

GetMessage() is a blocking function. It will not exit until the message queue of the calling thread has a message available for the caller to process.
But, you are running this code in a worker thread that has no UI of its own. And, although hooks do use messaging internally, they are private to the hook implementation and the caller of (Peek|Get)Message() will never see them. So, unless the other threads in your app are posting messages to this worker thread via PostThreadMessage(), there are simply no messages for GetMessage() to return to your code. That is why your loop doesn't break.
So, you need to either:
use PeekMessage() instead of GetMessage() so you can monitor your bContinue variable in between polls of the message queue:
#include <Windows.h>
bool bContinue = TRUE;
LRESULT CALLBACK KeyboardCallBack(int nCode, WPARAM wParam, LPARAM lParam)
{
if (nCode >= 0)
{
wchar_t vkTecla = ((KBDLLHOOKSTRUCT*)lParam)->vkCode;
if (wParam == WM_KEYDOWN)
{
// Do stuff here
}
// ...
}
return CallNextHookEx(hHook, nCode, wParam, lParam);
}
void MainThread(HANDLE hfile)
{
HHOOK hHook = SetWindowsHookExW(WH_KEYBOARD_LL, KeyboardCallBack, 0, 0);
MSG msg;
while (bContinue)
{
if (PeekMessageW(&msg, NULL, 0, 0, PM_REMOVE))
{
TranslateMessage(&msg);
DispatchMessage(&msg);
}
else
Sleep(10);
}
UnhookWindowsHookEx(hHook);
CloseHandle(hfile);
}
...
// to stop the thread:
bContinue = FALSE;
change bContinue to be a HANDLE returned by CreateEvent(), and then use MsgWaitForMultipleObjects() in a loop to monitor that event and the message queue at the same time, calling (Get|Peek)Message() only when it reports the message queue has messages to process. Then you can use SetEvent() when you want to trigger the loop to end.
#include <Windows.h>
HANDLE hStop = CreateEvent(NULL, TRUE, FALSE, NULL);
LRESULT CALLBACK KeyboardCallBack(int nCode, WPARAM wParam, LPARAM lParam)
{
if (nCode >= 0)
{
wchar_t vkTecla = ((KBDLLHOOKSTRUCT*)lParam)->vkCode;
if (wParam == WM_KEYDOWN)
{
// Do stuff here
}
// ...
}
return CallNextHookEx(hHook, nCode, wParam, lParam);
}
void MainThread(HANDLE hfile)
{
HHOOK hHook = SetWindowsHookExW(WH_KEYBOARD_LL, KeyboardCallBack, 0, 0);
MSG msg;
while (MsgWaitForMultipleObjects(1, &hStop, FALSE, INFINITE, QS_ALLINPUT) == (WAIT_OBJECT_0 + 1))
{
while (PeekMessageW(&msg, NULL, 0, 0, PM_REMOVE))
{
TranslateMessage(&msg);
DispatchMessage(&msg);
}
}
UnhookWindowsHookEx(hHook);
CloseHandle(hfile);
}
...
// to stop the thread:
SetEvent(hStop);