Why GetKeyState changed the behavior of ToUnicodeEx? - c++

Question

In the code below:-
BYTE ks[256];
auto keyboard_layout = GetKeyboardLayout(0);
GetKeyboardState(ks);
auto w = WCHAR(malloc(1));
ToUnicodeEx(wParam, MapVirtualKey(wParam, MAPVK_VK_TO_VSC), ks, LPWSTR(&w), 1, 0, keyboard_layout);
wcout << "KEY:" << w << endl;
The output only shows lowercase letters such as :-
KEY:a
KEY:b
KEY:2
Even when pressing SHIFT+A or SHIFT+2
But adding GetKeyState(VK_SHIFT) and/or GetKeyState(VK_CAPITAL) in the code below:-
auto shifted = false;
auto caps = false;
if (GetKeyState(VK_SHIFT) < 0)
{
shifted = true;
cout << "Shifted!" << endl;
}
if (GetKeyState(VK_CAPITAL) < 0)
{
shifted = true;
cout << "Caps!" << endl;
}
BYTE ks[256];
auto keyboard_layout = GetKeyboardLayout(0);
GetKeyboardState(ks);
auto w = WCHAR(malloc(1));
ToUnicodeEx(wParam, MapVirtualKey(wParam, MAPVK_VK_TO_VSC), ks, LPWSTR(&w), 1, 0, keyboard_layout);
wcout << "KEY:" << w << endl;
The behavior of the code changed directly when pressing SHIFT+A or SHIFT+2 to
KEY:A
KEY:B
KEY:#
I tried this on ToUnicode, ToAsciiEx and ToAscii and they shows the same situation as above.
I used WH_KEYBOARD hook in a separate DLL file called hook.dll and linked with a console application.
So my question is: why GetKeyState function enabled the detection of SHIFT and Caps Lock key? In addition,

Answer 1

Using auto w = WCHAR(malloc(1)) is wrong. malloc() dynamically allocates a block of bytes, not characters. WCHAR is 2 bytes in size, but you are allocating only 1 byte. Which doesn't matter since you don't use the pointer anyway. You are type-casting the pointer to a single WCHAR, truncating the pointer value. And then you are dismissing the value when passing &w to ToUnicodeEx() as it will overwrite the value of w. You are then leaking the allocated memory since you are not calling free() to deallocate it.
You don't need the malloc() at all:
WCHAR w;
ToUnicodeEx(..., &w, 1, ...);
wcout << "KEY:" << w << endl;
However, ToUnicodeEx() can potentially return more than 2 characters, so you should allocate extra room to account for that. Just use a local fixed array, like you do for GetKeyboardState(). And do pay attention to the return value, it contains important information.
As for the key states, since you are calling GetKeyboardState(), you don't need to use GetKeyState().
Try something more like this:
BYTE ks[256];
auto keyboard_layout = GetKeyboardLayout(0);
GetKeyboardState(ks);
if (ks[VK_SHIFT] & 0x80) wcout << L"Shifted!" << endl;
if (ks[VK_CAPITAL] & 0x80) wcout << L"Caps!" << endl;
WCHAR w[5] = {};
int ret = ToUnicodeEx(wParam, MapVirtualKey(wParam, MAPVK_VK_TO_VSC), ks, w, 4, 0, keyboard_layout);
switch (ret)
{
case -1:
wcout << L"DEAD KEY:" << w << endl;
break;
case 0:
wcout << L"NO TRANSLATION" << endl;
break;
case 1:
wcout << L"KEY:" << w << endl;
break;
case 2:
case 3:
case 4:
w[ret] = 0;
wcout << L"KEYS:" << w << endl;
break;
}