SetLayeredWindowAttributes not working on Windows 8+ (C++) - c++

Question

I'm writing a library that makes a color in the window client area invisible.
In the application half, first I call window_fix_transparent_color() to make the window layered. Then I use window_set_transparent_color() to make a color in the client area invisible.
Here's my library's code:
#define _WIN32_WINNT 0x0501
#include <windows.h>
extern "C"
{
void window_fix_transparent_color(double window_handle)
{
// sets the window flags to support RGB color transparency.
SetWindowLong((HWND)(DWORD)window_handle,GWL_EXSTYLE,
GetWindowLong((HWND)(DWORD)window_handle,GWL_EXSTYLE)|WS_EX_LAYERED);
}
void window_set_transparent_color(double window_handle,double red,double green,double blue)
{
// sets the RGB color to be transparent for the specified window.
SetLayeredWindowAttributes((HWND)(DWORD)window_handle,RGB(red,green,blue),255,LWA_COLORKEY);
}
}
I'm using the version of MinGW that is packaged with the latest Code::Blocks as my compiler. It works on Windows 7, but not on Windows 8, 8.1, or 10...
Any ideas as to why that is? Also, a weird thing worth noting - it used to work on Windows 8/8.1/10, which leads me to believe a certain Windows update for those platforms may have broken my code. I haven't made any changes to my code since the time it stopped working on platforms past Windows 7.
Thanks!

Answer 1

Why are you using strange types and casts? You should never cast a handle type to DWORD, use INT_PTR or UINT_PTR if you must. A double is actually larger than a HWND in 32-bit applications so you are actually wasting space in addition to making things harder for yourself. A double cannot be used to store a handle in a 64-bit application!
You are also not checking the return value of SetLayeredWindowAttributes so it is impossible to know what the problem really is.
Rewrite the function with correct types and error handling:
void display_error(DWORD error)
{
char buf[100];
wsprintfA(buf, "Error %u!", error);
MessageBoxA(NULL, buf, 0, 0); // Ideally you would pass a window handle here but I don't know if your handle is actually valid
}
void window_fix_transparent_color(HWND window_handle)
{
DWORD error;
// get the window flags to see if RGB color transparency is supported.
SetLastError(0);
LONG_PTR ExStyle = GetWindowLongPtr(window_handle, GWL_EXSTYLE);
if (ExStyle == 0)
{
error = GetLastError();
if (error != 0)
{
display_error(error);
return;
}
}
if ((ExStyle & WS_EX_LAYERED) == 0)
{
// set the window flags to support RGB color transparency.
SetLastError(0);
if (!SetWindowLongPtr(window_handle, GWL_EXSTYLE, ExStyle | WS_EX_LAYERED))
{
error = GetLastError();
if (error != 0)
display_error(error);
}
}
}
void window_set_transparent_color(HWND window_handle, BYTE red, BYTE green, BYTE blue)
{
// sets the RGB color to be transparent for the specified window.
if (!SetLayeredWindowAttributes(window_handle, RGB(red, green, blue), 255, LWA_COLORKEY))
{
display_error(GetLastError());
}
}
...
HWND mywindow = CreateWindowEx(...);
window_fix_transparent_color(mywindow);
window_set_transparent_color(mywindow, ...);

Answer 2

My guess is that you're using "basic" or "classic" theme on Windows 7. Although
undocumented, it activates Windows XP compatibility mode for the Desktop Window
Manager, and changes the way layered windows work. That doesn't happen in later
versions of Windows.