Partially transparent window OpenGL/Win32 - c++

Question

I have read every existing question on this tricky (niche) subject, but I am stuck. I have a Win32 window with an OpenGL context. I want my window to be partially transparent.
My result so far is that the entirety of the window is transparent. I want only the black area to be transparent, so that I can draw some 3D objects and they will look like they are coming of the window.
First, in my window class, I have set hbrBackground to black.
Windows::WindowClass windowClass;
windowClass.cbSize = sizeof(Windows::WindowClass);
windowClass.style = Windows::CS_VREDRAW | Windows::CS_HREDRAW;
windowClass.lpfnWndProc = WndProc;
windowClass.cbClsExtra = 0;
windowClass.cbWndExtra = 0;
windowClass.hInstance = moduleHandle;
windowClass.hIcon = Windows::LoadIcon(0u, (X*)Windows::IDI_QUESTION);
windowClass.hCursor = Windows::LoadCursor(0u, (X*)Windows::IDC_ARROW);
windowClass.hbrBackground = Windows::CreateSolidBrush(0x00000000);
windowClass.lpszMenuName = nullptr;
windowClass.lpszClassName = (X*)name;
windowClass.hIconSm = Windows::LoadIcon(0u, (X*)Windows::IDI_QUESTION);
I have created my window with the WS_EX_LAYERED flag.
windowHandle = Windows::CreateWindow(Windows::WS_EX_LAYERED, (X*)name, "", Windows::WS_POPUP, w / 4, h / 4, w / 2, h / 2, 0u, 0u, moduleHandle, 0u);
In my pixel format, I have enabled alpha and composition.
PixelFormatDescriptor format;
format.nSize = sizeof(PixelFormatDescriptor);
format.nVersion = 1;
format.dwFlags = PFD_SUPPORT_OPENGL | PFD_DRAW_TO_WINDOW | PFD_DOUBLEBUFFER | PFD_SUPPORT_COMPOSITION;
format.iPixelType = PFD_TYPE_RGBA;
format.cColorBits = 32;
format.cRedBits = 0;
format.cRedShift = 0;
format.cGreenBits = 0;
format.cGreenShift = 0;
format.cBlueBits = 0;
format.cBlueShift = 0;
format.cAlphaBits = 8;
format.cAlphaShift = 0;
format.cAccumBits = 0;
format.cAccumRedBits = 0;
format.cAccumGreenBits = 0;
format.cAccumBlueBits = 0;
format.cAccumAlphaBits = 0;
format.cDepthBits = 24;
format.cStencilBits = 8;
format.cAuxBuffers = 0;
format.iLayerType = PFD_MAIN_PLANE;
format.bReserved = 0;
format.dwLayerMask = 0;
format.dwVisibleMask = 0;
format.dwDamageMask = 0;
I have tried the blur region "trick", but it has no effect. My result is not related to this piece of code.
struct DwmBlurBehind
{
U4 dwFlags;
S4 fEnable;
X* blurRegionHandle;
S4 fTransitionOnMaximized;
};
DwmBlurBehind blurBehind;
blurBehind.dwFlags = Windows::DWM_BB_ENABLE | Windows::DWM_BB_BLURREGION;
blurBehind.fEnable = true;
blurBehind.blurRegionHandle = Windows::CreateRectRgn(0, 0, -1, -1);
blurBehind.fTransitionOnMaximized = false;
Windows::DwmEnableBlurBehindWindow(windowHandle, &blurBehind);
Finally, I have set the LWA_COLORKEY and LWA_ALPHA attributes. This is what gave me the effect displayed. However, the color key does not seem to be taken into account (I have tried non-zero values as well).
Windows::SetLayeredWindowAttributes(windowHandle, 0, 170, Windows::LWA_COLORKEY | Windows::LWA_ALPHA);
I did not forget to enable blending.
GL::Enable(GL::BLEND);
GL::BlendFunc(GL::SRC_ALPHA, GL::ONE_MINUS_SRC_ALPHA);

Answer 1

What you want to do requires window compositing which has been around since Windows Vista, so essentially every version of Windows you have to care about (Windows XP and earlier are at End of Life).
The key steps to take is, to enable DWM intra window compositing by enabling "Blur Behind Window" and use a WM_POPUP window; if you do not use WM_POPUP style the window manager will draw decorations and your OpenGL rendering will "hover" above that.
DWM_BLURBEHIND bb = {0};
bb.dwFlags = DWM_BB_ENABLE;
bb.fEnable = TRUE;
bb.hRgnBlur = NULL;
DwmEnableBlurBehindWindow(hWnd, &bb);
MARGINS margins = {-1};
impl_DwmExtendFrameIntoClientArea(hWnd, &margins);
Next, you must create an OpenGL context using the newer "attribute" API instead of using the pixelformat descriptor selection. With the attribute API you can select a transparent with alpha window format.
int attribs[] = {
WGL_DRAW_TO_WINDOW_ARB, TRUE,
WGL_DOUBLE_BUFFER_ARB, TRUE,
WGL_SUPPORT_OPENGL_ARB, TRUE,
WGL_PIXEL_TYPE_ARB, WGL_TYPE_RGBA_ARB,
WGL_TRANSPARENT_ARB, TRUE,
WGL_COLOR_BITS_ARB, 32,
WGL_RED_BITS_ARB, 8,
WGL_GREEN_BITS_ARB, 8,
WGL_BLUE_BITS_ARB, 8,
WGL_ALPHA_BITS_ARB, 8,
WGL_DEPTH_BITS_ARB, 24,
WGL_STENCIL_BITS_ARB, 8,
0, 0
};
INT iPF;
UINT num_formats_choosen;
if( !wglChoosePixelFormatARB(
hDC,
attribs,
NULL,
1,
&iPF,
&num_formats_choosen) ) {
fprintf(stderr, "error choosing proper pixel format\n");
return NULL;
}
if( !num_formats_choosen ) {
return NULL;
}
PIXELFORMATDESCRIPTOR pfd;
memset(&pfd, 0, sizeof(pfd));
/* now this is a kludge; we need to pass something in the PIXELFORMATDESCRIPTOR
* to SetPixelFormat; it will be ignored, mostly. OTOH we want to send something
* sane, we're nice people after all - it doesn't hurt if this fails. */
DescribePixelFormat(hDC, iPF, sizeof(pfd), &pfd);
if( !SetPixelFormat(hDC, iPF, &pfd) ) {
fprintf(stderr, "error setting proper pixel format\n");
ReleaseDC(hWnd, hDC);
DestroyWindow(hWnd);
return NULL;
}
I've got a complete working example for this in my wglarb wrapper repository over at GitHub:
https://github.com/datenwolf/wglarb/blob/master/test/layered.c

Answer 2

You might be missing the call to UpdateLayeredWindow . IIRC, not calling UpdateLayeredWindow results in some weird behavior. YMMV.
More details here

Answer 3

According with SetLayeredWindowAttributes doc you need to pass
a COLORREF structure that specifies the transparency color key to be
used when composing the layered window. All pixels painted by the
window in this color will be transparent. To generate a COLORREF, use
the RGB macro.
Review the second parameter of your call to this function.