Copying ID3D11Texture2D with proper alpha - c++

Question

I am new to Direct X, but I've been successfully able to use the Windows Desktop Duplication API to capture video. The API also allows you to retrieve mouse cursor information, including the position, height, width, and the raw pixel data (in system memory) of the cursor image. The mouse cursor is not drawn on the captured screen image by default, it needs to be handled manually.
I'm trying to "copy" this mouse cursor data to the main screen capture image to create a single image with a visible mouse cursor. So far I have been able to make the cursor show up by creating an ID3D11Texture2D from the cursor pixel data, then preforming an ID3D11DeviceContext::CopySubresourceRegion to copy the cursor to the main screen image, also stored as an ID3D11Texture2D. The main screen image texture is always in the DXGI_FORMAT_B8G8R8A8_UNORM format, and the raw cursor pixel data seems to be in the same format, at least for the DXGI_OUTDUPL_POINTER_SHAPE_TYPE_COLOR shape.
My current issue seems to be related to the alpha handling of this copy. The cursor shows up, but when the rectangle is copied the alpha surrounding the cursor is instead filled in black. Here is an example of what it looks like: Black border around mouse
Also, it is important to me that this happens in video memory, as the final texture goes straight from video memory into a video encoder.
I'm willing to change my method if CopySubresourceRegion is not the right tool for the job. Any ideas on how I can get this cursor onto the main screen image texture with proper alpha?

Answer 1

The only way to access the alpha blending capabilities of you GPU is with draw commands. Copy calls only do replacement, as you see.
You already have your mouse cursor in an 'ID3D11Texture2D', what you need now, is an 'ID3D11ShaderResourceView' to use it as a texture, an 'ID3DVertexShader' and 'ID3DPixelShader' pair to render in a surface. An 'ID3D11RenderTargetView' from your destination surface.
A set of 'ID3D11RasterizerState', 'ID3D11DepthStencilState' and 'ID3D11BlendState' to configure the gpu state with no depth test, alpha blend, and other meaning full setting, most of them at default for you should be ok.
Then you need to draw a quad with all that to display your cursor. Depending on how you write the shader, you will need either a constant buffer, a vertex buffer and an input layout or both too.
For that kind of quad blit, i usually prefer to only deal with a single constant buffer and rebuild the vertex position from SV_VertexID inside the vertex shader, but it is up to you.
This is how you can write the blit shader without a vertex buffer to manage, a single Draw(4,0) with a strip primitive topology is enough :
struct IA {
uint vid : SV_VertexID;
};
struct VSPS {
float4 pos : SV_Position;
float2 uv : COLOR;
};
struct Root {
float left;
float top;
float right;
float bottom;
};
ConstantBuffer<Root> root_ : register(b0);
Texture2D<float4> texture_ : register(t0);
SamplerState sampler_ : register(s0);
void mainvs( in IA input, out VSPS output ) {
float x = input.vid < 2 ? 0.f : 1.f;
float y = (input.vid & 1) ? 1.f : 0.f;
output.uv = float2(x, y);
float px = input.vid < 2 ? root_.left : root_.right;
float py = (input.vid & 1) ? root_.bottom : root_.top;;
output.pos = float4(px,py,0.f,1.f);
output.pos.y = 1 - output.pos.y;
output.pos.xy *= 2;
output.pos.xy -= 1;
}
float4 mainps( in VSPS input ) : SV_TARGET {
return texture_.Sample( sampler_, input.uv );
}