Porting OpenGL stencil functionality to DirectX 11 - c++

Question

I have some rendering code written in OpenGL. I use stencil buffer to implement clipping:
//Let's assume this is done in render loop.
if(!already_created())
{
create_stencil_attachment_and_bind_to_FB_as_depth_stencil_attachment();
}
glEnable(GL_STENCIL_TEST);
glColorMask(0,0,0,0);
glDepthMask(0);
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_ALWAYS,1,1);
glStencilOp(GL_REPLACE,GL_REPLACE,GL_REPLACE);
render_to_stencil();
glColorMask(1,1,1,1);
glDepthMask(1);
glStencilFunc(GL_EQUAL,1,1);
glStencilOp(GL_KEEP,GL_KEEP,GL_KEEP);
render_with_clipping();
glDisable(GL_STENCIL_TEST);
Now, the problem is, I need to port this code to DX11. I saw examples on MSDN and some nice tutorials. I end up with this logic:
1. Create ID3D11Texture2D with format = DXGI_FORMAT_D32_FLOAT_S8X24_UINT.
2. Create ID3D11DepthStencilState for rendering to stencil: //Let's call it DS_RENDER
- For both front and back faces:
- op = D3D11_STENCIL_OP_REPLACE for all 3 cases
- func = D3D11_COMPARISON_ALWAYS
- DepthEnable = FALSE
- StencilEnable = TRUE
- StencilReadMask = 0xFF
- StencilWriteMask = 0xFF
3. Create ID3D11DepthStencilView for state and texture created before. //Let's call it DSV
4. Create ID3D11DepthStencilState for using stencil as 'input': //Let's call it DS_CLIP
- For both front and back faces:
- op = D3D11_STENCIL_OP_KEEP for all 3 cases
- func = D3D11_COMPARISON_EQUAL
- DepthEnable = FALSE
- StencilEnable = TRUE
- StencilReadMask = 0xFF
- StencilWriteMask = 0xFF
Now, I'm not sure how do I set stencil as target or input.
MSDN says:
`pDevice->OMSetDepthStencilState(pDSState, 1);`
and
`pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, pDSV);`
If I understand these calls correctly, first one sets stencil state, while the second one binds pDSV as additional 'attachment' to render target. Is that correct?
If so, will this work as I expect?
pDevice->OMSetDepthStencilState(DS_RENDER, 1);
pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, DSV);
render_geometry_to_stencil_buffer();
pDevice->OMSetDepthStencilState(DS_CLIP, 1);
render_geometry_with_clipping();
pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, NULL); //Does this disable stencil testing?
Thanks in advance for every help or useful hint.

Answer 1

If you want to render only to stencil, use (setting your state for writing):
pd3dDeviceContext->OMSetRenderTargets(0, NULL, DSV);
You don't need to render to color buffer, so no need to bind it.
Then to render to your target and enabling stencil test, use:
pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, DSV);
When you use stencil as input, a very simple thing is also to set StencilWriteMask = 0;
So it will never write to it (which is what you want to render clipped geometry).
If you use:
pd3dDeviceContext->OMSetRenderTargets(1, &pRTV, NULL);
You will indeed disable any form of depth/stencil test (no more depth bound, so DepthStencilState will have no effect at all).
Also I would use DXGI_FORMAT_D24_UNORM_S8_UINT for you depth format (personal preference tho), it will prefectly fit your use case and consume less memory.
Hope that helps.