Deferred renderer with stencil buffer FBO - c++

Question

In my deferred renderer I create a FBO for my geometry pass where I store also a depth + stencil buffer into a texture with these parameters:
Attachment: GL_DEPTH_STENCIL_ATTACHMENT
Format: GL_DEPTH_STENCIL
Internal format: GL_DEPTH32F_STENCIL8
Data type: GL_UNSIGNED_INT_24_8
and I can successfully populate this texture with depth and stencil.
The problem I'm facing now is how to use the stencil buffer created in my geometry pass in order to use it somewhere else, like when I do a directional light pass I'd like to process only the pixels covered by the geometry visible on my geometry pass.
So, how can I use the stencil buffer stored during my geometry pass for further use?

Answer 1

You can of course always use the stencil buffer for stencil testing. To illustrate how to use stencil testing on a basic level, here are the critical calls for an example where you draw in two passes, where in the second pass we will exclude all pixels that were not covered in the first pass:
When you clear the framebuffer, make sure that you include the bit for clearing the stencil buffer:
glClear(... | GL_STENCIL_BUFFER_BIT);
Enable the stencil test:
glEnable(GL_STENCIL_TEST);
Set up the stencil function/operation to set the value in the stencil buffer to 1 for all rendered pixels:
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
GL_ALWAYS specifies that all pixels pass the stencil test (i.e. no pixels are rejected). GL_REPLACE specifies that the value in the stencil buffer is replaced by the reference value, which is the second argument to glStencilFunc(), for all rendered pixels.
Draw the geometry for the first pass.
Change the stencil function/operation to only render pixels where the value in the stencil buffer is 1:
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
GL_EQUAL specifies that only pixels at positions where the stencil buffer value is equal to the reference value 1 pass the stencil test, and all other pixels are rejected. GL_KEEP specifies that the current values in the stencil buffer are not modified in this pass.
Draw the geometry for the second pass.
But the way I read your question, you're looking for something different. You want to sample the stencil buffer from one rendering pass in a later rendering pass, similar to the way you can sample depth textures.
This is supported in OpenGL 4.3 and later. With texId the name of your depth/stencil texture, the key part is this texture parameter setting:
glBindTexture(GL_TEXTURE_2D, texId);
glTexParamteri(GL_TEXTURE_2D, GL_DEPTH_STENCIL_TEXTURE_MODE, GL_STENCIL_INDEX);
This specifies that the stencil part of the depth/stencil texture should be used for sampling. Only nearest sampling is supported for stencil textures, so make sure that you also have these values:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
Not surprisingly based on the nature of the value, stencil textures are treated as unsigned integer textures. This means that the sampler type in the shader code needs to be usampler2D:
uniform usampler2D MyStencilTexture;
And when you sample the texture, the result will also be of type unsigned:
uvec4 stencilValue = texture(MyStencilTexture, textureCoordinates);
The stencil value will then be in stencilValue.r.