Read depth values while stencil testing(same texture) - opengl

I know that it is a very bad idea to read/write from/to the same texture/location, because this would result in undefined behaviour.
But in my case, if depth testing is disabled and I read the depth values in a shader, is it ok to do the stencil testing at the same time as reading the depth values within the same texture?
In my opinion there should not be any problems, because i'm not reading the stencil buffer values in the shader. Or, could there by any hardware related problems when the texture is bound for reading in a shader and OpenGL uses it to do the stencil testing?
This texture is filled with depth/stencil values. I wan't to avoid some heavy BRDF lighting(Directional light) calculations on specific pixels(the sky).
Example code:
//Contains the depth/stencil texture(deferredDepthStencilTextureID).
//Two FBO's are sharing the same depth/stencil texture.
lightAccumulationFBO->bind();
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 1, 0xFF);
//Disable writing to the stencil buffer, i.e. all the bits is write-protected.
glStencilMask(0x00);
glDisable(GL_DEPTH_TEST);
glEnable(GL_BLEND);
glBlendFunc(GL_ONE, GL_ONE); //Additive blending. Light accumulation.
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, deferredDepthStencilTextureID); //GL_DEPTH24_STENCIL8
//Bind other textures here...
shader.bind();
//Uniforms here...
postProcessQuad->renderQuad();
shader.unbind();
//Unbind all the textures here.
...
glDisable(GL_BLEND);
glEnable(GL_DEPTH_TEST);
glDisable(GL_STENCIL_TEST);
lightAccumulationFBO->unbind();

But in my case, if depth testing is disabled and I read the depth values in a shader, is it ok to do the stencil testing at the same time as reading the depth values within the same texture?
No. Whether an operation is defined or not is based on the images attached to the FBO and read from. Not the components of said images. And no, write masking will not save you from undefined behavior.
So unless your stencil texture is separate from your depth texture, that's not going to work. And good luck finding hardware that will allow you to separate depth/stencil images.
Even with GL 4.5/NV/ARB_texture_barrier, the answer is still no. That functionality makes certain exceptions to the above rule, but only for operations that are due to fragment shader outputs. Stencil test operations are not fragment shader outputs, so they don't apply.

Related

glDisable(GL_DEPTH_TEST) makes nothing but Sky-Sphere Render

Edit: Rendering the skybox before all other objects in the scene fixed this problem.
I've seen the question here but adding
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
doesn't appear to help.
I'm trying to render a spherical Skybox for my scene and for some reason when I Disable depth testing before doing so, the Skybox is the only thing rendered.
[Render other objects..]
// Disable depth test & mask, faceculling
glDisable(GL_DEPTH_TEST); // Adding this makes everything else invisible
glDepthMask(GL_FALSE);
glCullFace(GL_FRONT);
[Render texture onto inside of sphere..]
// Re-enable faceculling, & depth
glDepthMask(GL_TRUE);
glEnable(GL_DEPTH_TEST);
glCullFace(GL_BACK);
Any idea why this might be happening?
I wasn't sure of what code to include to keep this clear, so don't hesitate to ask for more.
Just to let you know, this:
glDisable(GL_DEPTH_TEST);
Disables both depth test and writing, so you don't need both that and set the depth mask to GL_FALSE.
When you clear the depth buffer each frame by default it should clear it the maximum value, probably 1.0. By default the depth function is GL_LESS meaning any depth value coming out of the fragment shader less than the one in the depth buffer passes and is written to the framebuffer.
It seems to me that what you're doing is clearing the depth buffer to 1.0, disabling depth testing and writing, drawing your objects, then enabling depth testing and writing and drawing your skybox. The problem with this is that the drawing of your objects doesn't write anything to the depth buffer, and so when it comes time to draw your skybox (with depth testing enabled) all the pixel depth values in the buffer are 1.0 (because you never wrote anything to it), and because the depth function is GL_LESS every pixel you draw of your skybox passes the depth test and is written to the framebuffer.
If there is a special need to have your objects always drawn in front of the skybox, for example the skybox follows the camera position around, then:
1) Disable the depth writing.
2) Draw the skybox.
3) Enable the depth writing.
4) Draw your objects.
Well, yes. That's what the depth test is for. Without it, there's nothing to indicate to OpenGL that the skybox shouldn't be rendered on top of everything else.
If you don't want this to happen, don't disable the depth test… or draw the skybox before everything else, instead of afterwards.

Deferred renderer with stencil buffer FBO

In my deferred renderer I create a FBO for my geometry pass where I store also a depth + stencil buffer into a texture with these parameters:
Attachment: GL_DEPTH_STENCIL_ATTACHMENT
Format: GL_DEPTH_STENCIL
Internal format: GL_DEPTH32F_STENCIL8
Data type: GL_UNSIGNED_INT_24_8
and I can successfully populate this texture with depth and stencil.
The problem I'm facing now is how to use the stencil buffer created in my geometry pass in order to use it somewhere else, like when I do a directional light pass I'd like to process only the pixels covered by the geometry visible on my geometry pass.
So, how can I use the stencil buffer stored during my geometry pass for further use?
You can of course always use the stencil buffer for stencil testing. To illustrate how to use stencil testing on a basic level, here are the critical calls for an example where you draw in two passes, where in the second pass we will exclude all pixels that were not covered in the first pass:
When you clear the framebuffer, make sure that you include the bit for clearing the stencil buffer:
glClear(... | GL_STENCIL_BUFFER_BIT);
Enable the stencil test:
glEnable(GL_STENCIL_TEST);
Set up the stencil function/operation to set the value in the stencil buffer to 1 for all rendered pixels:
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
GL_ALWAYS specifies that all pixels pass the stencil test (i.e. no pixels are rejected). GL_REPLACE specifies that the value in the stencil buffer is replaced by the reference value, which is the second argument to glStencilFunc(), for all rendered pixels.
Draw the geometry for the first pass.
Change the stencil function/operation to only render pixels where the value in the stencil buffer is 1:
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
GL_EQUAL specifies that only pixels at positions where the stencil buffer value is equal to the reference value 1 pass the stencil test, and all other pixels are rejected. GL_KEEP specifies that the current values in the stencil buffer are not modified in this pass.
Draw the geometry for the second pass.
But the way I read your question, you're looking for something different. You want to sample the stencil buffer from one rendering pass in a later rendering pass, similar to the way you can sample depth textures.
This is supported in OpenGL 4.3 and later. With texId the name of your depth/stencil texture, the key part is this texture parameter setting:
glBindTexture(GL_TEXTURE_2D, texId);
glTexParamteri(GL_TEXTURE_2D, GL_DEPTH_STENCIL_TEXTURE_MODE, GL_STENCIL_INDEX);
This specifies that the stencil part of the depth/stencil texture should be used for sampling. Only nearest sampling is supported for stencil textures, so make sure that you also have these values:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
Not surprisingly based on the nature of the value, stencil textures are treated as unsigned integer textures. This means that the sampler type in the shader code needs to be usampler2D:
uniform usampler2D MyStencilTexture;
And when you sample the texture, the result will also be of type unsigned:
uvec4 stencilValue = texture(MyStencilTexture, textureCoordinates);
The stencil value will then be in stencilValue.r.

Access depth-stencil texture in a shader program

It seems to be difficult to find information about how to access depth and stencil buffers in shaders of successive render passes.
In a first render pass, I do not only render color and depth information but also make use of stencil operations to count objects. I use a multi render target FBO for this, with color buffers and a combined depth stencil buffer attached. All of them are in the form of textures (no render buffer objects involved).
In a second render pass (when rendering to the screen), I want to access the previously computed stencil index on a per-pixel basis (but not necessarily the same pixel I'm drawing then), similar like you would like to access the previously rendered color buffer to apply some post processing effect.
But I fail to bind the depth stencil texture in the second pass to my shader program as a uniform. At least only black values are read from it, so I guess it's not bound correctly.
Is it possible to bind a depth stencil texture to a texture unit for use in a shader program? Is it impossible to access depth and stencil textures using "normal" samplers? Is it possible with some "special" sampler? Does it depend on the interpolation mode set on the texture or a similar setting?
If not, what is the best (fastest) way to copy the stencil information into a separate color texture between these two render passes? Maybe involving a third render pass which draws a single color using stencil test (I only need a binary version of the stencil buffer in the final render pass, to be precise I need to test if the value is zero).
The setup for the textures being used by the intermediate FBO is as follows:
// The textures for color information (GL_COLOR_ATTACHMENT*):
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, w, h, 0, GL_RGBA, GL_UNSIGNED_BYTE, 0);
// The texture for depth and stencil information (GL_DEPTH_STENCIL_ATTACHMENT*):
glTexImage2D(GL_TEXTURE_2D, 0, GL_DEPTH24_STENCIL8, w, h, 0, GL_DEPTH_STENCIL, GL_UNSIGNED_INT_24_8, 0);
In the second render pass, I currently only try to "debug" the contents of all textures. Therefore I setup the shader with these values:
glActiveTexture(GL_TEXTURE0);
glBindTexture(GL_TEXTURE_2D, <texture>);
glUniform1i(texLocation, 0);
and let the shader program simply copy the texture to the screen:
uniform sampler2D tex;
in vec2 texCoord;
out vec4 fragColor;
void main() {
fragColor = texture2D(tex, texCoord);
}
The results are as followed:
When <texture> above refers to one of my color textures, I see the color output rendered in the first render pass, which is what I expect.
When <texture> above refers to the depth stencil texture, the shader doesn't do anything (I see the color with which I clear the screen).
When copying the depth stencil texture to the CPU and examine it, I see both the depth and stencil information in the packed 24 + 8 bit data as expected.
I have no experience with using stencil as a texture, but you may want to take a look at the following extension :
http://www.opengl.org/registry/specs/ARB/stencil_texturing.txt
Another option could be to create a view of the texture using
http://www.opengl.org/registry/specs/ARB/texture_view.txt
Or you could count objects without the stencil buffer, perhaps using MRT and additive blending on second render target using :
http://www.opengl.org/registry/specs/EXT/draw_buffers2.txt
But I'm afraid those options are not included in pure GL3.3...

How to render a mesh behind another mesh, like a mask?

I would like it so that when mesh A (the character), is behind mesh B (a wall), it is still rendered but with a solid gray color.
I'm beginning opengles 2.0 and I'm still unsure as to go about this. From what I understand the depth buffer allows meshes to fight out who will be seen in the fragments they encompass, also there are various blend functions that could possibly involved in this, finally the stencil buffer looks like it would also have this desirable functionality.
So is there a way to output different colors through the shader based on a failed depth test? Is there a way to do this through blending? Or must I use the stencil buffer some how?
And what is this technique called for future reference? I've seen it used in a lot of video games.
This can be done using the stencil buffer. The stencil buffer gives each pixel some additional bits which can be used as a bitmask or a counter. In your case you'd configure the stencil test unit to set a specific bitmask when the depth test for the character fails (because it's obstructed by the well). Then you switch the stencil test mode operation to pass the stencil test for this specific bitmask, and render a full viewport, solid quad in the desired color, with depth testing and depth writes disabled.
Code
I strongly recommend you dive deep into the documentation for the stencil test unit. It's a very powerful mechanism, often overlooked. Your particular problem would be solved by the following. I stuggest you take this example code, read it in parallel to the stencil test functions references glStencilFunc, glStencilOp.
You must add a stencil buffer to your frame buffer's pixel format – how you do that is platform dependent. For example, if you're using GLUT, then you'd add |GLUT_STENCIL to the format bitmask of glutInitDisplayMode; on iOS you'd set a property on your GLKView; etc. Once you've added a stencil buffer, you should clear it along with your other render buffers by adding |GL_STENCIL_BUFFER_BIT to the initial glClear call of each drawing.
GLint const silouhette_stencil_mask = 0x1;
void display()
{
/* ... */
glEnable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);
glDepthFunc(GL_LESS);
glDisable(GL_STENCIL_TEST);
/* The following two are not necessary according to specification.
* But drivers can be buggy and this makes sure we don't run into
* issues caused by not wanting to change the stencil buffer, but
* it happening anyway due to a buggy driver.
*/
glStencilFunc(GL_NEVER, 0, 0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
draw_the_wall();
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_REPLACE, GL_KEEP);
draw_the_character();
glStencilFunc(GL_EQUAL, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glDisable(GL_DEPTH_TEST);
glDepthMask(GL_FALSE);
draw_full_viewport_solid_color();
/* ... */
}

OpenGL, stencil buffer of FBO not working?

I try to use the stencil buffer of a FBO in OpenGL, but I can't get it to work. I bound a depth24_stencil8 texture to the FBO both for the depth and stencil targets. As a simple test, I tried:
/* Enable FBO */
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_NEVER, 1, 0xff);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glBegin(GL_TRIANGLE);
/* Draw some triangles */
glEnd();
glDisable(GL_STENCIL_TEST);
/* Disable FBO and render it on screen as a texture. */
As I use GL_NEVER, nothing should be rendered at all, but I can see the triangles. This is like if there were no stencil at all, but I cannot understand why. Trying this code without FBOs works, so I think I use the stencil functions correctly. I don't have any idea how to solve this problem. Did anyone already use a stencil with FBOs?
My bad, I was not attaching correctly the stencil buffer to my FBO. Strange thing is that my fbo status was not indicating any error or bad attachement, so I was persuaded it was OK... this problem has driven me crazy but now it seems to work.