Stencil buffer behaviour in simple case (GL_ALWAYS, GL_LEQUAL) - c++

I don't understand why those two codes doesn't yield the same result. I have enabled depth testing and stencil testing and my main loop looks like this :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
Obviously, my cube renders in this case.
However, if I use this code, nothing renders at all :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_LEQUAL, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
With only one object in my scene, both code should behave the same way. Yet, it doesn't.
I thought it had something to do with the default value in the stencil buffer. However, since I'm clearing the stencil buffer before doing this, the default value should be 0.
By the way, with this code, the cube renders as well :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_LEQUAL, 0, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
I don't understand what is happening.


glStencilFunc(GL_LEQUAL, 1, 0xFF) means if 1 <= 0, because the stencil buffer contains 0, since you cleared it. The reference value is 1, because you pass this value to the function. This always fails.
See the Khronos reference page (glStencilFunc):
GL_LEQUAL Passes if ( ref & mask ) <= ( stencil & mask ).

Related

why can't the contents of stencil buffer be replaced

I draw two triangles. Before draw the first, I set the contents of the stencil buffer to make sure the stencil test will fail. So the first triangle won't be rendered on the screen. At the same time, I call the func "glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP)" to replace the contents of the stencil buffer with new value ref. Then I draw the second triangle. Theoreticaly, the stencil test in drawing this triangle should pass and the second triangle should be rendered on the screen. While in fact, the second stencil test failed again! I can't find the reason. Can anyone help me ? Thanks.
glClearColor(0.2f, 0.3f, 0.3f, 1.0f);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); // replace stencil buffer when test failed
glStencilMask(10); // set mask 10
glClearStencil(13); // set clear value 13
// since mask is 10 and value is 13, so the contents of stencil buffer is 8
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glStencilMask(0xff); // enable the writing of stencil buffer
glStencilFunc(GL_EQUAL, 9, 0xf); // (0xf & 8) != (0xf & 9)，stencil test failed
// this triangle won't rendered
glUseProgram(shaderProgramOrange);
glBindVertexArray(VAOs[0]);
glDrawArrays(GL_TRIANGLES, 0, 3);
glStencilFunc(GL_EQUAL, 9, 0xf); // now ref is 9，theoreticaly, the stencil test should pass
// in fact, the test still failed! I didn't know why.
// this triangle should have been rendered
glUseProgram(shaderProgramYellow);
glBindVertexArray(VAOs[1]);
glDrawArrays(GL_TRIANGLES, 0, 3);
glBindVertexArray(0);
// Swap the screen buffers
glfwSwapBuffers(window);
glStencilMask(0xff);
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);

glStencilFunc(): why is GL_ALWAYS != GL_EQUAL union GL_NOTEQUAL?

I want to be able to clip my postprocessing image passes to specific regions so that effects such as a blur would only affect theses regions
In order to do that i use the stencil buffer and my pipeline is as follows :
Render some objects to the stencil buffer only, writing 1s
Render some objects where the stencil value equals 1 (this works)
Render some objects whatever there is in the stencil buffer
Run postprocess passes (by drawing a quad with the image resulting of the 3 previous step as a bound texture) where the stencil value equals 1 (or always, depending on an attribute of my effects)
The results i get :
Black image when postprocess involves stencil buffer
'Good' image when it does not
An image with non null values only outside the masks when i change `glStencilFunc(GL_EQUAL, 1, 0xFF);` to `glStencilFunc(GL_NOTEQUAL, 1, 0xFF);`
What strikes me is the fact that the image obtained with glStencilFunc(GL_ALWAYS, 1, 0xFF); is not even equal to the union of the other two : the one with glStencilFunc(GL_EQUAL, 1, 0xFF); is all black.
What is wrong with this code ?
gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, objectsTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);
// ================= Masks ===================
gl->glEnable(GL_STENCIL_TEST);
gl->glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Disable color buffer writing
gl->glStencilFunc(GL_ALWAYS, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
gl->glStencilMask(MASKSBITPLANE); // Write values as is in the stencil buffer
gl->glClear(GL_STENCIL_BUFFER_BIT);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskObjects()){
renderBlankSceneObject(scobjptr, gl, glext);
}
// ================= Masked ===================
gl->glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Enable color buffer writing
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskedObjects()){
renderSceneObject(scobjptr, gl, glext);
}
// ================= Raw objects ===================
gl->glDisable(GL_STENCIL_TEST);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getRawObjects()){
renderSceneObject(scobjptr, gl, glext);
}
// ================= Postprocess ===================
auto& shaderEffects(renderGroup->shaderEffects());
if (renderGroup->areShaderEffectsMasked()){
gl->glEnable(GL_STENCIL_TEST);
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
}
for (auto it(shaderEffects.begin()); it != shaderEffects.end(); ++it)
{
gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, currentImageTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);
// postprocess
gl->glUseProgram(shaderEffect->program().programId());
gl->glUniform1f(shaderEffect->m_timeLocation, m_time.elapsed());
gl->glActiveTexture(GL_TEXTURE0 + GLShaderEffect::PROCESSED_IMAGE_TEXTURE);
gl->glBindTexture(GL_TEXTURE_2D, processedTexture);
// some glUniform* calls
updateUniforms(gl, shaderEffect->ressourceClientsCollection());
// some glActiveTexture + glBindTexture calls
bindTextures(gl, shaderEffect->ressourceClientsCollection());
glext->glBindVertexArray(shaderEffect->vao());
gl->glDrawElements(GL_TRIANGLES, shaderEffect->elementsCount(), GL_UNSIGNED_INT, nullptr);
swap(currentImageTex, objectsTex);
}

The answer : I didn't restore the context after drawing, e.g. disable stencil testing at the end of my render pass.
I use Qt to blit my framebuffer into a widget and the stencil test was still active with another stencil buffer attached : that is why the screen got black.
Conclusion : Always restore the context to its previous state when you use a framework

How does gl_FragColor affect the stencil buffer?

I have set up my operations to draw to the stencil buffer, similar to the following:
void onDisplay() {
glClear(GL_DEPTH_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_NEVER, 1, 0xFF);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
// Draw to stencil buffer
glStencilMask(0xFF);
glClear(GL_STENCIL_BUFFER_BIT); // needs mask=0xFF
draw_circle();
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glStencilMask(0x00);
// draw where stencil's value is 0
glStencilFunc(GL_EQUAL, 0, 0xFF);
/* (nothing to draw) */
// draw only where stencil's value is 1
glStencilFunc(GL_EQUAL, 1, 0xFF);
draw_scene();
glDisable(GL_STENCIL_TEST);
}
Now, if I have the following fragment shader enabled when I call draw_circle() (above):
void main() {
gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
}
How will the values of the stencil buffer differ from if I were to use the following fragment shader?
void main() {
gl_FragColor = vec4(0.0, 1.0, 0.0, 1.0);
}
In other words, I'm wondering how the output of the fragment shader affects the stencil buffer when drawing to the stencil buffer.
Edit:
The point of my question is to correct some misunderstandings I know I have regarding the stencil buffer. One example I have that I think explains the stencil buffer fairly well is [1]. Here, the following is mentioned:
The glColorMask function allows you to specify which data is written to the color buffer during a drawing operation. In this case you would want to disable all color channels (red, green, blue, alpha). Writing to the depth buffer needs to be disabled separately as well with glDepthMask, so that cube drawing operation won't be affected by leftover depth values of the rectangle. This is cleaner than simply clearing the depth buffer again later.
So, it seems from this page, that, in order to write to the stencil buffer, one needs to enable/disable the appropriate modes (i.e. color and depth), and then go through the entire rasterization process, which will only write to the stencil buffer. Since the rasterization process includes the fragment shader, is the output of the fragment shader (i.e. gl_FragColor) simply ignored? How can I tell GL what to write to the stencil buffer position (x, y)?
[1] : https://open.gl/depthstencils

Unless you have access to AMD/ARB_shader_stencil_export, the fragment shader cannot directly affect the output to the stencil buffer. The only exception to this is discarding the fragment.
And according to this database, only AMD cards provide this extension. Also, that extension exposes an output specifically for the stencil. It modifies the stencil value of the fragment; the color values of the fragment never affect the fragment's stencil value.

OpenGL stencil buffer concave polygon

I'm trying to do this example.
I couldn't understand how it works still and I'm having difficulties in visualizing how the code would look like.
I stumbled upon this post which was the closest I think I could find.
From the redbook link, how can I do a "two-pass algorithm" to get the desired output? How do I use GL_INVERT and check if the pixel is covered even number of times?
I somehow managed to understand how the stencil works for the concave polygon but my problem now is the polygon is being rendered incorrectly.
http://img.photobucket.com/albums/v442/ardo/ScreenShot2013-12-30at24155PM.png
The location of where I am drawing this is in a framebuffer which is set up like this:
//multisample
glGenRenderbuffersEXT(1, &colorBuffer);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, colorBuffer);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, multisampling, GL_RGBA,800, 600);
//multi sample depth
glGenRenderbuffersEXT(1, &depthBuffer);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, depthBuffer);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, multisampling, GL_DEPTH24_STENCIL8, 800, 600);
//multisamplefbo
glGenFramebuffersEXT(1, &mFBO);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_RENDERBUFFER_EXT, colorBuffer);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER_EXT, depthBuffer);
Then my drawing code is:
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_INVERT, GL_INVERT);
glBegin(GL_TRIANGLE_FAN);
glVertex3f(1.396900,3.130690,0);
glVertex3f(2.034830,2.466900,0);
glVertex3f(2.486338,2.441036,0);
glVertex3f(2.802204,2.437803,0);
glVertex3f(2.910181,2.447098,0);
glVertex3f(2.957240,2.466900,0);
glVertex3f(3.019335,2.552570,0);
glVertex3f(3.130733,2.673799,0);
glVertex3f(3.284830,2.828970,0);
glVertex3f(3.474490,3.087590,0);
glVertex3f(4.319320,2.596210,0);
glVertex3f(4.508980,2.147930,0);
glVertex3f(4.714798,1.866680,0);
glVertex3f(4.907215,1.620586,0);
glVertex3f(5.101651,1.397930,0);
glVertex3f(5.186114,1.317364,0);
glVertex3f(5.254869,1.269024,0);
glVertex3f(5.302510,1.261700,0);
glVertex3f(5.323633,1.304180,0);
glVertex3f(5.312832,1.405254,0);
glVertex3f(5.264703,1.573711,0);
glVertex3f(5.173841,1.818340,0);
glVertex3f(5.034840,2.147930,0);
glVertex3f(4.875695,2.492816,0);
glVertex3f(4.727900,2.780675,0);
glVertex3f(4.591307,3.016304,0);
glVertex3f(4.465765,3.204503,0);
glVertex3f(4.351127,3.350070,0);
glVertex3f(4.247241,3.457803,0);
glVertex3f(4.153958,3.532502,0);
glVertex3f(4.071129,3.578965,0);
glVertex3f(3.998605,3.601990,0);
glVertex3f(3.936235,3.606376,0);
glVertex3f(3.841363,3.578427,0);
glVertex3f(3.766900,3.510000,0);
glVertex3f(2.974490,3.458280,0);
glVertex3f(2.672770,2.958280,0);
glVertex3f(2.379670,3.070350,0);
glVertex3f(1.853810,3.303110,0);
glVertex3f(1.396910,3.303110,0);
glVertex3f(1.396910,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glEnd();
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glColor3f(1,0,1);
glBegin(GL_TRIANGLE_FAN);
glVertex3f(1.396900,3.130690,0);
glVertex3f(2.034830,2.466900,0);
glVertex3f(2.486338,2.441036,0);
glVertex3f(2.802204,2.437803,0);
glVertex3f(2.910181,2.447098,0);
glVertex3f(2.957240,2.466900,0);
glVertex3f(3.019335,2.552570,0);
glVertex3f(3.130733,2.673799,0);
glVertex3f(3.284830,2.828970,0);
glVertex3f(3.474490,3.087590,0);
glVertex3f(4.319320,2.596210,0);
glVertex3f(4.508980,2.147930,0);
glVertex3f(4.714798,1.866680,0);
glVertex3f(4.907215,1.620586,0);
glVertex3f(5.101651,1.397930,0);
glVertex3f(5.186114,1.317364,0);
glVertex3f(5.254869,1.269024,0);
glVertex3f(5.302510,1.261700,0);
glVertex3f(5.323633,1.304180,0);
glVertex3f(5.312832,1.405254,0);
glVertex3f(5.264703,1.573711,0);
glVertex3f(5.173841,1.818340,0);
glVertex3f(5.034840,2.147930,0);
glVertex3f(4.875695,2.492816,0);
glVertex3f(4.727900,2.780675,0);
glVertex3f(4.591307,3.016304,0);
glVertex3f(4.465765,3.204503,0);
glVertex3f(4.351127,3.350070,0);
glVertex3f(4.247241,3.457803,0);
glVertex3f(4.153958,3.532502,0);
glVertex3f(4.071129,3.578965,0);
glVertex3f(3.998605,3.601990,0);
glVertex3f(3.936235,3.606376,0);
glVertex3f(3.841363,3.578427,0);
glVertex3f(3.766900,3.510000,0);
glVertex3f(2.974490,3.458280,0);
glVertex3f(2.672770,2.958280,0);
glVertex3f(2.379670,3.070350,0);
glVertex3f(1.853810,3.303110,0);
glVertex3f(1.396910,3.303110,0);
glVertex3f(1.396910,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glEnd();
glDisable(GL_STENCIL_TEST);
The way I have my multisampling set up is here (This is before the drawing is written)
[self checkError];
glEnable(GL_DEPTH_TEST);
[self checkError];
glDepthMask(GL_TRUE);
[self checkError];
glDepthFunc(GL_LESS);
[self checkError];
glClearDepth(10000.0);
[self checkError];
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
[self checkError];
if(multisampling != 0){
//Set multisampled framebuffer
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
//Antialiasing functions
glEnable (GL_POLYGON_SMOOTH);
glEnable (GL_BLEND);
glBlendFunc (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glHint (GL_POLYGON_SMOOTH_HINT, GL_NICEST);
glLineWidth (1.5);
}
glDisable(GL_CULL_FACE);
And my blit call:
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, mFBO);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, currentFramebuffer);
glBlitFramebuffer(0, 0, 800, 600, 0, 0, 800, 600, GL_COLOR_BUFFER_BIT, GL_NEAREST);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glReadBuffer(GL_COLOR_ATTACHMENT0_EXT);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, currentFramebuffer);
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, currentFramebuffer);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, currentFramebuffer);

What don't you get? The example starts out with a stencil buffer cleared to 0x00 and each time you draw a fragment (whether it passes or fails a depth/stencil test) it performs a bitwise inversion of the stencil buffer (e.g. ~0x00 --> 0xff). If you do this an odd number of times, the stencil buffer will be non-zero, but if you have an even number of fragments it will be zero.
Some things that might be confusing are the use of 1 as the bitmask for the stencil test and the stencil op in the second pass. This effectively limits the test to a single bit. In other words, the bit flipping I mentioned earlier will only happen for bit 1. Thus, you can actually simplify your test for even/odd to a test of a single bit... if the stencil buffer stores a value of 1 then you drew an odd number of fragments. If it stores a value of 0 then you drew an even number.
The second pass in your example actually does the exact stencil test I described. It tests the stencil buffer for 1 and fails the stencil test if the first bit != 1.
// The stencil op below is a convoluted way of clearing the stencil buffer
glStencilOp (GL_ZERO, GL_ZERO, GL_ZERO);
glStencilFunc (GL_EQUAL, 1, 1); // Test: (Stencil & 1) == 1

glStencilFunc(GL_NEVER, 0, 0) still renders?

I have problems utilizing the stencil buffer, and it's seems to boil down to not work at all.
Given the following code:
glEnable(GL_STENCIL_TEST);
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT | GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glColor4f(1, 1, 1, 1);
glStencilFuncSeparate(GL_FRONT_AND_BACK, GL_NEVER, 0, 0);
glStencilOpSeparate(GL_FRONT_AND_BACK, GL_INCR, GL_INCR, GL_INCR);
glBegin(GL_TRIANGLES);
{ draw something }
glEnd();
The triangles are still drawn!? Am I missing something trivial here?
Note:
I'm not rendering to a frame buffer, I'm using glStencil...Separate just to make sure it's not related to front\back, I've kept bits of code that to me doesn't seem related.

You don't have a stencil buffer. And by the specification (from 4.3 core, folio page 432):
If there is no stencil buffer, no stencil modification can occur, and it is as if the
stencil tests always pass, regardless of any calls to StencilFunc.