I want to be able to clip my postprocessing image passes to specific regions so that effects such as a blur would only affect theses regions
In order to do that i use the stencil buffer and my pipeline is as follows :
Render some objects to the stencil buffer only, writing 1s
Render some objects where the stencil value equals 1 (this works)
Render some objects whatever there is in the stencil buffer
Run postprocess passes (by drawing a quad with the image resulting of the 3 previous step as a bound texture) where the stencil value equals 1 (or always, depending on an attribute of my effects)
The results i get :
Black image when postprocess involves stencil buffer
'Good' image when it does not
An image with non null values only outside the masks when i change `glStencilFunc(GL_EQUAL, 1, 0xFF);` to `glStencilFunc(GL_NOTEQUAL, 1, 0xFF);`
What strikes me is the fact that the image obtained with glStencilFunc(GL_ALWAYS, 1, 0xFF); is not even equal to the union of the other two : the one with glStencilFunc(GL_EQUAL, 1, 0xFF); is all black.
What is wrong with this code ?
gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, objectsTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);
// ================= Masks ===================
gl->glEnable(GL_STENCIL_TEST);
gl->glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Disable color buffer writing
gl->glStencilFunc(GL_ALWAYS, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
gl->glStencilMask(MASKSBITPLANE); // Write values as is in the stencil buffer
gl->glClear(GL_STENCIL_BUFFER_BIT);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskObjects()){
renderBlankSceneObject(scobjptr, gl, glext);
}
// ================= Masked ===================
gl->glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Enable color buffer writing
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskedObjects()){
renderSceneObject(scobjptr, gl, glext);
}
// ================= Raw objects ===================
gl->glDisable(GL_STENCIL_TEST);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getRawObjects()){
renderSceneObject(scobjptr, gl, glext);
}
// ================= Postprocess ===================
auto& shaderEffects(renderGroup->shaderEffects());
if (renderGroup->areShaderEffectsMasked()){
gl->glEnable(GL_STENCIL_TEST);
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
}
for (auto it(shaderEffects.begin()); it != shaderEffects.end(); ++it)
{
gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, currentImageTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);
// postprocess
gl->glUseProgram(shaderEffect->program().programId());
gl->glUniform1f(shaderEffect->m_timeLocation, m_time.elapsed());
gl->glActiveTexture(GL_TEXTURE0 + GLShaderEffect::PROCESSED_IMAGE_TEXTURE);
gl->glBindTexture(GL_TEXTURE_2D, processedTexture);
// some glUniform* calls
updateUniforms(gl, shaderEffect->ressourceClientsCollection());
// some glActiveTexture + glBindTexture calls
bindTextures(gl, shaderEffect->ressourceClientsCollection());
glext->glBindVertexArray(shaderEffect->vao());
gl->glDrawElements(GL_TRIANGLES, shaderEffect->elementsCount(), GL_UNSIGNED_INT, nullptr);
swap(currentImageTex, objectsTex);
}
The answer : I didn't restore the context after drawing, e.g. disable stencil testing at the end of my render pass.
I use Qt to blit my framebuffer into a widget and the stencil test was still active with another stencil buffer attached : that is why the screen got black.
Conclusion : Always restore the context to its previous state when you use a framework
I don't understand why those two codes doesn't yield the same result. I have enabled depth testing and stencil testing and my main loop looks like this :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
Obviously, my cube renders in this case.
However, if I use this code, nothing renders at all :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_LEQUAL, 1, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
With only one object in my scene, both code should behave the same way. Yet, it doesn't.
I thought it had something to do with the default value in the stencil buffer. However, since I'm clearing the stencil buffer before doing this, the default value should be 0.
By the way, with this code, the cube renders as well :
myShader.Use() // Use shader program
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// define model, projection, view, ....
glBindVertexArray(VAO[0]);
glStencilMask(0xFF); // Enable stencil buffer writing
glStencilFunc(GL_LEQUAL, 0, 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
drawCube(&myShader, model, projection, view);
glBindVertexArray(0);
I don't understand what is happening.
glStencilFunc(GL_LEQUAL, 1, 0xFF) means if 1 <= 0, because the stencil buffer contains 0, since you cleared it. The reference value is 1, because you pass this value to the function. This always fails.
See the Khronos reference page (glStencilFunc):
GL_LEQUAL Passes if ( ref & mask ) <= ( stencil & mask ).
I'm trying to do this example.
I couldn't understand how it works still and I'm having difficulties in visualizing how the code would look like.
I stumbled upon this post which was the closest I think I could find.
From the redbook link, how can I do a "two-pass algorithm" to get the desired output? How do I use GL_INVERT and check if the pixel is covered even number of times?
I somehow managed to understand how the stencil works for the concave polygon but my problem now is the polygon is being rendered incorrectly.
http://img.photobucket.com/albums/v442/ardo/ScreenShot2013-12-30at24155PM.png
The location of where I am drawing this is in a framebuffer which is set up like this:
//multisample
glGenRenderbuffersEXT(1, &colorBuffer);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, colorBuffer);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, multisampling, GL_RGBA,800, 600);
//multi sample depth
glGenRenderbuffersEXT(1, &depthBuffer);
glBindRenderbufferEXT(GL_RENDERBUFFER_EXT, depthBuffer);
glRenderbufferStorageMultisampleEXT(GL_RENDERBUFFER_EXT, multisampling, GL_DEPTH24_STENCIL8, 800, 600);
//multisamplefbo
glGenFramebuffersEXT(1, &mFBO);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_COLOR_ATTACHMENT0_EXT, GL_RENDERBUFFER_EXT, colorBuffer);
glFramebufferRenderbufferEXT(GL_FRAMEBUFFER_EXT, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER_EXT, depthBuffer);
Then my drawing code is:
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_INVERT, GL_INVERT);
glBegin(GL_TRIANGLE_FAN);
glVertex3f(1.396900,3.130690,0);
glVertex3f(2.034830,2.466900,0);
glVertex3f(2.486338,2.441036,0);
glVertex3f(2.802204,2.437803,0);
glVertex3f(2.910181,2.447098,0);
glVertex3f(2.957240,2.466900,0);
glVertex3f(3.019335,2.552570,0);
glVertex3f(3.130733,2.673799,0);
glVertex3f(3.284830,2.828970,0);
glVertex3f(3.474490,3.087590,0);
glVertex3f(4.319320,2.596210,0);
glVertex3f(4.508980,2.147930,0);
glVertex3f(4.714798,1.866680,0);
glVertex3f(4.907215,1.620586,0);
glVertex3f(5.101651,1.397930,0);
glVertex3f(5.186114,1.317364,0);
glVertex3f(5.254869,1.269024,0);
glVertex3f(5.302510,1.261700,0);
glVertex3f(5.323633,1.304180,0);
glVertex3f(5.312832,1.405254,0);
glVertex3f(5.264703,1.573711,0);
glVertex3f(5.173841,1.818340,0);
glVertex3f(5.034840,2.147930,0);
glVertex3f(4.875695,2.492816,0);
glVertex3f(4.727900,2.780675,0);
glVertex3f(4.591307,3.016304,0);
glVertex3f(4.465765,3.204503,0);
glVertex3f(4.351127,3.350070,0);
glVertex3f(4.247241,3.457803,0);
glVertex3f(4.153958,3.532502,0);
glVertex3f(4.071129,3.578965,0);
glVertex3f(3.998605,3.601990,0);
glVertex3f(3.936235,3.606376,0);
glVertex3f(3.841363,3.578427,0);
glVertex3f(3.766900,3.510000,0);
glVertex3f(2.974490,3.458280,0);
glVertex3f(2.672770,2.958280,0);
glVertex3f(2.379670,3.070350,0);
glVertex3f(1.853810,3.303110,0);
glVertex3f(1.396910,3.303110,0);
glVertex3f(1.396910,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glEnd();
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 0x1, 0x1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glColor3f(1,0,1);
glBegin(GL_TRIANGLE_FAN);
glVertex3f(1.396900,3.130690,0);
glVertex3f(2.034830,2.466900,0);
glVertex3f(2.486338,2.441036,0);
glVertex3f(2.802204,2.437803,0);
glVertex3f(2.910181,2.447098,0);
glVertex3f(2.957240,2.466900,0);
glVertex3f(3.019335,2.552570,0);
glVertex3f(3.130733,2.673799,0);
glVertex3f(3.284830,2.828970,0);
glVertex3f(3.474490,3.087590,0);
glVertex3f(4.319320,2.596210,0);
glVertex3f(4.508980,2.147930,0);
glVertex3f(4.714798,1.866680,0);
glVertex3f(4.907215,1.620586,0);
glVertex3f(5.101651,1.397930,0);
glVertex3f(5.186114,1.317364,0);
glVertex3f(5.254869,1.269024,0);
glVertex3f(5.302510,1.261700,0);
glVertex3f(5.323633,1.304180,0);
glVertex3f(5.312832,1.405254,0);
glVertex3f(5.264703,1.573711,0);
glVertex3f(5.173841,1.818340,0);
glVertex3f(5.034840,2.147930,0);
glVertex3f(4.875695,2.492816,0);
glVertex3f(4.727900,2.780675,0);
glVertex3f(4.591307,3.016304,0);
glVertex3f(4.465765,3.204503,0);
glVertex3f(4.351127,3.350070,0);
glVertex3f(4.247241,3.457803,0);
glVertex3f(4.153958,3.532502,0);
glVertex3f(4.071129,3.578965,0);
glVertex3f(3.998605,3.601990,0);
glVertex3f(3.936235,3.606376,0);
glVertex3f(3.841363,3.578427,0);
glVertex3f(3.766900,3.510000,0);
glVertex3f(2.974490,3.458280,0);
glVertex3f(2.672770,2.958280,0);
glVertex3f(2.379670,3.070350,0);
glVertex3f(1.853810,3.303110,0);
glVertex3f(1.396910,3.303110,0);
glVertex3f(1.396910,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glVertex3f(1.396900,3.130690,0);
glEnd();
glDisable(GL_STENCIL_TEST);
The way I have my multisampling set up is here (This is before the drawing is written)
[self checkError];
glEnable(GL_DEPTH_TEST);
[self checkError];
glDepthMask(GL_TRUE);
[self checkError];
glDepthFunc(GL_LESS);
[self checkError];
glClearDepth(10000.0);
[self checkError];
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
[self checkError];
if(multisampling != 0){
//Set multisampled framebuffer
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
//Antialiasing functions
glEnable (GL_POLYGON_SMOOTH);
glEnable (GL_BLEND);
glBlendFunc (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glHint (GL_POLYGON_SMOOTH_HINT, GL_NICEST);
glLineWidth (1.5);
}
glDisable(GL_CULL_FACE);
And my blit call:
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, mFBO);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, currentFramebuffer);
glBlitFramebuffer(0, 0, 800, 600, 0, 0, 800, 600, GL_COLOR_BUFFER_BIT, GL_NEAREST);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, mFBO);
glReadBuffer(GL_COLOR_ATTACHMENT0_EXT);
glBindFramebufferEXT(GL_DRAW_FRAMEBUFFER_EXT, currentFramebuffer);
glBindFramebufferEXT(GL_READ_FRAMEBUFFER_EXT, currentFramebuffer);
glBindFramebufferEXT(GL_FRAMEBUFFER_EXT, currentFramebuffer);
What don't you get? The example starts out with a stencil buffer cleared to 0x00 and each time you draw a fragment (whether it passes or fails a depth/stencil test) it performs a bitwise inversion of the stencil buffer (e.g. ~0x00 --> 0xff). If you do this an odd number of times, the stencil buffer will be non-zero, but if you have an even number of fragments it will be zero.
Some things that might be confusing are the use of 1 as the bitmask for the stencil test and the stencil op in the second pass. This effectively limits the test to a single bit. In other words, the bit flipping I mentioned earlier will only happen for bit 1. Thus, you can actually simplify your test for even/odd to a test of a single bit... if the stencil buffer stores a value of 1 then you drew an odd number of fragments. If it stores a value of 0 then you drew an even number.
The second pass in your example actually does the exact stencil test I described. It tests the stencil buffer for 1 and fails the stencil test if the first bit != 1.
// The stencil op below is a convoluted way of clearing the stencil buffer
glStencilOp (GL_ZERO, GL_ZERO, GL_ZERO);
glStencilFunc (GL_EQUAL, 1, 1); // Test: (Stencil & 1) == 1
I have a simple UI widget system and I'm using the stencil buffer to act as a clipper so that children of a widget can't be drawn outside the widget. Basically, the stencil value for everything is inside the bounds for this widget is incremented. Then, anything drawn after I make the clipper must be within the box.
The clipper constructor looks like this:
// Stencil buffer and ClipperStack.size() start at 0
// Increment any pixel in the rect on my current recursion level
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
ClipperStack.push_back(Rect);
// only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
glVertex2f(Rect.Left, Rect.Top);
glVertex2f(Rect.Left, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Top);
glEnd();
// Stencil clipper drawn,
glColorMask(1, 1, 1, 1);
glStencilMask(0);
// now only draw stuff that's that has the right clipper value
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
When the clipper goes out of scope the destructor runs, which looks like this:
// Decrement anything we previously incremented
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_DECR, GL_DECR);
// Get the old rect
sf::FloatRect Rect = clipperStack.back();
ClipperStack.pop_back();
// Only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
glVertex2f(Rect.Left, Rect.Top);
glVertex2f(Rect.Left, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Top);
glEnd();
// now draw on regular color buffer again,
// stencil buffer should be the same as before constructor call
glColorMask(1, 1, 1, 1);
glStencilMask(0);
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
However, when I run this only the direct children of my root widget are drawn. The children of children aren't drawn at all. I've tried a bunch of variations of this and I keep doing something wrong. I don't know where I'm going wrong with this.
In both the constructor and destructor I think you need to set the glStencilMask() to set every bit of the stencil buffer. E.g. if you have an 8-bit stencil buffer you want to use glStencilMask(0xFF);
Of course, if you've only got a 1-bit stencil buffer your code won't work at all, since you're trying to increment the stencil value for each level of sub-widget.