The stencil buffer doesn’t draw a part of the tiles - c++

Well, I’m trying to draw tiles into a stencil buffer, but while drawing obviously happens something that I don’t understand and during the drawing disappearing (not drawn) part of the tiles.
So, how I draw:
// Enable blending.
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
// Enable testing.
glEnable(GL_STENCIL_TEST);
// Disable depth test.
glDisable(GL_DEPTH_TEST);
// Set stencil buff to 0.
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
// Here I got all visible tiles by a camera.
auto const visible_tiles = camera.visible_tiles();
// Draw tiles into a stencil.
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilMask(0xFF);
// Don't output the color.
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
// ...
// Here I had a loop where I go through each tile from all which was visible for the camera.
auto tile_id = 1;
visible_tiles.for_each_tile([this, &tile_id](auto const& tile_position) {
// Output tile ID into stencil buffer. I assume there will never be more than
// 255 tiles.
glStencilFunc(GL_ALWAYS, tile_id, 0xFF);
mesh_->draw(); // << Here I draw.
++tile_id;
});
// Disable testing.
glDisable(GL_STENCIL_TEST);
However, how my problem looks like?
]1
If I disable stencil testing at all, everything is OK.
And we draw it correctly.
UPD: With the help of the debug, I managed to narrow the circle of suspects )))
And I realized that the problem was somewhere in glEnable(GL_STENCIL_TEST), namely when I repeatedly call my method “draw(…)”.
So, I just did the following:
static bool flag = false;
if (!flag) {
glEnable(GL_STENCIL_TEST);
flag = true;
}
So, now I call glEnable(GL_STENCIL_TEST) only once, at first call of draw, and didn’t call glDisable(GL_STENCIL_TEST);
And it looks as if everything works correctly, at least now I didn’t see any defects.
But why this works?

Related

Stop two shadows overlapping using the stencil buffer in OpenGL

I have two planar shadows of the same object coming from the same light source - one that casts on the floor and one to cast on the wall when the object is close enough. Everything works just fine as far as the shadows being cast, I'm using the stencil buffer to make sure that the two shadows only cast on their respective surfaces without being rendered outside of the room.
The problem is that the two stencil buffers bleed into each other, specifically whichever shadow I render second bleeds into the stencil buffer for the first one. I figure it's some issue with the stencil function or something, using the wrong parameters, but I can't seem to figure it out.
// Generate the shadow using a shadow matrix (created using light position and vertices of
// the quad on which the shadow will be projected) and the object I'm making a shadow of
void createShadow(float shadowMat[16])
{
glDisable(GL_DEPTH_TEST);
glDisable(GL_LIGHTING);
glDisable(GL_TEXTURE_2D);
// Set the shadow color
glColor3f(0.1, 0.1, 0.1);
glPushMatrix();
// Create the shadow using the matrix and the object casting a shadow
glMultMatrixf((GLfloat*)shadowMat);
translate, rotate etc;
render object;
glPopMatrix();
// Reset values to render the rest of the scene
glColor3f(1.0, 1.0, 1.0);
glEnable(GL_DEPTH_TEST);
glEnable(GL_LIGHTING);
glEnable(GL_TEXTURE_2D);
}
// Set up the stencil buffer and render the shadow to it
void renderShadow(float shadowMat[16], float shadowQuad[12])
{
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glDisable(GL_DEPTH_TEST);
// Create a stencil for the shadow, using the vertices of the plane on which it will
// be projected
glPushMatrix();
translate, rotate etc;
glEnableClientState(GL_VERTEX_ARRAY);
// The shadow quad is the same vertices that I use to make the shadow matrix
glVertexPointer(3, GL_FLOAT, 0, shadowQuad);
glDrawArrays(GL_QUADS, 0, 4);
glDisableClientState(GL_VERTEX_ARRAY);
glPopMatrix();
glEnable(GL_DEPTH_TEST);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
// Render the shadow to the plane
createShadow(shadowMat);
glDisable(GL_STENCIL_TEST);
}
// In the render function:
Render floor/surrounding area;
Set up light using the same position used to make the shadow matrix;
renderShadow(wallShadowMatrix, wallVertices);
renderShadow(floorShadowMatrix, floorVertices);
Render rest of scene;
If I render the shadows on their own they work as intended, but when I render them together, whichever one rendered second shows up in the stencil of the first shadow.
I've included a few pictures; the first two show the individual Shadow on the wall and Shadow on the floor, and here is the floor shadow rendered after the wall shadow, and vice versa.
Fixed it, I needed to add the following code between the two renderShadow calls in the render function:
glClear(GL_STENCIL_BUFFER_BIT);

glStencilFunc(): why is GL_ALWAYS != GL_EQUAL union GL_NOTEQUAL?

I want to be able to clip my postprocessing image passes to specific regions so that effects such as a blur would only affect theses regions
In order to do that i use the stencil buffer and my pipeline is as follows :
Render some objects to the stencil buffer only, writing 1s
Render some objects where the stencil value equals 1 (this works)
Render some objects whatever there is in the stencil buffer
Run postprocess passes (by drawing a quad with the image resulting of the 3 previous step as a bound texture) where the stencil value equals 1 (or always, depending on an attribute of my effects)
The results i get :
Black image when postprocess involves stencil buffer
'Good' image when it does not
An image with non null values only outside the masks when i change `glStencilFunc(GL_EQUAL, 1, 0xFF);` to `glStencilFunc(GL_NOTEQUAL, 1, 0xFF);`
What strikes me is the fact that the image obtained with glStencilFunc(GL_ALWAYS, 1, 0xFF); is not even equal to the union of the other two : the one with glStencilFunc(GL_EQUAL, 1, 0xFF); is all black.
What is wrong with this code ?
gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, objectsTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);
// ================= Masks ===================
gl->glEnable(GL_STENCIL_TEST);
gl->glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Disable color buffer writing
gl->glStencilFunc(GL_ALWAYS, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
gl->glStencilMask(MASKSBITPLANE); // Write values as is in the stencil buffer
gl->glClear(GL_STENCIL_BUFFER_BIT);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskObjects()){
renderBlankSceneObject(scobjptr, gl, glext);
}
// ================= Masked ===================
gl->glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Enable color buffer writing
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getMaskedObjects()){
renderSceneObject(scobjptr, gl, glext);
}
// ================= Raw objects ===================
gl->glDisable(GL_STENCIL_TEST);
for (const auto& scobjptr : renderGroup->getRenderGroupObjects().getRawObjects()){
renderSceneObject(scobjptr, gl, glext);
}
// ================= Postprocess ===================
auto& shaderEffects(renderGroup->shaderEffects());
if (renderGroup->areShaderEffectsMasked()){
gl->glEnable(GL_STENCIL_TEST);
gl->glStencilFunc(GL_EQUAL, 1, MASKSBITPLANE);
gl->glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
gl->glStencilMask(0x00); // Disable writing to the stencil buffer
}
for (auto it(shaderEffects.begin()); it != shaderEffects.end(); ++it)
{
gl->glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, currentImageTex, 0);
gl->glClear(GL_COLOR_BUFFER_BIT);
// postprocess
gl->glUseProgram(shaderEffect->program().programId());
gl->glUniform1f(shaderEffect->m_timeLocation, m_time.elapsed());
gl->glActiveTexture(GL_TEXTURE0 + GLShaderEffect::PROCESSED_IMAGE_TEXTURE);
gl->glBindTexture(GL_TEXTURE_2D, processedTexture);
// some glUniform* calls
updateUniforms(gl, shaderEffect->ressourceClientsCollection());
// some glActiveTexture + glBindTexture calls
bindTextures(gl, shaderEffect->ressourceClientsCollection());
glext->glBindVertexArray(shaderEffect->vao());
gl->glDrawElements(GL_TRIANGLES, shaderEffect->elementsCount(), GL_UNSIGNED_INT, nullptr);
swap(currentImageTex, objectsTex);
}
The answer : I didn't restore the context after drawing, e.g. disable stencil testing at the end of my render pass.
I use Qt to blit my framebuffer into a widget and the stencil test was still active with another stencil buffer attached : that is why the screen got black.
Conclusion : Always restore the context to its previous state when you use a framework

Mask part of a texture on draw with OpenGL in a fixed pipeline

I am trying to figure out the best way to mask of sections of a texture when they ar drawn. My issue comes in the fact that I seem to have run our of alpha masks!
We are using openGL to draw a custom built 2D game engine. The game is built up off of sprites and simple block textures.
My desired outcome is like this:
A character sprite is drawn in place (using it's alpha color to not just be a box)
An item is drawn into the players hand (also using it's alpha color to draw into the scene without being a box)
The item should appear behind the characters arm/hand, but above the rest of the body.
For the moment the only way I can figure out how to accomplish this, is by drawing them in order (Body, Item, Arm) but I would like to avoid this to make art assets a bit easier to deal with. My idea solution would be to draw the character, then draw the item with an alpha mask that blocks out areas of the texture that should be "under" the arm.
Other solutions that I have seen are like this, where the glBlendFuncSeparate() function is used. I am trying to avoid bringing in extensions, as my current version of OpenGL doesn't support it. Not to say that I am opposed to the idea, but it seems a bit of a handle to brig it in just to draw an alpha mask?
I fully admit that this is a learning process for me, and I am using it as an excuse to really see how OpenGL handles. Any suggestions as to where I should head to get this to draw correctly? Is there a way for OpenGL in the fixed pipeline to take a texture, apply an alpha mask on top of it, and THEN draw it into the buffer? Should I give in and separate my character into several parts of its model?
[UPDATE: 8/12/12]
Tried to add the code suggested by Tim, but I seem to be having an issue. When I enable the stencil buffer, everything just gets blocked out, NOT just what I wanted. Here is my test example code.
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// Disable writing to any of the color fields
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);
glStencilFunc(GL_ALWAYS, 0,0);
// Draw our blocking poly
glBegin(GL_POLYGON);
glVertex2f( 50, 50 );
glVertex2f( 50, 50+128 );
glVertex2f( 50+128, 50+128 );
glEnd();
glStencilFunc(GL_GREATER, 0, -1);
glEnable(GL_STENCIL_TEST);
// Re enable drawing of colors
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
// Enable use of textures
glEnable(GL_TEXTURE_2D);
// Bind desired texture for drawing
glBindTexture(GL_TEXTURE_2D,(&texture)[0]);
// Draw the box with colors
glBegin(GL_QUADS);
glTexCoord2d( 0, 0 ); glVertex2f( 50, 50 );
glTexCoord2d( 0, 1 ); glVertex2f( 50, 50+128 );
glTexCoord2d( 1, 1 ); glVertex2f( 50+128, 50+128 );
glTexCoord2d( 1, 0 ); glVertex2f( 50+128, 50 );
glEnd();
// Swap buffers and display!
SDL_GL_SwapBuffers();
Just to be clear, here is my init code as well to set this system up.
When the code is run with stencil disabled, I get this:
When I use glEnable(GL_STENCIL_TEST), I get this:
I've tried playing around with various options, but I cannot see a clear reason why my stencil buffer is blocking everything.
[Update#2 8/12/12]
We got some working code, Thanks tim! Here is what I ended up running to work correctly.
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
// Disable writing to any of the color fields
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilOp(GL_INCR, GL_INCR, GL_INCR);
glEnable(GL_STENCIL_TEST);
// Draw our blocking poly
glBegin(GL_POLYGON);
glVertex2f( 50, 50 );
glVertex2f( 50, 50+128 );
glVertex2f( 50+128, 50+128 );
glEnd();
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
// Re enable drawing of colors
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
// Enable use of textures
glEnable(GL_TEXTURE_2D);
// Bind desired texture for drawing
glBindTexture(GL_TEXTURE_2D,(&texture)[0]);
// Draw the box with colors
glBegin(GL_QUADS);
glTexCoord2d( 0, 0 ); glVertex2f( 50, 50 );
glTexCoord2d( 0, 1 ); glVertex2f( 50, 50+128 );
glTexCoord2d( 1, 1 ); glVertex2f( 50+128, 50+128 );
glTexCoord2d( 1, 0 ); glVertex2f( 50+128, 50 );
glEnd();
glDisable(GL_STENCIL_TEST);
// Swap buffers and display!
SDL_GL_SwapBuffers();
Here's my idea for the situation where you have one texture and one alpha mask:
Draw the character onto the scene like normal.
Lock the RGB color channels so that it cannot be changed with glColorMask
Setup the stencil buffer with glStencilOp(GL_KEEP, GL_KEEP, GL_INCR); glStencilFunc(GL_ALWAYS, 0,0);
Draw the alpha mask with alpha testing enabled. This will increment the stencil buffer anywhere the alpha test passes (you may have to flip this based on your mask polarity)
At this point, you have a character texture in the framebuffer, and a mask outline in the stencil buffer.
Reenable the color channels with glColorMask
Setup the stencil buffer for the weapon with glStencilFunc(GL_GREATER, 0, -1); This will only draw the weapon texels where the stencil buffer is greater than zero, and reject pixels where the stencil is not updated.
Draw the weapon texture as normal.
Tim was pretty clear in his comment, but I want to present you the solution I find the most intuitive. It's 3D, so hold on... ;)
Basically, you can just use the Z coordinate of your images to create virtual "layers". It then doesnt' matter, in which order you draw them. Just alphatest every image individually, and draw it on correct Z value. If it still isn't enough, you could use separate texture containing "depth" of every pixel, and then use the 2nd texture to perform some sort of depth-testing.
Be sure to call glEnable(GL_DEPTH_TEST); if you want to use this approach.
As I see it, the problem is that you have one texture, but part of it represents the arm and part of it the rest of the character. The issue is that you want to draw the weapon over the character, but draw the arm over both.
This means, while drawing two objects, you want to put them into three different "layers". This fundamentally doesn't make sense, so you're kind of stuck.
Here's an idea though: use a fragment program (i.e., a shader).
I suggest you overload the character's texture's alpha channel to encode both transparency and layer. For example, let's use 0=transparent body, 64=opaque body, 128=transparent arm, 255=opaque arm.
From here, you draw your objects, but conditionally set the depth of your objects into three layers. Basically, you write a fragment program that draws your character into two different layers, the character gets pushed backward while the arm gets pulled forward. When the weapon is drawn, it is drawn without a shader, but it's tested against the characters' pixels' depths. It works something like this (untested, obviously).
Define a shader my_shader, which contains a fragment program:
uniform sampler2D character_texture;
void main(void) {
vec4 sample = texture2D(character_texture,gl_TexCoord[0].st);
int type; //Figure out what type of character texel we're looking at
if (fabs(sample.a-0.00)<0.01) type = 0; //transparent body
else if (fabs(sample.a-0.25)<0.01) type = 1; //opaque body
else if (fabs(sample.a-0.50)<0.01) type = 2; //transparent arm
else if (fabs(sample.a-1.00)<0.01) type = 3; //opaque arm
//Don't draw transparent pixels.
if (type==0 || type==2) discard;
gl_FragColor = vec4(sample.rgb,1.0);
//Normally, you (can) write "gl_FragDepth = gl_FragCoord.z". This
//is how OpenGL will draw your weapon. However, for the character,
//we alter that so that the arm is closer and the body is farther.
//Move body farther
if (type==1) gl_FragDepth = gl_FragCoord.z * 1.1;
//Move arm closer
else if (type==3) gl_FragDepth = gl_FragCoord.z * 0.9;
}
Here's some pseudocode for your draw function:
//...
//Algorithm to draw your character
glUseProgram(my_shader);
glBindTexture(GL_TEXTURE_2D,character.texture.texture_gl_id);
glUniform1i(glGetUniformLocation(my_shader,"character_texture"),1);
character.draw();
glUseProgram(0);
//Draw your weapon
glEnable(GL_DEPTH_TEST);
character.weapon.draw();
glDisable(GL_DEPTH_TEST);
//...

OperGL stencil buffer not acting how I expect

I have a simple UI widget system and I'm using the stencil buffer to act as a clipper so that children of a widget can't be drawn outside the widget. Basically, the stencil value for everything is inside the bounds for this widget is incremented. Then, anything drawn after I make the clipper must be within the box.
The clipper constructor looks like this:
// Stencil buffer and ClipperStack.size() start at 0
// Increment any pixel in the rect on my current recursion level
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
ClipperStack.push_back(Rect);
// only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
glVertex2f(Rect.Left, Rect.Top);
glVertex2f(Rect.Left, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Top);
glEnd();
// Stencil clipper drawn,
glColorMask(1, 1, 1, 1);
glStencilMask(0);
// now only draw stuff that's that has the right clipper value
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
When the clipper goes out of scope the destructor runs, which looks like this:
// Decrement anything we previously incremented
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_DECR, GL_DECR);
// Get the old rect
sf::FloatRect Rect = clipperStack.back();
ClipperStack.pop_back();
// Only draw to stencil buffer
glColorMask(0, 0, 0, 0);
glStencilMask(1);
glBegin(GL_QUADS);
glVertex2f(Rect.Left, Rect.Top);
glVertex2f(Rect.Left, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Bottom);
glVertex2f(Rect.Right, Rect.Top);
glEnd();
// now draw on regular color buffer again,
// stencil buffer should be the same as before constructor call
glColorMask(1, 1, 1, 1);
glStencilMask(0);
glStencilFunc(GL_EQUAL, ClipperStack.size(), 0xFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
However, when I run this only the direct children of my root widget are drawn. The children of children aren't drawn at all. I've tried a bunch of variations of this and I keep doing something wrong. I don't know where I'm going wrong with this.
In both the constructor and destructor I think you need to set the glStencilMask() to set every bit of the stencil buffer. E.g. if you have an 8-bit stencil buffer you want to use glStencilMask(0xFF);
Of course, if you've only got a 1-bit stencil buffer your code won't work at all, since you're trying to increment the stencil value for each level of sub-widget.

Pre Z buffer pass with OpenGL?

How exactly can I do a Z buffer prepass with openGL.
I'v tried this:
glcolormask(0,0,0,0); //disable color buffer
//draw scene
glcolormask(1,1,1,1); //reenable color buffer
//draw scene
//flip buffers
But it doesn't work. after doing this I do not see anything. What is the better way to do this?
Thanks
// clear everything
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
// z-prepass
glEnable(GL_DEPTH_TEST); // We want depth test !
glDepthFunc(GL_LESS); // We want to get the nearest pixels
glcolormask(0,0,0,0); // Disable color, it's useless, we only want depth.
glDepthMask(GL_TRUE); // Ask z writing
draw()
// real render
glEnable(GL_DEPTH_TEST); // We still want depth test
glDepthFunc(GL_LEQUAL); // EQUAL should work, too. (Only draw pixels if they are the closest ones)
glcolormask(1,1,1,1); // We want color this time
glDepthMask(GL_FALSE); // Writing the z component is useless now, we already have it
draw();
You're doing the right thing with glColorMask.
However, if you're not seeing anything, it's likely because you're using the wrong depth test function.
You need GL_LEQUAL, not GL_LESS (which happens to be the default).
glDepthFunc(GL_LEQUAL);
If i get you right, you are trying to disable the depth-test performed by OpenGL to determine culling. You are using color functions here, which does not make sense to me. I think you are trying to do the following:
glDisable(GL_DEPTH_TEST); // disable z-buffer
// draw scene
glEnable(GL_DEPTH_TEST); // enable z-buffer
// draw scene
// flip buffers
Do not forget to clear the depth buffer at the beginning of each pass.