I am trying to use the stencil buffer to cut two windows in a solid box. The windows are two rectangles (composed of 2 triangles).
But the OpenGL stencil buffer flags seem difficult to decipher.
Here is what I am doing
glClearStencil(0.0f);
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilMask(0xFF);
DrawWindows();
glStencilFunc(GL_EQUAL, 1, 0xFF);
glStencilMask(0x00);
glDisable(GL_STENCIL_TEST);
glDisable(GL_DEPTH_TEST);
DrawSolidBox();
However, this just give me a constant cutout in the box, on both sides, where I only want the windows to be cut out on the side where they are co-planer with the one wall
Any ideas?
Related
I have a serious problem understanding how stencil buffering in OpenGL works. I followed the examples of a tutorial and the code works of course, but I really don't get at what points I'm writing to the stencil buffer. To me it seems as I'm setting up the stencil tests, masks and everything, but then I just call glDrawArrays(...)
and write the triangles to the output without applying the stencil buffer at all.
I also don't understand why I clear the buffer using glClear(GL_STENCIL_BUFFER_BIT)
before I even draw the triangles. I would be very thankful if somebody could enlighten me.
// Draws the initial cube facing upwards
glDrawArrays(GL_TRIANGLES, 0, 36);
//enable stencil testing
glEnable(GL_STENCIL_TEST);
// Draws the floor between the upward cube and downward cube
glStencilFunc(GL_ALWAYS, 1, 0xFF); /*sets the test function, mask and ref value*/
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); /*defines what happens on restults*/
glStencilMask(0xFF); /*Mask that is applied on the bits before they are written to the stencil buffer*/
glDepthMask(GL_FALSE);
glClear(GL_STENCIL_BUFFER_BIT); /*Why do I clear the buffer here?!?*/
glDrawArrays(GL_TRIANGLES, 36, 6);
// Draws the downward cube (reflection)
glStencilFunc(GL_EQUAL, 1, 0xFF);
glDepthMask(GL_TRUE);
glStencilMask(0x00);
model = glm::scale(
glm::translate(model, glm::vec3(0, 0, -1)),
glm::vec3(1, 1, -1)
);
glUniformMatrix4fv(uniModel, 1, GL_FALSE, glm::value_ptr(model));
glUniform3f(uniColor, 0.3f, 0.3f, 0.3f);
glDrawArrays(GL_TRIANGLES, 0, 36);
glUniform3f(uniColor, 1.0f, 1.0f, 1.0f);
glDisable(GL_STENCIL_TEST);
The third value passed to glStencilOp (GL_REPLACE) tells OpenGL to set the stencil buffer to the ref value previously specified in glStencilFunc whenever both the stencil test and the depth test are passed. In this case: 1.
So then the call to glDrawArrays updates the stencil buffer to 1 everyplace where it draws successfully (i.e., passes both tests).
(And the reason you clear it first, is to get rid of any 1's that happened to be there when you started: you don't want them acting as false positives where you didn't draw triangles).
See https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/glStencilOp.xml for details.
Okay I think I finally get it. And just to provide the answer to anybody else who might have the same thinking barrier I had here is what was my problem.
In my head I thought I have to write the values somehow to the stencil buffer first and afterwards I will write them to the display. Which is wrong as #NicolBolas kindly stated.
The command glDrawArrays (while stencilTesting is active) writes to the display as well as to the stencil buffer. By that for everything following we have the stencil of the in my example floor in the buffer and can use it to evaluate whether or not a pixel will be drawn. By that we can effectively make the second cube be drawn only on the Floor and not outside of it. Due to setting glStencilMask to all zero before we draw the cube we make sure it doesn't affect the stencil buffer even though the stencil test is applied on the action. If anybody else stumbles upon this I hope this might help. And if this is complete nonsense, please anybody leave a comment with the mistake in my thinking.
I started out working on an OpenGL program written in C that loads a 3D model from the user's filesystem with ASSIMP and renders it into the center of a window created with GLUT. So far I've gotten that basic rendering step working just fine. Now I want to render that same model into an OpenGL stencil in the shape of a triangle, but I haven't been able to figure it out. I have included an image representing the desired effect:
To elaborate, I want to make a triangle shaped stencil over my model as demonstrated below:
My code is pretty basic, all I do is enable GL_STENCIL_TEST, setup the stencil function, draw my triangle, and then draw the model.
glEnable(GL_STENCIL_TEST);
// Fill stencil buffer with 0's
glClearStencil(0);
glClear(GL_STENCIL_BUFFER_BIT);
// Write 1's into stencil buffer where the hole will be
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 1, ~0);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
// Draw the triangle hole shape
glBegin(GL_TRIANGLES);
glVertex2f(-0.3f, -0.3f);
glVertex2f(0.f, 0.3f);
glVertex2f(0.3f, -0.3f);
glEnd();
// Draw scene, keeping fragments with 1's in the stencil buffer
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glStencilFunc(GL_EQUAL, 1, ~0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
// Renders the model
drawTheModel();
glDisable(GL_STENCIL_TEST);
My GLUT display mode is as follows:
glutInitDisplayMode(GLUT_RGB | GLUT_DOUBLE | GLUT_DEPTH | GLUT_STENCIL);
When I run the above code, all I get is an empty window. I've read the documentation and checked loads of examples, but I can't seem to figure out what I'm doing wrong.
Any ideas?
UPDATE: I downloaded and compiled this stencil example and added my triangle stencil. My triangle stencil seemingly draws into the depth buffer - investigating. How my stencil looks on the example:
I am writing an SDL2/modern OpenGL application that uses stencil buffers. I have written the following code in my renderer:
glEnable(GL_STENCIL_FUNC);
glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_NEVER, 1, 0xFF); //Always fail the stencil test
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); //Set the pixels which failed to 1
glStencilMask(0xFF);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
//Drawing small rectangle here
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 0xFF); //Only pass the stencil test where the pixel is 1 in the stencil buffer
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //Dont change the stencil buffer any further
//Drawing big rectangle here
glDisable(GL_STENCIL_FUNC);
The goal of the code above is to only draw the part of the big rectangle that fits in the small rectangle. Unfortunately, when I run the code the opposite happens, it renders the big rectangle with a hole in it the size of the small rectangle.
I have tried many more stencil functions, but they all result in the same, and this seems like it should work. So, does anybody have any ideas or can tell me where I am going wrong?
I apperantly don't have the reputation to embed pictures in my post but:
Intended result:
http://i.imgur.com/RpbHzCV.jpg
Actual result:
http://i.imgur.com/Z5qDqHk.jpg
The call glEnable(GL_STENCIL_FUNC); is just wrong, the correct enum is GL_STENCIL_TEST. So your code doesn't use the stencil buffer at all.
I can only guess why you get the result you got, then: Your code might draw the first rectangle into the depth buffer, so when you draw the second one, the fragments in that area might fail the depth test. So even when you correctly enable the stencil test, you still have to take care about the depht buffer here.
I would like it so that when mesh A (the character), is behind mesh B (a wall), it is still rendered but with a solid gray color.
I'm beginning opengles 2.0 and I'm still unsure as to go about this. From what I understand the depth buffer allows meshes to fight out who will be seen in the fragments they encompass, also there are various blend functions that could possibly involved in this, finally the stencil buffer looks like it would also have this desirable functionality.
So is there a way to output different colors through the shader based on a failed depth test? Is there a way to do this through blending? Or must I use the stencil buffer some how?
And what is this technique called for future reference? I've seen it used in a lot of video games.
This can be done using the stencil buffer. The stencil buffer gives each pixel some additional bits which can be used as a bitmask or a counter. In your case you'd configure the stencil test unit to set a specific bitmask when the depth test for the character fails (because it's obstructed by the well). Then you switch the stencil test mode operation to pass the stencil test for this specific bitmask, and render a full viewport, solid quad in the desired color, with depth testing and depth writes disabled.
Code
I strongly recommend you dive deep into the documentation for the stencil test unit. It's a very powerful mechanism, often overlooked. Your particular problem would be solved by the following. I stuggest you take this example code, read it in parallel to the stencil test functions references glStencilFunc, glStencilOp.
You must add a stencil buffer to your frame buffer's pixel format – how you do that is platform dependent. For example, if you're using GLUT, then you'd add |GLUT_STENCIL to the format bitmask of glutInitDisplayMode; on iOS you'd set a property on your GLKView; etc. Once you've added a stencil buffer, you should clear it along with your other render buffers by adding |GL_STENCIL_BUFFER_BIT to the initial glClear call of each drawing.
GLint const silouhette_stencil_mask = 0x1;
void display()
{
/* ... */
glEnable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);
glDepthFunc(GL_LESS);
glDisable(GL_STENCIL_TEST);
/* The following two are not necessary according to specification.
* But drivers can be buggy and this makes sure we don't run into
* issues caused by not wanting to change the stencil buffer, but
* it happening anyway due to a buggy driver.
*/
glStencilFunc(GL_NEVER, 0, 0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
draw_the_wall();
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_REPLACE, GL_KEEP);
draw_the_character();
glStencilFunc(GL_EQUAL, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glDisable(GL_DEPTH_TEST);
glDepthMask(GL_FALSE);
draw_full_viewport_solid_color();
/* ... */
}
I want to stencil out some objects with openGL.
glClear(GL_STENCIL_BUFFER_BIT);
glColorMask(false, false, false, false);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glDisable(GL_DEPTH_TEST);
glColor4f(0,0,1,1.0f);
gl::draw(mVBO); //a sphere
glEnable(GL_DEPTH_TEST);
glColorMask(true, true, true, true);
glStencilFunc(GL_ALWAYS,0, 1);
glStencilOp(GL_REPLACE,GL_KEEP, GL_ZERO);
mTexture.enableAndBind();
gl::drawCube(Vec3f(0,3,0),Vec3f(13,13,13) );
glDisable(GL_STENCIL_TEST)
I tried so many hours to achieve a simple boolean operation, but I dont get it.
I want something like this:
OpenGL's stencil buffer operates on the 2 dimensional pixel grid of the frame buffer. OpenGL itself has no notion of objects or a scene. It's merely drawing points, lines and triangles. So a real CSG is not possible with just OpenGL. However there are techniques that emulate CSG with stencil buffer operations, but they're quite complex. Google for "OpenGL CSG in the stencil buffer" to find some papers from the mid 1990-ies on the subject.