I want to stencil out some objects with openGL.
glClear(GL_STENCIL_BUFFER_BIT);
glColorMask(false, false, false, false);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glDisable(GL_DEPTH_TEST);
glColor4f(0,0,1,1.0f);
gl::draw(mVBO); //a sphere
glEnable(GL_DEPTH_TEST);
glColorMask(true, true, true, true);
glStencilFunc(GL_ALWAYS,0, 1);
glStencilOp(GL_REPLACE,GL_KEEP, GL_ZERO);
mTexture.enableAndBind();
gl::drawCube(Vec3f(0,3,0),Vec3f(13,13,13) );
glDisable(GL_STENCIL_TEST)
I tried so many hours to achieve a simple boolean operation, but I dont get it.
I want something like this:
OpenGL's stencil buffer operates on the 2 dimensional pixel grid of the frame buffer. OpenGL itself has no notion of objects or a scene. It's merely drawing points, lines and triangles. So a real CSG is not possible with just OpenGL. However there are techniques that emulate CSG with stencil buffer operations, but they're quite complex. Google for "OpenGL CSG in the stencil buffer" to find some papers from the mid 1990-ies on the subject.
Related
I need to draw a color with some shape onto an image. My thought was to supply a mask with the given shape (say, hearts), then fill the rectangular area with the color and use the mask to render it over the final image.
Masked by:
PLUS
EQUALS:
The rectangle color is decided at runtime - that's why I don't draw the colored heart on my own.
The black heart image is transparent (alpha is 0) anywhere except for the heart (alpha is 255).
I tried using:
glBlendFunc(GL_DST_ALPHA, GL_ZERO)
where the source is the solid color, and the destination is the alpha channel image.
I used https://www.andersriggelsen.dk/glblendfunc.php for help.
However the bottom image (tree) is being used as the DST image...
Seems like I need an intermediate buffer to first render the blue heart, then do a second render onto the tree.
What is the way to do it?
If the tree is drawn before, it will appear in the dest Color and change your final result.
You are right, you need an intermediate buffer to store which part of the quand should be rendered, with the shape of your heart.
OpenGL provide a perfect tool for this, it's called stencil buffer.
In your case i will render my scene like usual (the tree)
Then i will enable the stencil buffer glEnable(GL_STENCIL_TEST);
Disable the write to the colorBuffer glColorMask(false, false, false, false);,
Draw only the heart with the appropriate mask. glStencilMask(0xFF);
Then you draw your colored quad with stencil test enable with glStencilFunc(GL_EQUAL, 1, 0xFF)
Don't forget to clear your stencil buffer each frame glClear(GL_STENCIL_BUFFER_BIT);
You can find some good tutorials online: https://learnopengl.com/Advanced-OpenGL/Stencil-testing
Here's a very simple way to do this in legacy OpenGL (which I assume you're using) that does not require a stencil buffer:
public void render() {
glClearColor(0, 0, 0, 0);
glClear(GL_COLOR_BUFFER_BIT);
glLoadIdentity();
glOrtho(0, 1, 1, 0, 1, -1);
glEnable(GL_TEXTURE_2D);
glEnable(GL_BLEND);
// Regular blending
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_ALPHA_TEST);
// Discard transparent pixels. Not strictly necessary but good for performance in this case.
glAlphaFunc(GL_GREATER, 0.01f);
glColor3f(1,1,1);
glBindTexture(GL_TEXTURE_2D, treeTexture);
drawQuad();
glColor3f(1,0,1); // Your color goes here
glBindTexture(GL_TEXTURE_2D, maskTexture);
drawQuad();
}
private void drawQuad() {
glBegin(GL_QUADS);
glTexCoord2f(0,0);
glVertex2f(0,0);
glTexCoord2f(0,1);
glVertex2f(0,1);
glTexCoord2f(1,1);
glVertex2f(1,1);
glTexCoord2f(1,0);
glVertex2f(1,0);
glEnd();
}
Here, treeTexture is the tree texture, and maskTexture is the white-on-transparent heart shape.
Result:
The principle is that in the legacy OpenGL pipeline, you can use glColor* before glVertex* to specify a color that the texture color (in this case white or transparent) is multiplied by component-wise.
Note that with this method you can easily render multiple colored shapes in multiple different colors without needing any (relatively expensive) clears of the stencil buffer. I suggest cropping the mask texture to the boundaries of the actual mask shape, to save the GPU the small effort of discarding all the transparent fragments.
I have a serious problem understanding how stencil buffering in OpenGL works. I followed the examples of a tutorial and the code works of course, but I really don't get at what points I'm writing to the stencil buffer. To me it seems as I'm setting up the stencil tests, masks and everything, but then I just call glDrawArrays(...)
and write the triangles to the output without applying the stencil buffer at all.
I also don't understand why I clear the buffer using glClear(GL_STENCIL_BUFFER_BIT)
before I even draw the triangles. I would be very thankful if somebody could enlighten me.
// Draws the initial cube facing upwards
glDrawArrays(GL_TRIANGLES, 0, 36);
//enable stencil testing
glEnable(GL_STENCIL_TEST);
// Draws the floor between the upward cube and downward cube
glStencilFunc(GL_ALWAYS, 1, 0xFF); /*sets the test function, mask and ref value*/
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); /*defines what happens on restults*/
glStencilMask(0xFF); /*Mask that is applied on the bits before they are written to the stencil buffer*/
glDepthMask(GL_FALSE);
glClear(GL_STENCIL_BUFFER_BIT); /*Why do I clear the buffer here?!?*/
glDrawArrays(GL_TRIANGLES, 36, 6);
// Draws the downward cube (reflection)
glStencilFunc(GL_EQUAL, 1, 0xFF);
glDepthMask(GL_TRUE);
glStencilMask(0x00);
model = glm::scale(
glm::translate(model, glm::vec3(0, 0, -1)),
glm::vec3(1, 1, -1)
);
glUniformMatrix4fv(uniModel, 1, GL_FALSE, glm::value_ptr(model));
glUniform3f(uniColor, 0.3f, 0.3f, 0.3f);
glDrawArrays(GL_TRIANGLES, 0, 36);
glUniform3f(uniColor, 1.0f, 1.0f, 1.0f);
glDisable(GL_STENCIL_TEST);
The third value passed to glStencilOp (GL_REPLACE) tells OpenGL to set the stencil buffer to the ref value previously specified in glStencilFunc whenever both the stencil test and the depth test are passed. In this case: 1.
So then the call to glDrawArrays updates the stencil buffer to 1 everyplace where it draws successfully (i.e., passes both tests).
(And the reason you clear it first, is to get rid of any 1's that happened to be there when you started: you don't want them acting as false positives where you didn't draw triangles).
See https://www.khronos.org/registry/OpenGL-Refpages/gl2.1/xhtml/glStencilOp.xml for details.
Okay I think I finally get it. And just to provide the answer to anybody else who might have the same thinking barrier I had here is what was my problem.
In my head I thought I have to write the values somehow to the stencil buffer first and afterwards I will write them to the display. Which is wrong as #NicolBolas kindly stated.
The command glDrawArrays (while stencilTesting is active) writes to the display as well as to the stencil buffer. By that for everything following we have the stencil of the in my example floor in the buffer and can use it to evaluate whether or not a pixel will be drawn. By that we can effectively make the second cube be drawn only on the Floor and not outside of it. Due to setting glStencilMask to all zero before we draw the cube we make sure it doesn't affect the stencil buffer even though the stencil test is applied on the action. If anybody else stumbles upon this I hope this might help. And if this is complete nonsense, please anybody leave a comment with the mistake in my thinking.
I am writing an SDL2/modern OpenGL application that uses stencil buffers. I have written the following code in my renderer:
glEnable(GL_STENCIL_FUNC);
glClearStencil(0);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glStencilFunc(GL_NEVER, 1, 0xFF); //Always fail the stencil test
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP); //Set the pixels which failed to 1
glStencilMask(0xFF);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
//Drawing small rectangle here
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 0xFF); //Only pass the stencil test where the pixel is 1 in the stencil buffer
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //Dont change the stencil buffer any further
//Drawing big rectangle here
glDisable(GL_STENCIL_FUNC);
The goal of the code above is to only draw the part of the big rectangle that fits in the small rectangle. Unfortunately, when I run the code the opposite happens, it renders the big rectangle with a hole in it the size of the small rectangle.
I have tried many more stencil functions, but they all result in the same, and this seems like it should work. So, does anybody have any ideas or can tell me where I am going wrong?
I apperantly don't have the reputation to embed pictures in my post but:
Intended result:
http://i.imgur.com/RpbHzCV.jpg
Actual result:
http://i.imgur.com/Z5qDqHk.jpg
The call glEnable(GL_STENCIL_FUNC); is just wrong, the correct enum is GL_STENCIL_TEST. So your code doesn't use the stencil buffer at all.
I can only guess why you get the result you got, then: Your code might draw the first rectangle into the depth buffer, so when you draw the second one, the fragments in that area might fail the depth test. So even when you correctly enable the stencil test, you still have to take care about the depht buffer here.
I would like it so that when mesh A (the character), is behind mesh B (a wall), it is still rendered but with a solid gray color.
I'm beginning opengles 2.0 and I'm still unsure as to go about this. From what I understand the depth buffer allows meshes to fight out who will be seen in the fragments they encompass, also there are various blend functions that could possibly involved in this, finally the stencil buffer looks like it would also have this desirable functionality.
So is there a way to output different colors through the shader based on a failed depth test? Is there a way to do this through blending? Or must I use the stencil buffer some how?
And what is this technique called for future reference? I've seen it used in a lot of video games.
This can be done using the stencil buffer. The stencil buffer gives each pixel some additional bits which can be used as a bitmask or a counter. In your case you'd configure the stencil test unit to set a specific bitmask when the depth test for the character fails (because it's obstructed by the well). Then you switch the stencil test mode operation to pass the stencil test for this specific bitmask, and render a full viewport, solid quad in the desired color, with depth testing and depth writes disabled.
Code
I strongly recommend you dive deep into the documentation for the stencil test unit. It's a very powerful mechanism, often overlooked. Your particular problem would be solved by the following. I stuggest you take this example code, read it in parallel to the stencil test functions references glStencilFunc, glStencilOp.
You must add a stencil buffer to your frame buffer's pixel format – how you do that is platform dependent. For example, if you're using GLUT, then you'd add |GLUT_STENCIL to the format bitmask of glutInitDisplayMode; on iOS you'd set a property on your GLKView; etc. Once you've added a stencil buffer, you should clear it along with your other render buffers by adding |GL_STENCIL_BUFFER_BIT to the initial glClear call of each drawing.
GLint const silouhette_stencil_mask = 0x1;
void display()
{
/* ... */
glEnable(GL_DEPTH_TEST);
glDepthMask(GL_TRUE);
glDepthFunc(GL_LESS);
glDisable(GL_STENCIL_TEST);
/* The following two are not necessary according to specification.
* But drivers can be buggy and this makes sure we don't run into
* issues caused by not wanting to change the stencil buffer, but
* it happening anyway due to a buggy driver.
*/
glStencilFunc(GL_NEVER, 0, 0);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
draw_the_wall();
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_REPLACE, GL_KEEP);
draw_the_character();
glStencilFunc(GL_EQUAL, silouhette_stencil_mask, 0xffffffff);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
glDisable(GL_DEPTH_TEST);
glDepthMask(GL_FALSE);
draw_full_viewport_solid_color();
/* ... */
}
I'm rendering a quad-mesh to an off-screen framebuffer in OpenGL with possibly overlapping quads (more fragments into a single framebuffer pixel). All quads lie in the plane y=0.
I would like to know if there is an easy way to set a color (e.g. black) to pixels that have overlapping quads (preferrably without the need of an extra shader pass, using simple OpenGL functionality).
You could use the stencil buffer to count the "overdraw" per pixel with something like
glClear(GL_STENCIL_BUFFER_BIT)<
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
glStencilFunc(GL_ALWAYS, 0, 0xffffff);
which would simply clear the stencil buffer to 0 and increase it for every fragment you draw after that (and also for all fragments you generate, but fail the depth test). After you have drawn all quads, you could draw another fullscreen quad with the desired color, but the stencil test set to
glStencilFunc(GL_GREATER, 1, 0xffffff);
to only affect the pixels with more than one quad.
This still requires some kind of "extra pass", but the stencil test is quite efficient.