I want to set up multiple stencils in OpenGL and then draw into multiple combinations of them.
For example:
I have two rectangular stencils like this:
Then I want to draw into:
anywhere
left rectangle (blue+purple)
right rectangle (purple+red)
middle rectangle (purple)
whole colored area (blue+purple+red)
I have found that it is possible to declare multiple stencils in different bits of stencil buffer but I don't know how.
How do I setup glStencilOp and glStencilFunc for this to work?
Or can I (should I) use glScissor for that?
I don't currently know, if it's possible to configure the stencil buffer for being able to do all the above 5 steps without making any changes to the stencil buffer between them. It would be easy if glStencilOp provided bitwise OR, but it doesn't and with just using increment or decrement you would have to draw the rectangles multiple times.
But if the regions are always rectangles, why not just use the scissor test? So the first 3 steps (or actually 2 and 3) can be done by just setting the rectangle's region with glScissor and enabling the scissor test (glEnable(GL_SCISSOR_TEST)).
For the middle one (step 4) you either compute the purple intersection rectangle yourself and use the scissor test again, or you use the stencil test:
glEnable(GL_STENCIL_TEST);
glStencilFunc(/*whatever*/);
glStencilOp(GL_INCR, GL_INCR, GL_INCR); //increase the stencil value
//draw both rectangles
glStencilFunc(GL_EQUAL, 2, 0xFFFFFFFF); //only draw where both rectangles are
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //don't change the stencil buffer
//draw things
So we first draw both rectangles and increase the stencil value everywhere they are drawn. Then we draw our things everywhere the stencil value is 2, meaning both rectangles were drawn.
For the 5th step you use the same stencil buffer, but with
glStencilFunc(GL_LEQUAL, 1, 0xFFFFFFFF);
for the second pass. This way you draw something everywhere the stencil buffer is at least 1, meaning at least one rectangle was drawn.
For more than two rectangles it can get more complicated and you need to play around a bit to find the most optimal way.
Related
I am trying to make a specific VBO (vertex buffer object) to not be drawn in specific screen area (but be drawn in the rest of the screen).
I have two VBOs that must be drawn in the screen. They can appear independently, parts of VBO1 overlap VBO2 and VBO1 has priority over VBO2. When both objects are being drawn, I want VBO2 to not render in the overlap area, but render the rest of object. Since the size of VBO2 can change, trying to achieve this behaviour with EBOs (element buffer objects) can be tricky.
In some sense, what i am looking for is the inverse of GL_SCISSORS_TEST. If in GL_SCISSORS_TEST I can define an area where the symbol can only de drawn inside (and the part that is outside is clipped), I want to be able to define an area where a symbol will not be drawn inside (but the part of the symbol that is outside the area will be drawn).
Which strategies would be best to achieve this in Opengl?
There are several different ways how the desired result can be achieved:
For the general case, the stencil test seems to be a good option. Note, that your framebuffer needs to support stencil buffering.
Clear stencil buffer with 0
Draw VBO1 with the following settings which set the stencil buffer to 1 for all pixels covered by VBO1. If you need a larger range than just VBO1, consider rendering a special stencil shape with color writing disabled that marks the non-drawable area.
glStencilFunc(GL_ALWAYS, 1, 0xFF),
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE)
Draw VBO2 with following settings which only draw VBO2 where the stencil buffer has a value of 0 (aka, where VBO1 has not drawn):
glStencilFunc(GL_EQUAL, 0, 0xFF),
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
If the VBOs contain only 2-dimensional objects parallel to the screen plane (like sprites), then you could also use depth testing and render VBO2 further away than VBO1.
If you have a more specialized case like rectangles (as suggested by siccor testing which only works on rectangular areas), it is also an option to pass the boundary of the VBO1 rectangle to the fragment shader of VBO2 and discard fragments if they fall inside the boundary area.
I have a map, for simplicity let's say it's just a single texture. On top of this map, I have a polygon which indicates the route that the user must follow.
What I want is to draw everything outside the polygon black. Or, of course to only draw things inside the polygon.
To explain it better I made a picture. The blue lines define the polygon, with each corner being a point in the polygon. The red with yellow lines is the part I want to black out of the picture, and only leave the red with the purple lines. The polygon starts from A and ends at B.
The main challenge here is that you need to draw a non-convex polygon, which is not directly supported by OpenGL. One approach for drawing it is to break it down into triangles. Depending on how much you know about the shape of the polygon, and how constraint it is, this might be fairly easy. For a general non-convex polygon, it's slightly painful. But there are algorithms you can find if you search for keywords like "polygon triangulation".
OpenGL has another mechanism that works great for these kinds of use cases: stencil buffers. You can find an explanation of this approach in the Red Book under Drawing Filled, Concave Polygons Using the Stencil Buffer. The main idea is that you can draw a triangle fan with an arbitrary origin and your polygon vertices. The pixels that are inside the polygon will then be drawn an odd number of times, while the pixels outside the polygons are drawn an even number of times. The stencil buffer is used to track the odd/even count of how many times each pixel is drawn.
To outline the main steps:
While setting up your context and drawing surface, make sure that you request a configuration with a stencil buffer.
During drawing, clear the stencil buffer along with the color buffer, and enable the stencil test.
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
Set up state for the render pass that counts if pixels are rendered an odd/even number of times. Note that this must only write to the stencil buffer, so color writes are disabled. The key part is the GL_INVERT for the stencil op, which flips the stencil value each time a pixel is rendered, giving us the odd/even count.
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_INVERT);
glStencilMask(1);
Render a triangle fan with an arbitrary point, e.g. (0.0, 0.0), as the first vertex, and the polygon corners as the remaining vertices. The polygon must be closed, so the first and last polygon corner must be the same. If p1, p2, ... , pN are your polygon corners, the sequence of vertices for the GL_TRIANGLE_FAN draw call is:
(0.0f, 0.0f), p1, p2, ... , pN, p1
You can use a trivial shader for this pass since the color value is not even written.
Enable color writes again, and set up the stencil test attributes to render only pixels that were rendered an odd number of times in the previous pass.
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
Draw your entire content. Only the part within the polygon outline will be rendered, the rest is eliminated by the stencil test.
I was trying to configure my stencil buffer so that, when enabled, it would set when the pixel drawn is not transparent (thus creating a map of pixels that light can collide with). What I've done is:
glClearStencil(0); //clear stencil
glStencilFunc(GL_EQUAL, 0xFF, 0x000000FF); //only where alpha (mask : 0x000000FF) is 0xFF (opaque)
glStencilOp(GL_INCR, GL_KEEP, GL_KEEP); //increment if passes (if it is opaque)
render(); //withing this method I sometimes disable the whole thing to draw the floor, for example
Then, i use the following code to test:
/* TURN OFF STENCIL */
glEnable(GL_STENCIL_TEST); //re-enable
glStencilFunc(GL_EQUAL, 0, 1); //if the test is equal to 1
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //do not change stencil buffer
ImageInfo.drawColorSquare(0, 0, Configurations.SCREEN_WIDTH, Configurations.GAME_HEIGHT, Color.BLUE); //drwa blue square
glDisable(GL_STENCIL_TEST); //disable
However, there are two problems:
It doesn't seem to be ignoring transparent pixels, as it should;
If a region overlaps with another, then it reverses - for example, it sets to one, then another region is drawn in the same area, and it reset it to 0 again.
I don't know why that is happening. Probably something wrong with my mask, I guess - I wasn't absolutely sure how many pixels OpneGL used in the Color Buffer.
Also, GL_INCR should add up to the max, and not go back, according to the documentation. Since my stencil buffer size is one bit, it should set to one, try to increase again, fail, and keep on one (instead of reseting).
The Stencil Test is independent of what happens in the color buffer. Setting glStencilFunc
( http://www.opengl.org/sdk/docs/man/xhtml/glStencilFunc.xml )
you can specify how the Stencil Test interacts with what is already stored in the Stencil Buffer.
Setting glStencilOp ( http://www.opengl.org/sdk/docs/man/xhtml/glStencilOp.xml ) gives you the possibility of using the result of the depth test to perform the Stencil Test.
A good tutorial that explain the Stencil Test and a very instructive algorithm based upon it can be found here http://ogldev.atspace.co.uk/www/tutorial37/tutorial37.html
The stencil buffer is usually an 8-bit buffer that reserves a small portion of the memory normally used for the depth buffer and is used for advanced rejection of fragments; things like masking to arbitrary shapes rather than using rectangular scissor boxes. It has nothing to do with your color buffer, and to make sure that fragments that have a specific alpha value do not affect the pixels on screen you would use something called an alpha test.
In core OpenGL 3, the fixed-function alpha test is no longer supported, so you would have to implement it in a fragment shader and then discard if it failed to meet your condition.
I'm rendering a quad-mesh to an off-screen framebuffer in OpenGL with possibly overlapping quads (more fragments into a single framebuffer pixel). All quads lie in the plane y=0.
I would like to know if there is an easy way to set a color (e.g. black) to pixels that have overlapping quads (preferrably without the need of an extra shader pass, using simple OpenGL functionality).
You could use the stencil buffer to count the "overdraw" per pixel with something like
glClear(GL_STENCIL_BUFFER_BIT)<
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
glStencilFunc(GL_ALWAYS, 0, 0xffffff);
which would simply clear the stencil buffer to 0 and increase it for every fragment you draw after that (and also for all fragments you generate, but fail the depth test). After you have drawn all quads, you could draw another fullscreen quad with the desired color, but the stencil test set to
glStencilFunc(GL_GREATER, 1, 0xffffff);
to only affect the pixels with more than one quad.
This still requires some kind of "extra pass", but the stencil test is quite efficient.
I have drawn multiple different colored polygons on the screen, now I have to draw another polygon of different color, but this polygon should be drawn only on those pixels which have a specific color.
I render each of the different colored polygons at same time in their own "layers", (= one color at a time). They can cover each other; newest layer covers all previous layers. The black color in the image is the "no polygons" area: empty space, and it should ignore that too.
So, basically I just render polygons, and then the N'th (not first) layer of polygons must be masked with the next polygon layer, and nothing else under it should be affected.
Image of the method needed:
What method can I use to achieve this with OpenGL ? I would prefer non-shader solution for this, if possible(?).
The only method I can do currently is to render each of the layers separately into the memory, then go through the pixels myself and combine the layers "manually", but that seems like a very slow method, doable though, but the speed is important here.
To use the stencil buffer for this, what you can do is:
Make sure you request a context that has a stencil buffer, this is windowing system specific so I won't cover it here. Call glGet(GL_STENCIL_BITS) to make sure that you get a sufficient number of bits.
The stencil buffer maintains an integer alongside each pixel, and allows you to modify it as things are drawn. As you draw each layer, set up the stencil buffer to set to a specific value when you draw each layer. You do this with
//draw layer N
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, N, -1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
Now as you draw each layer, the last polygon that was drawn to the screen also stores it's layer number into the stencil buffer.
At this point when you want to go back and draw your green star, you just tell it to only draw on pixels where the stencil buffer is equal to N.
//draw only where stencil == N
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, N, -1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
drawStar();