I'm rendering a quad-mesh to an off-screen framebuffer in OpenGL with possibly overlapping quads (more fragments into a single framebuffer pixel). All quads lie in the plane y=0.
I would like to know if there is an easy way to set a color (e.g. black) to pixels that have overlapping quads (preferrably without the need of an extra shader pass, using simple OpenGL functionality).
You could use the stencil buffer to count the "overdraw" per pixel with something like
glClear(GL_STENCIL_BUFFER_BIT)<
glEnable(GL_STENCIL_TEST);
glStencilOp(GL_KEEP, GL_INCR, GL_INCR);
glStencilFunc(GL_ALWAYS, 0, 0xffffff);
which would simply clear the stencil buffer to 0 and increase it for every fragment you draw after that (and also for all fragments you generate, but fail the depth test). After you have drawn all quads, you could draw another fullscreen quad with the desired color, but the stencil test set to
glStencilFunc(GL_GREATER, 1, 0xffffff);
to only affect the pixels with more than one quad.
This still requires some kind of "extra pass", but the stencil test is quite efficient.
Related
I am trying to make a specific VBO (vertex buffer object) to not be drawn in specific screen area (but be drawn in the rest of the screen).
I have two VBOs that must be drawn in the screen. They can appear independently, parts of VBO1 overlap VBO2 and VBO1 has priority over VBO2. When both objects are being drawn, I want VBO2 to not render in the overlap area, but render the rest of object. Since the size of VBO2 can change, trying to achieve this behaviour with EBOs (element buffer objects) can be tricky.
In some sense, what i am looking for is the inverse of GL_SCISSORS_TEST. If in GL_SCISSORS_TEST I can define an area where the symbol can only de drawn inside (and the part that is outside is clipped), I want to be able to define an area where a symbol will not be drawn inside (but the part of the symbol that is outside the area will be drawn).
Which strategies would be best to achieve this in Opengl?
There are several different ways how the desired result can be achieved:
For the general case, the stencil test seems to be a good option. Note, that your framebuffer needs to support stencil buffering.
Clear stencil buffer with 0
Draw VBO1 with the following settings which set the stencil buffer to 1 for all pixels covered by VBO1. If you need a larger range than just VBO1, consider rendering a special stencil shape with color writing disabled that marks the non-drawable area.
glStencilFunc(GL_ALWAYS, 1, 0xFF),
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE)
Draw VBO2 with following settings which only draw VBO2 where the stencil buffer has a value of 0 (aka, where VBO1 has not drawn):
glStencilFunc(GL_EQUAL, 0, 0xFF),
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
If the VBOs contain only 2-dimensional objects parallel to the screen plane (like sprites), then you could also use depth testing and render VBO2 further away than VBO1.
If you have a more specialized case like rectangles (as suggested by siccor testing which only works on rectangular areas), it is also an option to pass the boundary of the VBO1 rectangle to the fragment shader of VBO2 and discard fragments if they fall inside the boundary area.
I have a map, for simplicity let's say it's just a single texture. On top of this map, I have a polygon which indicates the route that the user must follow.
What I want is to draw everything outside the polygon black. Or, of course to only draw things inside the polygon.
To explain it better I made a picture. The blue lines define the polygon, with each corner being a point in the polygon. The red with yellow lines is the part I want to black out of the picture, and only leave the red with the purple lines. The polygon starts from A and ends at B.
The main challenge here is that you need to draw a non-convex polygon, which is not directly supported by OpenGL. One approach for drawing it is to break it down into triangles. Depending on how much you know about the shape of the polygon, and how constraint it is, this might be fairly easy. For a general non-convex polygon, it's slightly painful. But there are algorithms you can find if you search for keywords like "polygon triangulation".
OpenGL has another mechanism that works great for these kinds of use cases: stencil buffers. You can find an explanation of this approach in the Red Book under Drawing Filled, Concave Polygons Using the Stencil Buffer. The main idea is that you can draw a triangle fan with an arbitrary origin and your polygon vertices. The pixels that are inside the polygon will then be drawn an odd number of times, while the pixels outside the polygons are drawn an even number of times. The stencil buffer is used to track the odd/even count of how many times each pixel is drawn.
To outline the main steps:
While setting up your context and drawing surface, make sure that you request a configuration with a stencil buffer.
During drawing, clear the stencil buffer along with the color buffer, and enable the stencil test.
glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
Set up state for the render pass that counts if pixels are rendered an odd/even number of times. Note that this must only write to the stencil buffer, so color writes are disabled. The key part is the GL_INVERT for the stencil op, which flips the stencil value each time a pixel is rendered, giving us the odd/even count.
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_ALWAYS, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_INVERT);
glStencilMask(1);
Render a triangle fan with an arbitrary point, e.g. (0.0, 0.0), as the first vertex, and the polygon corners as the remaining vertices. The polygon must be closed, so the first and last polygon corner must be the same. If p1, p2, ... , pN are your polygon corners, the sequence of vertices for the GL_TRIANGLE_FAN draw call is:
(0.0f, 0.0f), p1, p2, ... , pN, p1
You can use a trivial shader for this pass since the color value is not even written.
Enable color writes again, and set up the stencil test attributes to render only pixels that were rendered an odd number of times in the previous pass.
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
Draw your entire content. Only the part within the polygon outline will be rendered, the rest is eliminated by the stencil test.
In libGdx, i'm trying to create a shaped texture: Take a fully-visible rectangle texture and mask it to obtain a shaped textured, as shown here:
Here I test it on rectangle, but i will want to use it on any shape. I have looked into this tutorial and came with an idea to first draw the texture, and then the mask with blanding function:
batch.setBlendFunction(GL20.GL_ZERO, GL20.GL_SRC_ALPHA);
GL20.GL_ZERO - because i really don't want to paint any pixels from the mask
GL20.GL_SRC_ALPHA - from original texture i want to paint only those pixels, where mask was visible (= white).
Crucial part of the test code:
batch0.enableBlending();
batch0.begin();
batch0.draw(original, 0, 0); //to see the original
batch0.draw(mask, width1, 0); //and the mask
batch0.draw(original, 0, height1); //base for the result
batch0.setBlendFunction(GL20.GL_ZERO, GL20.GL_SRC_ALPHA);
batch0.draw(mask, 0, height1); //draw mask on result
batch0.setBlendFunction(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);
batch0.end();
The center ot the texture get's selected well, but instead of transparent color around, i see black:
Why is the result blank and not transparent?
(Full code - Warning: very messy)
What you're trying to do looks like a pretty clever use of blending. But I believe the exact way you apply it is "broken by design". Let's walk through the steps:
You render your background with red and green squares.
You render an opaque texture on top of you background.
You erase parts of the texture you rendered in step 2 by applying a mask.
The problem is that for the parts you erase in step 3, the previous background is not coming back. It really can't, because you wiped it out in step 2. The background of the whole texture area was replaced in step 2, and once it's gone there's no way to bring it back.
Now the question is of course how you can fix this. There are two conventional approaches I can think of:
You can combine the texture and mask by rendering them into an off-sreen framebuffer object (FBO). You perform steps 1 and 2 as you do now, but render into an FBO with a texture attachment. The texture you rendered into is then a texture with alpha values that reflect your mask, and you can use this texture to render into your default framebuffer with standard blending.
You can use a stencil buffer. Masking out parts of rendering is a primary application of stencil buffers, and using stencil would definitely be a very good solution for your use case. I won't elaborate on the details of how exactly to apply stencil buffers to your case in this answer. You should be able to find plenty of examples both online and in books, including in other answers on this site, if you search for "OpenGL stencil". For example this recent question deals with doing something similar using a stencil buffer: OpenGL stencil (Clip Entity).
So those would be the standard solutions. But inspired by the idea in your attempt, I think it's actually possible to get this to work with just blending. The approach that I came up with uses a slightly different sequence and different blend functions. I haven't tried this out, but I think it should work:
You render the background as before.
Render the mask. To prevent it from wiping out the background, disable writing to the color components of the framebuffer, and only write to the alpha component. This leaves the mask in the alpha component of the framebuffer.
Render the texture, using the alpha component from the framebuffer (DST_ALPHA) for blending.
You will need a framebuffer with an alpha component for this to work. Make sure that you request alpha bits for your framebuffer when setting up your context/surface.
The code sequence would look like this:
// Draw background.
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE);
glDisable(GL_BLEND);
// Draw mask.
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glEnable(GL_BLEND);
glBlendFunc(GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA);
// Draw texture.
A very late answer, but with the current version this is very easy. You simply draw the mask, set the blending mode to use the source color to the destination and draw the original. You'll only see the original image where the mask is.
//create batch with blending
SpriteBatch maskBatch = new SpriteBatch();
maskBatch.enableBlending();
maskBatch.begin();
//draw the mask
maskBatch.draw(mask);
//store original blending and set correct blending
int src = maskBatch.getBlendSrcFunc();
int dst = maskBatch.getBlendDstFunc();
maskBatch.setBlendFunction(GL20.GL_ZERO, GL20.GL_SRC_COLOR);
//draw original
maskBatch.draw(original);
//reset blending
maskBatch.setBlendFunction(src, dst);
//end batch
maskBatch.end();
If you want more info on the blending options, check How to do blending in LibGDX
I have drawn multiple different colored polygons on the screen, now I have to draw another polygon of different color, but this polygon should be drawn only on those pixels which have a specific color.
I render each of the different colored polygons at same time in their own "layers", (= one color at a time). They can cover each other; newest layer covers all previous layers. The black color in the image is the "no polygons" area: empty space, and it should ignore that too.
So, basically I just render polygons, and then the N'th (not first) layer of polygons must be masked with the next polygon layer, and nothing else under it should be affected.
Image of the method needed:
What method can I use to achieve this with OpenGL ? I would prefer non-shader solution for this, if possible(?).
The only method I can do currently is to render each of the layers separately into the memory, then go through the pixels myself and combine the layers "manually", but that seems like a very slow method, doable though, but the speed is important here.
To use the stencil buffer for this, what you can do is:
Make sure you request a context that has a stencil buffer, this is windowing system specific so I won't cover it here. Call glGet(GL_STENCIL_BITS) to make sure that you get a sufficient number of bits.
The stencil buffer maintains an integer alongside each pixel, and allows you to modify it as things are drawn. As you draw each layer, set up the stencil buffer to set to a specific value when you draw each layer. You do this with
//draw layer N
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, N, -1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
Now as you draw each layer, the last polygon that was drawn to the screen also stores it's layer number into the stencil buffer.
At this point when you want to go back and draw your green star, you just tell it to only draw on pixels where the stencil buffer is equal to N.
//draw only where stencil == N
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, N, -1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
drawStar();
I want to set up multiple stencils in OpenGL and then draw into multiple combinations of them.
For example:
I have two rectangular stencils like this:
Then I want to draw into:
anywhere
left rectangle (blue+purple)
right rectangle (purple+red)
middle rectangle (purple)
whole colored area (blue+purple+red)
I have found that it is possible to declare multiple stencils in different bits of stencil buffer but I don't know how.
How do I setup glStencilOp and glStencilFunc for this to work?
Or can I (should I) use glScissor for that?
I don't currently know, if it's possible to configure the stencil buffer for being able to do all the above 5 steps without making any changes to the stencil buffer between them. It would be easy if glStencilOp provided bitwise OR, but it doesn't and with just using increment or decrement you would have to draw the rectangles multiple times.
But if the regions are always rectangles, why not just use the scissor test? So the first 3 steps (or actually 2 and 3) can be done by just setting the rectangle's region with glScissor and enabling the scissor test (glEnable(GL_SCISSOR_TEST)).
For the middle one (step 4) you either compute the purple intersection rectangle yourself and use the scissor test again, or you use the stencil test:
glEnable(GL_STENCIL_TEST);
glStencilFunc(/*whatever*/);
glStencilOp(GL_INCR, GL_INCR, GL_INCR); //increase the stencil value
//draw both rectangles
glStencilFunc(GL_EQUAL, 2, 0xFFFFFFFF); //only draw where both rectangles are
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); //don't change the stencil buffer
//draw things
So we first draw both rectangles and increase the stencil value everywhere they are drawn. Then we draw our things everywhere the stencil value is 2, meaning both rectangles were drawn.
For the 5th step you use the same stencil buffer, but with
glStencilFunc(GL_LEQUAL, 1, 0xFFFFFFFF);
for the second pass. This way you draw something everywhere the stencil buffer is at least 1, meaning at least one rectangle was drawn.
For more than two rectangles it can get more complicated and you need to play around a bit to find the most optimal way.