OpenGL ignore object when blending? - c++

Question

I'm rendering a quad with an outline. I do this by rendering a quad of the required size in black, then rendering the "inner" quad smaller than the first. I also want this outlined quad to be transparent.
My current attempt renders as so:
However the problem I have is that because I'm essentially rendering two quads on top of each other, the black "main" quad blends through to the smaller "inner" quad. I don't want this to happen as I want the blending to only blend the background and the "inner" quad colors.
Other than rendering separate quads/triangles for the outline is there a way (ideally not using shaders as I'm using fixed pipeline) to make the blending ignore/not render the "main" quad when blending the "inner" quad?
Relevant code:
Blend mode:
glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA);
glEnable( GL_BLEND );
Rendering the box:
struct vertex
{
GLfloat x,y;
};
void draw_box()
{
glPushMatrix();
glTranslatef( -3.0f, 1.12f, 1.0f );
// Draw a huge black quad
const vertex vertices2[] =
{
{ -1.0f, -1.0f },
{ 0.0f, 0.0f },
{ -1.0f, 0.0f },
{ -1.0f, -1.0f },
{ 0.0f, 0.0f },
{ 0.0f, -1.0f },
};
glVertexPointer(2, GL_FLOAT, 0, vertices2);
glColor4f(0.0f, 0.0f, 0.0f, 0.5f);
glDrawArrays(GL_TRIANGLES, 0, 6);
// Draw a smaller grey quad
const GLfloat space = 0.04f;
const vertex vertices3[] =
{
{ -1.0f+space, -1.0f+space },
{ 0.0f-space, 0.0f-space },
{ -1.0f+space, 0.0f-space },
{ -1.0f+space, -1.0f+space },
{ 0.0f-space, 0.0f-space },
{ 0.0f-space, -1.0f+space },
};
glVertexPointer(2, GL_FLOAT, 0, vertices3);
glColor4f(0.5f, 0.5f, 0.5f, 0.5f);
glDrawArrays(GL_TRIANGLES, 0, 6);
glPopMatrix();
}

Answer 1

The best way to do this is probably to split up the border into four trapezoidal quads, but since you explicitly mentioned you didn't want to go down that path, there are a couple of other options you could pursue.
The next best method is probably to use the depth buffer: glEnable(GL_DEPTH_TEST). Draw the inside quad first (with blending enabled). Then draw the outer quad, but using coordinates such that it's depth is greater than the first quad (it is "farther away"). If depth testing is enabled when the border quad is drawn, it will fail for the quad you already drew, since those pixels have a smaller depth value. Note that using this method, if you are writing to the depth buffer when rendering the things behind these quads, you may run into some issues with parts of the background being considered closer than the new quads. You can solve this for the inner quad by configuring the depth test to always succeed, even when the test fails: glDepthFunc(GL_ALWAYS). This won't help for the outer quad, however. In that case the next method might be helpful:
You can use the stencil buffer in a similar manner to the depth buffer. When rendering the inside quad, you set the stencil value (or one bit of it at least) to a specific value:
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 1, GLuint(-1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
Then when you draw the outer quad, configure the stencil test to succeed only if a fragment does not have that value:
glStencilFunc(GL_NOT_EQUAL, 1, 1);
Essentially, this is the same thing the version above was doing with the depth buffer, but uses the stencil buffer instead. You may want to take a look at the documentation for glStencilFunc and glStencilOp.
There may be other ways to achieve this behavior, but all the other solutions I can think of involve using custom shaders, which you indicated is not acceptable. Personally, I'd just split up the border into 4 quads. Sure, it's 3 more quads than the other methods, but it doesn't require any extra OpenGL features or state changes.