I'm trying to use the stencil buffer to draw outline border behind a model.
I'm using 2 render passes, first rendering the model and using GL_ALWAYS to write 1 to the stencil buffer and then I render a slightly scaled up version of the model that uses GL_NOTEQUAL with ref value of 1 so basically to pass only fragments that are outside the original cube thus making the highlight.
I'm getting weird result, the outline is not fully rendered and when I move my camera enough it disappears completely as if the stencil test of the scaled up cube doesn't pass at all.
Here's the gist of the rendering loop (shader3 just outputs a color for the border):
while (!glfwWindowShouldClose(window))
{
glClearColor(0.2f, 0.3f, 0.3f, 1.0f);
glClearStencil(0);
renderer.Clear();
...
glm::mat4 projection = glm::perspective(glm::radians(45.0f), (float)SCREEN_WIDTH / (float)SCREEN_HEIGHT, 0.1f, 100.0f);
UniformMat4f projectionUniform("u_Projection");
projectionUniform.SetValues(projection);
glm::mat4 view = camera.GetView();
UniformMat4f viewUniform("u_View");
viewUniform.SetValues(view);
modelUniform.SetValues(glm::translate(glm::mat4(1.0f), glm::vec3(5.0f, 0.0f, 0.0f)));
shader1.SetUniform(projectionUniform);
shader1.SetUniform(viewUniform);
shader1.SetUniform(modelUniform);
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilMask(0xFF);
renderer.Draw(model, shader1);
modelUniform.SetValues(glm::scale(glm::translate(glm::mat4(1.0f), glm::vec3(5.0f, 0.0f, 0.0f)), glm::vec3(1.1f, 1.1f, 1.1f)));
shader3.SetUniform(projectionUniform);
shader3.SetUniform(viewUniform);
shader3.SetUniform(modelUniform);
glStencilFunc(GL_NOTEQUAL, 1, 0xFF);
glStencilMask(0x00);
glDisable(GL_DEPTH_TEST);
renderer.Draw(model, shader3);
glEnable(GL_DEPTH_TEST);
}
I first tried it with the nanosuit model from crysis and it work as I described not showing all the parts and eventually disappearing when moving the camera. I've decided to try a simple geometry to test this out and tried to outline a simple cube and got the same result:
The issue is the call to glStencilMask(0x00); at the end of the main loop. the default value of the stencil mask is 0xff (for 8 bit, see glStencilMask). OpenGL is a state engine. A state is kept, till it is changed again (even beyond frames).
glClear considers the state of the color, depth and stencil mask. If the stencil mask is 0x00, then the stencil buffer isn't cleared at all.
Set the stencil mask 0xff, before you clear the stencil buffer to solve the issue:
glStencilMask(0xFF);
renderer.Clear();
Related
I have two planar shadows of the same object coming from the same light source - one that casts on the floor and one to cast on the wall when the object is close enough. Everything works just fine as far as the shadows being cast, I'm using the stencil buffer to make sure that the two shadows only cast on their respective surfaces without being rendered outside of the room.
The problem is that the two stencil buffers bleed into each other, specifically whichever shadow I render second bleeds into the stencil buffer for the first one. I figure it's some issue with the stencil function or something, using the wrong parameters, but I can't seem to figure it out.
// Generate the shadow using a shadow matrix (created using light position and vertices of
// the quad on which the shadow will be projected) and the object I'm making a shadow of
void createShadow(float shadowMat[16])
{
glDisable(GL_DEPTH_TEST);
glDisable(GL_LIGHTING);
glDisable(GL_TEXTURE_2D);
// Set the shadow color
glColor3f(0.1, 0.1, 0.1);
glPushMatrix();
// Create the shadow using the matrix and the object casting a shadow
glMultMatrixf((GLfloat*)shadowMat);
translate, rotate etc;
render object;
glPopMatrix();
// Reset values to render the rest of the scene
glColor3f(1.0, 1.0, 1.0);
glEnable(GL_DEPTH_TEST);
glEnable(GL_LIGHTING);
glEnable(GL_TEXTURE_2D);
}
// Set up the stencil buffer and render the shadow to it
void renderShadow(float shadowMat[16], float shadowQuad[12])
{
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
glDisable(GL_DEPTH_TEST);
// Create a stencil for the shadow, using the vertices of the plane on which it will
// be projected
glPushMatrix();
translate, rotate etc;
glEnableClientState(GL_VERTEX_ARRAY);
// The shadow quad is the same vertices that I use to make the shadow matrix
glVertexPointer(3, GL_FLOAT, 0, shadowQuad);
glDrawArrays(GL_QUADS, 0, 4);
glDisableClientState(GL_VERTEX_ARRAY);
glPopMatrix();
glEnable(GL_DEPTH_TEST);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
// Render the shadow to the plane
createShadow(shadowMat);
glDisable(GL_STENCIL_TEST);
}
// In the render function:
Render floor/surrounding area;
Set up light using the same position used to make the shadow matrix;
renderShadow(wallShadowMatrix, wallVertices);
renderShadow(floorShadowMatrix, floorVertices);
Render rest of scene;
If I render the shadows on their own they work as intended, but when I render them together, whichever one rendered second shows up in the stencil of the first shadow.
I've included a few pictures; the first two show the individual Shadow on the wall and Shadow on the floor, and here is the floor shadow rendered after the wall shadow, and vice versa.
Fixed it, I needed to add the following code between the two renderShadow calls in the render function:
glClear(GL_STENCIL_BUFFER_BIT);
I have set up my operations to draw to the stencil buffer, similar to the following:
void onDisplay() {
glClear(GL_DEPTH_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_NEVER, 1, 0xFF);
glStencilOp(GL_REPLACE, GL_KEEP, GL_KEEP);
// Draw to stencil buffer
glStencilMask(0xFF);
glClear(GL_STENCIL_BUFFER_BIT); // needs mask=0xFF
draw_circle();
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
glStencilMask(0x00);
// draw where stencil's value is 0
glStencilFunc(GL_EQUAL, 0, 0xFF);
/* (nothing to draw) */
// draw only where stencil's value is 1
glStencilFunc(GL_EQUAL, 1, 0xFF);
draw_scene();
glDisable(GL_STENCIL_TEST);
}
Now, if I have the following fragment shader enabled when I call draw_circle() (above):
void main() {
gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
}
How will the values of the stencil buffer differ from if I were to use the following fragment shader?
void main() {
gl_FragColor = vec4(0.0, 1.0, 0.0, 1.0);
}
In other words, I'm wondering how the output of the fragment shader affects the stencil buffer when drawing to the stencil buffer.
Edit:
The point of my question is to correct some misunderstandings I know I have regarding the stencil buffer. One example I have that I think explains the stencil buffer fairly well is [1]. Here, the following is mentioned:
The glColorMask function allows you to specify which data is written to the color buffer during a drawing operation. In this case you would want to disable all color channels (red, green, blue, alpha). Writing to the depth buffer needs to be disabled separately as well with glDepthMask, so that cube drawing operation won't be affected by leftover depth values of the rectangle. This is cleaner than simply clearing the depth buffer again later.
So, it seems from this page, that, in order to write to the stencil buffer, one needs to enable/disable the appropriate modes (i.e. color and depth), and then go through the entire rasterization process, which will only write to the stencil buffer. Since the rasterization process includes the fragment shader, is the output of the fragment shader (i.e. gl_FragColor) simply ignored? How can I tell GL what to write to the stencil buffer position (x, y)?
[1] : https://open.gl/depthstencils
Unless you have access to AMD/ARB_shader_stencil_export, the fragment shader cannot directly affect the output to the stencil buffer. The only exception to this is discarding the fragment.
And according to this database, only AMD cards provide this extension. Also, that extension exposes an output specifically for the stencil. It modifies the stencil value of the fragment; the color values of the fragment never affect the fragment's stencil value.
The below code is nearly identical to the code retrieved from this NeHe tutorial. The only difference between my code and the code on the tutorial is that I am using SFML for window context, which should not be relevant. To view the entire source code, go here. A snippet of the relevant code is below (the comments are from NeHe):
// Clip Plane Equations
double eqr[] = {0.0f,-1.0f, 0.0f, 0.0f}; // Plane Equation glColorMask(0,0,0,0); // Set Color Mask
glEnable(GL_STENCIL_TEST); // Enable Stencil Buffer For "marking" The Floor
glStencilFunc(GL_ALWAYS, 1, 1); // Always Passes, 1 Bit Plane, 1 As Mask
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE); // We Set The Stencil Buffer To 1 Where We Draw Any Polygon
// Keep If Test Fails, Keep If Test Passes But Buffer Test Fails
// Replace If Test Passes
glDisable(GL_DEPTH_TEST); // Disable Depth Testing
DrawFloor(); // Draw The Floor (Draws To The Stencil Buffer)
// We Only Want To Mark It In The Stencil Buffer
glEnable(GL_DEPTH_TEST); // Enable Depth Testing
glColorMask(1,1,1,1); // Set Color Mask to TRUE, TRUE, TRUE, TRUE
glStencilFunc(GL_EQUAL, 1, 1); // We Draw Only Where The Stencil Is 1
// (I.E. Where The Floor Was Drawn)
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); // Don't Change The Stencil Buffer
glEnable(GL_CLIP_PLANE0); // Enable Clip Plane For Removing Artifacts
// (When The Object Crosses The Floor)
glClipPlane(GL_CLIP_PLANE0, eqr); // Equation For Reflected Objects
glPushMatrix(); // Push The Matrix Onto The Stack
glScalef(1.0f, -1.0f, 1.0f); // Mirror Y Axis
glLightfv(GL_LIGHT0, GL_POSITION, LightPos); // Set Up Light0
glTranslatef(0.0f, height, 0.0f); // Position The Object
DrawObject(); // Draw The Sphere (Reflection)
glPopMatrix(); // Pop The Matrix Off The Stack
glDisable(GL_CLIP_PLANE0); // Disable Clip Plane For Drawing The Floor
glDisable(GL_STENCIL_TEST); // We Don't Need The Stencil Buffer Any More (Disable)
glLightfv(GL_LIGHT0, GL_POSITION, LightPos); // Set Up Light0 Position
glEnable(GL_BLEND); // Enable Blending (Otherwise The Reflected Object Wont Show)
glDisable(GL_LIGHTING); // Since We Use Blending, We Disable Lighting
glColor4f(1.0f, 1.0f, 1.0f, 0.8f); // Set Color To White With 80% Alpha
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA); // Blending Based On Source Alpha And 1 Minus Dest Alpha
DrawFloor(); // Draw The Floor To The Screen
glEnable(GL_LIGHTING); // Enable Lighting
glDisable(GL_BLEND); // Disable Blending
glTranslatef(0.0f, height, 0.0f); // Position The Ball At Proper Height
DrawObject();
The final result of this code can be seen below:
How do I alter the above code to cause the bottom (reflected) sphere to appear only on the plane instead of outside of it.
Well, do you actually create a GL context with a stencil buffer? The only relevant line for context creation in your code seems to be
f::RenderWindow window(sf::VideoMode(800, 600, 32), "Test");
and that is not very specific. I don't know SFML, but why do you think changing the code for context creation isn't relevant here?
I need to render a sphere to a texture (done using a Framebuffer Object (FBO)), and then alpha blend that texture with the back buffer. So far I'm not doing any processing with the texture except clearing it at the beginning of every frame.
I should say that my scene consists of nothing but a planet in empty space, the sphere should appear next to or around the planet (kind of like a moon for now). When I render the sphere directly to the back buffer, it displays correctly; but when I do the intermediary step of rendering it to a texture and then blending that texture with the back buffer, the sphere only shows up when it is in front of the planet, the part that isn't in front is just "cut off":
I render the sphere using glutSolidSphere to a RGBA8 fullscreen texture that's bound to an FBO, making sure that every sphere pixel receives an alpha value of 1.0. I then pass the texture to a fragment shader program, and use this code to render a fullscreen quad - with the texture mapped onto it - to the backbuffer while alpha blending:
glBindFramebuffer(GL_FRAMEBUFFER, 0);
glBlendFunc(GL_SRC_ALPHA,GL_ONE_MINUS_SRC_ALPHA);
glEnable(GL_BLEND);
glDisable(GL_DEPTH_TEST);
glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glLoadIdentity();
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();
glBegin(GL_QUADS);
glTexCoord2i(0, 1);
glVertex3i(-1, 1, -1); // TOP LEFT
glTexCoord2i(0, 0);
glVertex3i(-1, -1, -1); // BOTTOM LEFT
glTexCoord2i(1, 0);
glVertex3i( 1, -1, -1); // BOTTOM RIGHT
glTexCoord2i(1, 1);
glVertex3i( 1, 1, -1); // TOP RIGHT
glEnd();
glPopMatrix();
glMatrixMode(GL_MODELVIEW);
glPopMatrix();
glEnable(GL_DEPTH_TEST);
glDisable(GL_BLEND);
This is the shader code (taken from an FX file written in Cg):
sampler2D BlitSamp = sampler_state
{
MinFilter = LINEAR;
MagFilter = LINEAR;
MipFilter = LINEAR;
AddressU = Clamp;
AddressV = Clamp;
};
float4 blendPS(float2 texcoords : TEXCOORD0) : COLOR
{
float4 outColor = tex2D(BlitSamp, texcoords);
return outColor;
}
I don't even know whether this is a problem with the depth buffer or with alpha blending, I've tried a lot of combinations of enabling and disabling depth testing (with a depth buffer attached to the FBO) and alpha blending.
EDIT: I tried just rendering a blank fullscreen quad straight to the back buffer and even that was cropped around the planet's edges. For some reason, enabling depth testing for rendering the quad (that is, removing the lines glDisable(GL_DEPTH_TEST) and glEnable(GL_DEPTH_TEST) in the code above) got rid of the problem, but now everything but the planet and the sphere appears white:
I made sure (and could confirm) that the alpha channel of the texture is 0 at every pixel but the sphere's, so I don't understand where the whiteness could be introduced. (Would also still be interested in an explanation why enabling depth testing has this effect.)
I see two possible sources of error here:
1. Rendering to the FBO
If the missing pixels are not even present in the FBO after rendering, there must be some mechanism which discarded the corresponding fragments. The OpenGL pipeline includes four different types of fragment tests which can lead to fragments being discarded:
Scissor Test: Unlikely to be the cause, as the scissor test only affects a rectangular portion of the screen.
Alpha Test: Equally unlikely, as your fragments should all have the same alpha value.
Stencil Test: Also unlikely, unless you use stencil operations when drawing the background planet and copy over the stencil buffer from the back buffer to the FBO.
Depth Test: Same as for stencil test.
So there's a good chance that rendering into FBO is not the issue here. But just to be absolutely sure, you should read back your color attachment texture and dump it into a file for inspection. You can use the following function for that:
void TextureToFile(GLuint texture, const char* filename) {
glBindTexture(GL_TEXTURE_2D, texture);
GLint width, height;
glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_WIDTH, &width);
glGetTexLevelParameteriv(GL_TEXTURE_2D, 0, GL_TEXTURE_HEIGHT, &height);
std::vector<GLubyte> pixels(3 * width * height);
glGetTexImage(GL_TEXTURE_2D, 0, GL_RGB, GL_UNSIGNED_BYTE, &pixels[0]);
std::ofstream out(filename, std::ios::out | std::ios::binary);
out << "P6\n"
<< width << '\n'
<< height << '\n'
<< 255 << '\n';
out.write(reinterpret_cast<const char*>(&pixels[0]), pixels.size());
}
The resulting file is a portable pixmap (.ppm). Be sure to unbind the FBO before reading back the texture.
2. Texture mapping
Assuming rendering into the FBO works as expected, the only other source of error is blending the texture over the previously rendered scene. There are two scenarios:
a) Fragments get discarded
The possible reasons for fragments to get discarded are the same as in 1.:
Scissor Test: Nope, affects rectangular areas only.
Alpha Test: Probably not, the texels covered sphere should all have the same alpha value.
Stencil Test: Might be the cause if you use stencil operations/stencil testing when drawing the background planet and the old stencil state is still active.
Depth Test: Might be the cause, but as you already disable it, it really shouldn't have any effect.
So you should make sure that all of these tests are disabled, especially the stencil test.
b) Wrong results from blending
Assuming all fragments reach the back buffer, blending is the only thing which could still cause the wrong result. With your blending function (GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) the values in the back buffer are irrelevant for blending, and we assume that the alpha values in the texture are correct. So I see no reason for why blending should be the root cause here.
Conclusion
In conclusion, the only sensible cause for the observed result seems to be stencil testing. If it's not, I'm out of options :)
I solved it or at least came up with a work around.
First off, the whiteness stems from the fact that glClearColor had been set to glClearColor(1.0f, 1.0f, 1.0f, 1000.0f), so everything but the planet wasn't even written to in the end. I now copy the contents of the back buffer (which is the planet, the atmosphere, and the space around it) to the texture before rendering the sphere, and I render the atmosphere and space before that copy/blit operation, so they are included in it. Previously, everything but the planet itself was rendered after my quad, which - when using depth testing - apparently placed everything behind the quad, making it invisible.
The reference implementation of the effect I'm trying to achieve has always used this kind of blit operation in its code but I didn't think it was necessary for the effect. Now I feel like there might be no other way...
I'm having an OpenGL texture that is binded to a simple quad.
My problem is: My texture is 128x128 pixels image. I'm only filling up about 100x60 pixels on that image, the other pixels are transparent. I saved it in a .png file. When I'm drawing, the transparent part of the binded texture is white.
Let's say I have a background. When I draw this new quad on this background I can't see the through the transparent part of my texture.
Any suggestions?
Code:
// Init code...
gl.glEnable(gl.GL_TEXTURE_2D);
gl.glDisable(gl.GL_DITHER);
gl.glDisable(gl.GL_LIGHTING);
gl.glDisable(gl.GL_DEPTH_TEST);
gl.glTexEnvi(gl.GL_TEXTURE_ENV, gl.GL_TEXTURE_ENV_MODE, gl.GL_MODULATE);
// Drawing code...
gl.glBegin(gl.GL_QUADS);
gl.glTexCoord2d(0.0, 0.0);
gl.glVertex3f(0.0f, 0.0f, 0.0f);
gl.glTexCoord2d(1.0, 0.0);
gl.glVertex3f(1.0f, 0.0f, 0.0f);
gl.glTexCoord2d(1.0, 1.0);
gl.glVertex3f(1.0f, 1.0f, 0.0f);
gl.glTexCoord2d(0.0, 1.0);
gl.glVertex3f(0.0f, 1.0f, 0.0f);
gl.glEnd();
I've tried almost everything, from enabling blending to change to GL_REPLACE, however I can't get it to work.
Edit:
// Texture. Have tested both gl.GL_RGBA and gl.GL_RGB8.
gl.glTexImage2D(gl.GL_TEXTURE_2D, 0, (int)gl.GL_RGBA, imgWidth, imgHeight,
0, gl.GL_BGR_EXT, gl.GL_UNSIGNED_BYTE, bitmapdata.Scan0);
Check that your texture is of RGBA format, and enable blending and set the blending func:
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
And draw the texture. If your texture is not RGBA, then there is no alpha and blending won't do anything.
EDIT: Since you posted your code, i can a spot a serious error:
glTexImage2D(gl.GL_TEXTURE_2D, 0, (int)gl.GL_RGBA, imgWidth, imgHeight, 0, gl.GL_BGR_EXT, gl.GL_UNSIGNED_BYTE, bitmapdata.Scan0);
You're telling GL that the texture has internalFormat RGBA, but the bitmap data has BGR format, so, no alpha from your texture data. This assumes alpha = 1.0.
To correct it, load your PNG with RGBA format and use GL_RGBA as internalFormat and format parameters for glTexImage2D.
When I'm drawing, the transparent part of the binded texture is white.
That means your PNG-parser converted transparent regions to the value of white. If you want to render transparent layers with OpenGL you dont typically depend on texture-files to hold the transparency but instead use the GLBlendFunc(). More information here:
http://www.opengl.org/resources/faq/technical/transparency.htm
Also, should you render to a frame buffer and copy the result into a texture, check that the frame buffer has alpha turned on. For example when using osgViewer this can be achived by (do this before calling setUpViewInWindow):
osg::DisplaySettings *pSet = myviewer.getDisplaySettings();
if(pSet == NULL)
pSet = new osg::DisplaySettings();
pSet->setMinimumNumAlphaBits(8);
myviewer.setDisplaySettings(pSet);
and under qt it should work with (from http://forum.openscenegraph.org/viewtopic.php?t=6411):
QGLFormat f;
f.setAlpha( true ); //enables alpha channel for this format
QGLFormat::setDefaultFormat( f ); //set it as default before instantiations
setupUi(this); //instantiates QGLWidget (ViewerQT)
Normally it is better to render directly into a frame buffer but I came alonge this while preparing some legacy code and in the beginning it was very hard to find this.