What's wrong with this simple OpenGL/JOGL stencil test? - opengl

Question

I'm learning how to use a stencil buffer, but so far have been unsuccessful at getting a even a simple example to work. In fact, despite trying various combinations of parameters for glStencilOp and glStencilFunc I have not been able to see any evidence that the stencil buffer is working at all. I'm starting to suspect my graphics driver (Mac Pro, Mac OS X 10.8.5) or JOGL (2.0.2) doesn't support it... or I'm missing something really basic.
Here's what I'm seeing:
I'm expecting to see the red diamond clipped by the green diamond. What am I doing wrong?
public class Test {
public static void main(String[] args) {
GLProfile glprofile = GLProfile.getDefault();
final GLCapabilities glcapabilities = new GLCapabilities(glprofile);
final GLCanvas glcanvas = new GLCanvas(glcapabilities);
final GLU glu = new GLU();
glcanvas.addGLEventListener(new GLEventListener() {
#Override
public void reshape(GLAutoDrawable glautodrawable, int x, int y, int width, int height) {}
#Override
public void init(GLAutoDrawable glautodrawable) {
GL2 gl = glautodrawable.getGL().getGL2();
glcapabilities.setStencilBits(8);
gl.glMatrixMode(GLMatrixFunc.GL_PROJECTION);
gl.glLoadIdentity();
glu.gluPerspective(45, 1, 1, 10000);
glu.gluLookAt(0, 0, 100, 0, 0, 0, 0, 1, 0);
gl.glMatrixMode(GLMatrixFunc.GL_MODELVIEW);
gl.glLoadIdentity();
}
#Override
public void dispose(GLAutoDrawable glautodrawable) {}
#Override
public void display(GLAutoDrawable glautodrawable) {
GL2 gl = glautodrawable.getGL().getGL2();
gl.glEnable(GL.GL_STENCIL_TEST);
gl.glClearStencil(0x0);
gl.glClear(GL.GL_COLOR_BUFFER_BIT | GL.GL_DEPTH_BUFFER_BIT | GL.GL_STENCIL_BUFFER_BIT);
gl.glStencilFunc(GL.GL_ALWAYS, 1, 1);
gl.glStencilOp(GL.GL_REPLACE, GL.GL_REPLACE, GL.GL_REPLACE);
gl.glStencilMask(0xFF);
//gl.glColorMask(false, false, false, false);
//gl.glDepthMask(false);
gl.glColor3f(0, 1, 0);
gl.glBegin(GL2.GL_QUADS);
gl.glVertex2f(-25.0f, 0.0f);
gl.glVertex2f(0.0f, 15.0f);
gl.glVertex2f(25.0f, 0.0f);
gl.glVertex2f(0.0f, -15.0f);
gl.glEnd();
gl.glStencilMask(0);
gl.glStencilFunc(GL2.GL_EQUAL, 1, 1);
gl.glStencilOp(GL2.GL_KEEP, GL2.GL_KEEP, GL2.GL_KEEP);
//gl.glColorMask(true, true, true, true);
//gl.glDepthMask(true);
gl.glColor3f(1, 0, 0);
gl.glBegin(GL2.GL_QUADS);
gl.glVertex2f(-20.0f, 0.0f);
gl.glVertex2f(0.0f, 20.0f);
gl.glVertex2f(20.0f, 0.0f);
gl.glVertex2f(0.0f, -20.0f);
gl.glEnd();
}
});
final JFrame jframe = new JFrame("One Triangle Swing GLCanvas");
jframe.addWindowListener(new WindowAdapter() {
#Override
public void windowClosing(WindowEvent windowevent) {
jframe.dispose();
System.exit(0);
}
});
jframe.getContentPane().add(glcanvas, BorderLayout.CENTER);
jframe.setSize(640, 480);
jframe.setVisible(true);
}
}

Answer 1

Zero298 has the right idea, though fails to explain why what you tried in your code does not work. This becomes more apparent when you understand how framebuffer pixel formats work in OpenGL; I will touch on this a little bit below, but first just to re-hash the proper solution:
public static void main(String[] args) {
GLProfile glprofile = GLProfile.getDefault ();
GLCapabilities glcapabilities = new GLCapabilities (glprofile);
// You must do this _BEFORE_ creating a render context
glcapabilities.setStencilBits (8);
final GLCanvas glcanvas = new GLCanvas (glcapabilities);
final GLU glu = new GLU ();
The important thing is that you do this before creating your render context ("canvas"). The stencil buffer is not something you can enable or disable whenever you need it -- you first have to select a pixel format that reserves storage for it. Since pixel formats are fixed from the time you create your render context onward, you need to do this before new GLCanvas (...).
You can actually use an FBO to do stencil operations in a render context that does not have a stencil buffer, but this is much more advanced than you should be considering at the moment. Something to consider if you ever want to do MSAA though, FBOs are a much nicer way of changing pixel formats at run-time than creating and destroying your render context ("canvas").

Answer 2

You need a call to glStencilMask() it's what controls what gets written or not. Set it to do or don't write, draw a stencil (in your case, the diamond), set the glStencilMask() again, and then draw what you want to get clipped.
This has a good sample: Stencil Buffer explanation
EDIT:
OK, I think I found the problem. You need to set your capabilities up at the top of the program.
final GLCapabilities glcapabilities = new GLCapabilities(glprofile);
glcapabilities.setStencilBits(8);
final GLCanvas glcanvas = new GLCanvas(glcapabilities);
The important part being:
glcapabilities.setStencilBits(8);
Thanks to: enabling stencil in jogl