How to clip with circles in OpenGL - opengl

Question

I'm wondering if it is possible to simulate the effect of looking through the keyhole in OpenGL.
I have my 3D scene drawn but I want to make everythig black everything except a central circle.
I tried this solution but its doing the completely opposite of what I want:
// here i draw my 3D scene
// Begin 2D orthographic mode
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();
GLint viewport [4];
glGetIntegerv(GL_VIEWPORT, viewport);
gluOrtho2D(0, viewport[2], viewport[3], 0);
glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glLoadIdentity();
// Here I draw a circle in the center of the screen
float radius=50;
glBegin(GL_TRIANGLE_FAN);
glVertex2f(x, y);
for( int n = 0; n <= 100; ++n )
{
float const t = 2*M_PI*(float)n/(float)100;
glVertex2f(x + sin(t)*r, y + cos(t)*r);
}
glEnd();
// end orthographic 2D mode
glMatrixMode(GL_PROJECTION);
glPopMatrix();
glMatrixMode(GL_MODELVIEW);
glPopMatrix();
What I get is a circle drawn in the center, but I would like to obtain its complementary...

Answer 1

Like everything else in OpenGL, there are a few ways to do this. Here are two off the top of my head.
Use a circle texture: (recommended)
Draw the scene.
Switch to an orthographic projection, and draw a quad over the entire screen using a texture which has a white circle at the center. Use the appropriate blending function:
glEnable(GL_BLEND);
glBlendFunc(GL_ZERO, GL_SRC_COLOR);
/* Draw a full-screen quad with a white circle at the center */
Alternatively, you can use a pixel shader to generate the circular shape.
Use a stencil test: (not recommended, but it may be easier if you don't have textures or shaders)
Clear the stencil buffer, and draw the circle into it.
glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
/* draw circle */
Enable the stencil test for the remainder of the scene.
glEnable(GL_STENCIL_TEST)
glStencilFunc(GL_EQUAL, 1, 1);
glStencileOp(GL_KEEP, GL_KEEP, GL_KEEP);
/* Draw the scene */
Footnote: I recommend avoiding use of immediate mode at any point in your code, and using arrays instead. This will improve the compatibility, maintainability, readibility, and performance of your code --- a win in all areas.