How to create glClipPlanes matching the current projection matrix? - c++

Question

I'm rendering a collection of points in space which the user has manually rotated, panned, zoomed, and finally drawn a selection rectangle over a desired section. When they do this I call gluPickMatrix and then gluPerspective to construct the projection matrix, before using the GL_SELECT render mode to find the points in their selected area.
I want to only show the points in the selected area. For testing each point has a boolean isSelected, and I know selection works because the correct ones are getting selected. There are too many points for this to be feasible, though, so I'd like to use glClipPlane to construct four planes corresponding to the rectangular prism (edit: I mean the frustum) defined by the user's selection rectangle. How can I derive the coefficients to pass to glClipPlane after setting up the projection matrix?
Right now I'm doing this, but it ends up clipping all the points:
// Set up projection matrix
int viewport[4];
glGetIntegerv(GL_VIEWPORT, viewport);
gluPickMatrix(x, y, w, h, viewport);
gluPerspective(FOV_Y, ASPECT_RATIO, NEAR_PLANE, FAR_PLANE);
// Derive clip planes - this doesn't work
// Based on the paper "Fast Extraction of Viewing Frustum Planes from the World-View-Projection Matrix"
// http://graphics.cs.ucf.edu/cap4720/fall2008/plane_extraction.pdf
double t[4], r[4], b[4], l[4];
double m[16];
glGetDoublev(GL_PROJECTION_MATRIX, m);
t[0] = m[3] - m[1]; t[1] = m[7] - m[5]; t[2] = m[11] - m[9]; t[3] = m[15] - m[13];
r[0] = m[3] - m[0]; r[1] = m[7] - m[4]; r[2] = m[11] - m[8]; r[3] = m[15] - m[12];
b[0] = m[3] + m[1]; b[1] = m[7] + m[5]; b[2] = m[11] + m[9]; b[3] = m[15] + m[13];
l[0] = m[3] + m[0]; l[1] = m[7] + m[4]; l[2] = m[11] + m[8]; l[3] = m[15] + m[12];
// ... Render points in GL_SELECT mode, then go back to GL_RENDER mode ...
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
// ... Apply user's pan/rotate/zoom choices here ...
// Apply clip planes
glEnable(GL_CLIP_PLANE0);
glClipPlane(GL_CLIP_PLANE0, t);
glEnable(GL_CLIP_PLANE1);
glClipPlane(GL_CLIP_PLANE1, r);
glEnable(GL_CLIP_PLANE2);
glClipPlane(GL_CLIP_PLANE2, b);
glEnable(GL_CLIP_PLANE3);
glClipPlane(GL_CLIP_PLANE3, l);
I also tried normalizing and/or negating the plane coordinates, which still seems to end up clipping all the points.

Answer 1

After reading that paper more carefully, I found the problem. I wasn't taking the model view matrix into account.
This:
double m[16];
glGetDoublev(GL_PROJECTION_MATRIX, m);
// ... derive t, r, b, and l from m
Should be this:
double vm[16], pm[16], m[16];
glGetDoublev(GL_MODELVIEW_MATRIX, vm);
glGetDoublev(GL_PROJECTION_MATRIX, pm);
matrix_mul(vm, pm, m); // m = vm * pm
// ... derive t, r, b, and l from m