I am trying to understand what the following code does:
glm::mat4 Projection = glm::perspective(35.0f, 1.0f, 0.1f, 100.0f);
Does it create a projection matrix? Clips off anything that is not in the user's view?
I wasn't able to find anything on the API page, and the only thing I could find in the pdf on their website was this:
gluPerspective:
glm::mat4 perspective(float fovy, float aspect, float zNear,
float zFar);
glm::dmat4 perspective(
double fovy, double aspect, double zNear,
double zFar);
From GLM_GTC_matrix_transform extension: <glm/gtc/matrix_transform.hpp>
But it doesn't explain the parameters. Maybe I missed something.
It creates a projection matrix, i.e. the matrix that describes the set of linear equations that transforms vectors from eye space into clip space. Matrices really are not black magic. In the case of OpenGL they happen to be a 4-by-4 arrangement of numbers:
X_x Y_x Z_x T_x
X_y Y_y Z_y T_y
X_z Y_z Z_z T_z
X_w Y_w Z_w W_w
You can multply a 4-vector by a 4×4 matrix:
v' = M * v
v'_x = M_xx * v_x + M_yx * v_y + M_zx * v_z + M_tx * v_w
v'_y = M_xy * v_x + M_yy * v_y + M_zy * v_z + M_ty * v_w
v'_z = M_xz * v_x + M_yz * v_y + M_zz * v_z + M_tz * v_w
v'_w = M_xw * v_x + M_yw * v_y + M_zw * v_z + M_tw * v_w
After reaching clip space (i.e. after the projection step), the primitives are clipped. The vertices resulting from the clipping are then undergoing the perspective divide, i.e.
v'_x = v_x / v_w
v'_y = v_y / v_w
v'_z = v_z / v_w
( v_w = 1 = v_w / v_w )
And that's it. There's really nothing more going on in all those transformation steps than ordinary matrix-vector multiplication.
Now the cool thing about this is, that matrices can be used to describe the relative alignment of a coordinate system within another coordinate system. What the perspective transform does is, that it let's the vertices z-values "slip" into their projected w-values as well. And by the perspective divide a non-unity w will cause "distortion" of the vertex coordinates. Vertices with small z will be divided by a small w, thus their coordinates "blow" up, whereas vertices with large z will be "squeezed", which is what's causing the perspective effect.
This is a c standalone version of the same function. This is roughly a copy paste version of the original.
# include <math.h>
# include <stdlib.h>
# include <string.h>
typedef struct s_mat {
float *array;
int width;
int height;
} t_mat;
t_mat *mat_new(int width, int height)
{
t_mat *to_return;
to_return = (t_mat*)malloc(sizeof(t_mat));
to_return->array = malloc(width * height * sizeof(float));
to_return->width = width;
to_return->height = height;
return (to_return);
}
void mat_zero(t_mat *dest)
{
bzero(dest->array, dest->width * dest->height * sizeof(float));
}
void mat_set(t_mat *m, int x, int y, float val)
{
if (m == NULL || x > m->width || y > m->height)
return ;
m->array[m->width * (y - 1) + (x - 1)] = val;
}
t_mat *mat_perspective(float angle, float ratio,
float near, float far)
{
t_mat *to_return;
float tan_half_angle;
to_return = mat_new(4, 4);
mat_zero(to_return);
tan_half_angle = tan(angle / 2);
mat_set(to_return, 1, 1, 1 / (ratio * tan_half_angle));
mat_set(to_return, 2, 2, 1 / (tan_half_angle));
mat_set(to_return, 3, 3, -(far + near) / (far - near));
mat_set(to_return, 4, 3, -1);
mat_set(to_return, 3, 4, -(2 * far * near) / (far - near));
return (to_return);
}
Related
According to the thread on this page, The equation given for calculating the depth buffer:
F_depth = 1/z - 1/n/(1/f - 1/n)
is non-linear only because of the perspective divide.(Note that this is a combination of from the view-space z coord to window coord directly)
So, as per my understanding:
to convert it to a linear depth buffer, the only thing we would do is to remove the perspective divide(?) and then perform the glDepthRange(a,b) given here.
In that case, the equation would be like this:
z_linear = z_NDC * W_clip = -(f+n)/(f-n)*z_eye + ( 2fn/(f-n) )
and, with depth range transformation:
z_[0,1] = ( z_linear + 1 ) /2
= ( (f+n)*z_eye - 2fn + f - n )/ ( 2(f-n) )
but, in the learnopenGL site for depth testing this is done:
First we transform the depth value to NDC which is not too difficult:
float ndc = depth * 2.0 - 1.0;
We then take the resulting ndc value and apply the inverse
transformation to retrieve its linear depth value:
float linearDepth = (2.0 * near * far) / (far + near - ndc * (far - near)
how is the non-linear to linear depth-buffer being computed?(i.e. equation being formed)?
Using glm in a right handed system I found the following solutions for converting from ndc depth [-1, 1] to eye depth [0-far].
perspective projection:
float screen_to_eye_depth_perspective(float d, float zNear, float zFar)
{
// [0; 1] -> [-1; 1]
float depth = d * 2.0 - 1.0;
return (2.0 * zNear * zFar) / (zFar + zNear - depth * (zFar - zNear));
}
orthographic projection:
float screen_to_eye_depth_ortho(float d, float zNear, float zFar)
{
// [0; 1] -> [-1; 1]
float depth = d * 2.0 - 1.0;
return (depth * (zFar - zNear) + (zFar + zNear)) / 2.0;
}
I advise you to test with your own near and far value to check the final result.
n = 10;
f = 100;
// -------------
z = 2 * f * n / (f + n) - ndc * (f - n);
z = 2000 / (110 - ndc * 90);
// -------------
ndc = -1;
z = 2000 / (110 + 90) = 10;
ndc = 1;
z = 2000 / (110 - 90) = 100;
I'm trying to implement terrain collision for my height map terrain, and I'm following this. The tutorial is for java but I'm using C++, though the principles are the same so it shouldn't be a problem.
To start off we need a function to get the height of the terrain based on the camera's position. WorldX and WorldZ is the camera's position (x, z) and heights is an 2D-array containing all the heights of the vertices.
float HeightMap::getHeightOfTerrain(float worldX, float worldZ, float heights[][256])
{
//Image is (256 x 256)
float gridLength = 256;
float terrainLength = 256;
float terrainX = worldX;
float terrainZ = worldZ;
float gridSquareLength = terrainLength / ((float)gridLength - 1);
int gridX = (int)std::floor(terrainX / gridSquareLength);
int gridZ = (int)std::floor(terrainZ / gridSquareLength);
//Check if position is on the terrain
if (gridX >= gridLength - 1 || gridZ >= gridLength - 1 || gridX < 0 || gridZ < 0)
{
return 0;
}
//Find out where the player is on the grid square
float xCoord = std::fmod(terrainX, gridSquareLength) / gridSquareLength;
float zCoord = std::fmod(terrainZ, gridSquareLength) / gridSquareLength;
float answer = 0.0;
//Top triangle of a square else the bottom
if (xCoord <= (1 - zCoord))
{
answer = barryCentric(glm::vec3(0, heights[gridX][gridZ], 0),
glm::vec3(1, heights[gridX + 1][gridZ], 0), glm::vec3(0, heights[gridX][gridZ + 1], 1),
glm::vec2(xCoord, zCoord));
}
else
{
answer = barryCentric(glm::vec3(1, heights[gridX + 1][gridZ], 0),
glm::vec3(1, heights[gridX + 1][gridZ + 1], 1), glm::vec3(0, heights[gridX][gridZ + 1], 1),
glm::vec2(xCoord, zCoord));
}
return answer;
}
To find the height of the triangle the camera is currently standing on we use the baryCentric interpolation function.
float HeightMap::barryCentric(glm::vec3 p1, glm::vec3 p2, glm::vec3 p3, glm::vec2 pos)
{
float det = (p2.z - p3.z) * (p1.x - p3.x) + (p3.x - p2.x) * (p1.z - p3.z);
float l1 = ((p2.z - p3.z) * (pos.x - p3.x) + (p3.x - p2.x) * (pos.y - p3.z)) / det;
float l2 = ((p3.z - p1.z) * (pos.x - p3.x) + (p1.x - p3.x) * (pos.y - p3.z)) / det;
float l3 = 1.0f - l1 - l2;
return l1 * p1.y + l2 * p2.y + l3 * p3.y;
}
Then we just have to use the height we have calculated to check for
collision during the game
float terrainHeight = heightMap.getHeightOfTerrain(camera.Position.x, camera.Position.z, heights);
if (camera.Position.y < terrainHeight)
{
camera.Position.y = terrainHeight;
};
Now according to the tutorial this should work perfectly fine, but the height is rather off and at some places it doesn't even work. I figured it might have something to do with the translation and scaling part of the terrain
glm::mat4 model;
model = glm::translate(model, glm::vec3(0.0f, -0.3f, -15.0f));
model = glm::scale(model, glm::vec3(0.1f, 0.1f, 0.1f));
and that I should multiply the values of the heights array by 0.1, as the scaling does that part for the terrain on the GPU side, but that didn't do the trick.
Note
In the tutorial the first lines in the getHeightOfTerrain function says
float terrainX = worldX - x;
float terrainZ = worldZ - z;
where x and z is the world position of the terrain. This is done to get the player position relative to the terrain's position. I tried with the values from the translation part, but it doensn't work either. I changed these lines because it doesn't seem necessary.
float terrainX = worldX - x;
float terrainZ = worldZ - z;
Those lines are, in fact, very necessary, unless your terrain is always at the origin.
Your code resource (tutorial) assumes that you haven't scaled or rotated the terrain in any way. The x and z variables are the XZ position of the terrain which take care of cases where the terrain is translated.
Ideally, you should transform the world position vector from world space to object space (using the inverse of the model matrix you use for the terrain), something like
vec3 localPosition = inverse(model) * vec4(worldPosition, 1)
And then use localPosition.x and localPosition.z instead of terrainX and terrainZ.
My Question
Can someone please link a good article/tutorial/anything or maybe even explain how to correctly cast a ray from the mouse coordinates to pick objects in 3D?
I already have the Ray and intersection works, now I only need to create the ray from the mouse click.
I would just like have something which I know actually should work, thats why I ask the professionals here, not something where I am unsure if it is even correct in the first place.
State right now
I have a ray class, which actually works and detects intersection if I set the origin and direction to be the same as the camera, so when I move the camera it actually selects the right thing.
Now I would like to actually have 3D picking with the mouse, not camera movement.
I have read so many other questions about this, 2 tutorials, and especially so much different math stuff, since I am really not good at it.
But that didn't help me much, because the people there often use some "unproject" functions, which seem to actually be deprecated and which I have no idea how to use and also don't have access to.
Right now I set the ray origin to the camera position and then try to get the direction of the ray from the calculations in this tutorial.
And it works a little bit, meaning the selection works when the camera is pointed at the object and also sometimes along the whole y-axis, I have no idea what is happening.
If someone wants to take a look at my code right now:
public Ray2(Camera cam, float mouseX, float mouseY) {
origin = cam.getEye();
float height = 600;
float width = 600;
float aspect = (float) width / (float) height;
float x = (2.0f * mouseX) / width - 1.0f;
float y = 1.0f - (2.0f * mouseX) / height;
float z = 1.0f;
Vector ray_nds = vecmath.vector(x, y, z);
Vector4f clip = new Vector4f(ray_nds.x(), ray_nds.y(), -1.0f, 1.0f);
Matrix proj = vecmath.perspectiveMatrix(60f, aspect, 0.1f, 100f);
proj = proj.invertRigid();
float tempX = proj.get(0, 0) * clip.x + proj.get(1, 0) * clip.y
+ proj.get(2, 0) * clip.z + proj.get(3, 0) * clip.w;
float tempY = proj.get(0, 1) * clip.x + proj.get(1, 1) * clip.y
+ proj.get(2, 1) * clip.z + proj.get(3, 1) * clip.w;
float tempZ = proj.get(0, 2) * clip.x + proj.get(1, 2) * clip.y
+ proj.get(2, 2) * clip.z + proj.get(3, 2) * clip.w;
float tempW = proj.get(0, 3) * clip.x + proj.get(1, 3) * clip.y
+ proj.get(2, 3) * clip.z + proj.get(3, 3) * clip.w;
Vector4f ray_eye = new Vector4f(tempX, tempY, tempZ, tempW);
ray_eye = new Vector4f(ray_eye.x, ray_eye.y, -1.0f, 0.0f);
Matrix view = cam.getTransformation();
view = view.invertRigid();
tempX = view.get(0, 0) * ray_eye.x + view.get(1, 0) * ray_eye.y
+ view.get(2, 0) * ray_eye.z + view.get(3, 0) * ray_eye.w;
tempY = view.get(0, 1) * ray_eye.x + view.get(1, 1) * ray_eye.y
+ view.get(2, 1) * ray_eye.z + view.get(3, 1) * ray_eye.w;
tempZ = view.get(0, 2) * ray_eye.x + view.get(1, 2) * ray_eye.y
+ view.get(2, 2) * ray_eye.z + view.get(3, 2) * ray_eye.w;
tempW = view.get(0, 3) * ray_eye.x + view.get(1, 3) * ray_eye.y
+ view.get(2, 3) * ray_eye.z + view.get(3, 3) * ray_eye.w;
Vector ray_wor = vecmath.vector(tempX, tempY, tempZ);
// don't forget to normalise the vector at some point
ray_wor = ray_wor.normalize();
direction = ray_wor;
}
First,unproject() method is the way to go.It is not deprecated at all.You can find it implemented in GLM math library for example.Here is my implementation of Ray based 3D picking:
// let's check if this renderable's AABB is clicked:
const glm::ivec2& mCoords = _inputManager->GetMouseCoords();
int mouseY = _viewportHeight - mCoords.y;
//unproject twice to build a ray from near to far plane"
glm::vec3 v0 = glm::unProject(glm::vec3(float(mCoords.x), float(mouseY), 0.0f),_camera->Transform().GetView(),_camera->Transform().GetProjection(), _viewport);
glm::vec3 v1 = glm::unProject(glm::vec3(float(mCoords.x), float(mouseY), 1.0f),_camera->Transform().GetView(),_camera->Transform().GetProjection(), _viewport);
glm::vec3 dir = (v1 - v0);
Ray r(_camera->Transform().GetPosition(),dir);
float ishit ;
//construct AABB:
glm::mat4 aabbMatr = glm::translate(glm::mat4(1.0),renderable->Transform().GetPosition());
aabbMatr = glm::scale(aabbMatr,renderable->Transform().GetScale());
//transforms AABB vertices(need it if the origianl bbox is not axis aligned as in this case)
renderable->GetBoundBox()->RecalcVertices(aabbMatr);
//this method makes typical Ray-AABB intersection test:
if(r.CheckIntersectAABB(*renderable->GetBoundBox().get(),&ishit)){
printf("HIT!\n");
}
But I would suggest you also to take a look at color based 3d picking which is pixel perfect and even easier to implement.
This function is supposed to give me the exact size of my near clipping plane.
public Vector2 NearplaneSize
{
get
{
float w = 2 * Mathf.Tan(Mathf.Deg2Rad(Fov) / 2) * ZNear;
return new Vector2(w, w / AspectRatio);
}
}
I'm creating a plane like this:
Vector2 s = cam.NearplaneSize;
Mesh = PrimitiveFactory.CreatePlane(s.X / -2, s.Y / -2, -(cam.ZNear + 0.1f), s.X, s.Y, 1, 1, Quaternion.FromAxisAngle(Vector3.UnitX, Mathf.Deg2Rad(90)));
in front of the camera, but its slightly larger than half the screen. So obviously the calculation is wrong. I can't seem to find a better formula though.
Any ideas? Thanks
I don't know OpenTK, but due to old gluPerspective call, "Fov" is generally understood as fov y, not fovx.
So I assume that
float h = 2 * Mathf.Tan(Mathf.Deg2Rad(Fov) / 2) * ZNear;
return new Vector2(h * AspectRatio, h);
should do the trick.
I have implemented frustum culling and am checking the bounding box for its intersection with the frustum planes. I added the ability to pause frustum updates which lets me see if the frustum culling has been working correctly. When I turn around after I have paused it, nothing renders behind me and to the left and right side, they taper off as well just as you would expect. Beyond the clip distance (far plane), they still render and I am not sure whether it is a problem with my frustum updating or bounding box checking code or I am using the wrong matrix or what. As I put the distance in the projection matrix at 3000.0f, it still says that bounding boxes well past that are still in the frustum, which isn't the case.
Here is where I create my modelview matrix:
projectionMatrix = glm::perspective(newFOV, 4.0f / 3.0f, 0.1f, 3000.0f);
viewMatrix = glm::mat4(1.0);
viewMatrix = glm::scale(viewMatrix, glm::vec3(1.0, 1.0, -1.0));
viewMatrix = glm::rotate(viewMatrix, anglePitch, glm::vec3(1.0, 0.0, 0.0));
viewMatrix = glm::rotate(viewMatrix, angleYaw, glm::vec3(0.0, 1.0, 0.0));
viewMatrix = glm::translate(viewMatrix, glm::vec3(-x, -y, -z));
modelViewProjectiomMatrix = projectionMatrix * viewMatrix;
The reason I scale it by -1 in the Z direction is because the levels were designed to be rendered with DirectX so I reverse the Z direction.
Here is where I update my frustum:
void CFrustum::calculateFrustum()
{
glm::mat4 mat = camera.getModelViewProjectionMatrix();
// Calculate the LEFT side
m_Frustum[LEFT][A] = (mat[0][3]) + (mat[0][0]);
m_Frustum[LEFT][B] = (mat[1][3]) + (mat[1][0]);
m_Frustum[LEFT][C] = (mat[2][3]) + (mat[2][0]);
m_Frustum[LEFT][D] = (mat[3][3]) + (mat[3][0]);
// Calculate the RIGHT side
m_Frustum[RIGHT][A] = (mat[0][3]) - (mat[0][0]);
m_Frustum[RIGHT][B] = (mat[1][3]) - (mat[1][0]);
m_Frustum[RIGHT][C] = (mat[2][3]) - (mat[2][0]);
m_Frustum[RIGHT][D] = (mat[3][3]) - (mat[3][0]);
// Calculate the TOP side
m_Frustum[TOP][A] = (mat[0][3]) - (mat[0][1]);
m_Frustum[TOP][B] = (mat[1][3]) - (mat[1][1]);
m_Frustum[TOP][C] = (mat[2][3]) - (mat[2][1]);
m_Frustum[TOP][D] = (mat[3][3]) - (mat[3][1]);
// Calculate the BOTTOM side
m_Frustum[BOTTOM][A] = (mat[0][3]) + (mat[0][1]);
m_Frustum[BOTTOM][B] = (mat[1][3]) + (mat[1][1]);
m_Frustum[BOTTOM][C] = (mat[2][3]) + (mat[2][1]);
m_Frustum[BOTTOM][D] = (mat[3][3]) + (mat[3][1]);
// Calculate the FRONT side
m_Frustum[FRONT][A] = (mat[0][3]) + (mat[0][2]);
m_Frustum[FRONT][B] = (mat[1][3]) + (mat[1][2]);
m_Frustum[FRONT][C] = (mat[2][3]) + (mat[2][2]);
m_Frustum[FRONT][D] = (mat[3][3]) + (mat[3][2]);
// Calculate the BACK side
m_Frustum[BACK][A] = (mat[0][3]) - (mat[0][2]);
m_Frustum[BACK][B] = (mat[1][3]) - (mat[1][2]);
m_Frustum[BACK][C] = (mat[2][3]) - (mat[2][2]);
m_Frustum[BACK][D] = (mat[3][3]) - (mat[3][2]);
// Normalize all the sides
NormalizePlane(m_Frustum, LEFT);
NormalizePlane(m_Frustum, RIGHT);
NormalizePlane(m_Frustum, TOP);
NormalizePlane(m_Frustum, BOTTOM);
NormalizePlane(m_Frustum, FRONT);
NormalizePlane(m_Frustum, BACK);
}
And finally, where I check the bounding box:
bool CFrustum::BoxInFrustum( float x, float y, float z, float x2, float y2, float z2)
{
// Go through all of the corners of the box and check then again each plane
// in the frustum. If all of them are behind one of the planes, then it most
// like is not in the frustum.
for(int i = 0; i < 6; i++ )
{
if(m_Frustum[i][A] * x + m_Frustum[i][B] * y + m_Frustum[i][C] * z + m_Frustum[i][D] > 0) continue;
if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y + m_Frustum[i][C] * z + m_Frustum[i][D] > 0) continue;
if(m_Frustum[i][A] * x + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z + m_Frustum[i][D] > 0) continue;
if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z + m_Frustum[i][D] > 0) continue;
if(m_Frustum[i][A] * x + m_Frustum[i][B] * y + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0) continue;
if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0) continue;
if(m_Frustum[i][A] * x + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0) continue;
if(m_Frustum[i][A] * x2 + m_Frustum[i][B] * y2 + m_Frustum[i][C] * z2 + m_Frustum[i][D] > 0) continue;
// If we get here, it isn't in the frustum
return false;
}
// Return a true for the box being inside of the frustum
return true;
}
I've noticed a few things, particularly with how you set up the projection matrix. For starters, gluProject doesn't return a value, unless you're using some kind of wrapper or weird api. gluLookAt is used more often.
Next, assuming the scale, rotate, and translate functions are intended to change the modelview matrix, you need to reverse their order. OpenGL doesn't actually move objects around; instead it effectively moves the origin around, and renders each object using the new definition of <0,0,0>. Thus you 'move' to where you want it to render, then you rotate the axes as needed, then you stretch out the grid.
As for the clipping problem, you may want to give glClipPlane() a good look over. If everything else mostly works, but there seems to be some rounding error, try changing the near clipping plane in your perspective(,,,) function from 0.1 to 1.0 (smaller values tend to mess with the z-buffer).
I see a lot of unfamiliar syntax, so I think you're using some kind of wrapper; but here are some (Qt) code fragments from my own GL project that I use. Might help, dunno:
//This gets called during resize, as well as once during initialization
void GLWidget::resizeGL(int width, int height) {
int side = qMin(width, height);
padX = (width-side)/2.0;
padY = (height-side)/2.0;
glViewport(padX, padY, side, side);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPerspective(60.0, 1.0, 1.0, 2400.0);
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
}
//This fragment gets called at the top of every paint event:
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glPushMatrix();
glLightfv(GL_LIGHT0, GL_POSITION, FV0001);
camMain.stepVars();
gluLookAt(camMain.Pos[0],camMain.Pos[1],camMain.Pos[2],
camMain.Aim[0],camMain.Aim[1],camMain.Aim[2],
0.0,1.0,0.0);
glPolygonMode(GL_FRONT_AND_BACK, drawMode);
//And this fragment represents a typical draw event
void GLWidget::drawFleet(tFleet* tIn) {
if (tIn->firstShip != 0){
glPushMatrix();
glTranslatef(tIn->Pos[0], tIn->Pos[1], tIn->Pos[2]);
glRotatef(tIn->Yaw, 0.0, 1.0, 0.0);
glRotatef(tIn->Pitch,0,0,1);
drawShip(tIn->firstShip);
glPopMatrix();
}
}
I'm working on the assumption that you're newish to GL, so my apologies if I come off as a little pedantic.
I had the same problem.
Given Vinny Rose's answer, I checked the function that creates a normalized plane, and found an error.
This is the corrected version, with the incorrect calculation commented out:
plane plane_normalized(float A, float B, float C, float D) {
// Wrong, this is not a 4D vector
// float nf = 1.0f / sqrtf(A * A + B * B + C * C + D * D);
// Correct
float nf = 1.0f / sqrtf(A * A + B * B + C * C);
return (plane) {{
nf * A,
nf * B,
nf * C,
nf * D
}};
}
My guess is that your NormalizePlane function does something similar.
The point of normalizing is to have a plane in Hessian normal form so that we can do easy half-space tests. If you normalize the plane as you would a four-dimensional vector, the normal direction [A, B, C] is still correct but the offset D is not.
I think you'd get correct results when testing points against the top, bottom, left and right planes because they pass through the origin, and the near plane might be close enough to not notice. (Bounding sphere tests would fail.)
The frustum cull worked as expected for me when I restored the correct normalization.
Here's what I think is happening: The far plane is getting defined correctly but in my testing the D value of that plane is coming out much too small. So objects are getting accepted as being on the correct side of the far plane because the math is forcing the far plane to actually be much farther away than you want.
Try a different approach: (http://www.lighthouse3d.com/tutorials/view-frustum-culling/geometric-approach-extracting-the-planes/)
float tang = tanf(fov * PI / 360.0f);
float nh = near * tang; // near height
float nw = nh * aspect; // near width
float fh = far * tang; // far height
float fw = fh * aspect; // far width
glm::vec3 p,nc,fc,X,Y,Z,Xnw,Ynh;
//camera position
p = glm::vec3(viewMatrix[3][0],viewMatrix[3][1],viewMatrix[3][2]);
// the left vector
glm::vec3 X = glm::vec3(viewMatrix[0][0], viewMatrix[1][0], viewMatrix[2][0]);
// the up vector
glm::vec3 Y = glm::vec3(viewMatrix[0][1], viewMatrix[1][1], viewMatrix[2][1]);
// the look vector
glm::vec3 Z = glm::vec3(viewMatrix[0][2], viewMatrix[1][2], viewMatrix[2][2]);
nc = p - Z * near; // center of the near plane
fc = p - Z * far; // center of the far plane
// the distance to get to the left or right edge of the near plane from nc
Xnw = X * nw;
// the distance to get to top or bottom of the near plane from nc
Ynh = Y * nh;
// the distance to get to the left or right edge of the far plane from fc
Xfw = X * fw;
// the distance to get to top or bottom of the far plane from fc
Yfh = Y * fh;
ntl = nc + Ynh - Xnw; // "near top left"
ntr = nc + Ynh + Xnw; // "near top right" and so on
nbl = nc - Ynh - Xnw;
nbr = nc - Ynh + Xnw;
ftl = fc + Yfh - Xfw;
ftr = fc + Yfh + Xfw;
fbl = fc - Yfh - Xfw;
fbr = fc - Yfh + Xfw;
m_Frustum[TOP] = planeWithPoints(ntr,ntl,ftl);
m_Frustum[BOTTOM] = planeWithPoints(nbl,nbr,fbr);
m_Frustum[LEFT] = planeWithPoints(ntl,nbl,fbl);
m_Frustum[RIGHT] = planeWithPoints(nbr,ntr,fbr);
m_Frustum[FRONT] = planeWithPoints(ntl,ntr,nbr);
m_Frustum[BACK] = planeWithPoints(ftr,ftl,fbl);
// Normalize all the sides
NormalizePlane(m_Frustum, LEFT);
NormalizePlane(m_Frustum, RIGHT);
NormalizePlane(m_Frustum, TOP);
NormalizePlane(m_Frustum, BOTTOM);
NormalizePlane(m_Frustum, FRONT);
NormalizePlane(m_Frustum, BACK);
Then planeWithPoints would be something like:
planeWithPoints(glm::vec3 a, glm::vec3 b, glm::vec3 c){
double A = a.y * (b.z - c.z) + b.y * (c.z - a.z) + c.y * (a.z - b.z);
double B = a.z * (b.x - c.x) + b.z * (c.x - a.x) + c.z * (a.x - b.x);
double C = a.x * (b.y - c.y) + b.x * (c.y - a.y) + c.x * (a.y - b.y);
double D = -(a.x * (b.y * c.z - c.y * b.z) + b.x * (c.y * a.z - a.y * c.z) + c.x * (a.y * b.z - b.y * a.z));
return glm::vec4(A,B,C,D);
}
I didn't test any of the above. But the original reference is there if you need it.
Previous Answer:
OpenGL and GLSL matrices are stored and accessed in column-major order when the matrix is represented by a 2D array. This is also true with GLM as they follow the GLSL standards.
You need to change your frustum creation to the following.
// Calculate the LEFT side (column1 + column4)
m_Frustum[LEFT][A] = (mat[3][0]) + (mat[0][0]);
m_Frustum[LEFT][B] = (mat[3][1]) + (mat[0][1]);
m_Frustum[LEFT][C] = (mat[3][2]) + (mat[0][2]);
m_Frustum[LEFT][D] = (mat[3][3]) + (mat[0][3]);
// Calculate the RIGHT side (-column1 + column4)
m_Frustum[RIGHT][A] = (mat[3][0]) - (mat[0][0]);
m_Frustum[RIGHT][B] = (mat[3][1]) - (mat[0][1]);
m_Frustum[RIGHT][C] = (mat[3][2]) - (mat[0][2]);
m_Frustum[RIGHT][D] = (mat[3][3]) - (mat[0][3]);
// Calculate the TOP side (-column2 + column4)
m_Frustum[TOP][A] = (mat[3][0]) - (mat[1][0]);
m_Frustum[TOP][B] = (mat[3][1]) - (mat[1][1]);
m_Frustum[TOP][C] = (mat[3][2]) - (mat[1][2]);
m_Frustum[TOP][D] = (mat[3][3]) - (mat[1][3]);
// Calculate the BOTTOM side (column2 + column4)
m_Frustum[BOTTOM][A] = (mat[3][0]) + (mat[1][0]);
m_Frustum[BOTTOM][B] = (mat[3][1]) + (mat[1][1]);
m_Frustum[BOTTOM][C] = (mat[3][2]) + (mat[1][2]);
m_Frustum[BOTTOM][D] = (mat[3][3]) + (mat[1][3]);
// Calculate the FRONT side (column3 + column4)
m_Frustum[FRONT][A] = (mat[3][0]) + (mat[2][0]);
m_Frustum[FRONT][B] = (mat[3][1]) + (mat[2][1]);
m_Frustum[FRONT][C] = (mat[3][2]) + (mat[2][2]);
m_Frustum[FRONT][D] = (mat[3][3]) + (mat[2][3]);
// Calculate the BACK side (-column3 + column4)
m_Frustum[BACK][A] = (mat[3][0]) - (mat[2][0]);
m_Frustum[BACK][B] = (mat[3][1]) - (mat[2][1]);
m_Frustum[BACK][C] = (mat[3][2]) - (mat[2][2]);
m_Frustum[BACK][D] = (mat[3][3]) - (mat[2][3]);