Normal Rotation in GLSL - opengl

Question

I have written a basic program that loads a model and renders it to the screen. I'm using GLSL to transform the model appropriately, but the normals always seem to be incorrect after rotating them with every combination of model matrix, view matrix, inverse, transpose, etc that I could think of. The model matrix is just a rotation around the y-axis using glm:
angle += deltaTime;
modelMat = glm::rotate(glm::mat4(), angle, glm::vec3(0.f, 1.f, 0.f));
My current vertex shader code (I've modified the normal line many many times):
#version 150 core
uniform mat4 projMat;
uniform mat4 viewMat;
uniform mat4 modelMat;
in vec3 inPosition;
in vec3 inNormal;
out vec3 passColor;
void main()
{
gl_Position = projMat * viewMat * modelMat * vec4(inPosition, 1.0);
vec3 normal = normalize(mat3(inverse(modelMat)) * inNormal);
passColor = normal;
}
And my fragment shader:
#version 150 core
in vec3 passColor;
out vec4 outColor;
void main()
{
outColor = vec4(passColor, 1.0);
}
I know for sure that the uniform variables are being passed to the shader properly, as the model itself gets transformed properly, and the initial normals are correct if I do calculations such as directional lighting.
I've created a GIF of the rotating model, sorry about the low quality:
http://i.imgur.com/LgLKHCb.gif?1
What confuses me the most is how the normals appear to rotate on multiple axis, which I don't think should happen when multiplied by a simple rotation matrix on one axis.
Edit:
I've added some more of the client code below.
This is where the buffers get bound for the model, in the Mesh class (vao is GLuint, defined in the class):
GLuint vbo[3];
glGenVertexArrays(1, &vao);
glBindVertexArray(vao);
glGenBuffers(normals? (uvcoords? 3 : 2) : (uvcoords? 2 : 1), vbo);
glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
glBufferData(GL_ARRAY_BUFFER, vcount * 3 * sizeof(GLfloat), vertices, GL_STATIC_DRAW);
glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, 0, 0);
glEnableVertexAttribArray(0);
if(normals)
{
glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
glBufferData(GL_ARRAY_BUFFER, vcount * 3 * sizeof(GLfloat), normals, GL_STATIC_DRAW);
glVertexAttribPointer(1, 3, GL_FLOAT, GL_TRUE, 0, 0);
glEnableVertexAttribArray(1);
}
if(uvcoords)
{
glBindBuffer(GL_ARRAY_BUFFER, vbo[2]);
glBufferData(GL_ARRAY_BUFFER, vcount * 2 * sizeof(GLfloat), uvcoords, GL_STATIC_DRAW);
glVertexAttribPointer(2, 2, GL_FLOAT, GL_FALSE, 0, 0);
glEnableVertexAttribArray(2);
}
glBindVertexArray(0);
glGenBuffers(1, &ib);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, ib);
glBufferData(GL_ELEMENT_ARRAY_BUFFER, icount * sizeof(GLushort), indices, GL_STATIC_DRAW);
This is where the shaders are compiled after being loaded into memory with a simple readf(), in the Material class:
u32 vertexShader = glCreateShader(GL_VERTEX_SHADER);
u32 fragmentShader = glCreateShader(GL_FRAGMENT_SHADER);
glShaderSource(vertexShader, 1, (const GLchar**)&vsContent, 0);
glCompileShader(vertexShader);
if(!validateShader(vertexShader)) return false;
glShaderSource(fragmentShader, 1, (const GLchar**)&fsContent, 0);
glCompileShader(fragmentShader);
if(!validateShader(fragmentShader)) return false;
programHandle = glCreateProgram();
glAttachShader(programHandle, vertexShader);
glAttachShader(programHandle, fragmentShader);
glBindAttribLocation(programHandle, 0, "inPosition");
glBindAttribLocation(programHandle, 1, "inNormal");
//glBindAttribLocation(programHandle, 2, "inUVCoords");
glLinkProgram(programHandle);
if(!validateProgram()) return false;
And the validateShader(GLuint) and validateProgram() functions:
bool Material::validateShader(GLuint shaderHandle)
{
char buffer[2048];
memset(buffer, 0, 2048);
GLsizei len = 0;
glGetShaderInfoLog(shaderHandle, 2048, &len, buffer);
if(len > 0)
{
Logger::log("ve::Material::validateShader: Failed to compile shader - %s", buffer);
return false;
}
return true;
}
bool Material::validateProgram()
{
char buffer[2048];
memset(buffer, 0, 2048);
GLsizei len = 0;
glGetProgramInfoLog(programHandle, 2048, &len, buffer);
if(len > 0)
{
Logger::log("ve::Material::validateProgram: Failed to link program - %s", buffer);
return false;
}
glValidateProgram(programHandle);
GLint status;
glGetProgramiv(programHandle, GL_VALIDATE_STATUS, &status);
if(status == GL_FALSE)
{
Logger::log("ve::Material::validateProgram: Failed to validate program");
return false;
}
return true;
}
Each Material instance has a std::map of Meshs, and get rendered as so:
void Material::render()
{
if(loaded)
{
glUseProgram(programHandle);
for(auto it = mmd->uniforms.begin(); it != mmd->uniforms.end(); ++it)
{
GLint loc = glGetUniformLocation(programHandle, (const GLchar*)it->first);
switch(it->second.type)
{
case E_UT_FLOAT3: glUniform3fv(loc, 1, it->second.f32ptr); break;
case E_UT_MAT4: glUniformMatrix4fv(loc, 1, GL_FALSE, it->second.f32ptr); break;
default: break;
}
}
for(Mesh* m : mmd->objects)
{
GLint loc = glGetUniformLocation(programHandle, "modelMat");
glUniformMatrix4fv(loc, 1, GL_FALSE, &m->getTransform()->getTransformMatrix()[0][0]);
m->render();
}
}
}
it->second.f32ptr would be a float pointer to &some_vec3[0] or &some_mat4[0][0].
I manually upload the model's transformation matrix before rendering, however (which is only a rotation matrix, the Transform class (returned by Mesh::getTransform()) will only do a glm::rotation() since I was trying to figure out the problem).
Lastly, the Mesh render code:
if(loaded)
{
glBindVertexArray(vao);
glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, ib);
glDrawElements(GL_TRIANGLES, indexCount, GL_UNSIGNED_SHORT, 0);
}
I think this is all the necessary code, but I can post more if needed.

Answer 1

Your nomal matrix calculation is just wrong. The correct normal matrix would be the transpose of the inverse of the upper-left 3x3 submatrix of the model or modelview matrix (depending on which space you want to do your lighting calculations).
What you do is just inverting the full 4x4 matrix and taking the upper-left 3x3 submatrix of that, which is just totally wrong.
You should calculate transpose(inverse(mat3(modelMat))), but you really shouldn't do this in the shader, but calulate this toghether with the model matrix on the CPU to avoid letting the GPU calculate a quite expensive matrix inversion per vertex.

Answer 2

As long as your transformations consist of only rotations, translations, and uniform scaling, you can simply apply the rotation part of your transformations to the normals.
In general, it's the transposed inverse matrix that needs to be applied to the normals, using only the regular 3x3 linear transformation matrix, without the translation part that extends the matrix to 4x4.
For rotations and uniform scaling, the inverse-transpose is identical to the original matrix. So the matrix operations to invert and transpose matrices are only needed if you apply other types of transformations, like non-uniform scaling, or shear transforms.

Answer 3

Apparently, if the vertex normals of a mesh are incorrect, then strange rotation artifacts will occur. In my case, I had transformed the mesh in my 3D modelling program (Blender) by 90 degrees on the X axis, as Blender uses the z-axis as its vertical axis, whereas my program uses the y-axis as the vertical axis. However, the method I used to transform/rotate the mesh in Blender in my export script did not properly transform the normals, but only the positions of the vertices. Without any prior transformations, the program works as expected. I initially found out that the normals were incorrect by comparing the normalized positions and normals in a symmetrical object (I used a cube with smoothed normals), and saw that the normals were rotated. Thank you to #derhass and #Solkar for guiding me to the answer.
However, if anyone still wants to contribute, I would like to know why the normals don't rotate in one axis when multiplied by a single axis rotation matrix, even if they are incorrect.