What is the expected rotation direction when using glm's rotate function? - c++

Question

I'm writing a static function that uses GLM's rotate() function to rotate a vector about an arbitrary axis.
I wrote a simple test to check my work, and I found that the rotations occur in the opposite direction than what I expected.
I rotate a unit vector (0,0,1) about the X axis (1,0,0) in steps of pi/4. I expected that since OpenGL (and GLM?) use a right-handed coordinate system, the rotations would occur in a counter-clockwise direction about the X axis. Instead, they're occurring in clockwise direction.
vec3& RotateVector(vec3& targetVector, float const& radians, vec3 const &axis)
{
mat4 rotation = glm::rotate(mat4(1.0f), radians, axis);
targetVector = (vec4(targetVector, 0.0f) * rotation).xyz();
return targetVector;
}
vec3 test(0, 0, 1);
cout << "test initial vals: " << test.x << " " << test.y << " " << test.z << "\n";
RotateVector(test, 3.14f / 4.0f, vec3(1, 0, 0) );
cout << "Rotated test: " << test.x << " " << test.y << " " << test.z << "\n";
RotateVector(test, 3.14 /4.0f, vec3(1, 0, 0));
cout << "Rotated test: " << test.x << " " << test.y << " " << test.z << "\n";
RotateVector(test, 3.14 / 4.0f, vec3(1, 0, 0));
cout << "Rotated test: " << test.x << " " << test.y << " " << test.z << "\n";
RotateVector(test, 3.14 / 4.0f, vec3(1, 0, 0));
cout << "Rotated test: " << test.x << " " << test.y << " " << test.z << "\n";
When I run the code above, I get the following output:
test initial vals: 0 0 1
Rotated test: 0 0.706825 0.707388
Rotated test: 0 1 0.000796229
Rotated test: 0 0.707951 -0.706262
Rotated test: 0 0.00159246 -0.999999
The output shows that the rotation is moving clockwise about the X axis.
Why is that? I'd expect that OpenGL's right handed coordinate system would adhere to the right hand rule? Am I missing something, or am I just confused?

Answer 1

You're using the matrices transposed, and since rotation matrices are orthogonal matrices, this has the effect of using the inverse of those matrices: R^-1 = R^T.
glm mimics classic GL conventions with mat4 * vec4 multiplication order, where vec4 is a column vector.
When you write vec4 * mat4, vec4 is interpreted as row vector, and since (A*B)^T = B^T * A^T, you get the same result as transpose(mat4) * vec4.