Rotate object from one coordinate system to another - opengl

Question

Using the glm library for the calculations. I have a mesh located in a local coordinate system where the axis are:
meshUp = glm::vec3(0, 1, 0);
meshForward = glm::vec3(0, 0, -1);
meshRight = glm::vec3(1, 0, 0);
and I need a matrix that will rotate its vertices to any other new coordinate system with 3 given axis, fx:
newUp = glm::vec3(-0.85, 0.51, -0.08);
newForward = glm::vec3(0.45, 0.65, -0.61);
newRight = glm::vec3(-0.26, -0.56, -0.79);
No translation needed, the coordinate systems share origin. I can get halfway there, like this:
glm::vec3 rotationVecForUpAxis = glm::normalize(glm::cross(meshUp, newUp));
float rotationRadiansForUpAxis = acos(glm::dot(meshUp, newUp));
glm::mat4 rotationMatrix = glm::rotate(glm::mat4(),
rotationRadiansForUpAxis,
rotationVecForUpAxis);
This works and rotates the mesh, so it's up axis aligns with the newUp axis. However, the mesh still needs to be rotated around the newUp axis, before the mesh's meshForward axis aligns with the newForward axis.
Does anyone know how to do this?

Answer 1

You could look up the math. But just for the fun of it, I'm going to derive it here.
Let's say that the new basis vectors, expressed in the original coordinate system, are:
[ xnx ] [ ynx ] [ znx ]
xn = [ xny ] yn = [ yny ] zn = [ zny ]
[ xnz ] [ ynz ] [ znz ]
You're looking for the matrix M that maps these vectors to the basis vectors in the new coordinate system:
[ xnx ] = [ 1 ] [ ynx ] = [ 0 ] [ znx ] = [ 0 ]
M * [ xny ] = [ 0 ] M * [ yny ] = [ 1 ] M * [ zny ] = [ 0 ]
[ xnz ] = [ 0 ] [ ynz ] = [ 0 ] [ znz ] = [ 1 ]
Writing this in matrix form gives:
[ xnx ynx znx ] = [ 1 0 0 ]
M * [ xny yny zny ] = [ 0 1 0 ]
[ xnz ynz znz ] = [ 0 0 1 ]
Which in turn gives for M:
[ xnx ynx znx ]
M = inverse( [ xny yny zny ] )
[ xnz ynz znz ]
In words, the matrix is the inverse of the matrix that has the new basis vectors as its columns.
For a rotation, this becomes particularly easy. The inverse of a rotation matrix is the transpose of the matrix. So M is the matrix with the new basis vectors as its rows:
[ xnx xny xnz ]
M = [ ynx yny ynz ]
[ znx zny znz ]
With this, all you need to get your rotation matrix is build a matrix that has the values of the new basis vectors as its rows. For example, if you use newRight as the x-axis, newUp as the y-axis, and newForward as the z-axis, the transformation matrix is:
[ newRight.x newRight.y newRight.z ]
[ newUp.x newUp.y newUp.z ]
[ newForward.x newForward.y newForward.z ]
When building the matrix, keep in mind that matrices for OpenGL are commonly stored in column major order.