Camera rotation and orientation in variable gravity - c++

Question

I'm attempting to implement a camera controller for a first person, mouse-look based camera for OpenGL. This is a simple problem when the camera is always oriented normally (camera up vector = world Y axis). However, I'm having real trouble getting everything working properly with a camera that can be used seamlessly for any orientation. The purpose is to allow a player to move around an entire planet. An additional requirement is that the direction remain the same relative to the orientation as the camera's orientation changes. An example would be, if you're walking around a planet, the direction remains the same relative to the ground, so as you go "down" along the side from a pole, the direction is also automatically rotated.
So far, I've attempted a number of different things to get this working, but as I see it, there should be two different ways of doing this. The first is to do regular camera rotation based on yaw and pitch angles from the world axes, and then transform the resulting look direction by the camera orientation to obtain the final look direction. The second approach is to rotate the camera with yaw and pitch angles based on calculated up and right vectors. The up vector is easy here; it's just the orientation. I haven't gotten any right vector I've found to work correctly though.
OK, here's the code for these two approaches.
Common code
// m_orientation calculated from planet center to current position
m_horizontal += horizontal;
m_vertical += vertical;
while (m_horizontal > TWO_PI) {
m_horizontal -= TWO_PI;
}
while (m_horizontal < -TWO_PI) {
m_horizontal += TWO_PI;
}
if (m_vertical > MAX_VERTICAL) {
m_vertical = MAX_VERTICAL;
}
else if (m_vertical < -MAX_VERTICAL) {
m_vertical = -MAX_VERTICAL;
}
// code from either implementation
m_view = glm::lookAt(m_position, m_position + m_direction, m_orientation);
First approach with yaw, pitch about world axes and then transform
// check for m_orientation != WORLD_UP...
glm::vec3 axis = glm::normalize(glm::cross(WORLD_UP, m_orientation));
float angle_degrees = acosf(m_orientation.y) * RADS_TO_DEGREES;
glm::mat4 trans = glm::rotate(glm::mat4(), angle_degrees, axis);
// can also be determined with two rotation matrices about world axes, end result is identical
m_direction = glm::vec3(cosf(m_vertical) * sinf(m_horizontal),
sinf(m_vertical),
cosf(m_vertical) * cosf(m_horizontal));
m_direction = glm::vec3(trans * glm::vec4(m_direction));
Second approach with yaw and pitch about appropriate up and right vectors
m_right = ??? // tried literally everything
glm::mat4 yaw = glm::rotate(glm::mat4(), m_horizontal, m_orientation);
glm::mat4 pitch = glm::rotate(glm::mat4(), m_vertical, m_right);
glm::mat4 trans = yaw * pitch;
m_direction = glm::vec3(trans[2]); // z axis
OK, so here's the problem. The first approach works almost perfectly, but near the south pole of a planet (within ~15 degrees of orientation=(0,-1,0), effect gets stronger closer you are), the camera is automatically rotated toward the south pole as the orientation changes. So if the camera orientation does not change, near the south pole, the camera works perfectly. Any change in orientation results in the camera rotating toward the south pole. The more orientation change, the more the camera rotates. Now I have tried removing either the pitch or yaw from the world axis camera rotation, and this effect appears only with the pitch calculation included. With only yaw, then the camera behaves perfectly (lacking any pitch control ofc). As far as I can tell, my transformation to go from regular up=(0,1,0) to the current orientation is incorrect. Any help on that?
Now the other way to do things appears to work somewhat correctly, but I simply have not found a good right vector. Everything I've tried results in strange behavior of both horizontal and vertical movements. The most obvious solution, cross product of previous frame's direction and current orientation to produce the right vector doesn't work. Any suggestions for a good right vector?
I'm also happy to see completely different solutions to this problem. I know it's possible, but no amount of searching has given me a good solution. Thanks very much in advance.
Edit 1: Tried a few more things in response to Paweł Stawarz
Results in incorrect orientation of camera and weird mouse movement. I made sure my matrix multiplication was in the correct order. I also tried the transpose.
m_view = glm::lookAt(glm::vec3(0.0f, 0.0f, 0.0f), m_direction, m_up);
m_view = trans * m_view; //trans is rotation from orientation=(0,1,0) to orientation=m_orientation
Results in the same problem as previously, with the camera rotating toward the south pole by itself. Also the vertical mouse rotation is not correct, causes camera to go in circles.
m_view = glm::lookAt(glm::vec3(0.0f, 0.0f, 0.0f), m_direction, m_up);
m_view = trans * m_view;
m_direction = glm::vec3(m_view[2]);
m_view = glm::lookAt(m_position, m_direction + m_position, m_orientation);
Edit 2: Using the RIGHT vector method, with no transformation between orientations is working a little better. However, it causes the camera yaw to oscillate wildly with pitch near to vertical (at least 5 degrees away from vertical). In addition, the range of motion for pitch is not adjusted by the orientation, so for example on the side of the planet, vertical motion is restricted to directly in front of you to behind you (~(0,1,0) to ~(0,-1,0)).
glm::mat4 yaw = glm::rotate(glm::mat4(), m_horizontal * ONEEIGHTY_PI, m_orientation);
glm::mat4 pitch = glm::rotate(glm::mat4(), m_vertical * -ONEEIGHTY_PI, m_right);
glm::mat4 cam = pitch * yaw;
m_right = glm::vec3(cam[0]);
m_up = glm::vec3(cam[1]);
m_direction = glm::vec3(cam[2]);
m_view = glm::lookAt(m_position, m_direction + m_position, m_up);
m_vp = m_perspective * m_view;

Answer 1

Solved it. Needed a different transformation. See here for a pretty good explanation.
glm::mat4 trans;
float factor = 1.0f;
float real_vertical = vertical;
m_horizontal += horizontal;
m_vertical += vertical;
while (m_horizontal > TWO_PI) {
m_horizontal -= TWO_PI;
}
while (m_horizontal < -TWO_PI) {
m_horizontal += TWO_PI;
}
if (m_vertical > MAX_VERTICAL) {
m_vertical = MAX_VERTICAL;
}
else if (m_vertical < -MAX_VERTICAL) {
m_vertical = -MAX_VERTICAL;
}
glm::quat world_axes_rotation = glm::angleAxis(m_horizontal * ONEEIGHTY_PI, glm::vec3(0.0f, 1.0f, 0.0f));
world_axes_rotation = glm::normalize(world_axes_rotation);
world_axes_rotation = glm::rotate(world_axes_rotation, m_vertical * ONEEIGHTY_PI, glm::vec3(1.0f, 0.0f, 0.0f));
m_pole = glm::normalize(m_pole - glm::dot(m_orientation, m_pole) * m_orientation);
glm::mat4 local_transform;
local_transform[0] = glm::vec4(m_pole.x, m_pole.y, m_pole.z, 0.0f);
local_transform[1] = glm::vec4(m_orientation.x, m_orientation.y, m_orientation.z, 0.0f);
glm::vec3 tmp = glm::cross(m_pole, m_orientation);
local_transform[2] = glm::vec4(tmp.x, tmp.y, tmp.z, 0.0f);
local_transform[3] = glm::vec4(m_position.x, m_position.y, m_position.z, 1.0f);
world_axes_rotation = glm::normalize(world_axes_rotation);
m_view = local_transform * glm::mat4_cast(world_axes_rotation);
m_direction = -1.0f * glm::vec3(m_view[2]);
m_up = glm::vec3(m_view[1]);
m_right = glm::vec3(m_view[0]);
m_view = glm::inverse(m_view);

Answer 2

If we keep things simple, by using the standard approach:
Move the camera to the current player position
Rotate it towards where the player is looking at
The camera rotation is described by:
The UP vector which is the normalized vector that starts at (p0x,p0y,p0z) (where p0 is the position of the center of planet) and goes thru (p1x,p1y,p1z) (where p1 describes the place the player is currently at),
the RIGHT vector is the vector perpendicular to the UP vector and perpendicular to the direction the player is looking - the LOOK vector (in the case where he's looking straight ahead - perpendicular to his direction).
Since the UP vector can be calulated straight from the player current position, you have to get the LOOK and RIGHT vectors. Both are cross products of corresponding other vectors.
Note also, that allowing the player to look up/down and pan his head, can (and probably will) in fact change the UP vector.