OpenGL flight simulator styled camera rotations not working - opengl

Question

I am trying to create a camera thats works like a flight-simulator (because I'm making a flight simulator) camera - I want to be able to perform pitch, yaw and roll, as well as translations. The translations work perfectly, but the rotations are causing me a very big headache.
The rotations are calculated using quaternions, using GLM, like so:
// Fuck quaternions
glm::fquat pitchQuat(cos(TO_RADIANS(pitch / 2.0f)), m_rightVector * (float)sin(TO_RADIANS(pitch / 2.0f)));
glm::fquat yawQuat(cos(TO_RADIANS(yaw / 2.0f)), m_upVector * (float)sin(TO_RADIANS(yaw / 2.0f)));
glm::fquat rollQuat(cos(TO_RADIANS(roll / 2.0f)), m_direction * (float)sin(TO_RADIANS(roll / 2.0f)));
m_rotation = yawQuat * pitchQuat * rollQuat;
m_direction = m_rotation * m_direction * glm::conjugate(m_rotation);
m_upVector = m_rotation * m_upVector * glm::conjugate(m_rotation);
m_rightVector = glm::cross(m_direction, m_upVector);
If I calculate pitch or roll or yaw alone, everything works fine, however, as soon as I introduce another rotation everything goes wrong. This video should be enough to show you what is going wrong:
https://www.youtube.com/watch?v=jlklem6t68I&feature=youtu.be
The translations work fine, the rotations not such much. When I move the mouse in a circular motion - which is yaw and pitch rotations - the house slowly begins the flip upside-down. You may have noticed in the video that rotating causes the house to stretch, which is also unwanted.
I cannot figure out what is wrong. Can anyone explain how I can create a camera with pitch, yaw and roll working?
If it is of any help the view matrix is calculated using:
m_viewMatrix = glm::lookAt(m_position, m_position + m_direction, m_upVector);
And the projection matrix is calculated using:
float t = tan(fov * 3.14159 / 360.0) * nPlane;
float r = aspectRatio * t;
float l = aspectRatio * -t;
m_projectionMatrix[0][0] = (2 * nPlane) / (r - l);
m_projectionMatrix[0][2] = (r + l) / (r - l);
m_projectionMatrix[1][1] = (2 * nPlane) / (t + t);
m_projectionMatrix[1][2] = (t - t) / (t + t);
m_projectionMatrix[2][2] = (nPlane + fPlane) / (nPlane - fPlane);
m_projectionMatrix[2][3] = (2 * fPlane * nPlane) / (nPlane - fPlane);
m_projectionMatrix[3][2] = -1;
If you would like to see all my code for the camera class you can find it in my Google Drive:
http://goo.gl/FFMPa0

Answer 1

What you're experiencing is normal for a quaternion camera. If you want to avoid the problem, simply use a fixed up-vector when yawing, but beware, that there's going to be an exception when you look straight up. You might want to handle that explicitly. And always reconstruct your view matrix, by crossing view with this static up.
EDIT:
Here's a step-by-step:
First, calculate your rotation around the static up-vector, while your direction is extracted from the view as usual (and right being the cross):
// Quaternions
glm::fquat pitchQuat(cos(TO_RADIANS(pitch / 2.0f)), cross(m_direction,vec3(0,1,0) * (float)sin(TO_RADIANS(pitch / 2.0f)));
glm::fquat yawQuat(cos(TO_RADIANS(yaw / 2.0f)), vec3(0,1,0) * (float)sin(TO_RADIANS(yaw / 2.0f)));
glm::fquat rollQuat(cos(TO_RADIANS(roll / 2.0f)), m_direction * (float)sin(TO_RADIANS(roll / 2.0f)));
m_rotation = yawQuat * pitchQuat * rollQuat;
Then reconstruct your view by using the quaternion, as shown in here (lm::lookAt will do too).
And, of course - repeat step every frame.