3d camera has unintended roll - c++

Question

I've been working on a 3d camera in opengl using C++.
When I look around with the camera, sometimes there will be unexpected roll in the camera, especially when I am rotating the camera in circles.
I suspect this is a floating point error, but I don't know how to detect it.
Here is the camera class:
#ifndef CAMERA_H
#define CAMERA_H
#include <GL/glew.h>
#include <GLFW/glfw3.h>
#include <glm/glm.hpp>
#include <glm/gtc/matrix_transform.hpp>
#include <glm/gtx/transform.hpp>
#include <glm/gtc/quaternion.hpp>
#include <glm/gtx/quaternion.hpp>
#include <glm/gtx/rotate_vector.hpp>
#include <glm/gtx/euler_angles.hpp>
#include <glm/gtx/string_cast.hpp>
#include <iostream>
using glm::vec3;
using glm::mat4;
using glm::quat;
enum CamDirection {
CAM_FORWARD,
CAM_BACKWARD,
CAM_LEFT,
CAM_RIGHT
};
class Camera {
public:
void cameraUpdate();
mat4 getViewMatrix();
Camera();
Camera(vec3 startPosition);
void move(CamDirection dir, GLfloat deltaTime);
void look(double xOffset, double yOffset);
void update();
private:
mat4 viewMatrix;
const GLfloat camSpeed = 5.05f;
};
mat4 Camera::getViewMatrix() {
return viewMatrix;
}
Camera::Camera(){}
Camera::Camera(vec3 startPos):
viewMatrix(glm::lookAt(startPos, vec3(0.0f, 0.0f, 0.0f), vec3(0.0f, 1.0f, 0.0f)))
{}
void Camera::move(CamDirection dir, GLfloat deltaTime) {
mat4 trans;
const vec3 camForward = vec3(viewMatrix[0][2], viewMatrix[1][2], viewMatrix[2][2]);
const vec3 camRight = vec3(viewMatrix[0][0], viewMatrix[1][0], viewMatrix[2][0]);
if (dir == CAM_FORWARD)
trans = glm::translate(trans, (camSpeed * deltaTime) * camForward);
else if (dir == CAM_BACKWARD)
trans = glm::translate(trans, -1 * (camSpeed * deltaTime) * camForward);
else if (dir == CAM_RIGHT)
trans = glm::translate(trans, -1 * (camSpeed * deltaTime) * camRight);
else
trans = glm::translate(trans, (camSpeed * deltaTime) * camRight);
viewMatrix *= trans;
}
void Camera::look(double xOffset, double yOffset) {
// 2 * acos(q[3])
quat rotation = glm::angleAxis((GLfloat)xOffset, vec3( 0.0f, 1.0f, 0.0f));
viewMatrix = glm::mat4_cast(rotation) * viewMatrix;
rotation = glm::angleAxis((GLfloat)yOffset, vec3(-1.0f, 0.0f, 0.0f));
mat4 rotMatrix = glm::mat4_cast(rotation);
viewMatrix = rotMatrix * viewMatrix;
}
void Camera::update() {
}
#endif // CAMERA_H

Answer 1

I managed to figure it out. Although I had to completely rewrite it to do it.
My problem was on these lines:
quat rotation = glm::angleAxis((GLfloat)xOffset, vec3( 0.0f, 1.0f, 0.0f));
viewMatrix = glm::mat4_cast(rotation) * viewMatrix;
rotation = glm::angleAxis((GLfloat)yOffset, vec3(-1.0f, 0.0f, 0.0f));
mat4 rotMatrix = glm::mat4_cast(rotation);
Building an intermediate quaternion to store orientation worked instead, and I could replace the look method with this:
quat pitch = quat(vec3(-yOffset, 0.0f, 0.0f));
quat yaw = quat(vec3(0.f, xOffset, 0.f));
orientation = pitch * orientation * yaw;
By multiplying the orientation the way on the last line, no unintended roll can happen.

Answer 2

There are two problems in that code:
First, if xOffset, yOffset are just screen pixel differences (obtained by mouse positions), you MUST set a factor that translates them to angles. There are better ways, for example form two vectors from center of window to mouse positions (previous and current) and calculate angle between them, by dot product. Depending on glm sets (degrees is default, but you can set radians) a non-factorized xOffset may be a huge angle, not smooth rotation.
Second accumlating rotations by newViewMatrix = thisMouseRotation * oldViewMatrix degenerates the matrix after some movements. This is due to limited numbers representation of computers: e.g. 10/3=3.333 but 3.333*3=9.999 != 10
Solutions:
A) Store the rotation in a quaternions. Initialize a quaternion and
update it for every rotation newQuater = thisMoveQuater * oldQuater. Time to time "normalize" the quaternion so as to minimize
numbers issue. The viewMatrix is calculated by viewMatrix = Mat4x4FromQuaternion * translationMatrix so we avoid the previous
viewMatrix and its issues.
B) Accumulate angles of rotation around each X,Y,Z axis. Calculate
each time it's needed the rotation matrix using these accumulated
angles. Perhaps you clamp an angle value to something like 0.2
degrees. This way the user can achieve the same position as several
rotations before.