So I've been having trouble with a camera I've implemented in OpenGL and C++ using the GLM library. The type of camera I'm aiming for is a fly around camera which will allow easy exploration of a 3D world. I have managed to get the camera pretty much working, it's nice and smooth, looks around and the movement seems to be nice and correct.
The only problem I seem to have is that the rotation along the camera's X and Y axis (looking up and down) introduces some rotation about it's Z axis. This has the result of causing the world to slightly roll whilst travelling about.
As an example... if I have a square quad in front of the camera and move the camera in a circular motion, so as if looking around in a circle with your head, once the motion is complete the quad will have rolled slightly as if you've tilted your head.
My camera is currently a component which I can attach to an object/entity in my scene. Each entity has a "Frame" which is basically the model matrix for that entity. The Frame contains the following attributes:
glm::mat4 m_Matrix;
glm::vec3 m_Position;
glm::vec3 m_Up;
glm::vec3 m_Forward;
These are then used by the camera to create the appropriate viewMatrix like this:
const glm::mat4& CameraComponent::GetViewMatrix()
{
//Get the transform of the object
const Frame& transform = GetOwnerGO()->GetTransform();
//Update the viewMatrix
m_ViewMatrix = glm::lookAt(transform.GetPosition(), //position of camera
transform.GetPosition() + transform.GetForward(), //position to look at
transform.GetUp()); //up vector
//return reference to the view matrix
return m_ViewMatrix;
}
And now... here are my rotate X and Y methods within the Frame object, which I'm guessing is the place of the problem:
void Frame::RotateX( float delta )
{
glm::vec3 cross = glm::normalize(glm::cross(m_Up, m_Forward)); //calculate x axis
glm::mat4 Rotation = glm::rotate(glm::mat4(1.0f), delta, cross);
m_Forward = glm::normalize(glm::vec3(Rotation * glm::vec4(m_Forward, 0.0f))); //Rotate forward vector by new rotation
m_Up = glm::normalize(glm::vec3(Rotation * glm::vec4(m_Up, 0.0f))); //Rotate up vector by new rotation
}
void Frame::RotateY( float delta )
{
glm::mat4 Rotation = glm::rotate(glm::mat4(1.0f), delta, m_Up);
//Rotate forward vector by new rotation
m_Forward = glm::normalize(glm::vec3(Rotation * glm::vec4(m_Forward, 0.0f)));
}
So somewhere in there, there's a problem which I've been searching around trying to fix. I've been messing with it for a few days now, trying random things but I either get the same result, or the z axis rotation is fixed but other bugs appear such as incorrect X, Y rotation and camera movement.
I had a look at gimbal lock but from what I understood of it, this problem didn't seem quite like gimbal lock to me. But I may be wrong.
Store the current pitch/yaw angles and generate the camera matrix on-the-fly instead of trying to accumulate small changes on the intermediate vectors.
In your RotateY function, change it from this:
glm::mat4 Rotation = glm::rotate(glm::mat4(1.0f), delta, m_Up);
to this:
glm::mat4 Rotation = glm::rotate(glm::mat4(1.0f), delta, glm::vec3(0,1,0));
Related
I'm trying to implement a camera that follows a moving object. I've implemented these functions:
void Camera::espheric_yaw(float degrees, glm::vec3 center_point)
{
float lim_yaw = glm::radians(89.0f);
float radians = glm::radians(degrees);
absoluteYaw += radians;
... clamp absoluteYaw
float radius = 10.0f;
float camX = cos(absoluteYaw) * cos(absoluteRoll) * radius;
float camY = sin(absoluteRoll)* radius;
float camZ = sin(absoluteYaw) * cos(absoluteRoll) * radius;
eyes.x = camX;
eyes.y = camY;
eyes.z = camZ;
lookAt = center_point;
view = glm::normalize(lookAt - eyes);
up = glm::vec3(0, 1, 0);
right = glm::normalize(glm::cross(view, up));
}
I want to use this function (and the pitch version) for a camera that follows a moving 3d model. Right now, it works when the center_point is the (0,1,0). I think i'm getting the position right but the up vector is clearly not always (0,1,0).
How can I get my up, view and right vector for the camera? And then, if I update the eyes position of the camera this way, how will my camera move when the other object (centered at center_position parameter) moves?
The idea is to update this each time I have mouse input with centered_value = center of the moving object. Then use gluLookAt with view, eyes and up values of my camera (and lookAt which will be eyes+view).
Following a moving object is matter of pointing the camera to that object. This is what typical lookAt function does. See the maths here and then use glm::lookAt().
The 'Arcball' technic is for rotating with the mouse. See some maths here.
The idea is to get two vectors (first, second) from positions on screen. For each vector, X,Y are taking depending on pixels "travelled" by mouse and the size of the window. Z is calculated by 'trackball' maths. With these two vectors (after normalizing them), its cross product gives the axis of rotation in camera coordinates, and its dot product gives the angle. Now, you can rotate the camera by glm::rotate()
If you go another route (e.g. calculating camera matrix on your own), then the "up" direction of the camera must be updated by yourself. Remember it's perpendicular to the other two axis of the camera.
I need to extract the transform matrix from my camera to assign it to a mesh.
I'm working in a computational graphics project in school, the objective is to simulate the arms of a character in first person perspective.
My camera implementation includes a vector3 for the camera position, so i can assign that to my mesh, the problem is that i can't extract the rotation of the camera from my view matrix yet.
I calculate my final pitch and yaw in the rotation function this way, x and y are the current mouse position in the screen
m_yaw += (x - m_mouseLastPosition.x) * m_rotateSpeed;
m_pitch -= (y - m_mouseLastPosition.y) * m_rotateSpeed;
This is how i update the view matrix when it changes
glm::vec3 newFront;
newFront.x = -cos(glm::radians(m_yaw)) * cos(glm::radians(m_pitch));
newFront.y = sin(glm::radians(m_yaw)) * cos(glm::radians(m_pitch));
newFront.z = sin(glm::radians(m_pitch));
m_front = glm::normalize(newFront);
m_right = glm::normalize(glm::cross(m_front, m_worldUp));
m_up = glm::normalize(glm::cross(m_right, m_front));
m_viewMatrix = glm::lookAt(m_position, (m_position + m_front), m_up);
Right now I can assign the position of the camera to my mesh, like this
m_mesh.m_transform = glm::translate(glm::mat4(1.0f), m_camera.m_position);
I can assign the camera position successfully, but not rotation.
What i expect is to assign the full camera transform to my mesh, or to extract the rotation independently and assign it to the mesh after.
The steps to setting up the model view projection matrix, that I have always followed (which doesn't mean it's 100% right), which appears to be what you are having problems with is:
// Eye position is in world coordinate system, as is scene_center. up_vector is normalized.
glm::dmat4 view = glm::lookat(eye_position, scene_center, up_vector);
glm::dmat4 proj = glm::perspective(field_of_view, aspect, near_x, far_x);
// This converts the model from it's units, to the units of the world coordinate system
glm::dmat4 model = glm::scale(glm::dmat4(1.0), glm::dvec3(1.0, 1.0, 1.0));
// Add model level rotations here utilizing glm::rotate
// offset is where the objects 0,0 should be mapped to in the world coordinate system
model = glm::translate(model, offset);
// Order of course matters here.
glm::dvec3 mvp = proj * view * model;
Hope that helps.
Thank you so much for your answers, they helped a lot.
I managed to solve my problem in a very simple way, i just had to directly assign the final transform to my mesh using the separate properties of my camera.
glm is so new to me, i wasn't familiar with the way it handles matrix multiplications.
The final code takes a translation matrix with the camera position, then i rotated the resulting matrix in the Y axis with my pitch and finaly the resulting one rotates in the X axis with my yaw.
m_mesh.m_transform = glm::rotate(glm::rotate(glm::translate(glm::mat4(1.0f), camera.m_position), glm::radians(-camera.m_yaw), glm::vec3(0.0f, 1.0f, 0.0f)), glm::radians(-camera.m_pitch), glm::vec3(1.0f, 0.0f, 0.0f));
I'm attempting to implement a camera controller for a first person, mouse-look based camera for OpenGL. This is a simple problem when the camera is always oriented normally (camera up vector = world Y axis). However, I'm having real trouble getting everything working properly with a camera that can be used seamlessly for any orientation. The purpose is to allow a player to move around an entire planet. An additional requirement is that the direction remain the same relative to the orientation as the camera's orientation changes. An example would be, if you're walking around a planet, the direction remains the same relative to the ground, so as you go "down" along the side from a pole, the direction is also automatically rotated.
So far, I've attempted a number of different things to get this working, but as I see it, there should be two different ways of doing this. The first is to do regular camera rotation based on yaw and pitch angles from the world axes, and then transform the resulting look direction by the camera orientation to obtain the final look direction. The second approach is to rotate the camera with yaw and pitch angles based on calculated up and right vectors. The up vector is easy here; it's just the orientation. I haven't gotten any right vector I've found to work correctly though.
OK, here's the code for these two approaches.
Common code
// m_orientation calculated from planet center to current position
m_horizontal += horizontal;
m_vertical += vertical;
while (m_horizontal > TWO_PI) {
m_horizontal -= TWO_PI;
}
while (m_horizontal < -TWO_PI) {
m_horizontal += TWO_PI;
}
if (m_vertical > MAX_VERTICAL) {
m_vertical = MAX_VERTICAL;
}
else if (m_vertical < -MAX_VERTICAL) {
m_vertical = -MAX_VERTICAL;
}
// code from either implementation
m_view = glm::lookAt(m_position, m_position + m_direction, m_orientation);
First approach with yaw, pitch about world axes and then transform
// check for m_orientation != WORLD_UP...
glm::vec3 axis = glm::normalize(glm::cross(WORLD_UP, m_orientation));
float angle_degrees = acosf(m_orientation.y) * RADS_TO_DEGREES;
glm::mat4 trans = glm::rotate(glm::mat4(), angle_degrees, axis);
// can also be determined with two rotation matrices about world axes, end result is identical
m_direction = glm::vec3(cosf(m_vertical) * sinf(m_horizontal),
sinf(m_vertical),
cosf(m_vertical) * cosf(m_horizontal));
m_direction = glm::vec3(trans * glm::vec4(m_direction));
Second approach with yaw and pitch about appropriate up and right vectors
m_right = ??? // tried literally everything
glm::mat4 yaw = glm::rotate(glm::mat4(), m_horizontal, m_orientation);
glm::mat4 pitch = glm::rotate(glm::mat4(), m_vertical, m_right);
glm::mat4 trans = yaw * pitch;
m_direction = glm::vec3(trans[2]); // z axis
OK, so here's the problem. The first approach works almost perfectly, but near the south pole of a planet (within ~15 degrees of orientation=(0,-1,0), effect gets stronger closer you are), the camera is automatically rotated toward the south pole as the orientation changes. So if the camera orientation does not change, near the south pole, the camera works perfectly. Any change in orientation results in the camera rotating toward the south pole. The more orientation change, the more the camera rotates. Now I have tried removing either the pitch or yaw from the world axis camera rotation, and this effect appears only with the pitch calculation included. With only yaw, then the camera behaves perfectly (lacking any pitch control ofc). As far as I can tell, my transformation to go from regular up=(0,1,0) to the current orientation is incorrect. Any help on that?
Now the other way to do things appears to work somewhat correctly, but I simply have not found a good right vector. Everything I've tried results in strange behavior of both horizontal and vertical movements. The most obvious solution, cross product of previous frame's direction and current orientation to produce the right vector doesn't work. Any suggestions for a good right vector?
I'm also happy to see completely different solutions to this problem. I know it's possible, but no amount of searching has given me a good solution. Thanks very much in advance.
Edit 1: Tried a few more things in response to Paweł Stawarz
Results in incorrect orientation of camera and weird mouse movement. I made sure my matrix multiplication was in the correct order. I also tried the transpose.
m_view = glm::lookAt(glm::vec3(0.0f, 0.0f, 0.0f), m_direction, m_up);
m_view = trans * m_view; //trans is rotation from orientation=(0,1,0) to orientation=m_orientation
Results in the same problem as previously, with the camera rotating toward the south pole by itself. Also the vertical mouse rotation is not correct, causes camera to go in circles.
m_view = glm::lookAt(glm::vec3(0.0f, 0.0f, 0.0f), m_direction, m_up);
m_view = trans * m_view;
m_direction = glm::vec3(m_view[2]);
m_view = glm::lookAt(m_position, m_direction + m_position, m_orientation);
Edit 2: Using the RIGHT vector method, with no transformation between orientations is working a little better. However, it causes the camera yaw to oscillate wildly with pitch near to vertical (at least 5 degrees away from vertical). In addition, the range of motion for pitch is not adjusted by the orientation, so for example on the side of the planet, vertical motion is restricted to directly in front of you to behind you (~(0,1,0) to ~(0,-1,0)).
glm::mat4 yaw = glm::rotate(glm::mat4(), m_horizontal * ONEEIGHTY_PI, m_orientation);
glm::mat4 pitch = glm::rotate(glm::mat4(), m_vertical * -ONEEIGHTY_PI, m_right);
glm::mat4 cam = pitch * yaw;
m_right = glm::vec3(cam[0]);
m_up = glm::vec3(cam[1]);
m_direction = glm::vec3(cam[2]);
m_view = glm::lookAt(m_position, m_direction + m_position, m_up);
m_vp = m_perspective * m_view;
Solved it. Needed a different transformation. See here for a pretty good explanation.
glm::mat4 trans;
float factor = 1.0f;
float real_vertical = vertical;
m_horizontal += horizontal;
m_vertical += vertical;
while (m_horizontal > TWO_PI) {
m_horizontal -= TWO_PI;
}
while (m_horizontal < -TWO_PI) {
m_horizontal += TWO_PI;
}
if (m_vertical > MAX_VERTICAL) {
m_vertical = MAX_VERTICAL;
}
else if (m_vertical < -MAX_VERTICAL) {
m_vertical = -MAX_VERTICAL;
}
glm::quat world_axes_rotation = glm::angleAxis(m_horizontal * ONEEIGHTY_PI, glm::vec3(0.0f, 1.0f, 0.0f));
world_axes_rotation = glm::normalize(world_axes_rotation);
world_axes_rotation = glm::rotate(world_axes_rotation, m_vertical * ONEEIGHTY_PI, glm::vec3(1.0f, 0.0f, 0.0f));
m_pole = glm::normalize(m_pole - glm::dot(m_orientation, m_pole) * m_orientation);
glm::mat4 local_transform;
local_transform[0] = glm::vec4(m_pole.x, m_pole.y, m_pole.z, 0.0f);
local_transform[1] = glm::vec4(m_orientation.x, m_orientation.y, m_orientation.z, 0.0f);
glm::vec3 tmp = glm::cross(m_pole, m_orientation);
local_transform[2] = glm::vec4(tmp.x, tmp.y, tmp.z, 0.0f);
local_transform[3] = glm::vec4(m_position.x, m_position.y, m_position.z, 1.0f);
world_axes_rotation = glm::normalize(world_axes_rotation);
m_view = local_transform * glm::mat4_cast(world_axes_rotation);
m_direction = -1.0f * glm::vec3(m_view[2]);
m_up = glm::vec3(m_view[1]);
m_right = glm::vec3(m_view[0]);
m_view = glm::inverse(m_view);
If we keep things simple, by using the standard approach:
Move the camera to the current player position
Rotate it towards where the player is looking at
The camera rotation is described by:
The UP vector which is the normalized vector that starts at (p0x,p0y,p0z) (where p0 is the position of the center of planet) and goes thru (p1x,p1y,p1z) (where p1 describes the place the player is currently at),
the RIGHT vector is the vector perpendicular to the UP vector and perpendicular to the direction the player is looking - the LOOK vector (in the case where he's looking straight ahead - perpendicular to his direction).
Since the UP vector can be calulated straight from the player current position, you have to get the LOOK and RIGHT vectors. Both are cross products of corresponding other vectors.
Note also, that allowing the player to look up/down and pan his head, can (and probably will) in fact change the UP vector.
I am using glm to create a camera class, and I am running into some problems with a lookat function. I am using a quaternion to represent rotation, but I want to use glm's prewritten lookat function to avoid duplicating code. This is my lookat function right now:
void Camera::LookAt(float x, float y, float z) {
glm::mat4 lookMat = glm::lookAt(position, glm::vec3(x, y, z), glm::vec3(0, 1, 0));
rotation = glm::toQuat(lookMat);
}
However when I call LookAt(0.0f,0.0f,0.0f), my camera is not rotated to that point. When I call glm::eulerangles(rotation) after the lookat call, I get a vec3 with the following values: (180.0f, 0.0f, 180.0f). position is (0.0f,0.0f,-10.0f), so I should not have any rotation at all to look at 0,0,0. This is the function which builds the view matrix:
glm::mat4 Camera::GetView() {
view = glm::toMat4(rotation) * glm::translate(glm::mat4(), position);
return view;
}
Why am I not getting the correct quaternion, and how can I fix my code?
Solution:
You have to invert the rotation of the quaternion by conjugating it:
using namespace glm;
quat orientation = conjugate(toQuat(lookAt(vecA, vecB, up)));
Explanation:
The lookAt function is a replacement for gluLookAt, which is used to construct a view matrix.
The view matrix is used to rotate the world around the viewer, and is therefore the inverse of the cameras transform.
By taking the inverse of the inverse, you can get the actual transform.
I ran into something similar, the short answer is your lookMat might need to be inverted/transposed, because it is a camera rotation (at least in my case), as opposed to a world rotation. Rotating the world would be a inverse of a camera rotation.
I have a m_current_quat which is a quaternion that stores the current camera rotation. I debugged the issue by printing out the matrix produced by glm::lookAt, and comparing with the resulting matrix that I get by applying m_current_quat and a translation by m_camera_position. Here is the relevant code for my test.
void PrintMatrix(const GLfloat m[16], const string &str)
{
printf("%s:\n", str.c_str());
for (int i=0; i<4; i++)
{
printf("[");
//for (int j=i*4+0; j<i*4+4; j++) // row major, 0, 1, 2, 3
for (int j=i+0; j<16; j+=4) // OpenGL is column major by default, 0, 4, 8, 12
{
//printf("%d, ", j); // print matrix index
printf("%.2f, ", m[j]);
}
printf("]\n");
}
printf("\n");
}
void CameraQuaternion::SetLookAt(glm::vec3 look_at)
{
m_camera_look_at = look_at;
// update the initial camera direction and up
//m_initial_camera_direction = glm::normalize(m_camera_look_at - m_camera_position);
//glm::vec3 initial_right_vector = glm::cross(m_initial_camera_direction, glm::vec3(0, 1, 0));
//m_initial_camera_up = glm::cross(initial_right_vector, m_initial_camera_direction);
m_camera_direction = glm::normalize(m_camera_look_at - m_camera_position);
glm::vec3 right_vector = glm::cross(m_camera_direction, glm::vec3(0, 1, 0));
m_camera_up = glm::cross(right_vector, m_camera_direction);
glm::mat4 lookat_matrix = glm::lookAt(m_camera_position, m_camera_look_at, m_camera_up);
// Note: m_current_quat quat stores the camera rotation with respect to the camera space
// The lookat_matrix produces a transformation for world space, where we rotate the world
// with the camera at the origin
// Our m_current_quat need to be an inverse, which is accompolished by transposing the lookat_matrix
// since the rotation matrix is orthonormal.
m_current_quat = glm::toQuat(glm::transpose(lookat_matrix));
// Testing: Make sure our model view matrix after gluLookAt, glmLookAt, and m_current_quat agrees
GLfloat current_model_view_matrix[16];
//Test 1: gluLookAt
gluLookAt(m_camera_position.x, m_camera_position.y, m_camera_position.z,
m_camera_look_at.x, m_camera_look_at.y, m_camera_look_at.z,
m_camera_up.x, m_camera_up.y, m_camera_up.z);
glGetFloatv(GL_MODELVIEW_MATRIX, current_model_view_matrix);
PrintMatrix(current_model_view_matrix, "Model view after gluLookAt");
//Test 2: glm::lookAt
lookat_matrix = glm::lookAt(m_camera_position, m_camera_look_at, m_camera_up);
PrintMatrix(glm::value_ptr(lookat_matrix), "Model view after glm::lookAt");
//Test 3: m_current_quat
glLoadIdentity();
glMultMatrixf( glm::value_ptr( glm::transpose(glm::mat4_cast(m_current_quat))) );
glTranslatef(-m_camera_position.x, -m_camera_position.y, -m_camera_position.z);
glGetFloatv(GL_MODELVIEW_MATRIX, current_model_view_matrix);
PrintMatrix(current_model_view_matrix, "Model view after quaternion transform");
return;
}
Hope this helps.
I want to use glm's prewritten lookat function to avoid duplicating code.
But it's not duplicating code. The matrix that comes out of glm::lookat is just a mat4. Going through the conversion from a quaternion to 3 vectors, only so that glm::lookat can convert it back into an orientation is just a waste of time. You've already done 85% of lookat's job; just do the rest.
You are getting the (or better: a) correct rotation.
When I call glm::eulerangles(rotation) after the lookat call, I get a
vec3 with the following values: (180.0f, 0.0f, 180.0f). position is
(0.0f,0.0f,-10.0f), so I should not have any rotation at all to look
at 0,0,0.
glm is following the conventions of the old fixed-function GL. And there, eye space was defined as the camera placed at origin, with x pointng to the right, y up and looking in -z direction. Since you want to look in positive z direction, the camera has to turn. Now, as a human, I would have described that as a rotation of 180 degrees around y, but a rotation of 180 degrees around x in combination with another 180 degrees rotation aroundz will have the same effect.
When multiplied by the LookAt view matrix, the world-space vectors are rotated (brought) into the camera's view while the camera's orientation is kept in place.
So an actual rotation of the camera by 45 degress to the right is achieved with a matrix which applies a 45 degree rotation to the left to all the world-space vertices.
For a Camera object you would need to get its local forward and up direction vectors in order to calculate a lookAt view matrix.
viewMatrix = glm::lookAtLH (position, position + camera_forward, camera_up);
When using quaternions to store the orientation of an object (be it a camera or anything else), usually this rotation quat is used to calculate the vectors which define its local-space (left-handed one in the below example):
glm::vec3 camera_forward = rotation * glm::vec3(0,0,1); // +Z is forward direction
glm::vec3 camera_right = rotation * glm::vec3(1,0,0); // +X is right direction
glm::vec3 camera_up = rotation * glm::vec3(0,1,0); // +Y is up direction
Thus, the world-space directions should be rotated 45 degress to the right in order to reflect the correct orientation of the camera.
This is why the lookMat or the quat obtained from it cannot be directly used for this purpose, since the orientation they describe is a reversed one.
Correct rotation can be done in two ways:
Calculate the inverse of the lookAt matrix and multiply the world-space direction vectors by this rotation matrix
(more efficient) Convert the LookAt matrix into a quaternion and conjugate it instead of applying glm::inverse, since the result is a unit quat and for such quats the inverse is equal to the conjugate.
Your LookAt should look like this:
void Camera::LookAt(float x, float y, float z) {
glm::mat4 lookMat = glm::lookAt(position, glm::vec3(x, y, z), glm::vec3(0, 1, 0));
rotation = glm::conjugate( glm::quat_cast(lookMat));
}
Hey guys I am trying to do a Camera class that uses lookAt from glm library. I have 4 points, the first one is eye, that is the camera position in space, the second one is the look, that is the point where the camera is looking at, the third one is the upp, that set the orientation of camera, and the forth is side, that is a cross product of look - eye and upp -eye.
So in the end I got a base of 3 vectors all of them with origin in the eye point. I got a coordinate system of the camera.
In my camera class, I want to be able to rotate about the coordinate system of the camera, not the coordinate system of the world. So what I am doing is rotate about one of the axis of the coordinate system of the camera.
I construct the class with initial values like this:
void Observer::initialize(glm::vec3 eye, glm::vec3 look, glm::vec3 upp, glm::vec3 side)
{
this->eye = eye; // (0.0, 0.0, 0.0)
this->look = look; // (0.0, 0.0, -1.0)
this->upp = upp; // (0.0, 1.0, 0.0)
this->side = side; // (1.0, 0.0, 0.0)
}
When I want to rotate the coordinate system about the x axis for example I call the function from glm like this:
void Observer::pitch(GLfloat pitch)
{
glm::mat4 rotate(1.0f);
rotate = glm::rotate(rotate, pitch, side - eye);
look = glm::vec3(rotate * glm::vec4(look, 1.0f));
upp = glm::vec3(rotate * glm::vec4(upp, 1.0f));
}
So far, I am understanding that all my points still form a coordinate system for the camera and all vector are perpendicular between each other.
But then I use these points I got with the lookAt function, to position the camera in the world.
glm::mat4 view = glm::lookAt(eye, look, upp);
And multiply this matrix with the modelview matrix from OpenGL
If I start to rotate a lot, the camera after a few rotations "reflect" the rotation, like I was rotating in the other way (I don't know how to describe what is really happening in a better way =s).
I really don't understand what is happening. I should normalize the vectors after I apply the rotation? Am I having a problem with gimbal lock (I don't know a lot about gimbal lock)?
When you do incremental rotations on vectors as you have done, numerical errors mount. When error causes the up and look-at vectors to point nearly in the same direction or opposite each other, the camera transformation calculation is unstable and whacky things can happen. Gimbal lock. Length changes cause different problems.
A solution that has worked for me is to re-orthogonalize the up and look-at vectors after each rotation. To do this, compute their cross-product L, then adjust (really replace) the up vector by crossing L with the lookAt. After all this re-normalize both up and look-at to unit length.
Though the orthogonalization-normalization is a fast operation, you don't really have to do it with every camera motion.
Note that when you correct the vectors like this you are actually doing part of the lookAt matrix calculation, so consider implementing your own to avoid an unnecessary cross product. See e.g. this prior SO article on this topic.