rotating object relative to camera in OpenGL - opengl

Question

I'm having problems in OpenGL getting my object (a planet) to rotate relative to the current camera rotation. It seems to work at first, but then after rotating a bit, the rotations are no longer correct/relative to the camera.
I'm calculating a delta (difference) in mouseX and mouseY movements on the screen. The rotation is stored in a Vector3D called 'planetRotation'.
Here is my code to calculate the rotation relative to the planetRotation:
Vector3D rotateAmount;
rotateAmount.x = deltaY;
rotateAmount.y = deltaX;
rotateAmount.z = 0.0;
glPushMatrix();
glLoadIdentity();
glRotatef(-planetRotation.z, 0.0, 0.0, 1.0);
glRotatef(-planetRotation.y, 0.0, 1.0, 0.0);
glRotatef(-planetRotation.x, 1.0, 0.0, 0.0);
GLfloat rotMatrix[16];
glGetFloatv(GL_MODELVIEW_MATRIX, rotMatrix);
glPopMatrix();
Vector3D transformedRot = vectorMultiplyWithMatrix(rotateAmount, rotMatrix);
planetRotation = vectorAdd(planetRotation, transformedRot);
In theory - what this does is, sets up a rotation in the 'rotateAmount' variable. It then gets this into model space, by multiplying this vector with the inverse model transform matrix (rotMatrix).
This transformed rotation is then added to the current rotation.
To render this is the transform being setup:
glPushMatrix();
glRotatef(planetRotation.x, 1.0, 0.0, 0.0);
glRotatef(planetRotation.y, 0.0, 1.0, 0.0);
glRotatef(planetRotation.z, 0.0, 0.0, 1.0);
//render stuff here
glPopMatrix();
The camera sort of wobbles around, the rotation I'm trying to perform, doesn't seem relative to the current transform.
What am I doing wrong?

Answer 1

GAH! Don't do that:
glPushMatrix();
glLoadIdentity();
glRotatef(-planetRotation.z, 0.0, 0.0, 1.0);
glRotatef(-planetRotation.y, 0.0, 1.0, 0.0);
glRotatef(-planetRotation.x, 1.0, 0.0, 0.0);
GLfloat rotMatrix[16];
glGetFloatv(GL_MODELVIEW_MATRIX, rotMatrix);
glPopMatrix();
OpenGL is not a math library. There are proper linear algebra libraries for that kind of job.
As for your problems. A vector is not fit to store a rotation. You need at least a Vector (axis of rotation) and the angle itself, or better yet a Quaternion.
Also rotations don't add. They're no commutative, however addition is a commutative operation. Rotations in fact multiply.
How to fix your code: Rewrite it from scratch using the proper mathematical methods. For this please read up the topics of "Rotation matrices" and "Quaternions" (Wikipedia has them).