Rotation: Quaternion to matrix - opengl

Question

I am trying to display a 360 panorama using an IMU for head tracking.
Yaw works correctly but the roll and pitch are reverse. I also notice that the pitch contains some roll (and maybe vice-versa).
I am receiving (W, X, Y, Z) coordinate from the IMU that I am storing in an array as X, Y, Z, W.
The next step is converting the quaternion to a rotation matrix. I have looked at many examples, and can't seem to find anything wrong with the following code:
static GLfloat rotation[16];
// Quaternion (x, y, z, w)
static void quaternionToRotation(float* quaternion)
{
// Normalize quaternion
float magnitude = sqrt(quaternion[0] * quaternion[0] +
quaternion[1] * quaternion[1] +
quaternion[2] * quaternion[2] +
quaternion[3] * quaternion[3]);
for (int i = 0; i < 4; ++i)
{
quaternion[i] /= magnitude;
}
double xx = quaternion[0] * quaternion[0], xy = quaternion[0] * quaternion[1],
xz = quaternion[0] * quaternion[2], xw = quaternion[0] * quaternion[3];
double yy = quaternion[1] * quaternion[1], yz = quaternion[1] * quaternion[2],
yw = quaternion[1] * quaternion[3];
double zz = quaternion[2] * quaternion[2], zw = quaternion[2] * quaternion[3];
// Column major order
rotation[0] = 1.0f - 2.0f * (yy + zz);
rotation[1] = 2.0f * (xy - zw);
rotation[2] = 2.0f * (xz + yw);
rotation[3] = 0;
rotation[4] = 2.0f * (xy + zw);
rotation[5] = 1.0f - 2.0f * (xx + zz);
rotation[6] = 2.0f * (yz - xw);
rotation[7] = 0;
rotation[8] = 2.0f * (xz - yw);
rotation[9] = 2.0f * (yz + xw);
rotation[10] = 1.0f - 2.0f * (xx + yy);
rotation[11] = 0;
rotation[12] = 0;
rotation[13] = 0;
rotation[14] = 0;
rotation[15] = 1;
}
The rotation matrix is then used in the draw call as such:
static void draw()
{
// Get IMU quaternion
float* quaternion = tracker.getTrackingData();
if (quaternion != NULL)
{
quaternionToRotation(quaternion);
}
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glLoadIdentity();
glPushMatrix();
// TODO: Multiply initialRotation quaternion with IMU quaternion
glMultMatrixf(initialRotation); // Initial rotation to point forward
glMultMatrixf(rotation); // Rotation based on IMU
glEnable(GL_TEXTURE_2D);
glBindTexture(GL_TEXTURE_2D, texture);
gluSphere(quad, 0.1, 50, 50);
glBindTexture(GL_TEXTURE_2D, 0);
glPopMatrix();
glFlush();
glutSwapBuffers();
}
I tried to set all but one fields in the quaternion to 0, and I notice that they all work individually, except roll and pitch is swapped around. I tried swapping X and Y but this does not seem to help.
Any help would be really appreciated. Please let me know as well if you have any steps that can let me debug my issue. Thanks!