Opengl Billboard matrix - opengl

Question

I am writing a viewer for a proprietary mesh & animation format in OpenGL.
During rendering a transformation matrix is created for each bone (node) and is applied to the vertices that bone is attached to.
It is possible for a bone to be marked as "Billboarded" which as most everyone knows, means it should always face the camera.
So the idea is to generate a matrix for that bone which when used to transform the vertices it's attached to, causes the vertices to be billboarded.
On my test model it should look like this:
However currently it looks like this:
Note, that despite its incorrect orientation, it is billboarded. As in no matter which direction the camera looks, those vertices are always facing that direction at that orientation.
My code for generating the matrix for bones marked as billboarded is:
mat4 view;
glGetFloatv(GL_MODELVIEW_MATRIX, (float*)&view);
vec4 camPos = vec4(-view[3].x, -view[3].y, -view[3].z,1);
vec3 camUp = vec3(view[0].y, view[1].y, view[2].y);
// zero the translation in the matrix, so we can use the matrix to transform
// camera postion to world coordinates using the view matrix
view[3].x = view[3].y = view[3].z = 0;
// the view matrix is how to get to the gluLookAt pos from what we gave as
// input for the camera position, so to go the other way we need to reverse
// the rotation. Transposing the matrix will do this.
{
float * matrix = (float*)&view;
float temp[16];
// copy this into temp
memcpy(temp, matrix, sizeof(float) * 16);
matrix[1] = temp[4]; matrix[4] = temp[1];
matrix[2] = temp[8]; matrix[8] = temp[2];
matrix[6] = temp[9]; matrix[9] = temp[6];
}
// get the correct position of the camera in world space
camPos = view * camPos;
//vec3 pos = pivot;
vec3 look = glm::normalize(vec3(camPos.x-pos.x,camPos.y-pos.y,camPos.z-pos.z));
vec3 right = glm::cross(camUp,look);
vec3 up = glm::cross(look,right);
mat4 bmatrix;
bmatrix[0].x = right.x;
bmatrix[0].y = right.y;
bmatrix[0].z = right.z;
bmatrix[0].w = 0;
bmatrix[1].x = up.x;
bmatrix[1].y = up.y;
bmatrix[1].z = up.z;
bmatrix[1].w = 0;
bmatrix[2].x = look.x;
bmatrix[2].y = look.y;
bmatrix[2].z = look.z;
bmatrix[2].w = 0;
bmatrix[3].x = pos.x;
bmatrix[3].y = pos.y;
bmatrix[3].z = pos.z;
bmatrix[3].w = 1;
I am using GLM to do the math involved.
Though this part of the code is based off of the tutorial here, other parts of the code are based off of an open source program similar to the one I'm building. However that program was written for DirectX and I haven't had much luck directly converting. The (working) directX code for billboarding looks like this:
D3DXMatrixRotationY(&CameraRotationMatrixY, -Camera.GetPitch());
D3DXMatrixRotationZ(&CameraRotationMatrixZ, Camera.GetYaw());
D3DXMatrixMultiply(&CameraRotationMatrix, &CameraRotationMatrixY, &CameraRotationMatrixZ);
D3DXQuaternionRotationMatrix(&CameraRotation, &CameraRotationMatrix);
D3DXMatrixTransformation(&CameraRotationMatrix, NULL, NULL, NULL, &ModelBaseData->PivotPoint, &CameraRotation, NULL);
D3DXMatrixDecompose(&Scaling, &Rotation, &Translation, &BaseMatrix);
D3DXMatrixTransformation(&RotationMatrix, NULL, NULL, NULL, &ModelBaseData->PivotPoint, &Rotation, NULL);
D3DXMatrixMultiply(&TempMatrix, &CameraRotationMatrix, &RotationMatrix);
D3DXMatrixMultiply(&BaseMatrix, &TempMatrix, &BaseMatrix);
Note the results are stored in baseMatrix in the directX version.
EDIT2: Here's the code I came up with when I tried to modify my code according to datenwolf's suggestions. I'm pretty sure I made some mistakes still. This attempt creates heavily distorted results with one end of the object directly in the camera.
mat4 view;
glGetFloatv(GL_MODELVIEW_MATRIX, (float*)&view);
vec3 pos = vec3(calculatedMatrix[3].x,calculatedMatrix[3].y,calculatedMatrix[3].z);
mat4 inverted = glm::inverse(view);
vec4 plook = inverted * vec4(0,0,0,1);
vec3 look = vec3(plook.x,plook.y,plook.z);
vec3 right = orthogonalize(vec3(view[0].x,view[1].x,view[2].x),look);
vec3 up = orthogonalize(vec3(view[0].y,view[1].y,view[2].y),look);
mat4 bmatrix;
bmatrix[0].x = right.x;
bmatrix[0].y = right.y;
bmatrix[0].z = right.z;
bmatrix[0].w = 0;
bmatrix[1].x = up.x;
bmatrix[1].y = up.y;
bmatrix[1].z = up.z;
bmatrix[1].w = 0;
bmatrix[2].x = look.x;
bmatrix[2].y = look.y;
bmatrix[2].z = look.z;
bmatrix[2].w = 0;
bmatrix[3].x = pos.x;
bmatrix[3].y = pos.y;
bmatrix[3].z = pos.z;
bmatrix[3].w = 1;
calculatedMatrix = bmatrix;
vec3 orthogonalize(vec3 toOrtho, vec3 orthoAgainst) {
float bottom = (orthoAgainst.x*orthoAgainst.x)+(orthoAgainst.y*orthoAgainst.y)+(orthoAgainst.z*orthoAgainst.z);
float top = (toOrtho.x*orthoAgainst.x)+(toOrtho.y*orthoAgainst.y)+(toOrtho.z*orthoAgainst.z);
return toOrtho - top/bottom*orthoAgainst;
}

Answer 1

Creating a parallel to view billboard matrix is as simple as setting the upper left 3×3 submatrix of the total modelview matrix to identity. There are only some cases where you actually require the actual look vector.
Anyway, you're thinking far too complicated. All your tinkering with the matrix completely misses the point. Namely that the modelview transformation assumes that the camera is always at (0,0,0) and moves world and models in opposite. What you try to do is finding the vector in model space that points towards the camera. Which is simply the vector that will point toward (0,0,0) after transformation.
So all we have to do is invert the modelview matrix and transform (0,0,0,1) with it. That's your look vector. For your calculations of right and up vectors orthogonalize the 1st (X) and 2nd (Y) column of the modelview matrix against that look vector.

Answer 2

Figured it out myself. It turns out the model format I'm using uses different axes for billboarding. Most billboarding implementations (including the one I used) use the X,Y coordinates to position the billboarded object. The format I was reading uses Y and Z.
The thing to look for is that there was a billboarding effect, but facing the wrong direction. To fix this I played with the different camera vectors until I arrived at the correct matrix calculation:
bmatrix[1].x = right.x;
bmatrix[1].y = right.y;
bmatrix[1].z = right.z;
bmatrix[1].w = 0;
bmatrix[2].x = up.x;
bmatrix[2].y = up.y;
bmatrix[2].z = up.z;
bmatrix[2].w = 0;
bmatrix[0].x = look.x;
bmatrix[0].y = look.y;
bmatrix[0].z = look.z;
bmatrix[0].w = 0;
bmatrix[3].x = pos.x;
bmatrix[3].y = pos.y;
bmatrix[3].z = pos.z;
bmatrix[3].w = 1;
My attempts to follow datenwolf's advice did not succeed and at this time he hasn't offered any additional explanation so I'm unsure why. Thanks anyways!