AR with OpenCV & OpenGL - c++

Here's the problem: I wrote a code to display the OpenGL teapot on a sheet of paper with drawing. For this, I track the 4 corners of the paper (using SURF detection & matching followed by computing the homography matrix, then moving average of the corners position to reduce the jitter). The corners coordinates are used to compute the intrinsic & extrinsic matrices of the camera (using calibrateCamera() and solvePnP(), respectively). The rotation matrix is then computed using Rodrigues(). Afterwards, I computed the rotation angles using decomposeProjectionMatrix(). Here's the OpenCV part of the code:
...
objPoints.push_back(objCorners);
scenePoints.push_back(sceneCorners);
calibrateCamera(objPoints, scenePoints, Size(640,480), camMtx, distortCoeff, RVecs, tVecs);
solvePnP(objCorners, sceneCorners, camMtx, distortCoeff, RVec, tVec);
Rodrigues(RVec, rotMtx);
getAngles(rotMtx, rotAngles);
objCorners are the corners coordinates in the template image ([1 1], [img width 1], [img width img height], [1 img height]). sceneCorners are the corners coordinates in the webcam frame, computed using the homography matrix. The function getAngles() is as follows:
void getAngles(Mat &rotCamMtx, Vec3d &angles)
{
Mat camMtx, rotMtx, transVec, rotMtxX, rotMtxY, rotMtxZ;
double *r = rotCamMtx.ptr<double>();
double projMtx[12] = {r[0], r[1], r[2], 0,
r[3], r[4], r[5], 0,
r[6], r[7], r[8], 0};
decomposeProjectionMatrix(Mat(3,4,CV_64FC1,projMtx), camMtx, rotMtx, transVec, rotMtxX, rotMtxY, rotMtxZ, angles);
}
Then I set the element of the OpenGL model view matrix as follows:
modelViewMat[0] = 1.0;
modelViewMat[1] = 0.0;
modelViewMat[2] = 0.0;
modelViewMat[3] = 0.0;
modelViewMat[4] = 0.0;
modelViewMat[5] = 1.0;
modelViewMat[6] = 0.0;
modelViewMat[7] = 0.0;
modelViewMat[8] = 0.0;
modelViewMat[9] = 0.0;
modelViewMat[10] = 1.0;
modelViewMat[11] = 0.0;
modelViewMat[12] = 2*matCenter.x/639 - 641/639;
modelViewMat[13] = 481/479 - 2*matCenter.y/479;
modelViewMat[14] = -0.25;
modelViewMat[15] = 1.0;
matCenter is the center coordinate of the paper, obtained by taking the average of the 4 corners. The values in modelViewMat[12] and modelViewMat[13] are obtained by mapping the pixel coordinates ([1 640], [1 480]) to ([-1 1], [1 -1]). The OpenGL part of the code:
...
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
glMatrixMode(GL_MODELVIEW);
glLoadMatrixd(modelViewMat);
glRotated(-45, 1.0, 0.0, 0.0);
glRotated(rotAngles[2], 0.0, 1.0, 0.0);
glShadeModel(GL_SMOOTH);
glColor3f(1.0, 1.0, 1.0);
glutSolidTeapot(0.3);
I rotated the teapot -45 degrees around x-axis to make it appears "sitting" on the paper.
The result is this: if I translate the paper on the desk, the location of the teapot on the paper is more or less correct (on the same spot). If I rotate the paper, the teapot will follow the rotation correctly (around y-axis), but the location is no more correct. The question is: how to "pin" the teapot always on the same spot of the paper? I've tried using the result of Rodrigues() and solvePnP() directly in the OpenGL model view matrix (as suggested in OpenCV + OpenGL: proper camera pose using solvePnP), but the result is incorrect.

Solved this problem several days ago, based on the code from http://blog.yarrago.com/2011/08/introduction-to-augmented-reality.html. To display the 3D object correctly, the OpenGL projection matrix is set first, followed by the OpenGL model view matrix. The elements of the projection matrix are computed from the intrinsic matrix of the camera as follows:
calibrateCamera(objPoints, scenePoints, Size(640,480), camMtx, distortCoeff, RVecs, tVecs);
...
projectionMat[0] = 2*camMtx.at<double>(0,0)/frameW;
projectionMat[1] = 0;
projectionMat[2] = 0;
projectionMat[3] = 0;
projectionMat[4] = 0;
projectionMat[5] = 2*camMtx.at<double>(1,1)/frameH;
projectionMat[6] = 0;
projectionMat[7] = 0;
projectionMat[8] = 1 - 2*camMtx.at<double>(0,2)/frameW;
projectionMat[9] = -1 + (2*camMtx.at<double>(1,2) + 2)/frameH;
projectionMat[10] = (zNear + zFar)/(zNear - zFar);
projectionMat[11] = -1;
projectionMat[12] = 0;
projectionMat[13] = 0;
projectionMat[14] = 2*zNear*zFar/(zNear - zFar);
projectionMat[15] = 0;
frameW and frameH are 640 and 480, respectively. zNear is 0.1 and zFar is 100.
The elements of the OpenGL model view matrix are computed from the rotation matrix and the translation vector (obtained from solvePnP() and Rodrigues()). To get a correct positioning of the 3D object, the translation vector needs to be transformed before computing the model view matrix.
// Offset value to move the translation vector
double offsetC[3][1] = {424, 600, 0};
Mat offset(3, 1, CV_64F, offsetC);
...
solvePnP(objCorners, sceneCorners, camMtx, distortCoeff, RVec, tVec);
Rodrigues(RVec, rotMtx);
tVec = tVec + rotMtx*offset; // Move tVec to refer to the center of the paper
tVec = tVec / 250.0; // Converting pixel coordinates to OpenGL world coordinates
...
modelviewMat[0] = rotMtx.at<double>(0,0);
modelviewMat[1] = -rotMtx.at<double>(1,0);
modelviewMat[2] = -rotMtx.at<double>(2,0);
modelviewMat[3] = 0;
modelviewMat[4] = rotMtx.at<double>(0,1);
modelviewMat[5] = -rotMtx.at<double>(1,1);
modelviewMat[6] = -rotMtx.at<double>(2,1);
modelviewMat[7] = 0;
modelviewMat[8] = rotMtx.at<double>(0,2);
modelviewMat[9] = -rotMtx.at<double>(1,2);
modelviewMat[10] = -rotMtx.at<double>(2,2);
modelviewMat[11] = 0;
modelviewMat[12] = tVec.at<double>(0,0);
modelviewMat[13] = -tVec.at<double>(1,0);
modelviewMat[14] = -tVec.at<double>(2,0);
modelviewMat[15] = 1;
The numerical values for offsetC is the pixel coordinate of the center of the paper. The OpenGL part of the code is then:
glMatrixMode(GL_PROJECTION);
glLoadMatrixf(projectionMat);
glMatrixMode(GL_MODELVIEW);
glLoadMatrixf(modelviewMat);
glRotatef(90, -1.0, 0.0, 0.0); // Rotate the teapot first so that it will be displayed correctly on the paper
glutSolidTeapot(1.0);
One important thing for the correct positioning of the teapot is the transformation of tVec.

Related

camera matrix to OpenSceneGraph camera

I have a problem.
I Have a OpenGl project and i want to set the camera matrix of the OpenGl into a OpenSceneGraph camera to have the same view.
I've this code to get OpenGl camera :
GLdouble camera_pos[3];
double rationZoom = // RatioZoom
int* iViewport = // the ViewPort
double* projetMatrix = // Projection Matrix
double* modelMatrix = // ModelView matrix
// screenCenter is the position of my model.
double screenCenterX = // The center in x Axis
double screenCenterY = // The center in y Axis
gluUnProject((iViewport[2] - iViewport[0]) / 2, (iViewport[3] - iViewport[1]) / 2,
0.0, modelMatrix, projetMatrix, iViewport, &camera_pos[0], &camera_pos[1], &camera_pos[2]);`
//Camera_pos is the position X,Y,Z of my camera.
And in OpenSceneGraph i make this code to set the camera with eye/center/up to LookAt of the camera (to have the same view as OpenGl) :
// i use a zoom ration to have the same distance.
double X = ((camera_pos[0]/2) - ((screenCenterX)))/rationZoom;
double Y = ((camera_pos[1]/2) - ((screenCenterY)))/rationZoom;
double Z = ((camera_pos[2]/2)) / rationZoom;
osg::Vec3 updateCameraPosition(X, Y, Z);
osg::Matrix matrixCameraPosition;
matrixCameraPosition.setTrans(updateCameraPosition);
// Yes, for the center i invert the position matrix of the camera
osg::Matrix matrixCameraCenter;
matrixCameraCenter.invert(matrixCameraPosition);
osg::Vec3f eye(matrixCameraPosition.getTrans());
osg::Vec3f up(0, 0, 1);
osg::Vec3f centre = osg::Vec3f(matrixCameraCenter.getTrans().x(),
matrixCameraCenter.getTrans().y(),
matrixCameraCenter.getTrans().z());
// And a set the view into the camera
nodeCamera->setViewMatrixAsLookAt(eye, centre, up);
For the initialisation of the position i've no problem but if i panning the model of the OpenGl project i don't have the same view.
If I'm not mistaken for OpenGl the coordinate system is : X-left, Y-up and Z-backward, and for OpenSceneGraph this is : X-left, Y-backward, Z-up.
Maybe this is the problem and i have to set the Y up instead of Z in OpenSceneGraph ?
I have solved my problem.
I don't need to calculate the camera or set the setViewMatrixAsLookAt but just to get the modelview matrix and the projection matrix of OpenGl and set to the OpenScenGraph camera, like this:
double* projetMatrix = // the Projection matrix
double* modelMatrix = // The modelView matrix
osg::Matrixd modelViewMatrix(modelMatrix[0], modelMatrix[1], modelMatrix[2], modelMatrix[3],
modelMatrix[4], modelMatrix[5], modelMatrix[6], modelMatrix[7],
modelMatrix[8], modelMatrix[9], modelMatrix[10], modelMatrix[11],
modelMatrix[12], modelMatrix[13], modelMatrix[14], modelMatrix[15]
);
osg::Matrixd projectionMatrix(projetMatrix[0], projetMatrix[1], projetMatrix[2], projetMatrix[3],
projetMatrix[4], projetMatrix[5], projetMatrix[6], projetMatrix[7],
projetMatrix[8], projetMatrix[9], projetMatrix[10], projetMatrix[11],
projetMatrix[12], projetMatrix[13], projetMatrix[14], projetMatrix[15]
);
And :
camera->setViewMatrix(modelViewMatrix);
camera->setProjectionMatrix(projectionMatrix);
camera->setViewport(new osg::Viewport(iViewport[0], iViewport[1], iViewport[2], iViewport[3]));

GLUT torus colliding with camera

I want to implement collision of 6 torus which are randomly disturbed in the game area. It is a simple 3D space game using the perspective view and in first person. I saw some stack overflow answer suggesting to compute distance of whatever (player) to torus cell and if bigger than half or whole cell size you are colliding +/- your coordinate system and map topology tweaks. But if we take the distance that means we're only considering the z co-ordinates so if the camera moved to that distance (without considering x,y coordinates) it's always taking as a collision which is wrong right?
I'm hoping to do this using AABB algorithm. Is it ok to consider camera position and torus position as 2 boxes and check the collision (box to box collision) or camera as a point and torus as a box (point to box)? Or can somebody suggest best way to do that?
Below is the code that I've tried so far.
float im[16], m[16], znear = 0.1, zfar = 100.0, fovx = 45.0 * M_PI / 180.0;
glm::vec3 p0, p1, p2, p3, o, u, v;
//p0, p1, p2, p3 holds your znear camera screen corners in world coordinates
void ChangeSize(int w, int h)
{
GLfloat fAspect;
// Prevent a divide by zero
if(h == 0)
h = 1;
// Set Viewport to window dimensions
glViewport(0, 0, w, h);
// Calculate aspect ratio of the window
fAspect = (GLfloat)w*1.0/(GLfloat)h;
// Set the perspective coordinate system
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
// field of view of 45 degrees, near and far planes 1.0 and 1000
//that znear and zfar should typically have a ratio of 1000:1 to make sorting out z depth easier for the GPU
gluPerspective(45.0f, fAspect, 0.1f, 300.0f); //may need to make larger depending on project
// Modelview matrix reset
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
// get camera matrix (must be in right place in code before model transformations)
glGetFloatv(GL_MODELVIEW_MATRIX, im); // get camera inverse matrix
matrix_inv(m, im); // m = inverse(im)
u = glm::vec3(m[0], m[1], m[2]); // x axis
v = glm::vec3(m[4], m[5], m[6]); // y axis
o = glm::vec3(m[12], m[13], m[14]); // origin
o -= glm::vec3(m[8], m[9], m[10]) * znear; // z axis offset
// scale by FOV
u *= znear * tan(0.5 * fovx);
v *= znear * tan(0.5 * fovx / fAspect);
// get rectangle coorners
p0 = o - u - v;
p1 = o + u - v;
p2 = o + u + v;
p3 = o - u + v;
}
void matrix_inv(float* a, float* b) // a[16] = Inverse(b[16])
{
float x, y, z;
// transpose of rotation matrix
a[0] = b[0];
a[5] = b[5];
a[10] = b[10];
x = b[1]; a[1] = b[4]; a[4] = x;
x = b[2]; a[2] = b[8]; a[8] = x;
x = b[6]; a[6] = b[9]; a[9] = x;
// copy projection part
a[3] = b[3];
a[7] = b[7];
a[11] = b[11];
a[15] = b[15];
// convert origin: new_pos = - new_rotation_matrix * old_pos
x = (a[0] * b[12]) + (a[4] * b[13]) + (a[8] * b[14]);
y = (a[1] * b[12]) + (a[5] * b[13]) + (a[9] * b[14]);
z = (a[2] * b[12]) + (a[6] * b[13]) + (a[10] * b[14]);
a[12] = -x;
a[13] = -y;
a[14] = -z;
}
//Store torus coordinates
std::vector<std::vector<GLfloat>> translateTorus = { { 0.0, 1.0, -10.0, 1 }, { 0.0, 4.0, -6.0, 1 } , { -1.0, 0.0, -4.0, 1 },
{ 3.0, 1.0, -6.0, 1 }, { 1.0, -1.0, -9.0, 1 } , { 4.0, 1.0, -4.0, 1 } };
GLfloat xpos, ypos, zpos, flagToDisplayCrystal;
//Looping through 6 Torus
for (int i = 0; i < translateTorus.size(); i++) {
//Get the torus coordinates
xpos = translateTorus[i][0];
ypos = translateTorus[i][1];
zpos = translateTorus[i][2];
//This variable will work as a variable to display crystal after collision
flagToDisplayCrystal = translateTorus[i][3];
//p0 min, p2 max
//Creating a square using Torus index coordinates and radius
double halfside = 1.0 / 2;
//This (xpos+halfside), (xpos-halfside), (ypos+halfside), (ypos-halfside) are //created using Torus index and radius
float d1x = p0[0] - (xpos + halfside);
float d1y = p0[1] - (ypos + halfside);
float d2x = (xpos - halfside) - p2[0];
float d2y = (ypos - halfside) - p2[1];
//Collision is checking here
//For square's min z and max z is checking whether equal to camera's min //z and max z
if ((d1x > 0.0f || d1y > 0.0f || d2x > 0.0f || d2y > 0.0f) && p2[2] == zpos && p0[2] == zpos) {
//If there is collision update the variable as 0
translateTorus[i][3] = 0;
}
else {
if (flagToDisplayCrystal == 1) {
glPushMatrix();
glEnable(GL_TEXTURE_2D);
glTranslatef(xpos, ypos, zpos);
glRotatef(fPlanetRot, 0.0f, -1.0f, 0.0f);
glColor3f(0.0, 0.0, 0.0);
// Select the texture object
glBindTexture(GL_TEXTURE_2D, textures[3]);
glutSolidTorus(0.1, 1.0, 30, 30);
glDisable(GL_TEXTURE_2D);
glPopMatrix();
}
}
}
as I mentioned in the comments you got 2 options either use OpenGL rendering or compute entirely on CPU side without it. Let start with rendering first:
render your scene
but instead of color of torus and stuff use integer indexes (for example 0 empty space, 1 obstacle, 2 torus ...) you can even have separate indexes for each object in the world so you know exactly which one is hit etc ...
so: clear screen with empty color, render your scene (using indexes instead of color with glColor??(???)) without lighting or shading or whatever. But Do not swap buffers !!! as that would show the stuff on screen and cause flickering.
read rendered screen and depth buffers
you simply use glReadPixels to copy your screen and depth buffers into CPU side memory (1D arrays) lets call them scr[],zed[].
scan the scr[] for color matching torus indexes
simply loop through all pixels and if torus pixel found check its depth. If it is close enough to camera you found your collision.
render normally
now clear screen again and render your screen with colors and lighting... now you can swap buffers too.
Beware depth buffer will be non linear which requires linearization to obtain original depth in world units. For more about it and example of reading both scr,zed see:
depth buffer got by glReadPixels is always 1
OpenGL 3D-raypicking with high poly meshes
The other approach is is much faster in case you have not too many torus'es. You simply compute intersection between camera znear plane and torus. Which boils down to either AABB vs rectangle intersection or cylinder vs. rectangle intersection.
However if you not familiar with 3D vector math you might get lost quickly.
let assume the torus is described by AABB. Then intersection between that and rectangle boils down to checking intersection between line (each edge of AABB) and rectangle. So simply finding instersection between plane and line and checking if the point is inside rectangle.
if our rectangle is defined by its vertexes in CW or CCW order (p0,p1,p2,p3) and line by endpoints q0,q1 then:
n = normalize(cross(p1-p0,p2-p1)) // is rectangle normal
dq = normalize(q1-q0) // is line direction
q = q0 + dq*dot(dq,p1-p0) // is plane/line intersection
So now just check if q is inside rectangle. There are 2 ways either test if all crosses between q-edge_start and edge_end-edge_start have the same direction or all dots between all edge_normal and q-edge_point has the same sign or zero.
The problem is that both AABB and rectangle must be in the same coordinate system so either transform AABB into camera coordinates by using modelview matrix or transform the rectangle into world coordinates using inverse of modelview. The latter is better as you do it just once instead of transforming each torus'es AABB ...
For more info about math side see:
Cone to box collision
Understanding 4x4 homogenous transform matrices
The rectangle itself is just extracted from your camera matrix (part of modelviev) position, and x,y basis vectors gives you the "center" and axises of your rectangle... The size must be derived from the perspective matrix (or parameters you passed to it especially aspect ratio, FOV and znear)
Well first you need to obtain camera (view) matrix. The GL_MODELVIEW usually holds:
GL_MODELVIEW = Inverse(Camera)*Rendered_Object
so you need to find the place in your code where your GL_MODELVIEW holds just the Inverse(Camera) transformation and there place:
float aspect=float(xs)/float(ys); // aspect from OpenGL window resolution
float im[16],m[16],znear=0.1,zfar=100.0,fovx=60.0*M_PI/180.0;
vec3 p0,p1,p2,p3,o,u,v; // 3D vectors
// this is how my perspective is set
// glMatrixMode(GL_PROJECTION);
// glLoadIdentity();
// gluPerspective(fovx*180.0/(M_PI*aspect),aspect,znear,zfar);
// get camera matrix (must be in right place in code before model transformations)
glGetFloatv(GL_MODELVIEW_MATRIX,im); // get camera inverse matrix
matrix_inv(m,im); // m = inverse(im)
u =vec3(m[ 0],m[ 1],m[ 2]); // x axis
v =vec3(m[ 4],m[ 5],m[ 6]); // y axis
o =vec3(m[12],m[13],m[14]); // origin
o-=vec3(m[ 8],m[ 9],m[10])*znear; // z axis offset
// scale by FOV
u*=znear*tan(0.5*fovx);
v*=znear*tan(0.5*fovx/aspect);
// get rectangle coorners
p0=o-u-v;
p1=o+u-v;
p2=o+u+v;
p3=o-u+v;
// render it for debug
glColor3f(1.0,1.0,0.0);
glBegin(GL_QUADS);
glColor3f(1.0,0.0,0.0); glVertex3fv(p0.dat);
glColor3f(0.0,0.0,0.0); glVertex3fv(p1.dat);
glColor3f(0.0,0.0,1.0); glVertex3fv(p2.dat);
glColor3f(1.0,1.0,1.0); glVertex3fv(p3.dat);
glEnd();
Which basicaly loads the matrix into CPU side variables inverse it like this:
void matrix_inv(float *a,float *b) // a[16] = Inverse(b[16])
{
float x,y,z;
// transpose of rotation matrix
a[ 0]=b[ 0];
a[ 5]=b[ 5];
a[10]=b[10];
x=b[1]; a[1]=b[4]; a[4]=x;
x=b[2]; a[2]=b[8]; a[8]=x;
x=b[6]; a[6]=b[9]; a[9]=x;
// copy projection part
a[ 3]=b[ 3];
a[ 7]=b[ 7];
a[11]=b[11];
a[15]=b[15];
// convert origin: new_pos = - new_rotation_matrix * old_pos
x=(a[ 0]*b[12])+(a[ 4]*b[13])+(a[ 8]*b[14]);
y=(a[ 1]*b[12])+(a[ 5]*b[13])+(a[ 9]*b[14]);
z=(a[ 2]*b[12])+(a[ 6]*b[13])+(a[10]*b[14]);
a[12]=-x;
a[13]=-y;
a[14]=-z;
}
And compute the corners with perspective in mind as described above...
I used GLSL like vec3 but you can use any 3D math even own like float p0[3],.... You just need +,- and multiplying by constant.
Now the p0,p1,p2,p3 holds your znear camera screen corners in world coordinates.
[Edit1] example
I managed to put together simple example for this. Here support functiosn used first:
//---------------------------------------------------------------------------
void glutSolidTorus(float r,float R,int na,int nb) // render torus(r,R)
{
float *pnt=new float[(na+1)*(nb+1)*3*2]; if (pnt==NULL) return;
float *nor=pnt+((na+1)*(nb+1)*3);
float ca,sa,cb,sb,a,b,da,db,x,y,z,nx,ny,nz;
int ia,ib,i,j;
da=2.0*M_PI/float(na);
db=2.0*M_PI/float(nb);
glBegin(GL_LINES);
for (i=0,a=0.0,ia=0;ia<=na;ia++,a+=da){ ca=cos(a); sa=sin(a);
for ( b=0.0,ib=0;ib<=nb;ib++,b+=db){ cb=cos(b); sb=sin(b);
z=r*ca;
x=(R+z)*cb; nx=(x-(R*cb))/r;
y=(R+z)*sb; ny=(y-(R*sb))/r;
z=r*sa; nz=sa;
pnt[i]=x; nor[i]=nx; i++;
pnt[i]=y; nor[i]=ny; i++;
pnt[i]=z; nor[i]=nz; i++;
}}
glEnd();
for (ia=0;ia<na;ia++)
{
i=(ia+0)*(nb+1)*3;
j=(ia+1)*(nb+1)*3;
glBegin(GL_QUAD_STRIP);
for (ib=0;ib<=nb;ib++)
{
glNormal3fv(nor+i); glVertex3fv(pnt+i); i+=3;
glNormal3fv(nor+j); glVertex3fv(pnt+j); j+=3;
}
glEnd();
}
delete[] pnt;
}
//---------------------------------------------------------------------------
const int AABB_lin[]= // AABB lines
{
0,1,
1,2,
2,3,
3,0,
4,5,
5,6,
6,7,
7,4,
0,4,
1,5,
2,6,
3,7,
-1
};
const int AABB_fac[]= // AABB quads
{
3,2,1,0,
4,5,6,7,
0,1,5,4,
1,2,6,5,
2,3,7,6,
3,0,4,7,
-1
};
void AABBSolidTorus(vec3 *aabb,float r,float R) // aabb[8] = AABB of torus(r,R)
{
R+=r;
aabb[0]=vec3(-R,-R,-r);
aabb[1]=vec3(+R,-R,-r);
aabb[2]=vec3(+R,+R,-r);
aabb[3]=vec3(-R,+R,-r);
aabb[4]=vec3(-R,-R,+r);
aabb[5]=vec3(+R,-R,+r);
aabb[6]=vec3(+R,+R,+r);
aabb[7]=vec3(-R,+R,+r);
}
//---------------------------------------------------------------------------
void matrix_inv(float *a,float *b) // a[16] = Inverse(b[16])
{
float x,y,z;
// transpose of rotation matrix
a[ 0]=b[ 0];
a[ 5]=b[ 5];
a[10]=b[10];
x=b[1]; a[1]=b[4]; a[4]=x;
x=b[2]; a[2]=b[8]; a[8]=x;
x=b[6]; a[6]=b[9]; a[9]=x;
// copy projection part
a[ 3]=b[ 3];
a[ 7]=b[ 7];
a[11]=b[11];
a[15]=b[15];
// convert origin: new_pos = - new_rotation_matrix * old_pos
x=(a[ 0]*b[12])+(a[ 4]*b[13])+(a[ 8]*b[14]);
y=(a[ 1]*b[12])+(a[ 5]*b[13])+(a[ 9]*b[14]);
z=(a[ 2]*b[12])+(a[ 6]*b[13])+(a[10]*b[14]);
a[12]=-x;
a[13]=-y;
a[14]=-z;
}
//---------------------------------------------------------------------------
const int QUAD_lin[]= // quad lines
{
0,1,
1,2,
2,3,
3,0,
-1
};
const int QUAD_fac[]= // quad quads
{
0,1,2,3,
-1
};
void get_perspective_znear(vec3 *quad) // quad[4] = world coordinates of 4 corners of screen at znear distance from camera
{
vec3 o,u,v; // 3D vectors
float im[16],m[16],znear,zfar,aspect,fovx;
// get stuff from perspective
glGetFloatv(GL_PROJECTION_MATRIX,m); // get perspective projection matrix
zfar =0.5*m[14]*(1.0-((m[10]-1.0)/(m[10]+1.0)));// compute zfar from perspective matrix
znear=zfar*(m[10]+1.0)/(m[10]-1.0); // compute znear from perspective matrix
aspect=m[5]/m[0];
fovx=2.0*atan(1.0/m[5])*aspect;
// get stuff from camera matrix (must be in right place in code before model transformations)
glGetFloatv(GL_MODELVIEW_MATRIX,im); // get camera inverse matrix
matrix_inv(m,im); // m = inverse(im)
u =vec3(m[ 0],m[ 1],m[ 2]); // x axis
v =vec3(m[ 4],m[ 5],m[ 6]); // y axis
o =vec3(m[12],m[13],m[14]); // origin
o-=vec3(m[ 8],m[ 9],m[10])*znear; // z axis offset
// scale by FOV
u*=znear*tan(0.5*fovx);
v*=znear*tan(0.5*fovx/aspect);
// get rectangle coorners
quad[0]=o-u-v;
quad[1]=o+u-v;
quad[2]=o+u+v;
quad[3]=o-u+v;
}
//---------------------------------------------------------------------------
bool collideLineQuad(vec3 *lin,vec3 *quad) // return if lin[2] is colliding quad[4]
{
float t,l,u,v;
vec3 p,p0,p1,dp;
vec3 U,V,W;
// quad (rectangle) basis vectors
U=quad[1]-quad[0]; u=length(U); u*=u;
V=quad[3]-quad[0]; v=length(V); v*=v;
W=normalize(cross(U,V));
// convert line from world coordinates to quad local ones
p0=lin[0]-quad[0]; p0=vec3(dot(p0,U)/u,dot(p0,V)/v,dot(p0,W));
p1=lin[1]-quad[0]; p1=vec3(dot(p1,U)/u,dot(p1,V)/v,dot(p1,W));
dp=p1-p0;
// test if crossing the plane
if (fabs(dp.z)<1e-10) return false;
t=-p0.z/dp.z;
p=p0+(t*dp);
// test inside 2D quad (rectangle)
if ((p.x<0.0)||(p.x>1.0)) return false;
if ((p.y<0.0)||(p.y>1.0)) return false;
// inside line
if ((t<0.0)||(t>1.0)) return false;
return true;
}
//---------------------------------------------------------------------------
bool collideQuadQuad(vec3 *quad0,vec3 *quad1) // return if quad0[4] is colliding quad1[4]
{
int i;
vec3 l[2];
// lines vs. quads
for (i=0;QUAD_lin[i]>=0;)
{
l[0]=quad0[QUAD_lin[i]]; i++;
l[1]=quad0[QUAD_lin[i]]; i++;
if (collideLineQuad(l,quad1)) return true;
}
for (i=0;QUAD_lin[i]>=0;)
{
l[0]=quad1[QUAD_lin[i]]; i++;
l[1]=quad1[QUAD_lin[i]]; i++;
if (collideLineQuad(l,quad0)) return true;
}
// ToDo coplanar quads tests (not needed for AABB test)
return false;
}
//---------------------------------------------------------------------------
bool collideAABBQuad(vec3 *aabb,vec3 *quad) // return if aabb[8] is colliding quad[4]
{
int i;
vec3 q[4],n,p;
// test all AABB faces (rectangle) for intersection with quad (rectangle)
for (i=0;AABB_fac[i]>=0;)
{
q[0]=aabb[AABB_fac[i]]; i++;
q[1]=aabb[AABB_fac[i]]; i++;
q[2]=aabb[AABB_fac[i]]; i++;
q[3]=aabb[AABB_fac[i]]; i++;
if (collideQuadQuad(q,quad)) return true;
}
// test if one point of quad is fully inside AABB
for (i=0;AABB_fac[i]>=0;i+=4)
{
n=cross(aabb[AABB_fac[i+1]]-aabb[AABB_fac[i+0]],
aabb[AABB_fac[i+2]]-aabb[AABB_fac[i+1]]);
if (dot(n,quad[0]-aabb[AABB_fac[i+0]])>0.0) return false;
}
return true;
}
//---------------------------------------------------------------------------
And here the usage (during rendering):
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
int i;
float m[16];
mat4 m0,m1;
vec4 v4;
float aspect=float(xs)/float(ys);
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPerspective(60.0/aspect,aspect,0.1,20.0);
glMatrixMode(GL_TEXTURE);
glLoadIdentity();
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
static float anim=180.0; anim+=0.1; if (anim>=360.0) anim-=360.0;
glEnable(GL_DEPTH_TEST);
glDisable(GL_CULL_FACE);
vec3 line[2],quad[4],aabb[8]; // 3D vectors
get_perspective_znear(quad);
// store view matrix for latter
glMatrixMode(GL_MODELVIEW);
glGetFloatv(GL_MODELVIEW_MATRIX,m);
m0=mat4(m[0],m[1],m[2],m[3],m[4],m[5],m[6],m[7],m[8],m[9],m[10],m[11],m[12],m[13],m[14],m[15]);
m0=inverse(m0);
// <<-- here should be for start that loop through your toruses
glMatrixMode(GL_MODELVIEW);
glPushMatrix();
// set/animate torus position
glTranslatef(0.3,0.3,3.5*(-1.0-cos(anim)));
glRotatef(+75.0,0.5,0.5,0.0);
// get actual matrix and convert it to the change
glGetFloatv(GL_MODELVIEW_MATRIX,m);
m1=m0*mat4(m[0],m[1],m[2],m[3],m[4],m[5],m[6],m[7],m[8],m[9],m[10],m[11],m[12],m[13],m[14],m[15]);
// render torus and compute its AABB
glEnable(GL_LIGHTING);
glEnable(GL_LIGHT0);
glColor3f(1.0,1.0,1.0);
glutSolidTorus(0.1,0.5,36,36);
AABBSolidTorus(aabb,0.1,0.5);
glDisable(GL_LIGHT0);
glDisable(GL_LIGHTING);
// convert AABB to the same coordinates as quad
for (i=0;i<8;i++) aabb[i]=(m1*vec4(aabb[i],1.0)).xyz;
// restore original view matrix
glPopMatrix();
// render wireframe AABB
glColor3f(0.0,1.0,0.0);
glBegin(GL_LINES);
for (i=0;AABB_lin[i]>=0;i++)
glVertex3fv(aabb[AABB_lin[i]].dat);
glEnd();
/*
// render filled AABB for debug
glBegin(GL_QUADS);
for (i=0;AABB_fac[i]>=0;i++)
glVertex3fv(aabb[AABB_fac[i]].dat);
glEnd();
// render quad for debug
glBegin(GL_QUADS);
glColor3f(1.0,1.0,1.0);
for (i=0;QUAD_fac[i]>=0;i++)
glVertex3fv(quad[QUAD_fac[i]].dat);
glEnd();
*/
// render X on colision
if (collideAABBQuad(aabb,quad))
{
glColor3f(1.0,0.0,0.0);
glBegin(GL_LINES);
glVertex3fv(quad[0].dat);
glVertex3fv(quad[2].dat);
glVertex3fv(quad[1].dat);
glVertex3fv(quad[3].dat);
glEnd();
}
// <<-- here should be end of the for that loop through your toruses
glFlush();
SwapBuffers(hdc);
just ignore the GLUT solid torus function as you already got it ... Here preview:
The red cross indicates collision with screen ...

UnProject fails to get the world coordinate when scaling

In my SharpGL project (C#) I have used the Unproject function in order to get the world coordinates from mouse coordinates.
This procedure, quite trivial, fails when the drawing is scaled. I found many articles about this issue, but no one suited me.
When I say scaled means that in draw main proc i apply this code:
_gl.Scale(_params.ScaleFactor, _params.ScaleFactor, _params.ScaleFactor);
Then, when I intercept the mouse move I want to visualize the world coords. These coordinates are precise when the scale factor is 1, but when I change it these are wrong.
for example:
a world point (10, 10)
scaled 1 is detected (10, 10)
scaled 1,25 is detected (8, 8)
scaled 1,25 is detected (6.65, 6.65)
This is my simple code, consider that scale_factor is just passed for debugging.
public static XglVertex GetWorldCoords(this OpenGL gl, int x, int y, float scale_factor)
{
double worldX = 0;
double worldY = 0;
double worldZ = 0;
int[] viewport = new int[4];
double[] modelview = new double[16];
double[] projection = new double[16];
gl.GetDouble(OpenGL.GL_MODELVIEW_MATRIX, modelview); //get the modelview info
gl.GetDouble(OpenGL.GL_PROJECTION_MATRIX, projection); //get the projection matrix info
gl.GetInteger(OpenGL.GL_VIEWPORT, viewport); //get the viewport info
float winX = (float)x;
float winY = (float)viewport[3] - (float)y;
float winZ = 0;
//get the world coordinates from the screen coordinates
gl.UnProject(winX, winY, winZ, modelview, projection, viewport, ref worldX, ref worldY, ref worldZ);
XglVertex vres = new XglVertex((float)worldX, (float)worldY, (float)worldZ);
Debug.Print(string.Format("World Coordinate: x = {0}, y = {1}, z = {2}, sf = {3}", vres.X, vres.Y, vres.Z, scale_factor));
return vres;
}
I found a solution!
there were in the main draw procedure a portion of code that alterate results of UnProject function.
gl.PushMatrix();
.Translate(_mouse.CursorPosition.X * _params.ScaleFactor, _mouse.CursorPosition.Y * _params.ScaleFactor, _mouse.CursorPosition.Z * _params.ScaleFactor);
foreach (var el in _cursor.Elements) //objects to be drawn
{
gl.LineWidth(1f);
gl.Begin(el.Mode);
foreach (var v in el.Vertex)
{
gl.Color(v.Color.R, v.Color.G, v.Color.B, v.Color.A);
gl.Vertex(v.X, v.Y, v.Z);
}
gl.End();
}
gl.PopMatrix();

ray casting from mouse with opengl 2

I'm attempting to do ray casting on mouse click with the eventual goal of finding the collision point with a plane. However I'm unable to create the ray. The world is rendered using a frustum and another matrix I'm using as a camera, in the order of frustum * camera * vertex_position. With the top left of the screen as 0,0 I'm able to get the X,Y of the click in pixels. I then use the below code to convert this to the ray:
float x = (2.0f * x_screen_position) / width - 1.0f;
float y = 1.0f - (2.0f * y_screen_position) / height;
Vector4 screen_click = Vector4 (x, y, 1.0f, 1.0f);
Vector4 ray_origin_world = get_camera_matrix() * screen_click;
Vector4 tmp = (inverse(get_view_frustum()) * screen_click;
tmp = get_camera_matrix() * tmp;
Vector4 ray_direction = normalize(tmp);
view_frustum matrix:
Matrix4 view_frustum(float angle_of_view, float aspect_ratio, float z_near, float z_far) {
return Matrix4(
Vector4(1.0/tan(angle_of_view), 0.0, 0.0, 0.0),
Vector4(0.0, aspect_ratio/tan(angle_of_view), 0.0, 0.0),
Vector4(0.0, 0.0, (z_far+z_near)/(z_far-z_near), 1.0),
Vector4(0.0, 0.0, -2.0*z_far*z_near/(z_far-z_near), 0.0)
);
}
When the "camera" matrix is at 0,0,0 this gives the expected results however once I change to a fixed camera position in another location the results returned are not correct at all. The fixed "camera" matrix:
Matrix4(
Vector4(1.0, 0.0, 0.0, 0.0),
Vector4(0.0, 0.70710678118, -0.70710678118, 0.000),
Vector4(0.0, 0.70710678118, 0.70710678118, 0.0),
Vector4(0.0, 8.0, 20.0, 1.000)
);
Because many examples I have found online do not implement a camera in such a way I am unable to found much information to help in this case. Can anyone offer any insight into this or point me in a better direction?
Vector4 tmp = (inverse(get_view_frustum() * get_camera_matrix()) * screen_click; //take the inverse of the camera matrix as well
tmp /= tmp.w; //homogeneous coordinate "normalize" (different to typical normalization), needed with perspective projection or non-linear depth
Vector3 ray_direction = normalize(Vector3(tmp.x, tmp.y, tmp.z)); //make sure to normalize just the direction without w
[EDIT]
A more lengthy and similar post is here: https://stackoverflow.com/a/20143963/1888983
If you only have matrices, a start point and point in the ray direction should be used. It's common to use points on the near and far plane for this (an advantage is if you only want the ray to intersect things that are visible). That is,
(x, y, -1, 1) to (x, y, 1, 1)
These points are in normalized device coordinates (NDC, a -1 to 1 cube that is your viewing volume). All you need to do is move both points all the way to world space and normalize...
ndcPoint4 = /* from above */;
eyespacePoint4 = inverseProjectionMatrix * ndcPoint4;
worldSpacePoint4 = inverseCameraMatrix * eyespacePoint4;
worldSpacePoint3 = worldSpacePoint4.xyz / worldSpacePoint4.w;
//alternatively, with combined matrices
worldToClipMatrix = projectionMatrix * cameraMatrix; //called "clip" space before normalization
clipToWorldMatrix = inverse(worldToClipMatrix);
worldSpacePoint4 = clipToWorldMatrix * ndcPoint4;
worldSpacePoint3 = worldSpacePoint4.xyz / worldSpacePoint4.w;
//then for the ray, after transforming both start/end points
rayStart = worldPointOnNearPlane;
rayEnd = worldPointOnFarPlane;
rayDir = rayEnd - rayStart;
If you have the camera's world space position, you can drop either start or end point since all rays pass through the camera's origin.

OpenGL scene coordinates to screen coordinates

I'm having trouble converting my OpenGL scene coordiates to my screen coordinates.
I thought I needed to multiply my coordinates with the modelview matrix then with the projection matrix to get the ndc. But i'm getting weird coordinates.
Here is my piece of code
GLKVector3 coor = GLKVector3Make(point.x, point.y, 0);
GLKMatrix4 modelview = GLKMatrix4MakeWithArray(glProjectionMatrix);
GLKMatrix4 projetion = GLKMatrix4MakeWithArray(modelViewMatrix.data);
GLKVector3 eyeCoor = GLKMatrix4MultiplyVector3(modelview, coor);
GLKVector3 ndcCoor = GLKMatrix4MultiplyVector3(projetion,eyeCoor);
CGPoint p = CGPointMake(ndcCoor.x, ndcCoor.y);
Any idea ?
The code seems perfectly valid, but you should use 4D vectors for these homogeneous transforms.
So,
GLKVector4 coor = GLKVector4Make(point.x, point.y, 0, 1);
/// I hope those matrices are fine
GLKMatrix4 modelview = GLKMatrix4MakeWithArray(glProjectionMatrix);
GLKMatrix4 projetion = GLKMatrix4MakeWithArray(modelViewMatrix.data);
GLKVector4 eyeCoor = GLKMatrix4MultiplyVector4(modelview, coor);
GLKVector4 ndcCoor = GLKMatrix4MultiplyVector4(projetion,eyeCoor);
float XScr = ndcCoor.x / ndcCoor.w;
float YScr = ndcCoor.y / ndcCoor.w;
CGPoint p = CGPointMake(XScr, YScr);
If you want XScr and YScr to be in [0..1] range, then add
XScr = (XScr + 1.0f) * 0.5f;
YScr = (YScr + 1.0f) * 0.5f;
conversion.
Even easier: use the GLKit Math function GLKMathProject.
GLKVector3 GLKMathProject (
GLKVector3 object,
GLKMatrix4 model,
GLKMatrix4 projection,
int *viewport
);
So, in your case, e.g.
int viewport[] = {0, 0, 320, 480};
GLKVector3 windowVector = GLKMathProject(coor, modelview, projetion, viewport);
CGPoint p = CGPointMake(windowVector.x, windowVector.y);
Note that the origin is at lower left, so if you're using UIKit coordinates where the origin is at the upper left, then switch the y coordinate, e.g.
CGPoint p = CGPointMake(windowVector.x, window.bounds.size.height - windowVector.y);