Problem converting screen coordinates to OpenGL world coordinates - c++

Question

I'm using C++/Qt/OpenGL 4.3 to implement an OpenGL viewer and I'm stuck on converting mouse coordinates to world coordinates.
Update: After reading the comments and answer, I found this code to work correctly:
float depth;
double mouseX;
double mouseY;
_app->getCurPos(mouseX, mouseY);
glReadPixels(mouseX, _h - mouseY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, (void *)&depth);
float x = (2.0f * mouseX) / _w - 1.0f;
float y = 1.0f - (2.0f * mouseY) / _h;
float z = depth * 2.0 - 1.0; // convert to NDC
QVector4D pos(x, y, z, 1.0);
QMatrix4x4 invVM(_app->camera()->ViewMatrix().inverted());
QMatrix4x4 invPM(_app->camera()->ProjectionMatrix().inverted());
QVector4D world = invVM * invPM * pos;
world /= world.w();
qDebug() << "world" << world;
(from the original question...)
Where does this code go wrong? Its output is included below.
Code:
// mouse coordinates to world coordinates
QVector3D GLCamera::transformScreen(float mouseX, float mouseY)
{
// our 3D view has Y as "UP"
float x = (2.0f * mouseX) / _w - 1.0f;
float y = 0.0f;
float z = 1.0f - (2.0f * mouseY) / _h;
// hard code NDC to upper-right of screen
x = 1.0;
y = 0.0;
z = 1.0;
QVector4D ndc = QVector4D(x, y, z, 1);
QVector4D point1 = ndc * mProjectionMatrix.inverted();
qDebug() << point1;
QVector3D point3D = QVector3D(point1) / point1.w();
qDebug() << point3D;
QVector3D point2 = point3D * mViewMatrix.inverted();
qDebug() << point2;
qDebug() << "mViewMatrix: " << mViewMatrix;
qDebug() << "inv : " << mViewMatrix.inverted();
qDebug() << "mProjMatrix: " << mProjectionMatrix;
qDebug() << "inv : " << mProjectionMatrix.inverted();
return point2;
}
Output:
qDebug: QVector4D(1, 0, 1, 1)
qDebug: QVector4D(0.742599, 0, -49.995, 49.005)
qDebug: QVector3D(0.0151535, 0, -1.0202)
# This is the returned value x seems so much smaller than z
qDebug: QVector3D(-0.000938916, -0.0418856, 0.0473429)
qDebug: mViewMatrix: QMatrix4x4(type:Translation,Rotation
1 0 0 0
0 0.748955 -0.662621 -0.626549
0 0.662621 0.748955 -22.9856
0 0 0 1
)
qDebug: inv : QMatrix4x4(type:Translation,Rotation
1 0 0 0
0 0.748955 0.662621 15.7
0 -0.662621 0.748955 16.8
0 0 0 1
)
qDebug: mProjMatrix: QMatrix4x4(type:General
1.34662 0 0 0
0 2.41421 0 0
0 0 -1.0002 -0.020002
0 0 -1 0
)
qDebug: inv : QMatrix4x4(type:General
0.742599 0 0 0
0 0.414214 0 0
0 0 0 -1
0 0 -49.995 50.005
)

Answer 1

The top right corner of the screen is (1, 1, depth) in NDC. You need a reasonable depth value to get the point that you want. Otherwise, you will only get a ray.
The matrices you show are supposed to be used for right-multiplication:
QVector4D point1 = mProjectionMatrix.inverted() * ndc;
QVector3D point2 = mViewMatrix.inverted() * point3D;
The perspective divide should happen at the very end and not in between. Up to this point, continue working with 4D vectors. Otherwise, passing the view-space position (point1) as a QVector3D will set a w-component of 0 and you will lose any translations. You can pass it as a QPoint3D, which would set w=1, but keeping the 4D vector is the safest choice.
QVector3D point3D = QVector3D(point1) / point1.w();
I would also suggest, naming the intermediate results reasonably (not just pointX). Name them pointClipSpace, pointViewSpace, pointWorldSpace or similar.
Again, deciding for an NDC depth is very important to get an interpretable result.