UnProjected mouse coordinates are between 0-1 - c++

Question

I'm trying to create a ray from my mouse location out into 3D space, and apparently in order to do that I need to "UnProject()" it.
Doing so will give me a value between 0 & 1 for each axis.
This can't be right for drawing a "Ray" or a line from the viewport, can it? All this is, is a percentage essentially of my mouse to viewport size.
If this is actually right, then I don't understand the following:
I draw triangles that have vertices that are not constrained from 0-1, rather they are coordinates like (0,100,0), (100,100,0), (100,0,0), And these draw perfectly fine
But also, drawing the vertices that are unprojected from my mouse coordinates as lines/points also draw perfectly fine.
How the heck would I then compare my mouse coordinates to the coordinates of my objects?
If this is actually wrong, then what can cause such an error?
I tried unprojecting my own object's vertices, and those aren't constrained from 0-1.
I don't know whether or not the way I handle my "projections" when rendering is even compatible with gluUnproject. I've been just doing it the way these tutorials here show it (near bottom): http://qt-project.org/wiki/Developer-Guides#28810c65dd0f273a567b83a48839d275
This is the way I try to get my mouse coordinates:
GLdouble modelViewMatrix[16];
GLdouble projectionMatrix[16];
GLint viewport[4];
GLfloat winX, winY, winZ;
glGetDoublev(GL_MODELVIEW_MATRIX, modelViewMatrix);
glGetDoublev(GL_PROJECTION_MATRIX, projectionMatrix);
glGetIntegerv(GL_VIEWPORT, viewport);
winX = (float)x;
winY = (float)viewport[3] - (float)y;
glReadPixels( winX, winY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &winZ );
GLdouble nearPlaneLocation[3];
gluUnProject(winX, winY, 0, modelViewMatrix, projectionMatrix,
viewport, &nearPlaneLocation[0], &nearPlaneLocation[1],
&nearPlaneLocation[2]);
GLdouble farPlaneLocation[3];
gluUnProject(winX, winY, 1, modelViewMatrix, projectionMatrix,
viewport, &farPlaneLocation[0], &farPlaneLocation[1],
&farPlaneLocation[2]);
QVector3D nearP = QVector3D(nearPlaneLocation[0], nearPlaneLocation[1],
nearPlaneLocation[2]);
QVector3D farP = QVector3D(farPlaneLocation[0], farPlaneLocation[1],
farPlaneLocation[2]);
Perhaps my actual projections are off?
void oglWidget::paintGL()
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
QMatrix4x4 mMatrix;
QMatrix4x4 vMatrix;
QMatrix4x4 cameraTransformation;
cameraTransformation.rotate(alpha, 0, 1, 0);
cameraTransformation.rotate(beta, 1, 0, 0);
QVector3D cameraPosition = cameraTransformation * QVector3D(camX, camY, distance);
QVector3D cameraUpDirection = cameraTransformation * QVector3D(0, 1, 0);
vMatrix.lookAt(cameraPosition, QVector3D(camX, camY, 0), cameraUpDirection);
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
gluLookAt(cameraPosition.x(), cameraPosition.y(), cameraPosition.z(), camX, camY, 0, cameraUpDirection.x(), cameraUpDirection.y(), cameraUpDirection.z());
shaderProgram.bind();
shaderProgram.setUniformValue("mvpMatrix", pMatrix * vMatrix * mMatrix);
shaderProgram.setUniformValue("texture", 0);
for (int x = 0; x < tileCount; x++)
{
shaderProgram.setAttributeArray("vertex", tiles[x]->vertices.constData());
shaderProgram.enableAttributeArray("vertex");
shaderProgram.setAttributeArray("textureCoordinate", textureCoordinates.constData());
shaderProgram.enableAttributeArray("textureCoordinate");
//Triangle Drawing
glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, tiles[x]->image.width(), tiles[x]->image.height(), 0, GL_RGBA, GL_UNSIGNED_BYTE, tiles[x]->image.bits());
glDrawArrays(GL_TRIANGLES, 0, tiles[x]->vertices.size());
}
shaderProgram.release();
}
Where as, pMatrix is a 4x4 matrix, controlled during resize events like:
pMatrix.setToIdentity();
pMatrix.perspective(fov, (float) width / (float) height, 0.001, 10000);
glViewport(0, 0, width, height);
and my vertex shader is set up like this:
uniform mat4 mvpMatrix;
in vec4 vertex;
in vec2 textureCoordinate;
out vec2 varyingTextureCoordinate;
void main(void)
{
varyingTextureCoordinate = textureCoordinate;
gl_Position = mvpMatrix * vertex;
}

Answer 1

glReadPixels takes integers (x and y) and you don't seem to be using winZ for some reason in gluUnProject.
Try it like this:
gluUnProject(winX, winY, winZ, glView, glProjection, viewport, &posX, &posY, &posZ);
Also, if you want the ray to stop when it meets something in the depth buffer then don't clear the depth buffer after rendering. If you do a glClear(GL_DEPTH_BUFFER_BIT) then the ray should go as far as the far clip you set in your projection matrix.
I also have no idea why you need to call it more than once. The last three floats will be the target vector and you can just use your camera position as the source of the ray (depending on what you are doing).

Answer 2

Part of my problem here was poorly describing it. I accidentally left residual code from frantically testing, resulting in bits of "read Pixel" functions and related nonsense which wasn't useful for solving the problem.
The rest of my problem was due to inconsistent data types for the matrices, and trying to pull matrices from OpenGL when it never had them stored in the first place.
The problem was solved by:
Using GLM to hold all my matrices
performing the calculations myself (inverse view matrix * inverse model matrix * inverse projection matrix) * vector holding NDC converted screen space coordinates (range of -1 to 1: x or y divided by width or height, * 2 - 1), which also has a z of -1 or 1 for the far or near planes, and a w of 1.
Divide result by the fourth spot of the vector.
I still do not know why unprojecting doesn't work for me, as I got the wrong results with GLU as well as GLM's unproject function, but doing it manually worked for me.
Since my problem extended over quite a great length of time, and took up several questions, I owe credit to a few individuals who helped me along the way:
srobins of facepunch, in this thread
derhass from here, in this question, and this discussion