How to get object coordinates from screen coordinates? - opengl

Question

I'm trying to get object space coordinates from the mouse position. I have some standard rendering code, which works well.
The problem is with the mouse picking code. I have tried lots of things and gone through similar questions but I can't seem to understand why it's not working.
I expect the result to return a x, y coordinates within [-1, 1] based on the position of the mouse over the object. I do get points within [-1, 1], but they are extremely skewed, such as (2.63813e-012, -1, 300).
Unproject code:
int z;
glReadPixels(mouse_pos_[0], int( navWidget->height() - mouse_pos_[1]), 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &z);
glm::vec3 win(mouse_pos_[0], navWidget->height() - mouse_pos_[1], z);
glm::vec4 viewport(0, 0, navWidget->width(), navWidget->height());
auto result_vec3 = glm::unProject(win, view * model1, proj, viewport);
auto result = glm::normalize(glm::vec2(result_vec3.x, result_vec3.y)); // < -- I normalize here since that gave good results without the translate
bool left_image = true;
if (!(result.x <= length_per_side && result.x >= -length_per_side &&
result.y <= length_per_side && result.y >= -length_per_side)) {
// do stuff
}
}
Rendering code:
float fov = 2*(atan((camProjModule->camResY()/2*camProjModule->camPixSizeY()) /
camProjModule->camFocalLength()) / M_PI * 180.0);
float znear = 1.0f;
float zfar = 6000.0f;
//float aspect = 1024.f / 683.f;
float aspect = navWidget->width() / navWidget->height();
glm::mat4 proj = glm::perspective(fov, aspect, znear, zfar);
float required_height =(float)( znear * tan((fov / 2.f) * M_PI / 180.f));
float eye_distance = znear / required_height * ((float)(navWidget->height()) / 2.f);
eye_distance = 300.f;
glm::mat4 view = glm::lookAt(glm::vec3(0.f, 0.f, 1.f * eye_distance), glm::vec3(0.f, 0.f, 0.f), glm::vec3(0.f, 1.f, 0.f));
glUseProgram(correspond_shader_);
glBindVertexArray(quad_vao_);
glUniform3f(colorLoc, 1.0f, 1.0f, 1.0f);
// draw left
if (left_correspond_texture_) {
glEnable(GL_TEXTURE_2D);
glActiveTexture(GL_TEXTURE0 + 0);
glBindTexture(GL_TEXTURE_2D, left_correspond_texture_);
glUniform1i(drawTexLoc, left_correspond_texture_);
}
GLint proj_loc = glGetUniformLocation(correspond_shader_, "proj");
GLint view_loc = glGetUniformLocation(correspond_shader_, "view");
GLint draw_tex_loc = glGetUniformLocation(correspond_shader_, "drawTex");
glUniformMatrix4fv(proj_loc, 1, GL_FALSE, glm::value_ptr(proj));
glUniformMatrix4fv(view_loc, 1, GL_FALSE, glm::value_ptr(view));
float ratio = 1024.f / 683.f;
float height = navWidget->height() / 2.f;
float ratio_to_multiply = height / 2.f;
glm::vec3 translation_vector = glm::vec3(0.f, height / 2.f, 0.f); // < --- If I remove this translation I get results that seem to be correct, and can be used after normalizing the x and y
glm::mat4 left_model = glm::scale(glm::translate(glm::mat4(1.f), translation_vector), glm::vec3(ratio * ratio_to_multiply, ratio_to_multiply, 1.f));
glm::mat4 right_model = glm::scale(glm::translate(glm::mat4(1.f), -1.f * translation_vector), glm::vec3(ratio * ratio_to_multiply, ratio_to_multiply, 1.f));
glUniformMatrix4fv(glGetUniformLocation(correspond_shader_, "model"), 1, GL_FALSE, glm::value_ptr(left_model));
glDrawArrays(GL_TRIANGLES, 0, 6); //, GL_UNSIGNED_INT, NULL);
EDIT: I think my question needs to be improved. I'm drawing two quads and rendering separate textures to it. What I want to do is get the mouse coordinates as normalized texture coordinates depending on which quad it is.

Answer 1

I see that you are using glm library. You can get mouse coordinate/ray direction using unprojection method.
glm::vec2 screenPos(mousePos.x, mousePos.y);
screenPos.y = height - screenPos.y;
float aspect = width / height;
glm::vec4 viewport = glm::vec4(0.0f, 0.0f, width , height);
glm::mat4 proj = glm::perspective(75.0f, aspect, 0.1f, 10000.0f);
glm::vec3 a (screenPos.x, screenPos.y, 0);
glm::vec3 b (screenPos.x, screenPos.y, 1);
glm::vec3 result = glm::unProject(a, viewMatrix, proj, viewport);
glm::vec3 result2 = glm::unProject(b, viewMatrix, proj, viewport);
glm::vec3 pickingPos = result;
glm::vec3 pickingDir = result2 - result;
After that you can use direction and position to check for collisions

Answer 2

I think CrSe's answer is right too. I have done this and I can pick any point on model:
I shoot a ray from these two points (p1 and p2):
Glu.gluUnProject(tempx, viewport[3] - tempy, 0, modelMatrix, projMatrix, viewport, out x1, out y1, out z1);
p = new Point(x1, y1, z1);
Glu.gluUnProject(tempx, viewport[3] - tempy, 1, modelMatrix, projMatrix, viewport, out x1, out y1, out z1);
p1 = new Point(x1, y1, z1);
if the distance btw this ray and a vertex is less than a threshold, I pick that point. I hope it is useful.