OpenGL picking - Ray/sphere intersection wrong - c++

Question

I have problem with my ray picking code. My code
I am using this code for picking calulation:
/*-----------------------------------------------------------
Function: GetViewportSystem
Returns:
viewportCoordSystem
Get viewport coordinate system (only for reading)
Forward ray goes through origin
-------------------------------------------------------------*/
ViewportCoordSystem Camera::GetViewportSystem() const
{
ViewportCoordSystem viewportCoord;
viewportCoord.w = this->cameraPos;
viewportCoord.w -= this->lookAt;
viewportCoord.w.Normalize();
viewportCoord.u = MyMath::Vector3::Cross(MyMath::Vector3::UnitY(), viewportCoord.w);
viewportCoord.v = MyMath::Vector3::Cross(viewportCoord.w, viewportCoord.u);
float d = (this->viewport.Height / 2.0f) * (1.0f / tanf(this->viewport.fov / 2.0f));
viewportCoord.origin = this->cameraPos;
viewportCoord.origin -= d * viewportCoord.w;
return viewportCoord;
}
/*-----------------------------------------------------------
Function: MapViewport2Dto3D
Parametrs:
[in] viewportSystem - cameras viewport coordinate system
[in] point - 2D point on image
Returns:
3D mapped point in space
Map 2D image point to 3D space
Info about mapping 2D to 3D: http://meatfighter.com/juggler/
-------------------------------------------------------------*/
MyMath::Vector3 Camera::MapViewport2Dto3D(const ViewportCoordSystem & viewportSystem, const MyMath::Vector2 & point) const
{
MyMath::Vector3 res = viewportSystem.origin;
res += (point.X - this->viewport.Width * 0.5f) * viewportSystem.u;
res += (this->viewport.Height * 0.5f - point.Y) * viewportSystem.v;
return res;
}
Picking itself
ViewportCoordSystem vpSystem = this->camera->GetViewportSystem();
MyMath::Vector3 pos = this->camera->MapViewport2Dto3D(vpSystem, MyMath::Vector2(mouseX, mouseY));
this->ray.dir = pos - this->camera->GetPosition();
this->ray.dir.Normalize();
this->ray.origin = this->camera->GetPosition();
With this ray, I calculate ray - sphere intersection test.
bool BoundingSphere::RayIntersection(const MyMath::Ray & ray) const
{
MyMath::Vector3 Q = this->sphereCenter - ray.origin;
double c = Q.LengthSquared();
double v = MyMath::Vector3::Dot(Q, ray.dir);
double d = this->sphereRadius * this->sphereRadius - (c - v * v);
if (d < 0.0) return false;
return true;
}
Problem is, that my code works incorrect. If I visualuse my spheres, and click inside them, I got correct answer only for half of the sphere. When I move camera, than its all messed up and picking reacts outside spheres.
My world is not transformed (all world matrices are Identity). Only camera is moving. I calculate mouse position within OpenGL window correctly (upper left corner has [0, 0] and goes to [width, height]).
PS: I am using this code succesfully in DirectX for raycasting / raytracing. And I cant see anything wrong with it. My OpenGL renderer is using Left-Handed system (its not natural for OpenGL, but I want it that way)
Edit:
After visualizing the ray, problem is apearing, when I mov cameraleft / right. Center of ray is not cohherent with mouse position.

Answer 1

Ok.. found the problem... for anybody else, who might be interessted
Those wo lines are incorrect
viewportCoord.u = MyMath::Vector3::Cross(MyMath::Vector3::UnitY(), viewportCoord.w);
viewportCoord.v = MyMath::Vector3::Cross(viewportCoord.w, viewportCoord.u);
Working solution is
viewportCoord.u = MyMath::Vector3::Cross(viewportCoord.w, MyMath::Vector3::UnitY());
viewportCoord.u.Normalize();
viewportCoord.v = MyMath::Vector3::Cross(viewportCoord.u, viewportCoord.w);
viewportCoord.v.Normalize();