I need to do a limited form of ray tracing. I do not need reflections. I only need to change the color of a pixel, depending on how it passes by an object, and refraction. I also only need to test for intersections between the ray and spheres and disks, nothing else.
This is the main function in my shader:
void main(void)
{
Ray ray;
ray.origin=vec3(0.5,0.5,.75);
ray.direction=vec3(gl_FragCoord.x/width,gl_FragCoord.y/height,-gl_FragCoord.z)-ray.origin;
ray.direction=normalize(ray.direction);
gl_FragColor=trace(ray);
}
My first question is regarding the origin of the ray. How do I get its location? Right now, I just fiddle around until it looks right, but if I change the width or height of the screen I have to play around until it looks right.
My second question is about the intersection between a ray and a disk. I do this by first checking to see if the ray intersects a plane and then if the intersection point is within the radius of the disk.
My code looks like this
float intersectPlane(Ray ray,vec3 point,vec3 normal)
{
return dot(point-ray.origin,normal)/dot(ray.direction,normal);
}
...
det=intersectPlane(ray,bodies[count].position,vec3(0,0,1));
if(det>0)
{
if(distance(det*ray.direction,bodies[count].position)<=bodies[count].radius)
{
return vec4(1.0,0.0,0.0,1.0);
}
}
The problem is that if bodies[count].radius is less than or equal to the z-position of the ray's origin then nothing shows up. So
if(det>0)
{
if(distance(det*ray.direction,bodies[count].position)<=.76)
{
return vec4(1.0,0.0,0.0,1.0);
}
}
results in visible disks, while using the actual radius results in nothing.
As to your second question: don't use a distance, use a squared distance. It's faster processing, and I suspect it may solve your problem.
Origin of the ray really depends on you however I recommend you to specify the origin point such that the pixel positions are approximately equidistant from the origin and the objects.
Be careful about the direction of the ray meaning that the objects you are trying to see must be in front of the camera. (The rays that are sent must hit the objects.)
The intersection point of a ray and a plane is calculated as follows:
dist = dot( plane_origin - ray.origin, plane_NV ) / dot( ray.direction, plane_NV );
plane_isect = ray.origin + ray.direction * dist;
Your function intersectPlane calculates the distance from the origin of the ray to the intersection point at the plane correctly, but you do not calculate the intersection point before you compare it to the center of the disks.
To test if the intersection point is within the radius you have to do the following:
vec3 plane_isect = ray.origin + det * ray.direction;
if ( distance( plane_isect, bodies[count].position ) <= bodies[count].radius )
Adapt your code like this:
det = intersectPlane( ray, bodies[count].position, vec3(0,0,1) );
if ( det>0 )
{
vec3 plane_isect = ray.origin + det * ray.direction;
if ( distance( plane_isect, bodies[count].position ) <= bodies[count].radius )
{
return vec4(1.0,0.0,0.0,1.0);
}
}
Related
The situation si as follows. I am trying to implement a linear voxel search in a glsl shader for efficient voxel ray tracing. In toehr words, I have a 3D texture and I am ray tracing on it but I am trying to ray trace such that I only ever check voxels intersected by the ray once.
To this effect I have written a program with the following results:
Not efficient but correct:
The above image was obtained by adding a small epsilon ray multiple times and sampling from the texture on each iteration. Which produces the correct results but it's very inefficient.
That would look like:
loop{
start += direction*0.01;
sample(start);
}
To make it efficient I decided to instead implement the following lookup function:
float bound(float val)
{
if(val >= 0)
return voxel_size;
return 0;
}
float planeIntersection(vec3 ray, vec3 origin, vec3 n, vec3 q)
{
n = normalize(n);
if(dot(ray,n)!=0)
return (dot(q,n)-dot(n,origin))/dot(ray,n);
return -1;
}
vec3 get_voxel(vec3 start, vec3 direction)
{
direction = normalize(direction);
vec3 discretized_pos = ivec3((start*1.f/(voxel_size))) * voxel_size;
vec3 n_x = vec3(sign(direction.x), 0,0);
vec3 n_y = vec3(0, sign(direction.y),0);
vec3 n_z = vec3(0, 0,sign(direction.z));
float bound_x, bound_y, bound_z;
bound_x = bound(direction.x);
bound_y = bound(direction.y);
bound_z = bound(direction.z);
float t_x, t_y, t_z;
t_x = planeIntersection(direction, start, n_x,
discretized_pos+vec3(bound_x,0,0));
t_y = planeIntersection(direction, start, n_y,
discretized_pos+vec3(0,bound_y,0));
t_z = planeIntersection(direction, start, n_z,
discretized_pos+vec3(0,0,bound_z));
if(t_x < 0)
t_x = 1.f/0.f;
if(t_y < 0)
t_y = 1.f/0.f;
if(t_z < 0)
t_z = 1.f/0.f;
float t = min(t_x, t_y);
t = min(t, t_z);
return start + direction*t;
}
Which produces the following result:
Notice the triangle aliasing on the left side of some surfaces.
It seems this aliasing occurs because some coordinates are not being set to their correct voxel.
For example modifying the truncation part as follows:
vec3 discretized_pos = ivec3((start*1.f/(voxel_size)) - vec3(0.1)) * voxel_size;
Creates:
So it has fixed the issue for some surfaces and caused it for others.
I wanted to know if there is a way in which I can correct this truncation so that this error does not happen.
Update:
I have narrowed down the issue a bit. Observe the following image:
The numbers represent the order in which I expect the boxes to be visited.
As you can see for some of the points the sampling of the fifth box seems to be ommitted.
The following is the sampling code:
vec4 grabVoxel(vec3 pos)
{
pos *= 1.f/base_voxel_size;
pos.x /= (width-1);
pos.y /= (depth-1);
pos.z /= (height-1);
vec4 voxelVal = texture(voxel_map, pos);
return voxelVal;
}
yep that was the +/- rounding I was talking about in my comments somewhere in your previous questions related to this. What you need to do is having step equal to grid size in one of the axises (and test 3 times once for |dx|=1 then for |dy|=1 and lastly |dz|=1).
Also you should create a debug draw 2D slice through your map to actually see where the hits for a single specific test ray occurred. Now based on direction of ray in each axis you set the rounding rules separately. Without this you are just blindly patching one case and corrupting other two ...
Now actually Look at this (I linked it to your before but you clearly did not):
Wolf and Doom ray casting techniques
especially pay attention to:
On the right It shows you how to compute the ray step (your epsilon). You simply scale the ray direction so one of the coordinate is +/-1. For simplicity start with 2D slice through your map. The red dot is ray start position. Green is ray step vector for vertical grid lines hits and red is for horizontal grid lines hits (z will be analogically the same).
Now you should add the 2D overview of your map through some height slice that is visible (like on the image on the left) add a dot or marker to each intersection detected but distinguish between x,y and z hits by color. Do this for single ray only (I use the center of view ray). Fist handle view when you look at X+ directions than X- and when done move to Y,Z ...
In my GLSL volumetric 3D back raytracer I also linked you before look at these lines:
if (dir.x<0.0) { p+=dir*(((floor(p.x*n)-_zero)*_n)-ray_pos.x)/dir.x; nnor=vec3(+1.0,0.0,0.0); }
if (dir.x>0.0) { p+=dir*((( ceil(p.x*n)+_zero)*_n)-ray_pos.x)/dir.x; nnor=vec3(-1.0,0.0,0.0); }
if (dir.y<0.0) { p+=dir*(((floor(p.y*n)-_zero)*_n)-ray_pos.y)/dir.y; nnor=vec3(0.0,+1.0,0.0); }
if (dir.y>0.0) { p+=dir*((( ceil(p.y*n)+_zero)*_n)-ray_pos.y)/dir.y; nnor=vec3(0.0,-1.0,0.0); }
if (dir.z<0.0) { p+=dir*(((floor(p.z*n)-_zero)*_n)-ray_pos.z)/dir.z; nnor=vec3(0.0,0.0,+1.0); }
if (dir.z>0.0) { p+=dir*((( ceil(p.z*n)+_zero)*_n)-ray_pos.z)/dir.z; nnor=vec3(0.0,0.0,-1.0); }
they are how I did this. As you can see I use different rounding/flooring rule for each of the 6 cases. This way you handle case without corrupting the other. The rounding rule depends on a lot of stuff like how is your coordinate system offseted to (0,0,0) and more so it might be different in your code but the if conditions should be the same. Also as you can see I am handling this by offsetting the ray start position a bit instead of having these conditions inside the ray traversal loop castray.
That macro cast ray and look for intersections with grid and on top of that actually zsorts the intersections and use the first valid one (that is what l,ll are for and no other conditions or combination of ray results are needed). So my way of deal with this is cast ray for each type of intersection (x,y,z) starting on the first intersection with the grid for the same axis. You need to take into account the starting offset so the l,ll resembles the intersection distance to real start of ray not to offseted one ...
Also a good idea is to do this on CPU side first and when 100% working port to GLSL as in GLSL is very hard to debug things like this.
For picking objects, I've implemented a ray casting algorithm similar to what's described here. After converting the mouse click to a ray (with origin and direction) the next task is to intersect this ray with all triangles in the scene to determine hit points for each mesh.
I have also implemented the triangle intersection test algorithm based on the one described here. My question is, how should we account for the objects' transforms when performing the intersection? Obviously, I don't want to apply the transformation matrix to all vertices and then do the intersection test (too slow).
EDIT:
Here is the UnProject implementation I'm using (I'm using OpenTK by the way). I compared the results, they match what GluUnProject gives me:
private Vector3d UnProject(Vector3d screen)
{
int[] viewport = new int[4];
OpenTK.Graphics.OpenGL.GL.GetInteger(OpenTK.Graphics.OpenGL.GetPName.Viewport, viewport);
Vector4d pos = new Vector4d();
// Map x and y from window coordinates, map to range -1 to 1
pos.X = (screen.X - (float)viewport[0]) / (float)viewport[2] * 2.0f - 1.0f;
pos.Y = 1 - (screen.Y - (float)viewport[1]) / (float)viewport[3] * 2.0f;
pos.Z = screen.Z * 2.0f - 1.0f;
pos.W = 1.0f;
Vector4d pos2 = Vector4d.Transform(pos, Matrix4d.Invert(GetModelViewMatrix() * GetProjectionMatrix()));
Vector3d pos_out = new Vector3d(pos2.X, pos2.Y, pos2.Z);
return pos_out / pos2.W;
}
Then I'm using this function to create a ray (with origin and direction):
private Ray ScreenPointToRay(Point mouseLocation)
{
Vector3d near = UnProject(new Vector3d(mouseLocation.X, mouseLocation.Y, 0));
Vector3d far = UnProject(new Vector3d(mouseLocation.X, mouseLocation.Y, 1));
Vector3d origin = near;
Vector3d direction = (far - near).Normalized();
return new Ray(origin, direction);
}
You can apply the reverse transformation of each object to the ray instead.
I don't know if this is the best/most efficient approach, but I recently implemented something similar like this:
In world space, the origin of the ray is the camera position. In order to get the direction of the ray, I assumed the user had clicked on the near plane of the camera and thus applied the 'reverse transformation' - from screen space to world space - to the screen space position
( mouseClick.x, viewportHeight - mouseClick.y, 0 )
and then subtracted the origin of the ray, i.e. the camera position, from
the now transformed mouse click position.
In my case, there was no object-specific transformation, meaning I was done once I had my ray in world space. However, transforming origin & direction with the inverse model matrix would have been easy enough after that.
You mentioned that you tried to apply the reverse transformation, but that it didn't work - maybe there's a bug in there? I used a GLM - i.e. glm::unProject - for this.
I'm now working on the ray tracer, reflection part. I have everything working correctly, including creating a sphere with shadow. Now, I'm implementing the reflection part. However, I couldn't get it. My algorithm is below:
traceRay(Ray ray, int counter){
// look through the intersection between ray and list of objects
// find the final index aka the winning index, (if final index == -1, return background color)
// then calculate the intersection point
// perform reflection calculation here
if(counter > 1 && winning object's reflectivity > 1 ){
//get the intersection normal, vector N
//Calculate the reflection ray, R
// let I is the inverse of direction of incoming ray
//Calculate R = 2aN - I (a = N dotProduct I)
// the reflection ray is origin at the point of intersection between incoming ray and sphere with the R direction
Ray reflecRay (intersection_poisition, R);
Color reflection = traceRay(reflecRay, counter + 1);
// multiply by fraction ks
reflection = reflection * ks;
}
// the color of the sphere calculated using phong formula in shadeRay function
Color prefinal = shadeRay();
// return the total color of prefinal + reflection
}
I trying to get the reflection but couldn't get it, can anyone please let me know if my algorithm for traceRay function is correct?
When reflecting a ray, you need to move it along the reflector's normal to avoid intersection with the reflector itself. For example:
const double ERR = 1e-12;
Ray reflecRay (intersection_poisition + normal*ERR, R);
I am developing a small tool for 3D visualization of molecules.
For my project i choose to make a thing in the way of what Mr "Brad Larson" did with his Apple software "Molecules". A link where you can find a small presentation of the technique used : Brad Larsson software presentation
For doing my job i must compute sphere impostor and cylinder impostor.
For the moment I have succeed to do the "Sphere Impostor" with the help of another tutorial Lies and Impostors
for summarize the computing of the sphere impostor : first we send a "sphere position" and the "sphere radius" to the "vertex shader" which will create in the camera-space an square which always face the camera, after that we send our square to the fragment shader where we use a simple ray tracing to find which fragment of the square is included in the sphere, and finally we compute the normal and the position of the fragment to compute lighting. (another thing we also write the gl_fragdepth for giving a good depth to our impostor sphere !)
But now i am blocked in the computing of the cylinder impostor, i try to do a parallel between the sphere impostor and the cylinder impostor but i don't find anything, my problem is that for the sphere it was some easy because the sphere is always the same no matter how we see it, we will always see the same thing : "a circle" and another thing is that the sphere was perfectly defined by Math then we can find easily the position and the normal for computing lighting and create our impostor.
For the cylinder it's not the same thing, and i failed to find a hint to modeling a form which can be used as "cylinder impostor", because the cylinder shows many different forms depending on the angle we see it !
so my request is to ask you about a solution or an indication for my problem of "cylinder impostor".
In addition to pygabriels answer I want to share a standalone implementation using the mentioned shader code from Blaine Bell (PyMOL, Schrödinger, Inc.).
The approach, explained by pygabriel, also can be improved. The bounding box can be aligned in such a way, that it always faces to the viewer. Only two faces are visible at most. Hence, only 6 vertices (ie. two faces made up of 4 triangles) are needed.
See picture here, the box (its direction vector) always faces to the viewer:
Image: Aligned bounding box
For source code, download: cylinder impostor source code
The code does not cover round caps and orthographic projections. It uses geometry shader for vertex generation. You can use the shader code under the PyMOL license agreement.
I know this question is more than one-year old, but I'd still like to give my 2 cents.
I was able to produce cylinder impostors with another technique, I took inspiration from pymol's code. Here's the basic strategy:
1) You want to draw a bounding box (a cuboid) for the cylinder. To do that you need 6 faces, that translates in 18 triangles that translates in 36 triangle vertices. Assuming that you don't have access to geometry shaders, you pass to a vertex shader 36 times the starting point of the cylinder, 36 times the direction of the cylinder, and for each of those vertex you pass the corresponding point of the bounding box. For example a vertex associated with point (0, 0, 0) means that it will be transformed in the lower-left-back corner of the bounding box, (1,1,1) means the diagonally opposite point etc..
2) In the vertex shader, you can construct the points of the cylinder, by displacing each vertex (you passed 36 equal vertices) according to the corresponding points you passed in.
At the end of this step you should have a bounding box for the cylinder.
3) Here you have to reconstruct the points on the visible surface of the bounding box. From the point you obtain, you have to perform a ray-cylinder intersection.
4) From the intersection point you can reconstruct the depth and the normal. You also have to discard intersection points that are found outside of the bounding box (this can happen when you view the cylinder along its axis, the intersection point will go infinitely far).
By the way it's a very hard task, if somebody is interested here's the source code:
https://github.com/chemlab/chemlab/blob/master/chemlab/graphics/renderers/shaders/cylinderimp.frag
https://github.com/chemlab/chemlab/blob/master/chemlab/graphics/renderers/shaders/cylinderimp.vert
A cylinder impostor can actually be done just the same way as a sphere, like Nicol Bolas did it in his tutorial. You can make a square facing the camera and colour it that it will look like a cylinder, just the same way as Nicol did it for spheres. And it's not that hard.
The way it is done is ray-tracing of course. Notice that a cylinder facing upwards in camera space is kinda easy to implement. For example intersection with the side can be projected to the xz plain, it's a 2D problem of a line intersecting with a circle. Getting the top and bottom isn't harder either, the z coordinate of the intersection is given, so you actually know the intersection point of the ray and the circle's plain, all you have to do is to check if its inside the circle. And basically, that's it, you get two points, and return the closer one (the normals are pretty trivial too).
And when it comes to an arbitrary axis, it turns out to be almost the same problem. When you solve equations at the fixed axis cylinder, you are solving them for a parameter that describes how long do you have to go from a given point in a given direction to reach the cylinder. From the "definition" of it, you should notice that this parameter doesn't change if you rotate the world. So you can rotate the arbitrary axis to become the y axis, solve the problem in a space where equations are easier, get the parameter for the line equation in that space, but return the result in camera space.
You can download the shaderfiles from here. Just an image of it in action:
The code where the magic happens (It's only long 'cos it's full of comments, but the code itself is max 50 lines):
void CylinderImpostor(out vec3 cameraPos, out vec3 cameraNormal)
{
// First get the camera space direction of the ray.
vec3 cameraPlanePos = vec3(mapping * max(cylRadius, cylHeight), 0.0) + cameraCylCenter;
vec3 cameraRayDirection = normalize(cameraPlanePos);
// Now transform data into Cylinder space wherethe cyl's symetry axis is up.
vec3 cylCenter = cameraToCylinder * cameraCylCenter;
vec3 rayDirection = normalize(cameraToCylinder * cameraPlanePos);
// We will have to return the one from the intersection of the ray and circles,
// and the ray and the side, that is closer to the camera. For that, we need to
// store the results of the computations.
vec3 circlePos, sidePos;
vec3 circleNormal, sideNormal;
bool circleIntersection = false, sideIntersection = false;
// First check if the ray intersects with the top or bottom circle
// Note that if the ray is parallel with the circles then we
// definitely won't get any intersection (but we would divide with 0).
if(rayDirection.y != 0.0){
// What we know here is that the distance of the point's y coord
// and the cylCenter is cylHeight, and the distance from the
// y axis is less than cylRadius. So we have to find a point
// which is on the line, and match these conditions.
// The equation for the y axis distances:
// rayDirection.y * t - cylCenter.y = +- cylHeight
// So t = (+-cylHeight + cylCenter.y) / rayDirection.y
// About selecting the one we need:
// - Both has to be positive, or no intersection is visible.
// - If both are positive, we need the smaller one.
float topT = (+cylHeight + cylCenter.y) / rayDirection.y;
float bottomT = (-cylHeight + cylCenter.y) / rayDirection.y;
if(topT > 0.0 && bottomT > 0.0){
float t = min(topT,bottomT);
// Now check for the x and z axis:
// If the intersection is inside the circle (so the distance on the xz plain of the point,
// and the center of circle is less than the radius), then its a point of the cylinder.
// But we can't yet return because we might get a point from the the cylinder side
// intersection that is closer to the camera.
vec3 intersection = rayDirection * t;
if( length(intersection.xz - cylCenter.xz) <= cylRadius ) {
// The value we will (optianally) return is in camera space.
circlePos = cameraRayDirection * t;
// This one is ugly, but i didn't have better idea.
circleNormal = length(circlePos - cameraCylCenter) <
length((circlePos - cameraCylCenter) + cylAxis) ? cylAxis : -cylAxis;
circleIntersection = true;
}
}
}
// Find the intersection of the ray and the cylinder's side
// The distance of the point and the y axis is sqrt(x^2 + z^2), which has to be equal to cylradius
// (rayDirection.x*t - cylCenter.x)^2 + (rayDirection.z*t - cylCenter.z)^2 = cylRadius^2
// So its a quadratic for t (A*t^2 + B*t + C = 0) where:
// A = rayDirection.x^2 + rayDirection.z^2 - if this is 0, we won't get any intersection
// B = -2*rayDirection.x*cylCenter.x - 2*rayDirection.z*cylCenter.z
// C = cylCenter.x^2 + cylCenter.z^2 - cylRadius^2
// It will give two results, we need the smaller one
float A = rayDirection.x*rayDirection.x + rayDirection.z*rayDirection.z;
if(A != 0.0) {
float B = -2*(rayDirection.x*cylCenter.x + rayDirection.z*cylCenter.z);
float C = cylCenter.x*cylCenter.x + cylCenter.z*cylCenter.z - cylRadius*cylRadius;
float det = (B * B) - (4 * A * C);
if(det >= 0.0){
float sqrtDet = sqrt(det);
float posT = (-B + sqrtDet)/(2*A);
float negT = (-B - sqrtDet)/(2*A);
float IntersectionT = min(posT, negT);
vec3 Intersect = rayDirection * IntersectionT;
if(abs(Intersect.y - cylCenter.y) < cylHeight){
// Again it's in camera space
sidePos = cameraRayDirection * IntersectionT;
sideNormal = normalize(sidePos - cameraCylCenter);
sideIntersection = true;
}
}
}
// Now get the results together:
if(sideIntersection && circleIntersection){
bool circle = length(circlePos) < length(sidePos);
cameraPos = circle ? circlePos : sidePos;
cameraNormal = circle ? circleNormal : sideNormal;
} else if(sideIntersection){
cameraPos = sidePos;
cameraNormal = sideNormal;
} else if(circleIntersection){
cameraPos = circlePos;
cameraNormal = circleNormal;
} else
discard;
}
From what I can understand of the paper, I would interpret it as follows.
An impostor cylinder, viewed from any angle has the following characteristics.
From the top, it is a circle. So considering you'll never need to view a cylinder top down, you don't need to render anything.
From the side, it is a rectangle. The pixel shader only needs to compute illumination as normal.
From any other angle, it is a rectangle (the same one computed in step 2) that curves. Its curvature can be modeled inside the pixel shader as the curvature of the top ellipse. This curvature can be considered as simply an offset of each "column" in texture space, depending on viewing angle. The minor axis of this ellipse can be computed by multiplying the major axis (thickness of the cylinder) with a factor of the current viewing angle (angle / 90), assuming that 0 means you're viewing the cylinder side-on.
Viewing angles. I have only taken the 0-90 case into account in the math below, but the other cases are trivially different.
Given the viewing angle (phi) and the diameter of the cylinder (a) here's how the shader needs to warp the Y-Axis in texture space Y = b' sin(phi). And b' = a * (phi / 90). The cases phi = 0 and phi = 90 should never be rendered.
Of course, I haven't taken the length of this cylinder into account - which would depend on your particular projection and is not an image-space problem.
So I decided to write a ray tracer the other day, but I got stuck because I forgot all my vector math.
I've got a point behind the screen (the eye/camera, 400,300,-1000) and then a point on the screen (a plane, from 0,0,0 to 800,600,0), which I'm getting just by using the x and y values of the current pixel I'm looking for (using SFML for rendering, so it's something like 267,409,0)
Problem is, I have no idea how to cast the ray correctly. I'm using this for testing sphere intersection(C++):
bool SphereCheck(Ray& ray, Sphere& sphere, float& t)
{ //operator * between 2 vec3s is a dot product
Vec3 dist = ray.start - sphere.pos; //both vec3s
float B = -1 * (ray.dir * dist);
float D = B*B - dist * dist + sphere.radius * sphere.radius; //radius is float
if(D < 0.0f)
return false;
float t0 = B - sqrtf(D);
float t1 = B + sqrtf(D);
bool ret = false;
if((t0 > 0.1f) && (t0 < t))
{
t = t0;
ret = true;
}
if((t1 > 0.1f) && (t1 < t))
{
t = t1;
ret = true;
}
return ret;
}
So I get that the start of the ray would be the eye position, but what is the direction?
Or, failing that, is there a better way of doing this? I've heard of some people using the ray start as (x, y, -1000) and the direction as (0,0,1) but I don't know how that would work.
On a side note, how would you do transformations? I'm assuming that to change the camera angle you just adjust the x and y of the camera (or the screen if you need a drastic change)
The parameter "ray" in the function,
bool SphereCheck(Ray& ray, Sphere& sphere, float& t)
{
...
}
should already contain the direction information and with this direction you need to check if the ray intersects the sphere or not. (The incoming "ray" parameter is the vector between the camera point and the pixel the ray is sent.)
Therefore the local "dist" variable seems obsolete.
One thing I can see is that when you create your rays you are not using the center of each pixel in the screen as the point for building the direction vector. You do not want to use just the (x, y) coordinates on the grid for building those vectors.
I've taken a look at your sample code and the calculation is indeed incorrect. This is what you want.
http://www.csee.umbc.edu/~olano/435f02/ray-sphere.html (I took this course in college, this guy knows his stuff)
Essentially it means you have this ray, which has an origin and direction. You have a sphere with a point and a radius. You use the ray equation and plug it into the sphere equation and solve for t. That t is the distance between the ray origin and the intersection point on the spheres surface. I do not think your code does this.
So I get that the start of the ray would be the eye position, but what is the direction?
You have camera defined by vectors front, up, and right (perpendicular to each other and normalized) and "position" (eye position).
You also have width and height of viewport (pixels), vertical field of view (vfov) and horizontal field of view (hfov) in degrees or radians.
There are also 2D x and y coordinates of pixel. X axis (2D) points to the right, Y axis (2D) points down.
For a flat screen ray can be calculated like this:
startVector = eyePos;
endVector = startVector
+ front
+ right * tan(hfov/2) * (((x + 0.5)/width)*2.0 - 1.0)
+ up * tan(vfov/2) * (1.0 - ((y + 0.5f)/height)*2.0);
rayStart = startVector;
rayDir = normalize(endVector - startVector);
That assumes that screen plane is flat. For extreme field of view angles (fov >= 180 degreess) you might want to make screen plane spherical, and use different formulas.
how would you do transformations
Matrices.