Inside-Outside-Test of an Triangle - c++

I try to write an method bool intersect(const Ray& ray, Intersection& intersection) that returns true, when the Intersection is inside the Triangle.
What i've done so far , is check if there are Points on the Plane, that is created by 2 Vectors of the Triangle.
The Problem is now to check, if the Point is inside the Triangle.I use barycentric Coordinates
Vec3 AB = b_-a_;
Vec3 AC = c_-a_;
double areaABC = vec_normal_triangle.dot(AB.cross(AC));
Vec3 PB = b_-intersection.pos;
Vec3 PC = c_-intersection.pos;
double alpha = vec_normal_triangle.dot(PB.cross(PC));
Vec3 PA = a_-position.pos;
double beta = vec_normal_triangle.dot(PC.cross(PA));
double gamma = 1-alpha-beta;
if((beta+gamma) < 1 && beta > 0 && gamma > 0) {
return true;
}
Actually its not even a triangle, just about random Points.
Can someone explain me or knows how i compute the barycentric Coordinates for 3 given Vectors?

Assuming vec_normal_triangle is the vector computed as AB.cross(AC) normalized (in other words, the triangle's normal), you should divide alpha and beta by areaABC to get the barycentric coordinates of the intersecton point.
double alpha = vec_normal_triangle.dot(PB.cross(PC)) / areaABC;
and
double beta = vec_normal_triangle.dot(PC.cross(PA)) / areaABC;
This normalizes alpha and beta so that your computation of gamma and comparison against 1 make sense.
I'd also like to make a suggestion. To avoid recomputation and make the code a bit cleaner you could replace your test with the following.
if(alpha > 0 && beta > 0 && gamma > 0) {
return true;
}
Aside from that, I see that you first use intersection.pos and then position.pos. Is this intentional? My guess is that you need to use intersection.pos both times.

Related

Best way to interpolate triangle surface using 3 positions and normals for ray tracing

I am working on conventional Whitted ray tracing, and trying to interpolate surface of hitted triangle as if it was convex instead of flat.
The idea is to treat triangle as a parametric surface s(u,v) once the barycentric coordinates (u,v) of hit point p are known.
This surface equation should be calculated using triangle's positions p0, p1, p2 and normals n0, n1, n2.
The hit point itself is calculated as
p = (1-u-v)*p0 + u*p1 + v*p2;
I have found three different solutions till now.
Solution 1. Projection
The first solution I came to. It is to project hit point on planes that come through each of vertexes p0, p1, p2 perpendicular to corresponding normals, and then interpolate the result.
vec3 r0 = p0 + dot( p0 - p, n0 ) * n0;
vec3 r1 = p1 + dot( p1 - p, n1 ) * n1;
vec3 r2 = p2 + dot( p2 - p, n2 ) * n2;
p = (1-u-v)*r0 + u*r1 + v*r2;
Solution 2. Curvature
Suggested in a paper of Takashi Nagata "Simple local interpolation of surfaces using normal vectors" and discussed in question "Local interpolation of surfaces using normal vectors", but it seems to be overcomplicated and not very fast for real-time ray tracing (unless you precompute all necessary coefficients). Triangle here is treated as a surface of the second order.
Solution 3. Bezier curves
This solution is inspired by Brett Hale's answer. It is about using some interpolation of the higher order, cubic Bezier curves in my case.
E.g., for an edge p0p1 Bezier curve should look like
B(t) = (1-t)^3*p0 + 3(1-t)^2*t*(p0+n0*adj) + 3*(1-t)*t^2*(p1+n1*adj) + t^3*p1,
where adj is some adjustment parameter.
Computing Bezier curves for edges p0p1 and p0p2 and interpolating them gives the final code:
float u1 = 1 - u;
float v1 = 1 - v;
vec3 b1 = u1*u1*(3-2*u1)*p0 + u*u*(3-2*u)*p1 + 3*u*u1*(u1*n0 + u*n1)*adj;
vec3 b2 = v1*v1*(3-2*v1)*p0 + v*v*(3-2*v)*p2 + 3*v*v1*(v1*n0 + v*n2)*adj;
float w = abs(u-v) < 0.0001 ? 0.5 : ( 1 + (u-v)/(u+v) ) * 0.5;
p = (1-w)*b1 + w*b2;
Alternatively, one can interpolate between three edges:
float u1 = 1.0 - u;
float v1 = 1.0 - v;
float w = abs(u-v) < 0.0001 ? 0.5 : ( 1 + (u-v)/(u+v) ) * 0.5;
float w1 = 1.0 - w;
vec3 b1 = u1*u1*(3-2*u1)*p0 + u*u*(3-2*u)*p1 + 3*u*u1*( u1*n0 + u*n1 )*adj;
vec3 b2 = v1*v1*(3-2*v1)*p0 + v*v*(3-2*v)*p2 + 3*v*v1*( v1*n0 + v*n2 )*adj;
vec3 b0 = w1*w1*(3-2*w1)*p1 + w*w*(3-2*w)*p2 + 3*w*w1*( w1*n1 + w*n2 )*adj;
p = (1-u-v)*b0 + u*b1 + v*b2;
Maybe I messed something in code above, but this option does not seem to be very robust inside shader.
P.S. The intention is to get more correct origins for shadow rays when they are casted from low-poly models. Here you can find the resulted images from test scene. Big white numbers indicates number of solution (zero for original image).
P.P.S. I still wonder if there is another efficient solution which can give better result.
Keeping triangles 'flat' has many benefits and simplifies several stages required during rendering. Approximating a higher order surface on the other hand introduces quite significant tracing overhead and requires adjustments to your BVH structure.
When the geometry is being treated as a collection of facets on the other hand, the shading information can still be interpolated to achieve smooth shading while still being very efficient to process.
There are adaptive tessellation techniques which approximate the limit surface (OpenSubdiv is a great example). Pixar's Photorealistic RenderMan has a long history using subdivision surfaces. When they switched their rendering algorithm to path tracing, they've also introduced a pretessellation step for their subdivision surfaces. This stage is executed right before rendering begins and builds an adaptive triangulated approximation of the limit surface. This seems to be more efficient to trace and tends to use less resources, especially for the high-quality assets used in this industry.
So, to answer your question. I think the most efficient way to achieve what you're after is to use an adaptive subdivision scheme which spits out triangles instead of tracing against a higher order surface.
Dan Sunday describes an algorithm that calculates the barycentric coordinates on the triangle once the ray-plane intersection has been calculated. The point lies inside the triangle if:
(s >= 0) && (t >= 0) && (s + t <= 1)
You can then use, say, n(s, t) = nu * s + nv * t + nw * (1 - s - t) to interpolate a normal, as well as the point of intersection, though n(s, t) will not, in general, be normalized, even if (nu, nv, nw) are. You might find higher order interpolation necessary. PN-triangles were a similar hack for visual appeal rather than mathematical precision. For example, true rational quadratic Bezier triangles can describe conic sections.

Ray Tracing - Geometric Sphere Intersection - Intersection function returns true for all rays despite no intersection

I am writing a ray tracing project with C++ and OpenGL and am running into some obstacles with my sphere intersection function: I've checked multiple sources and the math looks right, but for some reason for every single ray, the intersection method is returning true. Here is the code to the sphere intersection function as well as some other code for clarification:
bool intersect(Vertex & origin, Vertex & rayDirection, float intersection)
{
bool insideSphere = false;
Vertex oc = position - origin;
float tca = 0.0;
float thcSquared = 0.0;
if (oc.length() < radius)
insideSphere = true;
tca = oc.dot(rayDirection);
if (tca < 0 && !insideSphere)
return false;
thcSquared = pow(radius, 2) - pow(oc.length(), 2) + pow(tca, 2);
if (thcSquared < 0)
return false;
insideSphere ? intersection = tca + sqrt(thcSquared) : intersection = tca - sqrt(thcSquared);
return true;
}
Here is some context from the ray tracing function that calls the intersection function. FYI my camera is at (0, 0, 0) and that is what is in my "origin" variable in the ray tracing function:
#define WINDOW_WIDTH 640
#define WINDOW_HEIGHT 480
#define WINDOW_METERS_WIDTH 30
#define WINDOW_METERS_HEIGHT 20
#define FOCAL_LENGTH 25
rayDirection.z = FOCAL_LENGTH * -1;
for (int r = 0; r < WINDOW_HEIGHT; r++)
{
rayDirection.y = (WINDOW_METERS_HEIGHT / 2 * -1) + (r * ((float)WINDOW_METERS_HEIGHT / (float)WINDOW_HEIGHT));
for (int c = 0; c < WINDOW_WIDTH; c++)
{
intersection = false;
t = 0.0;
rayDirection.x = (WINDOW_METERS_WIDTH / 2 * -1) + (c * ((float)WINDOW_METERS_WIDTH / (float)WINDOW_WIDTH));
rayDirection = rayDirection - origin;
for (int i = 0; i < NUM_SPHERES; i++)
{
if (spheres[i].intersect(CAM_POS, rayDirection, t))
{
intersection = true;
}
}
Thanks for taking a look and let me know if there is any other code that may help!
It seems you got your math a bit mixed. The first part of the function, ie until the first return false, is ok and will return false if the ray start outside of the sphere and don't go toward it. However, I think you put the camera outside all your spheres in such a manner that all spheres are visible, that's why this part never return false.
thcSquared is really wrong and I don't know what it is supposed to represent.
Let's do the intersection mathematically. We have:
origin : the start of the ray, let's call this A
rayDirection : the direction of the infinite ray, let's call this d.
position : the center of the sphere, called P
radius : self-explanatory, called r
What you want is a point on both the sphere and the line, let's call it M:
M = A + t * d because it is on the line
|M - P| = r because it is on the sphere
The second equation can be changed to be |A + t * d - P|² = r², which gives (A - P)² + 2 * t * (A - P).dot(d) + t²d² = r². This is a simple quadratic equation. Once solved, you have 0, 1 or 2 solutions, select the closest to the ray origin (but which is positive).
edit: You are forced to use another approach that I will detail here:
Compute the distance between the center of the sphere and the line (calling it l). This is done by 'projecting' the center on the line. So:
tca = ( (P - A) dot d ) / |d|, or with your variable names, tca = (OC dot rd) / |rd|. The projection is H = A + tca * d, and l = |H - P|.
If l > R then return false, there is no intersection.
Let's call M one intersection point. The triangle MHP have a right angle, so MH² + HP² = MP², in other terms thc² + l² = r², so we now have thc, the distance from H to the sphere.
With all that, t = tca +- thc, simply take the lowest non-negative of the two.
The paper you linked explain this, but without saying that it assumes the norm of the ray direction to be 1. I don't see a normalization in your code, that may be why your code fails (not verified).
Side note: the name Vertex for a 3d vector is really badly chosen, something like Vector3 or vec3 would be way better.

Linear/Nonlinear Texture Mapping a Distorted Quad

In my previous question, it was established that, when texturing a quad, the face is broken down into triangles and the texture coordinates interpolated in an affine manner.
Unfortunately, I do not know how to fix that. The provided link was useful, but it doesn't give the desired effect. The author concludes: "Note that the image looks as if it's a long rectangular quad extending into the distance. . . . It can become quite confusing . . . because of the "false depth perception" that this produces."
What I would like to do is to have the texturing preserve the original scaling of the texture. For example, in the trapezoidal case, I want the vertical spacing of the texels to be the same (example created with paint program):
Notice that, by virtue of the vertical spacing being identical, yet the quad's obvious distortion, straight lines in texture space are no longer straight lines in world space. Thus, I believe the required mapping to be nonlinear.
The question is: is this even possible in the fixed function pipeline? I'm not even sure exactly what the "right answer" is for more general quads; I imagine that the interpolation functions could get very complicated very fast, and I realize that "preserve the original scaling" isn't exactly an algorithm. World-space triangles are no longer linear in texture space.
As an aside, I do not really understand the 3rd and 4th texture coordinates; if someone could point me to a resource, that would be great.
The best approach to a solution working with modern gpu-api's can be found on Nathan Reed's blog.
But you end up with a problem similar to the original problem with the perspective :( I try to solve it and will post a solution once i have it.
Edit:
There is a working sample of a Quad-Texture on shadertoy.
Here is a simplified modified version I did, all honors deserved to Inigo Quilez:
float cross2d( in vec2 a, in vec2 b )
{
return a.x*b.y - a.y*b.x;
}
// given a point p and a quad defined by four points {a,b,c,d}, return the bilinear
// coordinates of p in the quad. Returns (-1,-1) if the point is outside of the quad.
vec2 invBilinear( in vec2 p, in vec2 a, in vec2 b, in vec2 c, in vec2 d )
{
vec2 e = b-a;
vec2 f = d-a;
vec2 g = a-b+c-d;
vec2 h = p-a;
float k2 = cross2d( g, f );
float k1 = cross2d( e, f ) + cross2d( h, g );
float k0 = cross2d( h, e );
float k2u = cross2d( e, g );
float k1u = cross2d( e, f ) + cross2d( g, h );
float k0u = cross2d( h, f);
float v1, u1, v2, u2;
float w = k1*k1 - 4.0*k0*k2;
w = sqrt( w );
v1 = (-k1 - w)/(2.0*k2);
u1 = (-k1u - w)/(2.0*k2u);
bool b1 = v1>0.0 && v1<1.0 && u1>0.0 && u1<1.0;
if( b1 )
return vec2( u1, v1 );
v2 = (-k1 + w)/(2.0*k2);
u2 = (-k1u + w)/(2.0*k2u);
bool b2 = v2>0.0 && v2<1.0 && u2>0.0 && u2<1.0;
if( b2 )
return vec2( u2, v2 )*.5;
return vec2(-1.0);
}
float sdSegment( in vec2 p, in vec2 a, in vec2 b )
{
vec2 pa = p - a;
vec2 ba = b - a;
float h = clamp( dot(pa,ba)/dot(ba,ba), 0.0, 1.0 );
return length( pa - ba*h );
}
vec3 hash3( float n )
{
return fract(sin(vec3(n,n+1.0,n+2.0))*43758.5453123);
}
//added for dithering
bool even(float a)
{
return fract(a/2.0) <= 0.5;
}
void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
vec2 p = (-iResolution.xy + 2.0*fragCoord.xy)/iResolution.y;
// background
vec3 bg = vec3(sin(iTime+(fragCoord.y * .01)),sin(3.0*iTime+2.0+(fragCoord.y * .01)),sin(5.0*iTime+4.0+(fragCoord.y * .01)));
vec3 col = bg;
// move points
vec2 a = sin( 0.11*iTime + vec2(0.1,4.0) );
vec2 b = sin( 0.13*iTime + vec2(1.0,3.0) );
vec2 c = cos( 0.17*iTime + vec2(2.0,2.0) );
vec2 d = cos( 0.15*iTime + vec2(3.0,1.0) );
// area of the quad
vec2 uv = invBilinear( p, a, b, c, d );
if( uv.x>-0.5 )
{
col = texture( iChannel0, uv ).xyz;
}
//mesh or screen door or dithering like in many sega saturn games
fragColor = vec4( col, 1.0 );
}
Wow this is quite a complex problem. Basically you aren't bringing perspective into the equation. Your final x,y coord are divided by the z value you get when you rotate. This means you get a smaller change in texture space (s,t) the further you get from the camera.
However by doing this you, effectively, interpolate linearly as z increases. I wrote an ANCIENT demo that did this back in 1997. This is called "affine" texture mapping.
Thing is because you are dividing x and y by z you actually need to interpolate your values using "1/z". This properly takes into account the perspective that is being applied (when you divide x & y) by z. Hence you end up with "perspective correct" texture mapping.
I wish I could go into more detail than that but over the last 15 odd years of alcohol abuse my memory has got a bit hazy ;) One of these days I'd love to re-visit software rendering as I'm convinced that with the likes of OpenCL taking off it will soon end up being a better way to write a rendering engine!
Anyway, hope thats some help and apologies I can't be more help.
(As an aside when I was figuring all that rendering out 15 years ago I'd loved to have had all the resources available to me now, the internet makes my job and life so much easier. Working everything out from first principles was incredibly painful. I recall initially trying the 1/z interpolation but it made thins smaller as it got closer and I gave up on it when I should've inverted things ... to this day, as my knowledge has increased exponentially, i STILL really wish i'd written a "slow" but fully perspective correct renderer ... one day I'll get round to it ;))
Good luck!

3D Line Segment and Plane Intersection

I'm trying to implement a line segment and plane intersection test that will return true or false depending on whether or not it intersects the plane. It also will return the contact point on the plane where the line intersects, if the line does not intersect, the function should still return the intersection point had the line segmenent had been a ray. I used the information and code from Christer Ericson's Real-time Collision Detection but I don't think im implementing it correctly.
The plane im using is derived from the normal and vertice of a triangle. Finding the location of intersection on the plane is what i want, regardless of whether or not it is located on the triangle i used to derive the plane.
The parameters of the function are as follows:
contact = the contact point on the plane, this is what i want calculated
ray = B - A, simply the line from A to B
rayOrigin = A, the origin of the line segement
normal = normal of the plane (normal of a triangle)
coord = a point on the plane (vertice of a triangle)
Here's the code im using:
bool linePlaneIntersection(Vector& contact, Vector ray, Vector rayOrigin, Vector normal, Vector coord) {
// calculate plane
float d = Dot(normal, coord);
if (Dot(normal, ray)) {
return false; // avoid divide by zero
}
// Compute the t value for the directed line ray intersecting the plane
float t = (d - Dot(normal, rayOrigin)) / Dot(normal, ray);
// scale the ray by t
Vector newRay = ray * t;
// calc contact point
contact = rayOrigin + newRay;
if (t >= 0.0f && t <= 1.0f) {
return true; // line intersects plane
}
return false; // line does not
}
In my tests, it never returns true... any ideas?
I am answering this because it came up first on Google when asked for a c++ example of ray intersection :)
The code always returns false because you enter the if here :
if (Dot(normal, ray)) {
return false; // avoid divide by zero
}
And a dot product is only zero if the vectors are perpendicular, which is the case you want to avoid (no intersection), and non-zero numbers are true in C.
Thus the solution is to negate ( ! ) or do Dot(...) == 0.
In all other cases there will be an intersection.
On to the intersection computation :
All points X of a plane follow the equation
Dot(N, X) = d
Where N is the normal and d can be found by putting a known point of the plane in the equation.
float d = Dot(normal, coord);
Onto the ray, all points s of a line can be expressed as a point p and a vector giving the direction D :
s = p + x*D
So if we search for which x s is in the plane, we have
Dot(N, s) = d
Dot(N, p + x*D) = d
The dot product a.b is transpose(a)*b.Let transpose(N) be Nt.
Nt*(p + x*D) = d
Nt*p + Nt*D*x = d (x scalar)
x = (d - Nt*p) / (Nt*D)
x = (d - Dot(N, p)) / Dot(N, D)
Which gives us :
float x = (d - Dot(normal, rayOrigin)) / Dot(normal, ray);
We can now get the intersection point by putting x in the line equation
s = p + x*D
Vector intersection = rayOrigin + x*ray;
The above code updated :
bool linePlaneIntersection(Vector& contact, Vector ray, Vector rayOrigin,
Vector normal, Vector coord) {
// get d value
float d = Dot(normal, coord);
if (Dot(normal, ray) == 0) {
return false; // No intersection, the line is parallel to the plane
}
// Compute the X value for the directed line ray intersecting the plane
float x = (d - Dot(normal, rayOrigin)) / Dot(normal, ray);
// output contact point
*contact = rayOrigin + normalize(ray)*x; //Make sure your ray vector is normalized
return true;
}
Aside 1:
What does the d value mean ?
For two vectors a and b a dot product actually returns the length of the orthogonal projection of one vector on the other times this other vector.
But if a is normalized (length = 1), Dot(a, b) is then the length of the projection of b on a. In case of our plane, d gives us the directional distance all points of the plane in the normal direction to the origin (a is the normal). We can then get whether a point is on this plane by comparing the length of the projection on the normal (Dot product).
Aside 2:
How to check if a ray intersects a triangle ? (Used for raytracing)
In order to test if a ray comes into a triangle given by 3 vertices, you first have to do what is showed here, get the intersection with the plane formed by the triangle.
The next step is to look if this point lies in the triangle. This can be achieved using the barycentric coordinates, which express a point in a plane as a combination of three points in it. See Barycentric Coordinates and converting from Cartesian coordinates
I could be wrong about this, but there are a few spots in the code that seem very suspicious. To begin, consider this line:
// calculate plane
float d = Dot(normal, coord);
Here, your value d corresponds to the dot product between the plane normal (a vector) and a point in space (a point on the plane). This seems wrong. In particular, if you have any plane passing through the origin and use the origin as the coordinate point, you will end up computing
d = Dot(normal, (0, 0, 0)) = 0
And immediately returning false. I'm not sure what you intended to do here, but I'm pretty sure that this isn't what you meant.
Another spot in the code that seems suspicious is this line:
// Compute the t value for the directed line ray intersecting the plane
float t = (d - Dot(normal, rayOrigin)) / Dot(normal, ray);
Note that you're computing the dot product between the plane's normal vector (a vector) and the ray's origin point (a point in space). This seems weird because it means that depending on where the ray originates in space, the scaling factor you use for the ray changes. I would suggest looking at this code one more time to see if this is really what you meant.
Hope this helps!
This all looks fine to me. I've independently checked the algebra and this looks fine for me.
As an example test case:
A = (0,0,1)
B = (0,0,-1)
coord = (0,0,0)
normal = (0,0,1)
This gives:
d = Dot( (0,0,1), (0,0,0)) = 0
Dot( (0,0,1), (0,0,-2)) = -2 // so trap for the line being in the plane passes.
t = (0 - Dot( (0,0,1), (0,0,1) ) / Dot( (0,0,1), (0,0,-2)) = ( 0 - 1) / -2 = 1/2
contact = (0,0,1) + 1/2 (0,0,-2) = (0,0,0) // as expected.
So given the emendation following #templatetypedef's answer, the only area where I can see a problem is with the implementation of one of the other operations, be it Dot(), or the Vector operators.
This version worked for me in OpenGL C# application.
bool GetLinePlaneIntersection(out vec3 contact, vec3 ray_origin, vec3 ray_end, vec3 normal, vec3 coord)
{
contact = new vec3();
vec3 ray = ray_end - ray_origin;
float d = glm.dot(normal, coord);
if (glm.dot(normal, ray) == 0)
{
return false;
}
float t = (d - glm.dot(normal, ray_origin)) / glm.dot(normal, ray);
contact = ray_origin + ray * t;
return true;
}

Generating a normal map from a height map?

I'm working on procedurally generating patches of dirt using randomized fractals for a video game. I've already generated a height map using the midpoint displacement algorithm and saved it to a texture. I have some ideas for how to turn that into a texture of normals, but some feedback would be much appreciated.
My height texture is currently a 257 x 257 gray-scale image (height values are scaled for visibility purposes):
My thinking is that each pixel of the image represents a lattice coordinate in a 256 x 256 grid (hence, why there are 257 x 257 heights). That would mean that the normal at coordinate (i, j) is determined by the heights at (i, j), (i, j + 1), (i + 1, j), and (i + 1, j + 1) (call those A, B, C, and D, respectively).
So given the 3D coordinates of A, B, C, and D, would it make sense to:
split the four into two triangles: ABC and BCD
calculate the normals of those two faces via cross product
split into two triangles: ACD and ABD
calculate the normals of those two faces
average the four normals
...or is there a much easier method that I'm missing?
Example GLSL code from my water surface rendering shader:
#version 130
uniform sampler2D unit_wave
noperspective in vec2 tex_coord;
const vec2 size = vec2(2.0,0.0);
const ivec3 off = ivec3(-1,0,1);
vec4 wave = texture(unit_wave, tex_coord);
float s11 = wave.x;
float s01 = textureOffset(unit_wave, tex_coord, off.xy).x;
float s21 = textureOffset(unit_wave, tex_coord, off.zy).x;
float s10 = textureOffset(unit_wave, tex_coord, off.yx).x;
float s12 = textureOffset(unit_wave, tex_coord, off.yz).x;
vec3 va = normalize(vec3(size.xy,s21-s01));
vec3 vb = normalize(vec3(size.yx,s12-s10));
vec4 bump = vec4( cross(va,vb), s11 );
The result is a bump vector: xyz=normal, a=height
My thinking is that each pixel of the image represents a lattice coordinate in a 256 x 256 grid (hence, why there are 257 x 257 heights). That would mean that the normal at coordinate (i, j) is determined by the heights at (i, j), (i, j + 1), (i + 1, j), and (i + 1, j + 1) (call those A, B, C, and D, respectively).
No. Each pixel of the image represents a vertex of the grid, so intuitively, from symmetry, its normal is determined by heights of neighboring pixels (i-1,j), (i+1,j), (i,j-1), (i,j+1).
Given a function f : ℝ2 → ℝ that describes a surface in ℝ3, a unit normal at (x,y) is given by
v = (−∂f/∂x, −∂f/∂y, 1) and n = v/|v|.
It can be proven that the best approximation to ∂f/∂x by two samples is archived by:
∂f/∂x(x,y) = (f(x+ε,y) − f(x−ε,y))/(2ε)
To get a better approximation you need to use at least four points, thus adding a third point (i.e. (x,y)) doesn't improve the result.
Your hightmap is a sampling of some function f on a regular grid. Taking ε=1 you get:
2v = (f(x−1,y) − f(x+1,y), f(x,y−1) − f(x,y+1), 2)
Putting it into code would look like:
// sample the height map:
float fx0 = f(x-1,y), fx1 = f(x+1,y);
float fy0 = f(x,y-1), fy1 = f(x,y+1);
// the spacing of the grid in same units as the height map
float eps = ... ;
// plug into the formulae above:
vec3 n = normalize(vec3((fx0 - fx1)/(2*eps), (fy0 - fy1)/(2*eps), 1));
A common method is using a Sobel filter for a weighted/smooth derivative in each direction.
Start by sampling a 3x3 area of heights around each texel (here, [4] is the pixel we want the normal for).
[6][7][8]
[3][4][5]
[0][1][2]
Then,
//float s[9] contains above samples
vec3 n;
n.x = scale * -(s[2]-s[0]+2*(s[5]-s[3])+s[8]-s[6]);
n.y = scale * -(s[6]-s[0]+2*(s[7]-s[1])+s[8]-s[2]);
n.z = 1.0;
n = normalize(n);
Where scale can be adjusted to match the heightmap real world depth relative to its size.
If you think of each pixel as a vertex rather than a face, you can generate a simple triangular mesh.
+--+--+
|\ |\ |
| \| \|
+--+--+
|\ |\ |
| \| \|
+--+--+
Each vertex has an x and y coordinate corresponding to the x and y of the pixel in the map. The z coordinate is based on the value in the map at that location. Triangles can be generated explicitly or implicitly by their position in the grid.
What you need is the normal at each vertex.
A vertex normal can be computed by taking an area-weighted average of the surface normals for each of the triangles that meet at that point.
If you have a triangle with vertices v0, v1, v2, then you can use a vector cross product (of two vectors that lie on two of the sides of the triangle) to compute a vector in the direction of the normal and scaled proportionally to the area of the triangle.
Vector3 contribution = Cross(v1 - v0, v2 - v1);
Each of your vertices that aren't on the edge will be shared by six triangles. You can loop through those triangles, summing up the contributions, and then normalize the vector sum.
Note: You have to compute the cross products in a consistent way to make sure the normals are all pointing in the same direction. Always pick two sides in the same order (clockwise or counterclockwise). If you mix some of them up, those contributions will be pointing in the opposite direction.
For vertices on the edge, you end up with a shorter loop and a lot of special cases. It's probably easier to create a border around your grid of fake vertices and then compute the normals for the interior ones and discard the fake borders.
for each interior vertex V {
Vector3 sum(0.0, 0.0, 0.0);
for each of the six triangles T that share V {
const Vector3 side1 = T.v1 - T.v0;
const Vector3 side2 = T.v2 - T.v1;
const Vector3 contribution = Cross(side1, side2);
sum += contribution;
}
sum.Normalize();
V.normal = sum;
}
If you need the normal at a particular point on a triangle (other than one of the vertices), you can interpolate by weighing the normals of the three vertices by the barycentric coordinates of your point. This is how graphics rasterizers treat the normal for shading. It allows a triangle mesh to appear like smooth, curved surface rather than a bunch of adjacent flat triangles.
Tip: For your first test, use a perfectly flat grid and make sure all of the computed normals are pointing straight up.