OpenCL producing QNaN on NVidia hardware - c++

Question

I'm programming in OpenCL using the C++ bindings. I have a problem where on NVidia hardware, my OpenCL code is spontaneously producing very large numbers, and then (on the next run) a "1.#QNaN". My code is pretty much a simple physics simulation using the equation x = vt * .5at^2. The only thing I've noticed odd about it is that the velocities of the particles suddenly shoot up to about 6e+34, which I'm guessing is the maximum floating point value on that machine. However, the velocities/forces before then are quite small, often with values less than 1.
The specific GPU I'm using is the Tesla M2050, with latest drivers. I prototype on my laptop using my AMD Fusion CPU as a platform (it does not have a dedicated GPU) and the problem does not occur there. I am not sure if this is an NVidia driver problem, a problem with my computation, or something else entirely.
Here is the kernel code (Note: I'm reasonably sure mass is always nonzero):
_kernel void update_atom(__global float4 *pos, __global float4 *vel, __global float4 *force,
__global const float *mass, __global const float *radius, const float timestep, const int wall)
{
// Get the index of the current element to be processed
int i = get_global_id(0);
float constMult;
float4 accel;
float4 part;
//Update the position, velocity and force
accel = (float4)(force[i].x/mass[i],
force[i].y/mass[i],
force[i].z/mass[i],
0.0f);
constMult = .5*timestep*timestep;
part = (float4)(constMult*accel.x,
constMult*accel.y,
constMult*accel.z, 0.0f);
pos[i] = (float4)(pos[i].x + vel[i].x*timestep + part.x,
pos[i].y + vel[i].y*timestep + part.y,
pos[i].z + vel[i].z*timestep + part.z,
0.0f);
vel[i] = (float4)(vel[i].x + accel.x*timestep,
vel[i].y + accel.y*timestep,
vel[i].z + accel.z*timestep,
0.0f);
force[i] = (float4)(force[i].x,
force[i].y,
force[i].z,
0.0f);
//Do reflections off the wall
//http://www.3dkingdoms.com/weekly/weekly.php?a=2
float4 norm;
float bouncePos = wall - radius[i];
float bounceNeg = radius[i] - wall;
norm = (float4)(0.0f, 0.0f, 0.0f, 0.0f);
if(pos[i].x >= bouncePos)
{
//Normal is unit YZ vector
pos[i].x = bouncePos;
norm.x = 1.0f;
}
else if(pos[i].x <= bounceNeg)
{
pos[i].x = bounceNeg;
norm.x = -1.0f;
}
if(pos[i].y >= bouncePos)
{
//Normal is unit XZ vector
pos[i].y = bouncePos;
norm.y = 1.0f;
}
else if(pos[i].y <= bounceNeg)
{
//etc
pos[i].y = bounceNeg;
norm.y = -1.0f;
}
if(pos[i].z >= bouncePos)
{
pos[i].z = bouncePos;
norm.z = 1.0f;
}
else if(pos[i].z <= bounceNeg)
{
pos[i].z = bounceNeg;
norm.z = -1.0f;
}
float dot = 2 * (vel[i].x * norm.x + vel[i].y * norm.y + vel[i].z * norm.z);
vel[i].x = vel[i].x - dot * norm.x;
vel[i].y = vel[i].y - dot * norm.y;
vel[i].z = vel[i].z - dot * norm.z;
}
And here's how I store the information into the kernel. PutData just uses the std::vector::push_back function on the positions, velocities and forces of the atoms into the corresponding vectors, and kernel is just a wrapper class I wrote for the OpenCL libraries (you can trust me that I put the right parameters into the right places for the enqueue functions).
void LoadAtoms(int kernelNum, bool blocking)
{
std::vector<cl_float4> atomPos;
std::vector<cl_float4> atomVel;
std::vector<cl_float4> atomForce;
for(int i = 0; i < numParticles; i++)
atomList[i].PutData(atomPos, atomVel, atomForce);
kernel.EnqueueWriteBuffer(kernelNum, posBuf, blocking, 0, numParticles*sizeof(cl_float4), &atomPos[0]);
kernel.EnqueueWriteBuffer(kernelNum, velBuf, blocking, 0, numParticles*sizeof(cl_float4), &atomVel[0]);
kernel.EnqueueWriteBuffer(kernelNum, forceBuf, blocking, 0, numParticles*sizeof(cl_float4), &atomForce[0]);
}
void LoadAtomTypes(int kernelNum, bool blocking)
{
std::vector<cl_float> mass;
std::vector<cl_float> radius;
int type;
for(int i = 0; i < numParticles; i++)
{
type = atomList[i].GetType();
mass.push_back(atomTypes[type].mass);
radius.push_back(atomTypes[type].radius);
}
kernel.EnqueueWriteBuffer(kernelNum, massBuf, blocking, 0, numParticles*sizeof(cl_float), &mass[0]);
kernel.EnqueueWriteBuffer(kernelNum, radiusBuf, blocking, 0, numParticles*sizeof(cl_float), &radius[0]);
}
There is more to my code, as always, but this is what's related to the kernel.
I saw this question, which is similar, but I use cl_float4 everywhere I can so I don't believe it's an issue of alignment. There aren't really any other related questions.
I realize this probably isn't a simple question, but I've run out of ideas until we can get new hardware in the office to test on. Can anyone help me out?

Answer 1

Since no one answered, I suppose I'll just contribute what I've learned so far.
I don't have a definitive conclusion, but at the very least, I figured out why it was doing it so often. Since I'm running this (and other similar kernels) to figure out an estimate for the order of time, I was clearing the lists, resizing them and then re-running the calculations. What I wasn't doing, however, was resizing the buffers. This resulted in some undefined numbers getting pulled by the threads.
However, this doesn't solve the QNaNs I get from running the program over long periods of time. Those simply appear spontaneously. Maybe it's a similar issue that I'm overlooking, but I can't say. If anyone has further input on the issue, it'd be appreciated.