C++: Simple CUDA volume reconstruction code crashing - c++

Question

I am currently working on a more comprehensive project involving CUDA. During the recent days I have been encountering errors that I have been desperately trying to bugfix. However, I couldn't figure it out, so I made up a minimal example now that shows the same behaviour. I have to say I am kind of new to CUDA. I am using Visual Studio 2015 and the CUDA Toolkit 7.5.
The program involves creating a 3D-volume on the GPU memory and then calculating values and writing them to the volume. I have tried to make the code as simple as possible:
First ist the main.cpp file:
#include "cuda_test.h"
int main() {
size_t const xDimension = 500;
size_t const yDimension = 500;
size_t const zDimension = 1000;
//allocate volume part memory on gpu
cudaPitchedPtr volume = ct::cuda::create3dVolumeOnGPU(xDimension, yDimension, zDimension);
//start reconstruction
ct::cuda::startReconstruction(volume,
xDimension,
yDimension,
zDimension);
return 0;
}
Then the cuda_test.h that is the header file for the actual .cu file:
#ifndef CT_CUDA
#define CT_CUDA
#include <cstdlib>
#include <stdio.h>
#include <cmath>
//CUDA
#include <cuda_runtime.h>
namespace ct {
namespace cuda {
cudaPitchedPtr create3dVolumeOnGPU(size_t xSize, size_t ySize, size_t zSize);
void startReconstruction(cudaPitchedPtr volume,
size_t xSize,
size_t ySize,
size_t zSize);
}
}
#endif
And then the cuda_test.cu file that contains the actual function implementations:
#include "cuda_test.h"
namespace ct {
namespace cuda {
cudaPitchedPtr create3dVolumeOnGPU(size_t xSize, size_t ySize, size_t zSize) {
cudaExtent extent = make_cudaExtent(xSize * sizeof(float), ySize, zSize);
cudaPitchedPtr ptr;
cudaMalloc3D(&ptr, extent);
printf("malloc3D: %s\n", cudaGetErrorString(cudaGetLastError()));
cudaMemset3D(ptr, 0, extent);
printf("memset: %s\n", cudaGetErrorString(cudaGetLastError()));
return ptr;
}
__device__ void addToVolumeElement(cudaPitchedPtr volumePtr, size_t ySize, size_t xCoord, size_t yCoord, size_t zCoord, float value) {
char* devicePtr = (char*)(volumePtr.ptr);
//z * xSize * ySize + y * xSize + x
size_t pitch = volumePtr.pitch;
size_t slicePitch = pitch * ySize;
char* slice = devicePtr + zCoord*slicePitch;
float* row = (float*)(slice + yCoord * pitch);
row[xCoord] += value;
}
__global__ void reconstructionKernel(cudaPitchedPtr volumePtr, size_t xSize, size_t ySize, size_t zSize) {
size_t xIndex = blockIdx.x;
size_t yIndex = blockIdx.y;
size_t zIndex = blockIdx.z;
if (xIndex == 0 && yIndex == 0 && zIndex == 0) {
printf("kernel start\n");
}
//just make sure we're inside the volume bounds
if (xIndex < xSize && yIndex < ySize && zIndex < zSize) {
//float value = z;
float value = sqrt(sqrt(sqrt(5.3))) * sqrt(sqrt(sqrt(1.2))) * sqrt(sqrt(sqrt(10.8))) + 501 * 0.125 * 0.786 / 5.3;
addToVolumeElement(volumePtr, ySize, xIndex, yIndex, zIndex, value);
}
if (xIndex == 0 && yIndex == 0 && zIndex == 0) {
printf("kernel end\n");
}
}
void startReconstruction(cudaPitchedPtr volumePtr, size_t xSize, size_t ySize, size_t zSize) {
dim3 blocks(xSize, ySize, zSize);
reconstructionKernel <<< blocks, 1 >>>(volumePtr,
xSize,
ySize,
zSize);
printf("Kernel launch: %s\n", cudaGetErrorString(cudaGetLastError()));
cudaDeviceSynchronize();
printf("Device synchronise: %s\n", cudaGetErrorString(cudaGetLastError()));
}
}
}
The function create3dVolumeOnGPU allocates a 3-dimensional "volume" in the gpu memory and returns a pointer to it. This is a host function. The second host function is startReconstruction. The only thing it does is launching the actual kernel with as many blocks as there are voxels in the volume. The kernel function is reconstructionKernel. It just calculates an arbitrary value out of some constants and then calls addToVolumeElement (device function) to write the result in the corresponding voxel (adding it).
Now, the problem is that it crashes. If I launch with debugger (NSight), NSight interrupts giving the error message:
CUDA grid launch failed: CUcontext: 2358451327088 CUmodule: 2358541519888 Function: _ZN2ct4cuda20reconstructionKernelE14cudaPitchedPtryyy
The console outputs:
malloc3D: no error
memset: no error
kernel started
kernel end
If I launch in release mode the whole machine resets.
However, if I change the volume dimensions to be smaller it works, for example:
size_t const xDimension = 100;
size_t const yDimension = 100;
size_t const zDimension = 100;
However, the amount of free GPU memory should not be the problem (card has 4GB VRAM).
It would be nice if someone could have a look at it and maybe give me a tip what could cause the problem.

Answer 1

Now, the problem is that it crashes
It would be nice if someone could have a look at it and maybe give me a tip what could cause the problem.
I think it's likely you are running into a WDDM TDR issue. On windows, any time a kernel running on a WDDM GPU takes more than about 2 seconds to execute, you may run into the WDDM TDR watchdog (assuming you haven't made any changes to the watchdog).
Furthermore, launching kernels like this:
reconstructionKernel <<< blocks, 1 >>>(...);
where the threads-per-block number is 1, means that only one thread in each warp (and in each block) is active. But the GPU likes to have 32 active threads per warp. So the net effect is inefficient utilization of the GPU resources; perhaps as much as 97% of the GPU horsepower sits idle when you run kernels this way.
So if your code is flexible enough to allow this:
reconstructionKernel <<< blocks, 1 >>>(...);
or equivalently this:
reconstructionKernel <<< blocks/256, 256 >>>(...);
(this is just a representative example; I realize you have a multidimensional grid, and the above probably isn't exactly relevant for your case)
then the second invocation method will almost certainly be more efficient, leading to a shorter execution time for the same work.
So I believe when you tested your code with multiple threads per block, you did something like the above, and it reduced the execution time below the TDR limit.
That's a perfectly fine solution, but if you end up adding more work to your kernel (more total threads, or more work per thread) then you may run into the limit again. In that case, the linked article explains a possible work-around.
As an aside, kernel launch configurations like this:
kernel<<<1, ?>>>(...);
or this:
kernel<<<?, 1>>>(...);
are never recommended for high performance code on the GPU.