C++/CUDA: Calculating maximum gridSize and blockSize dynamically - c++

Question

I'm wanting to find a way to dynamically calculate the necessary grid and block size for a calculation. I have run into the issue that the problem that I am wanting to handle is simply too large to handle in a single run of the GPU from a thread limit perspective. Here is a sample kernel setup which runs into the error that I am having:
__global__ void populateMatrixKernel(char * outMatrix, const int pointsToPopulate)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < pointsToPopulate)
{
outMatrix[i] = 'A';
}
}
cudaError_t populateMatrixCUDA(char * outMatrix, const int pointsToPopulate, cudaDeviceProp &deviceProp)
{
//Device arrays to be used
char * dev_outMatrix = 0;
cudaError_t cudaStatus;
//THIS IS THE CODE HERE I'M WANTING TO REPLACE
//Calculate the block and grid parameters
auto gridDiv = div(pointsToPopulate, deviceProp.maxThreadsPerBlock);
auto gridX = gridDiv.quot;
if (gridDiv.rem != 0)
gridX++; //Round up if we have stragling points to populate
auto blockSize = deviceProp.maxThreadsPerBlock;
int gridSize = min(16 * deviceProp.multiProcessorCount, gridX);
//END REPLACE CODE
//Allocate GPU buffers
cudaStatus = cudaMalloc((void**)&dev_outMatrix, pointsToPopulate * sizeof(char));
if (cudaStatus != cudaSuccess)
{
cerr << "cudaMalloc failed!" << endl;
goto Error;
}
populateMatrixKernel << <gridSize, blockSize >> > (dev_outMatrix, pointsToPopulate);
//Check for errors launching the kernel
cudaStatus = cudaGetLastError();
if (cudaStatus != cudaSuccess)
{
cerr << "Population launch failed: " << cudaGetErrorString(cudaStatus) << endl;
goto Error;
}
//Wait for threads to finish
cudaStatus = cudaDeviceSynchronize();
if (cudaStatus != cudaSuccess) {
cerr << "cudaDeviceSynchronize returned error code " << cudaStatus << " after launching visit and bridger analysis kernel!" << endl;
cout << "Cuda failure " << __FILE__ << ":" << __LINE__ << " '" << cudaGetErrorString(cudaStatus);
goto Error;
}
//Copy output to host memory
cudaStatus = cudaMemcpy(outMatrix, dev_outMatrix, pointsToPopulate * sizeof(char), cudaMemcpyDeviceToHost);
if (cudaStatus != cudaSuccess) {
cerr << "cudaMemcpy failed!" << endl;
goto Error;
}
Error:
cudaFree(dev_outMatrix);
return cudaStatus;
}
Now, when I test this code using the following testing setup:
//Make sure we can use the graphics card (This calculation would be unresonable otherwise)
if (cudaSetDevice(0) != cudaSuccess) {
cerr << "cudaSetDevice failed! Do you have a CUDA-capable GPU installed?" << endl;
}
cudaDeviceProp deviceProp;
cudaError_t cudaResult;
cudaResult = cudaGetDeviceProperties(&deviceProp, 0);
if (cudaResult != cudaSuccess)
{
cerr << "cudaGetDeviceProperties failed!" << endl;
}
int pointsToPopulate = 250000 * 300;
auto gpuMatrix = new char[pointsToPopulate];
fill(gpuMatrix, gpuMatrix + pointsToPopulate, 'B');
populateMatrixCUDA(gpuMatrix, pointsToPopulate, deviceProp);
for (int i = 0; i < pointsToPopulate; ++i)
{
if (gpuMatrix[i] != 'A')
{
cout << "ERROR: " << i << endl;
cin.get();
}
}
I get an error at i=81920. Moreover, if I check the memory before and after the execution, all of the memory values after 81920 go from 'B' to null. It seems that this error is originating from this line in the kernel execution parameter code:
int gridSize = min(16 * deviceProp.multiProcessorCount, gridX);
For my graphics card (GTX 980M) I get out a value for deviceProp.multiProcessorCount of 5, and if I multiply this by 16 and 1024 (for max blocks per grid) I get out the 81920. It seems that, while I am fine on the memory space side of things, I am getting choked by how many threads I can run. Now, this 16 is just being set as an arbitrary value (after looking at some example code my friend made), I was wondering if there was a way to actually calculate "What 16 should be" based on the GPUs properties instead of setting it arbitrarily. I'm wanting to write an iterative code that is able to determine the maximum amount of calculations that are able to be performed at one point in time, and then fill the matrix piece by piece accordingly, but I need to know the maximum calculation value to do this. Does anyone know of a way to calculate these parameters? If any more information is needed, I'm happy to oblige. Thank you!

Answer 1

There is fundamentally nothing wrong with the code you have posted. It is probably close to best practice. But it isn't compatible with the design idiom of your kernel.
As you can see here, your GPU is capable of running 2^31 - 1 or 2147483647 blocks. So you could change the code in question to this:
unsigned int gridSize = min(2147483647u, gridX);
and it should probably work. Better still, don't change that code at all, but change your kernel to something like this:
__global__ void populateMatrixKernel(char * outMatrix, const int pointsToPopulate)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
for(; i < pointsToPopulate; i += blockDim.x * gridDim.x)
{
outMatrix[i] = 'A';
}
}
That way your kernel will emit multiple outputs per thread and everything should should just work as it is intended.