Performance optimization with different blocks and threads in CUDA - c++

Question

I've written a program to compute a histogram, where each of the 256 values for a char byte is counted:
#include "cuda_runtime.h"
#include "device_launch_parameters.h"
#include "..\..\common\book.h"
#include <stdio.h>
#include <cuda.h>
#include <conio.h>
#define SIZE (100*1024*1024)
__global__ void histo_kernel(unsigned char *buffer, long size, unsigned int *histo){
__shared__ unsigned int temp[256];
temp[threadIdx.x] = 0;
__syncthreads();
int i = threadIdx.x + blockIdx.x * blockDim.x;
int offset = blockDim.x * gridDim.x;
while (i < size) {
atomicAdd(&temp[buffer[i]], 1);
i += offset;}
__syncthreads();
atomicAdd(&(histo[threadIdx.x]), temp[threadIdx.x]);
}
int main()
{
unsigned char *buffer = (unsigned char*)big_random_block(SIZE);
cudaEvent_t start, stop;
cudaEventCreate(&start);
cudaEventCreate(&stop);
cudaEventRecord(start, 0);
unsigned char *dev_buffer;
unsigned int *dev_histo;
cudaMalloc((void**)&dev_buffer, SIZE);
cudaMemcpy(dev_buffer, buffer, SIZE, cudaMemcpyHostToDevice);
cudaMalloc((void**)&dev_histo, 256 * sizeof(long));
cudaMemset(dev_histo, 0, 256 * sizeof(int));
cudaDeviceProp prop;
cudaGetDeviceProperties(&prop, 0);
int blocks = prop.multiProcessorCount;
histo_kernel << <blocks * 256 , 256>> >(dev_buffer, SIZE, dev_histo);
unsigned int histo[256];
cudaMemcpy(&histo, dev_histo, 256 * sizeof(int), cudaMemcpyDeviceToHost);
cudaEventRecord(stop, 0);
cudaEventSynchronize(stop);
float elapsed_time;
cudaEventElapsedTime(&elapsed_time, start, stop);
printf("Time to generate: %f ms\n", elapsed_time);
long sum = 0;
for (int i = 0; i < 256; i++)
sum += histo[i];
printf("The sum is %ld", sum);
cudaFree(dev_buffer);
cudaFree(dev_histo);
free(buffer);
getch();
return 0;
}
I'ves read in the book, CUDA by example, that launching the kernel with number of blocks twice the number of multiprocessors is empirically found to be the most optimal solution. Yet, when I launch it with 8 times the number of blocks, the running time is cut down.
I've run the kernel with: 1.Blocks same as the number of multiprocessors, 2.Blocks twice the number of multiprocessors, 3.Blocks 4 times, and so on.
With (1), I got the running time to be 112ms
With (2) I got the running time to be 73ms
With (3) I got the running time to be 52ms
Funnily, after the number of blocks being 8 times the number of multiprocessors, the running time did not vary by a significant amount. Like it was the same for block being 8 times and 256 times and 1024 times the number of multiprocessors.
How can this be explained?

Answer 1

This behavior is typical. The GPU is a latency-hiding machine. In order to hide latency, when it hits a stall, it needs additional new work available. You can maximize the amount of additional new work available by giving the GPU a large number of blocks and threads.
Once you have given it enough work to hide latency as best it can, giving it additional work does not help. The machine is saturated. However, having additional work available is generally/typically not much of a detriment either. There is little overhead associated with blocks and threads.
Whatever you read in CUDA by Example may have been true for a specific case, but it is certainly not generally true that the correct number of blocks to launch is equal to twice the number of multiprocessors. A better target (typically) would be 4-8 blocks per multiprocessor.
When it comes to blocks and threads, more is usually better, and it's rarely the case that having arbitrarily large numbers of blocks and threads will actually cause a significant degradation in performance. This is contrary to typical CPU thread programming, where having large numbers of OMP threads, for example, may lead to a significant reduction in performance, when you exceed the core count.
When you are tuning the code for the last 10% in performance, then you will see people limit the amount of blocks they launch, to some number that is typically 4-8 times the number of SMs, and construct their threadblocks to loop over the data set. But this normally only yields a few percent performance improvement, in most cases. As a reasonable CUDA programming starting point, aim for tens of thousands of threads, and hundreds of blocks, at least. A carefully tuned code may be able to saturate the machine with fewer blocks and threads, but it will become GPU-dependent at that point. And as I've stated already, there's rarely much of a performance detriment to having millions of threads and thousands of blocks.