Measuring OpenCL kernel's memory throughput - profiling

Question

I read about global memory optimization in OpenCL. In one of the slide-shows, a very simple kernel (below) has been used to demonstrate the importance of memory coalescing.
__kernel void measure(__global float* idata, __global float* odata, int offset) {
int xid = get_global_id(0) + offset;
odata[xid] = idata[xid];
}
Please see my code below which measures the running time of the kernel
ret = clFinish(command_queue);
size_t local_item_size = MAX_THREADS;
size_t global_item_size = INPUTSIZE;
struct timeval t0,t1;
gettimeofday(&t0, 0 );
//ret = clFinish(command_queue);
ret = clEnqueueNDRangeKernel(command_queue, measure, 1, NULL,
&global_item_size, &local_item_size, 0, NULL, NULL);
ret = clFlush(command_queue);
ret = clFinish(command_queue);
gettimeofday(&t1,0);
double elapsed = (t1.tv_sec-t0.tv_sec)*1000000 + (t1.tv_usec-t0.tv_usec);
printf("time taken = %lf microseconds\n", elapsed);
I transfer around 0.5 GB of data:
#define INPUTSIZE 1024 * 1024 * 128
int main (int argc, char *argv[])
{
int offset = atoi(argv[1]);
float* input = (float*) malloc(sizeof(float) * INPUTSIZE);
Now, the results are a bit random. With offset =0, I get times as low as 21 usecs. With offset = 1, I get times ranging between 53 usecs to 24400 usecs.
Can someone please tell me what is going on. I thought that offset=0 will be the fastest, because all the threads will access consecutive locations, hence the minimum number of memory transactions will take place.

Answer 1

Bandwidth is a measure of how fast data can be transferred, and is typically measured in bytes/second in these situations (usually GB/s for GPU memory bandwidth).
To compute the bandwidth of a compute kernel, you just need to know how much data the kernel reads/writes from/to memory, and then divide that by the time your kernel took to execute.
Your example kernel has each work-item (or CUDA thread) read a single float, and write a single float. If you launch this kernel to copy 2^10 floats, then you will be reading 2^10 * sizeof(float) bytes, and writing the same amount (so 8MB in total). If this kernel takes 1ms to execute, then you have achieved bandwidth of 8MB / 0.001s = 8GB/s.
Your new code snippet that shows your kernel timing approach indicates that you are only timing the kernel enqueue, not the amount of time it actually takes to run the kernel. This is why you are getting very low kernel timings (0.5GB / 0.007ms ~= 71TB/s!). You should add calls to clFinish() to obtain proper timing. I typically also take timings over several runs, to allow the device to warm-up, which usually gives more consistent timing:
// Warm-up run (not timed)
clEnqueueNDRangeKernel(command_queue, ...);
clFinish(command_queue);
// start timing
start = ...
for (int i = 0; i < NUM_RUNS; i++)
{
clEnqueueNDRangeKernel(command_queue, ...);
}
clFinish(command_queue);
// stop timing
end = ...
// Compute time taken, bandwidth etc
average_time = (end-start)/NUM_RUNS;
...
Question from comment:
Why does offset=0 perform better than offset=1,4 or 6?
On NVIDIA GPUs, work-items are grouped into 'warps' of size 32, which execute in lockstep (other devices have similar approaches, just with a different sizes). Memory transactions are aligned to multiples of the cacheline size (e.g. 64 bytes, 128 bytes etc). Consider what happens when each work-item in a warp attempts to read a single 4-byte value (assuming they are contiguous, as per your example), with a cacheline size of 64 bytes.
This warp is reading a total of 128 bytes of data. If the start of this 128-byte chunk is aligned to a 64-byte boundary (i.e. if offset=0), then this can serviced in two 64-byte transactions. However, if this chunk is not aligned to the a 64-byte boundary (offset=1,4,6,etc), then this will require three memory transactions to fetch all of the data. This is where your performance difference comes from.
If you set the offset to be a multiple of the cacheline size (e.g. 64), then you will likely get performance equivalent to offset=0.