Kepler -Concurrent kernel launches not overlapping - concurrency

Question

I am trying to overlap kernel execution on Kepler device, but from NVVP layout it seems that they are not overlapping. here is the code,
#include<stdio.h>
#include<sys/time.h>
#include<time.h>
#define NY 1024
#define NX 1024
__global__ void kernel1(int j,int *A,int *b)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
b[j*NY+i] = A[i*NY+j];
}
__global__ void kernel2(int j,int *A,int *b)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
for(int time=0;time<100;time++)
b[j*NY+i] += 10;
}
int main()
{
int nstreams=4;
int *a, *b;
struct timeval t1,t2;
cudaMalloc((void**)&a,NX*NY*sizeof(int));
cudaMalloc((void**)&b,NX*NY*sizeof(int));
cudaStream_t *streams = (cudaStream_t *) malloc(nstreams * sizeof(cudaStream_t));
for (int i = 0; i < nstreams; i++)
{
cudaStreamCreate(&(streams[i]));
}
gettimeofday(&t1, NULL);
for(int newvar=0;newvar<NX;newvar++)
{
kernel1<<<1,NY,0,streams[newvar%nstreams]>>>(newvar,a,b);
}
for(int newvar=0;newvar<NX;newvar++)
{
kernel2<<<1,NY,0,streams[newvar%nstreams]>>>(newvar,a,b);
}
cudaDeviceSynchronize();
gettimeofday(&t2, NULL);
return 0;
}
Please suggest some tips.
CUDA version 5.5
NVVP version 5.5 Linux machine Ubuntu 12.10

Answer 1

Fundamentally I think the problem is that your kernels are not executing long enough. The execution time of your kernels is a few microseconds, and the kernel launch overhead is also a few microseconds, so you're not seeing any overlap. By the time the API has completed the setup of the new kernel launch, the previous kernel has finished.
I modified your kernel1 as follows:
__global__ void kernel1(int j,int *A,int *b)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
for (int q = 0; q < 1000; q++)
b[j*NY+i] = A[i*NY+j] + q/j;
}
There's nothing magical or special about these modifications, I'm just looking for a way to increase the kernel duration execution (from a few microseconds to a few milliseconds).
With the above changes, I saw good overlap of your kernel1 in the profiler.
I imagine something similar could be done with your kernel2.
You should also make sure you have not deselected the "enable concurrent kernel profiling" checkbox when you start a profiling session in nvvp.