CUDA Optimizations for the RTX A5000 [closed] - c++

Question

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 months ago.
Improve this question
I am a bit rusty on my CUDA skills. I am attempting to generate a 5120x5120 image that consists entirely of random noise generated via cuRand. I am using a single NVIDIA RTX A5000 who's compute compatibility is 8.6. My question is, what should be the ideal grid dimensions and threads per block to squeeze the highest amount of efficiency out of this noise generation. For context, here are some of the hardware specs of the A5000:
-64 SM's
-8192 CUDA cores
-128 cores per SM
-16 blocks per SM
-48 active warps per SM (1536 threads)

Answer 1

Both for general considerations of optimizing performance based on maximizing occupancy as well as considerations on CURAND generator initialization, you would want to choose a number of blocks and threads per block so that the product is equal to at most 64 SMs x 1536 threads per SM. Ideally you would target that number.
You would start by writing the kernel that takes generator state that has already been initialized, and runs a grid-stride loop to use random number generation to write image points.
If the occupancy analysis (probably based on registers per thread) suggests that maximum thread load per SM is possible (for that kernel that you just wrote), then you would size your grids that way.
If the occupancy analysis indicated that your kernel cannot have the full 1536 threads per SM, then you would reduce your grid launch (size) accordingly.
Since the cc8.6 SM has a maximum of 1536 threads, don't size your threadblocks at 1024 threads. Choose 512, or some other number like 256.
With CURAND, in my view, the best practice is to launch the generator state initialization kernel (separately, first) before your image update kernel. Don't try to do generator state initialization in the same kernel that is doing the image generation.
Once you figure out what the maximum occupancy is, then you will size your generator state array to match that, and you will launch a grid-stride kernel to fill that array as your CURAND init kernel.
Then launch your image creation kernel.
Here is a simple example:
$ cat t2115.cu
#include <curand_kernel.h>
#include <curand.h>
const int imageW = 5120;
const int imageH = 5120;
using mt = float;
__global__ void setup_kernel(curandState *state, size_t N, const unsigned long long seed = 1, const unsigned long long offset = 0){
for (size_t id = blockIdx.x * blockDim.x + threadIdx.x; id < N; id += gridDim.x*blockDim.x)
curand_init(seed, id, offset, state+id);
}
template <typename T>
__global__ void image_gen(T *img, curandState *state, const size_t img_size){
size_t id = blockIdx.x * blockDim.x + threadIdx.x;
curandState s = state[id];
for (;id < img_size; id += gridDim.x*blockDim.x)
img[id] = curand_uniform(&s);
}
int main(){
size_t is = ((size_t)imageW)*imageH;
int grid_dim = 64 * 1536;
int bs = 512;
int gs = grid_dim/bs;
mt *img;
curandState *s;
cudaMalloc(&s, sizeof(curandState)*grid_dim);
cudaMallocManaged(&img, sizeof(mt)*is);
setup_kernel<<<gs, bs>>>(s, grid_dim);
image_gen<<<gs, bs>>>(img, s, is);
cudaDeviceSynchronize();
}
$ nvcc -Xptxas=-v -arch=sm_86 -o t2115 t2115.cu
ptxas info : 218048 bytes gmem, 72 bytes cmem[3]
ptxas info : Compiling entry function '_Z9image_genIfEvPT_P17curandStateXORWOWm' for 'sm_86'
ptxas info : Function properties for _Z9image_genIfEvPT_P17curandStateXORWOWm
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 18 registers, 376 bytes cmem[0]
ptxas info : Compiling entry function '_Z12setup_kernelP17curandStateXORWOWmyy' for 'sm_86'
ptxas info : Function properties for _Z12setup_kernelP17curandStateXORWOWmyy
0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info : Used 39 registers, 384 bytes cmem[0]
$
We've compiled with the -Xptxas=-v switch which causes the compiler to indicate register utilization. We see that the image_gen function uses 18 registers per thread. This is well below the limit for what will allow full occupancy, so we should be fine with the indicated launch sizes and should expect full occupancy. The sm_86 SM has support for up to 65536 registers, so when considered across 1536 threads, this implies a limit of about 65536/1536 = 42 register per thread. If we had a number larger than that, we would reduce the grid_dim variable accordingly.
This can be largely "automated" with the occupancy API and launch bounds functionality, but these basic ideas should be understood first.