RuntimeError 'CUDA error' when trying the Google Cloud AI Platform Tutorial for PyTorch - google-cloud-platform

Question

I've been trying to get started with Google Cloud's AI Platform. I have been following this tutorial: https://cloud.google.com/ai-platform/training/docs/getting-started-pytorch#gpu_1
My own models are written in PyTorch, hence the choice to get started with PyTorch. Why I want to use the GPU I guess goes without saying.
I've tried to follow the instructions to the letter, and I've used the provided sample code. Yet I still run into errors. I can create a job without problems, but the job ends up failing with the following error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
I'm relatively new to PyTorch and completely new to GCP, so I have no idea how I would go about fixing this and any help would be much appreciated.
Full trace:
The replica master 0 exited with a non-zero status of 1.
Traceback (most recent call last):
File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 123, in <module>
main()
File "/root/.local/lib/python3.7/site-packages/trainer/task.py", line 119, in main
experiment.run(args)
File "/root/.local/lib/python3.7/site-packages/trainer/experiment.py", line 132, in run
train(sequential_model, train_loader, criterion, optimizer, epoch)
File "/root/.local/lib/python3.7/site-packages/trainer/experiment.py", line 37, in train
for batch_index, data in enumerate(train_loader):
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 347, in __next__
data = self._next_data()
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 387, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 74, in <dictcomp>
return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate
return torch.stack(batch, 0, out=out)
RuntimeError: CUDA error: no kernel image is available for execution on the device

Answer 1

I have managed to reproduce the same issue with pytorch-gpu.1-6.
As a workaround, it works with the pytorch-gpu.1-4
I suppose that something in the code changed from 1.6 but not sure what as I am not familiar with Pytorch.
Besides, up to version 1.7, it seems that the code to select the device hasn’t changed compared to our sample code.
Additionally, our GPU basic tier Nvidia Tesla K80 seems to be supported by CUDA 9.0, 9.2, 10.0 OR 11
In any case, I have created a public issue on issuetracker so the AI Platform engineering team can investigate it.