CUDNN_STATUS_INTERNAL_ERROR in tensorflow 2.1 c++ - c++

Question

I face a problem as the title says when I load a pre-trained model(.pb model of YOLOv3) and infer with this model in tensorflow 2.1 c++. Error messages are as the following:
2020-10-30 21:36:20.245492: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2020-10-30 21:36:20.269906: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
[InferCC] Model infer failed(2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node yolov3/yolo_darknet/conv2d/Conv2D}}]]
[[StatefulPartitionedCall/_791]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[{{node yolov3/yolo_darknet/conv2d/Conv2D}}]]
0 successful operations.
Here is my configuration:
Ubuntu 18.04
Tensorflow 2.1 c++
Cuda 10.1
cuDNN 7.6.5
GPU memory ~6G
$ nvidia-smi
Fri Oct 30 21:50:24 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2060 Off | 00000000:01:00.0 On | N/A |
| N/A 46C P8 6W / N/A | 553MiB / 5931MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1242 G /usr/lib/xorg/Xorg 18MiB |
| 0 N/A N/A 1886 G /usr/bin/gnome-shell 50MiB |
| 0 N/A N/A 9708 G /usr/lib/xorg/Xorg 321MiB |
| 0 N/A N/A 9876 G /usr/bin/gnome-shell 128MiB |
| 0 N/A N/A 12591 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 14848 G ...s/QtCreator/bin/qtcreator 3MiB |
| 0 N/A N/A 15696 G ...AAAAAAAAA= --shared-files 21MiB |
+-----------------------------------------------------------------------------+
I searched it on internet and it seems that my GPU memory is ran out(I'm not sure about it). So I add following codes to set GPU memory growth before load model:
tensorflow::ConfigProto config;
config.mutable_gpu_options()->set_allow_growth(true);
But with no luck, errors are still there.
Or I want to know if it is indeed lack of GPU memory(~6G is not enough for YOLOv3 model)?
Please some one helps me out. Thanks.

Answer 1

Finally, I find a way to correct that error. Just pass the SessionOptions variable(call set_allow_growth with it) into LoadSavedModel function:
if(tensorflow::MaybeSavedModelDirectory(modelName.toStdString()))
{
// set gpu memory growth true here
tensorflow::SessionOptions sessionOpt;
sessionOpt.config.mutable_gpu_options()->set_allow_growth(true);
// then pass that variable into LoadSavedModel
tensorflow::Status status = tensorflow::LoadSavedModel(
sessionOpt,
tensorflow::RunOptions(),
modelName.toStdString(),
{tensorflow::kSavedModelTagServe},
_model);
if(!status.ok())
{
qCritical("[InferCC] Load %s model failed - %s.", modelName.toStdString().c_str(), status.error_message().c_str());
_loaded = false;
return false;
}
}
Then run program while keep calling nvidia-smi in command line, showing that GPU memory is increasing and the last state(used 5278 MiB)as :
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.57 Driver Version: 450.57 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2060 Off | 00000000:01:00.0 On | N/A |
| N/A 58C P0 63W / N/A | 5832MiB / 5931MiB | 37% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 656 C ...VisTrack/release/VisTrack 5278MiB |
| 0 N/A N/A 1242 G /usr/lib/xorg/Xorg 18MiB |
| 0 N/A N/A 1886 G /usr/bin/gnome-shell 50MiB |
| 0 N/A N/A 4843 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 5879 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 9708 G /usr/lib/xorg/Xorg 293MiB |
| 0 N/A N/A 9876 G /usr/bin/gnome-shell 143MiB |
| 0 N/A N/A 12591 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 13952 G /usr/lib/firefox/firefox 3MiB |
| 0 N/A N/A 14848 G ...s/QtCreator/bin/qtcreator 3MiB |
| 0 N/A N/A 15696 G ...AAAAAAAAA= --shared-files 21MiB |
+-----------------------------------------------------------------------------+