Significant RAM cost while running TensorFlow in C++ - c++

Question

I'm currently writing some inference code using a trained TensorFlow graph on GPU machine with C++ APIs.
Here are my settings:
Platform: CentOS 7
TensorFlow Version: TensorFlow 1.5
CUDA Version: CUDA 9.0
C++ Version: C++11
There are a couple of questions that I'm struggling with.
1) First, I followed this tutorial to learn the basic template to load a graph in C++. The example in this tutorial is quite simple, but the program takes almost 0.9G in RAM when I run it (on GPU machine).
2) My graph is way more complicated than the one in that tutorial. There are approximately 20 layers and the numbers of nodes in layers vary from 300 to 5000.
My (pseudo) code snippet is here. For simplicity, I only keep the code that causes (potential) memory issue:
tensorflow::Tensor input = getDataFromSomewhere(...);
int length = size of the input;
int g_batch_size = 50;
// 1) Create session...
// 2) Load graph...
// 3) Inference
for (int x = 0; x < length; x += g_batch_size) {
tensorflow::Tensor out;
auto cur_slice = input.Slice(x, std::min(x + g_batch_size, length));
inference(cur_slice, out);
// doSomethingWithOutput(out);
}
// 4) Close session and free session memory
// Inference helper function
tensorflow::Status inference(tensorflow::Tensor& input_tensors, tensorflow::Tensor& out) {
// This line increases a lot more memory usage
TensorDict feed_dict = {{"IteratorGetNext:0", input_tensors}};
std::vector<tensorflow::Tensor> outputs;
tensorflow::Status status = session->Run(feed_dict, {"final_dense:0"}, {}, &outputs);
// UpdateOutWithOutputs();
return tensorflow::Status::OK();
}
After I created session and loaded graph, the memory cost is around 1.2G.
Then, as I noted in my code, when the program reached session->Run(...), the memory usage went up to more than 2G.
I'm not sure if this is a normal behavior of TensorFlow. I've checked this and this thread, but I don't quite know if I created redundant ops in my code.
Any comments or suggestions are appreciated! Thanks in advance!

Answer 1

The issue that I found was that Tensorflow dynamic libraries would take about 200MB and CUDA dynamic libraries would take more than 500MB memory. So loading those libraries already takes a great amount of memory.