How to use Keras SavedModel from C++ - c++

Question

I have a trained a classic CNN(pre-trained mobile net) for image classification. I want to now use this model from c++. From my understanding, I need to create a library of the model, that can accept the input and return its outputs. I have the model saved in format .pb (SavedModel).
I have already tried, CppFlow, where the error shows that it can't read my model. I assume it's due to incompatibility with TF 2.0.
I have also got the command line interface of SavedModel working, but I don't know how to use it in my cpp application.
I want to know how I can build a library of my model and use this library such that it can make predictions on the fly. Any guidance will be helpful. Please let me know if any additional information is required.

Answer 1

One way of using keras model in C++ is to convert it to TensorFlow .pb format. I've just composed a script for doing this, down below.
Usage: python script.py keras_model.hdf5
It outputs tensorflow model as input file name appended by .pb.
Then you can use TF C++ api for reading model and doing inference. Nice detailed example of using image recognition model to label images in C++ TF is located here.
Another option - you may use Keras directly by calling Python API from C++, it is not that difficult, there is standalone python which is compiled statically meaning having no dll/shared libs dependencies at all hence python interpreter can be fully compiled into C++ single binary. There are also many libraries in Internet that help you to easily run Python from C++.
import sys, os
from keras import backend as K
from keras.models import load_model
import tensorflow as tf
def freeze_session(session, keep_var_names=None, output_names=None, clear_devices=True):
"""
Freezes the state of a session into a pruned computation graph.
Creates a new computation graph where variable nodes are replaced by
constants taking their current value in the session. The new graph will be
pruned so subgraphs that are not necessary to compute the requested
outputs are removed.
#param session The TensorFlow session to be frozen.
#param keep_var_names A list of variable names that should not be frozen,
or None to freeze all the variables in the graph.
#param output_names Names of the relevant graph outputs.
#param clear_devices Remove the device directives from the graph for better portability.
#return The frozen graph definition.
"""
from tensorflow.python.framework.graph_util import convert_variables_to_constants
graph = session.graph
with graph.as_default():
freeze_var_names = list(set(v.op.name for v in tf.global_variables()).difference(keep_var_names or []))
output_names = output_names or []
output_names += [v.op.name for v in tf.global_variables()]
# Graph -> GraphDef ProtoBuf
input_graph_def = graph.as_graph_def()
if clear_devices:
for node in input_graph_def.node:
node.device = ""
frozen_graph = convert_variables_to_constants(session, input_graph_def,
output_names, freeze_var_names)
return frozen_graph
if len(sys.argv) <= 1:
print('Usage: python script.py keras_model.hdf5')
sys.exit(0)
else:
ifname = sys.argv[1]
model = load_model(ifname)
frozen_graph = freeze_session(
K.get_session(),
output_names = [out.op.name for out in model.outputs],
)
tf.io.write_graph(frozen_graph, os.path.dirname(ifname), ifname + '.pb', as_text = False)

Answer 2

There are standalone third-party libraries that can import keras model into c++ for inference without doing much of work from our side.
Examples are
Multithreaded library for image segmentation models such as U-Net etc. - https://github.com/upashu1/keras2cpp_multithreading_image_segmentation
A library that supports the most of the layers
https://github.com/Dobiasd/frugally-deep