PyTorch model to C++ - c++

I have trained the detection algorithm and saved my best model. Now I want to convert my model (pretrained) to C++ and use it in my app. I wanted to know what are the possible ways to convert a pyTorch model to c++?
Thanks!


You can use TorchScript intermediate representation of a PyTorch model, through tracing and scripting, that can be run in C++ environment. For this, you'll probably have to modify the model itself in order for it to be traced or scripted.
You can use ONNX (Open Neural Network Exchange), through which you can export your model and load it in another C++ framework such as Caffe. It comes with its own implications though.
The easiest is to try Embedding Python, through which you can run your python (pytorch) model in C++ environment. Note that the model will still run in python, but only through C++, so there won't be any speed gains that you might be expecting in C++.
Also, with the release of torchvision 0.5, all models in torchvision have native support for TorchScript and ONNX.

Related

Is it possible to train ONNX models developed in tensorflow and pytorch with C++?

I wonder if its possible to use tensorflow and pytorch models converted to onnx models to train them with the C++ Api like it is done in e.g. https://gist.github.com/asimshankar/5c96acd1280507940bad9083370fe8dc with a tensorflow model. I just found examples for inference with onnx. The idea is to be able to prototype with tensorflow and pytorch in python, convert to onnx models and to have a unified API in C++ to do inference and training. It would help quite a lot to get some (links to get) informaton.

ONNX's GitHub page suggests that it can be used for inference, but it doesn't seem reasonable to be able to train all models with it (from the development perspective).
Currently we focus on the capabilities needed for inferencing (scoring).
Although there are some difficulties, such as always writing backpropagation is more difficult than feedforwarding, and supporting it would double the framework size, which is not what ONNX is aiming for since there are already so many frameworks for this. To train you will need all the parameters, the derivatives of functions in the GPU and CPU(if its performance is lower than other frameworks, it will be a big problem since nobody will use it). And there are many other things that make a unified framework difficult(Supporting training on multiple GPUs over a network, for example).(So In our perspective, it's great, but in theirs it's so difficult)
But we can see that some functionality for training has been added to the framework, in this case it can train transformer models
Also, to training transformers in PyTorch you could see this link

ONNX Runtime does support training but not in C++. You can train an ONNX model using ORT and Pytorch. Please see here https://onnxruntime.ai/docs/get-started/training-pytorch.html.

What is the best way of deploying a neural network trained in tensorflow to torch C++?

I'm working on a project where I need to forward pass the network in C++ with PyTorch. The weights of the network need to be imported from a model trained in Tensorflow. I will write the C++ version of the network using Torch but I can not train the network in Torch because of some issues. Which methods can I use to do this?

Things to try:
Convert TensorFlow model to ONNX/TensorRT/whatever using some converter. ONNX can be then converted to PyTorch (not sure about Torch), but you may find it unnecessary, as both ONNX and TensorRT are good for inference with their own runtimes. But in fact, you may face a lot of issues with conversion.
Reproduce model code with Torch, then copy TensorFlow tensors to their analogs in your code. You can see an example of such TF->PyTorch conversion in HuggingFace/Transformers code 1, 2. As you can see, it is quite tricky.

Use Keras to set up Neural Network, then train it using the TF C++ API

I am using Keras with TF backend. I want to set up a neural network with keras using python, then save it, convert it to tensorflow and train it using TF's C++ API.
What I have found so far was already pretty helpful:
1) Run your Keras models in C++ Tensorflow
2) Convert Keras model to C++
Unfortunately both approaches use Keras to train the NN and convert it to TF afterwards. Basically, I just want to set up the architecture using Keras and then train the network using TF's C++ API. Is there a way to do so?
Thanks for help!

I guess the second link you noted, is just enough what you need. You may need to search on "Save and Load Your Keras Deep Learning Models" and then open it with the TF's C++ API.

Object detection using python

I am working on my college project and for that I have to recognize different hand gestures, so can any one tell me that how can I learn this image recognization quickly using python?

If you're using Python, I think you would better use Tensorflow.
Check https://github.com/tensorflow/models/tree/master/object_detection.
It is easy to follow instructions and they provide convenient script for retraining a detection model.
If you need to train a model with custom data, you have to prepared images dataset annotated with bounding boxes.

Convert Keras model to C++ [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I am using Keras (with Theano) to train my CNN model. Does anyone has idea how can I use it in my C++ application? Does anyone tried something similar? I have idea to write some python code that will generate a c++ code with network functions - any suggestion on it?
I found a similar question here how to use Tensorflow Keras model in C++ but without answer.

To answer my own question and have a solution - I wrote a plain c++ solution called keras2cpp (its code available on github).
In this solution you store network architecture (in json) and weights (in hdf5). Then you can dump a network to a plain text file with provided script. You can use obtained text file with network in pure c++ code. There are no dependencies on python libraries or hdf5. It should work for theano and tensorflow backend.

I found myself in a similar situation but needed to not only support forward passes of sequential Keras models in C++ but also of more complex models build with the functional API.
So I wrote a new library called frugally-deep. You can find it on GitHub and it is published under the MIT License: https://github.com/Dobiasd/frugally-deep
Additionally to supporting many common layer types it can keep up with (and sometimes even beat) the performance of TensorFlow on a single CPU. You can find up-to-date benchmark results for some common model in the repo.
By automatic testing frugally-deep guarantees that the output of a model used with it in C++ is exactly the same as if run with Keras in Python.

If your keras model is trained using tensorflow backend, you can save the keras model as a tensorflow model following this code:
https://github.com/amir-abdi/keras_to_tensorflow
Here is a shorter version of the code:
from keras import backend as K
from tensorflow.python.framework import graph_util
from tensorflow.python.framework import graph_io
weight_file_path = 'path to your keras model'
net_model = load_model(weight_file_path)
sess = K.get_session()
constant_graph = graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), 'name of the output tensor')
graph_io.write_graph(constant_graph, 'output_folder_path', 'output.pb', as_text=False)
print('saved the constant graph (ready for inference) at: ', osp.join('output_folder_path', 'output.pb'))

You Can try this one
https://github.com/gosha20777/keras2cpp
Keras2cpp is a small library for running trained Keras models from a C++ application without any dependencies.
Supported Keras layers:
- Dense
- Convolution1D
- Convolution2D
- Convolution3D
- Flatten
- ELU
- Activation
- MaxPooling2D
- Embedding
- LocallyConnected1D
- LocallyConnected2D
- LSTM
- GRU
- CNN
- BatchNormalization
Supported activation:
- linear
- relu
- softplus
- tanh
- sigmoid
- hard_sigmoid
- elu
- softsign
- softmax
Design goals:
Compatibility with networks generated by Keras using TensorFlow backend.
CPU only.
No external dependencies, standard library, C++17.
Model stored in memory.

The solutions found here are quite good, but if your model has some different types of layers not supported by these libraries, I would recommend doing the following:
Converting the Keras model to a tensorflow model.
Freeze the model and use Tranform graph tool provided by tensorflow (you'll have to build it from source with bazel)
Compile the C++ API tensorflow library to use it in your project.
Use the C++ API tensorflow library and link the libraries to your project.
If you want to use a something differentcompiler than bazel (like g++ for example) you can follow this great tuturial:
http://tuatini.me/building-tensorflow-as-a-standalone-project/

The easiest way is probably to make a system call to a Python script that writes the predictions to a binary or HDF5 file, which can be read in from C++. You can also directly integrate Python into C++.
If you need to deploy and distribute this easily, you can look into self-contained Python installations like Anaconda, but your best bet may be to avoid Keras and use the C++ interface to Caffe or Tensorflow. I wouldn't recommend Tensorflow since using it from C++ isn't standard; see this discussion. Caffe is arguably the second most-popular deep learning library so you can't really go wrong.

I had a similar need--I wanted to embed Keras models in a C++ application--and decided to write my own library: Kerasify
Design goals of Kerasify:
Compatibility with image processing Sequential networks generated by Keras using Theano backend. (Could work with Tensorflow if you switch around matrix col/row ordering).
No external dependencies, standard library, C++11 features OK.
Model stored on disk in binary format that can be quickly read.
Model stored in memory in contiguous block for better cache performance.
Doesn't throw exceptions, returns only bool on error.
CPU only, no GPU
Example code, unit tests, etc. at the github link. It's not fully complete, it only supports the narrow subset of Keras functions I'm using, but it should be extensible with a little effort.