I need to load and run an ONNX-model in a C++ environment using Libtorch on Windows 10 (Visual Studio 2015, v140). Searching the web, there seem to be almost exclusivly instructions for how to do it in Python. Is there a well documented way/does anyone know how to this in C++?
Here's two C++ based resources that might be relevant:
The ONNX Runtime C++ API enables inference and loading ONNX models with C++.
Windows ML C++ APIs can be leveraged to load ONNX models in C++ Windows desktop applications.
Related
Has anyone been successful in building/using the c++ API for TensorFlow on windows (withing Visual Studio)? The tutorials I found online or on TensorFlow's website have only shown building from source for python, or are outdated (3+ years old or for TensorFlow 1.x)
I see that there are docs for the functions of the built C++ API on TensorFlow's website, but there's no mention of where to get it or how to build it, the build from source section builds the python package that can be installed using pip, not anything that can be used with C++.
I've looked at the C api, but that seems to be different from what is mentioned in the C++ API section of the docs, and I was unable to get the C API working within VS due an error in mixing C++ and C.
If it helps, the end goal is to run inferences for pre-trained models (built using the python API) using C++.
Any help is appreciated.
What I want to do
First of all, my goal is using Tensorflow C++ API as a library on Windows, which is part of my project, instead of building my project inside Tensorflow.
Background
I had achieved this by building Tensorflow with CMake. However, from Tensorflow 1.10, building with CMake was deprecated and Bazel is recommended instead. But the official way to use C++ API is building project inside Tensorflow with Bazel. Thus, this way is not good for me.
What I have done
To use a newer version of Tensorflow, I have been trying to build Tensorflow with Bazel as a standalone library.
Some maintainer denoted that it is possible by substituting //tensorflow/tools/pip_package:build_pip_package to //tensorflow:libtensorflow_cc.so in the official tutorial. But in fact I encountered some problems and solved them by reading this tutorial. Now I have successfully built libtensorflow_cc.so.
What the problem is
However, I have no idea what should be done next to use the built result. And it is exactly what my problem is. There is no documentation of course. Only some incomplete ideas on it I have found, and I will show all of them, trying to give you more information:
There is somebody already successfully linking built *.so and having solved the problems he has encountered.
There is a repo doing the what I want to do on Ubuntu and Arch Linux. I have contacted with the maintainer and he told me that they have no plan for supporting Windows now.
A related issue: Building a .dll on Windows.
A related issue: Packaged TensorFlow C++ library for bazel-independent use.
A related issue: Feature request: provide a means to configure, build, and install that includes cc.
A related question: How to build and use Google TensorFlow C++ api. The scope of this question is a little larger without 'using bazel' and 'on Windows' restrictions.
A related pull request: C++ API
There must be someone struggling with similar problems like me. I hope this question can build a reservoir of ways to solve the problem.
It's over 2 years since this question was asked, and the news is not good: it seems there are insufficient people with Windows skills in a position to provide the support to integrate Tensorflow into Windows applications using the familiar headers + library model. And Tensorflow advances week by week, meaning that the Windows support falls further behind.
In my assessment, the path to building on Windows is currently blocked due to inadequate documentation. It's not so much that "There is no documentation of course" as the OP asserts, it's that the sparse documentation is distributed throughout dozens of separate posts, each of which dates rapidly with the continuing development of the Tensorflow along paths other than Windows C++.
I originally gave this answer to a similar question, but updated it with advice along the following lines yesterday:
Windows is a Microsoft product, so watch what Microsoft is doing
Hint: Microsoft is investing in the ONNX format
you can convert Tensorflow to ONNX, or Keras to ONNX
You can implement your (ONNX) model on Windows in C++ in at least 3 ways:
Windows ML (uses Onnx runtime)
Onnx runtime (supports DirectML as an execution provider)
DirectML (how Microsoft uses graphics cards to boost performance)
We don't have the latest or best hardware (e.g. we have Intel graphics cards), but have been able to get a solution based on Onnx runtime that classifies 224 x 224 RGB images in about 20 milliseconds for us. We found the Windows ML path much more difficult to work with legacy code, and also slower to run.
I would like to use a model trained with Tensorflow in a Windows standalone desktop application. I only need to perform predictions, I can train the model with Tensorflow Python API. What is the recommended approach?
I know there is a C++ API, but it is really hard to compile it, especially on Windows. Can I find any prebuilt C++ Tensorflow binaries for Windows?
Is there an easy way to distribute Python with Tensorflow as a Windows installer prerequisite?
Can I import the Tensorflow model in another technology and use it for inference? OpenCv DNN module has a function which imports data from Tensorflow, but I understood it has many limitations, and I was not able to import and use a model with OpenCv.
Thanks for help!
I was challenging the same issues as you.
You should at least try to compile it (try CMake, it might be easier)
If you still having trouble:
Compiler is out of Heap Space
Standalone Windows Lib
Basic Tensorflow Handling with C++
I asked a similar question and eventually found my own way to the answer. In the end, I found the Tensorflow instructions were actually pretty good (it was my reading them that was bad!). I have not tried using Bazel for Windows, but building Tensorflow using CMake ended up working fine.
The main issue was the compiler heap space issue. This always seems to occur in some random place if you are using the MS Visual Studio 32-bit compiler (default). The key is to make sure you run vcvarsall.bat or vcvars64.bat or whatever it takes to invoke the 64-bit compiler (in Task Manager, it should show up as cl.exe, not cl.exe *32) I found it hard (read: impossible) to get Visual Studio to use the 64-bit compiler, but using the MSBuild tool to compile on the command line worked fine.
Once you can build the example program, you have an example of an application that links to a static tensorflow library to do its stuff. You can just make your own application link to this library for what you want.
I have to create an application for testing that is relatively simple. If a user plugs in a device into a USB (hid library), the form will say "Pass". If no device is detected, it will say "Fail".
I want to create the app to run on WinXP+. It must be a forms application as it needs to be visual and have colors. The biggest requirement for me is that it must be stand-alone (a user just double clicks, the file and it opens, no installation). I need to avoid installing any extra .dll files and want to avoid the .net framework all together. It has to be written in C++.
So my question is, what are my options? I am new to Forms applications and the only forms application I have written required the .net framework.
From my research, Visual C++ seems out of the question as it requires .net for Forms apps. What about Turbo C++ by Borland? It seems outdated, but can it do the job?
I would suggest Borland C++ Builder 5 or 6. Visual C++ lacks the functionality to create C++ form apps, if we dismiss MFC
I'd say Visual C++ would probably still be your best option. A standard simple Win32 C/C++ program using only the native API will should suite your needs just fine. If you're targeting XP+, then the only dependency would be "msvcrt.dll" which is included with Windows by default.
Edit: This article on MSDN should get you started in the right direction.
What you're talking about is C++/CLI, it's microsoft's dialect of C++ that requires a .Net framework. Please distinguish it from real (native) C++, which can be used to create what you want to. Visual C++ is perfectly fine, just make sure you don't create a .Net project.
Forms are a .NET library. It's not Windows or anything like that, you can use the native GUI libraries in Win32 with no problem or any native wrapping of them. You can implement this in native C++ easily in Visual Studio.
If you're looking to create the application in a rapid fashion (i.e. stock UI components that you can drag and drop onto forms), my experience is that Embarcardero C++ Builder effectively solves that problem space.
I like Visual C++ but it's not a great environment for rapidly creating forms based applications. I want to emphasize that I'm referring to Rapid Application Development (RAD) tools. There are plenty of great C++ IDEs and libraries for creating forms based applications and all of them should be able to create standalone exes.
Here is a list of related questions that might help in your decision:
https://stackoverflow.com/q/1754020/4916
https://stackoverflow.com/q/93326/4916
Best C++ IDE or Editor for Windows
Native exe!
that means my program can run easily without any requirement?
even if I use .net classes?
You know I want to write a program that is so light and I don't like to use C# or any other .net programing language because all of them need .net-framework 4.5.
Just think a 2.5 MB programm needs a +250 MB .netframework.
New Update - 12/01/2016:
It's almost 4 years ago when I asked this question. As you know Dotnet Native is announced. It's an interesting feature which compile IL into native code.
Compiling Apps with .NET Native
.NET Native is a precompilation technology for building and deploying
Windows apps that is included with Visual Studio 2015. It
automatically compiles the release version of apps that are written in
managed code (C# or Visual Basic) and that target the .NET Framework
and Windows 10 to native code. Typically, apps that target the .NET
Framework are compiled to intermediate language (IL). At run time, the
just-in-time (JIT) compiler translates the IL to native code. In
contrast, .NET Native compiles Windows apps directly to native code.
For developers, this means:
Your apps will provide the superior performance of native code.
You can continue to program in C# or Visual Basic.
You can continue to take advantage of the resources provided by the .NET Framework, including its class library, automatic memory
management and garbage collection, and exception handling.
Last I checked none of the .NET frameworks were 250+ MB! Yes, the offline installer for .NET Framework 3.5 SP1 is 231MB but it contains x86 and x64 versions of .NET 2, 3 and 3.5 sp1.
You should read this http://www.smallestdotnet.com for details on sizes of various versions of the installers.
Now on to your question:
Yes, It is a little annoying to have your clients install a big framework, even 20-40MB does get annoying. With .NET, the advantage is the ease of programming (In my opionion) compared to other Native options.
Your native options are:
MFC - You need only the VS runtimes installed, which is 1-2MB and is usually installed on newer pcs. Also, you can ship your application with the MFC libraries packaged into a dll which is again <2MB
The trade of here is you need to program in C++, the libraries overall are a very thin layer over the native libraries. and people have had harsh opinions about MFC. I've barely just tried it.
Win32 API - This is going all bare bones, and quite difficult, you could use C or C++ but you'd really have to know a lot about the Win32 API and how windows itself works (Stuff like windows messages, hwnds etc) Its not fun, believe me. But during deployment you would not need any external libraries.
There are tons more options, see here:
Native Windows Application Development Options
https://stackoverflow.com/questions/2711599/what-programming-language-should-i-use-to-create-small-native-windows-applicatio
Here are some links on MFC that might help:
Want to learn Windows Programming,some suggestions?
How do I decide whether to use ATL, MFC, Win32 or CLR for a new C++ project?
C++ MFC vs .NET?
https://stackoverflow.com/questions/557114/stick-with-mfc-or-go-to-net
you are able to create native exe by using c++ Win 32 projects.
Alas, nearly everything requires a download runtime library and even if you have one installed, you'll need to download updates for them almost continually. Even Microsoft C++ apps nowadays come with security updates that have to be installed if you've compiled your app with them.
But.. there is a solution of sorts. If you use C++, it has a feature where only the things you use are compiled into the final app. Normally, this would require linking with all the library dlls, but if you statically link with the library, you will end up with a single .exe that is as small as can be, and you will not need any dlls (as all the code contained in the library will be compiled into the .exe).
The benefits are debatable compared to dlls, but as MS has pretty much broken the idea of shared dlls in .NET (ie, you practically have to put all the shared dlls in the same directory as your running app, giving you a nightmare in maintenance if you have these shared dlls spread around all your apps) then there's not much of a difference anymore. Static linking is getting a little bit of a comeback and sounds like its what you want.
For modern C++ development, you'll probably want to take a look at Qt instead of MFC. Its a lot nicer to use and is cross-platform so you can run Qt apps on your Android or Linux platforms as well as Windows.