I have a demo application that is written in Python. It uses a lot of existing C++ code (written by me) that relies on OpenCV for image processing. Currently, communication between Python and C++ is being done through file I/O and subprocess calls, which isn't very efficient. What is the best way to wrap the C++ code so that it can be called from Python?
There is too much C++ code to think about porting it to Python, so that's not really an option.
A long time ago, the Python OpenCV wrappers were written in SWIG, but it looks like the most recent version of the wrappers is completely different. Can anyone point me in the right direction?
There are two ways that you can make your python program interact directly with your C/C++ program:
Wrap your C/C++ code in a DLL with C-API only. Then, use ctypes
to call C-function within the DLL. The advantage of this way is that you don't need to include/link any other library.
Extend python by adding new python module. You may use boost python to easily create a python module. The advantage of this way is that you don't need to wrap your code to C-API.
Without knowing code complexity, variety of C++ code and style of it... I would recommend "Extending Python"
It's not an immediate solution (you should change the C++ code, prototype some new functions or add a simple wrapper layer in C). But, if you plan to do a complex project (and are also a bit worried on performance)... it seems the best way to do it.
Porting C++ code to Python seems a step backwards, doing new code in Python is ok (I'm a fan of it) but C++ will (often) be more efficient.
Edit: Take also a look on ctypes module. Maybe it suits your needs. If you are more comfortable doing the wrapping in python language, then it may be better. If you don't mind playing with the C code, then extend Python by doing a module with your existing code.
Related
I'm a Python guy building a Linux-based web service for a client who wants me to interface with a small C++ library that they're currently using with a bunch of Windows based VB applications.
They have assured me that the library is fairly simple (as far as they go I guess), and that they just need to know how best to compile and deliver it to me so that I can use it in Python under Linux.
I've read a bit about the ctypes library and other options (SWIG, etc), but for some reason I haven't really been able to wrap my head around the concept and still don't know how to tell them what I need.
I'm pretty sure having them re-write it with Python.h, etc is out, so I'm hoping there's a way I can simply have them compile it on Linux as a .so and just import it into Python. Is such a thing possible? How does one accomplish this?
No, such a thing is not possible.
Either they have to provide Python bindings, or you do. Either one of you can do this in any of the following ways:
Using <Python.h> directly to write a C extension module.
Using boost::python to make writing the extension module much easier (especially when they're using C++ rather than C).
Using SWIG, SIP, or similar tools to partly automate the writing of the extension module.
Using Cython to write the extension module in a Python-like language, instead of in C or C++.
Using ctypes from within Python.
For very simple cases (especially if they're actually exporting a C interface to their C++ code), ctypes probably is the easiest solution. Otherwise, I'd suggest looking at Cython first. But at any rate, you're going to have to wrap your head around one of the solutions—or convince them to do it instead.
Due to complexities of the C++ ABI (such as name mangling), it's generally difficult and platform-specific to load a C++ library directly from Python using ctypes.
I'd recommend you either create a simple C API which can be easily wrapped with ctypes, or use SWIG to generate wrapper types and a proper extension module for Python.
I have a scenario where I have some functions in C++ classes and I want to be able to call them using a python script. Let's say I have a function
void greet(_msg);
std::cout >> _msg >> std::endl;
I want to be able to call it trough a custom Python call and pass arguments to it, for example using
saySomething("Hello")
As a .py file I want it to call the greet function and pass "Hello" as an argument.
I know it's a subject that has been throughly discussed, and I've done a share of research on embedding python in C++, I've managed to read values from a python script using the standard Python/C API and run a function in Python from C++ and pass argument to it, but I can't seem to get my head around how to achieve this specific outcome.
I've had a look at ctypes and various wrappin libraries such as boost:python or swig, but I can't seem to understand to which degree they could help me achieve want I want.
Depending on which version of Python you are interested in, 2.x or 3.x,
read through the Extending and Embedding the Python Interpreter chapter for 2.x or 3.x. You are interested only in extending Python, so section the 1. Extending Python with C or C++ will provide you with complete explanation how to implement what you need in order to be able to call your functions implemented in C++ from Python script.
Certainly, there are numerous libraries and generators which allow you to wrap C/C++ APIs for Python (e.g. Boost.Python or SWIG), but your case sounds simple enough, that for the purpose of learning it is IMO better to get familiar with Python C API. Even if you use these tools, you will frequently have to get down to Python C API anyway or at least understand it.
I recently needed to do this very thing. Boost.Python does what we're looking for (and more) but personally (as much as I love Boost) I find it a little overkill to have to drag in half the Boost library to get one feature. SWIG also wasn't really an option for me as code generation always becomes a pain to maintain while class structures change (Don't get me wrong, these are BRILLIANT solutions!, just not what I was looking for).
So, the only thing left for me was to implement it from first principles (Python/C API). Hense, "ECS:Python" was born. ECS:Python (Embedded C++ Scripting with Python) is a simple C++ Python wrapper library I designed specifically for C++ developers. It allows you to expose objects from a C++ application to an embedded Python interpreter for interactive scripting, and it's very light-weight and easy to use.
Its free (BSD) and open source. If you're interested here it is:
http://sourceforge.net/projects/ecspython
You can use the weave.inline() function, which is part of the scipy package, to compile and execute C/C++ files and get their output from within your python script.
I want to execute a code helloword.cpp which takes in some argument from console parses those arguments and then prints "hello world" in the console.
Now, I want to parse these arguments from a python scripts parsearguments.py
So for example:
def parse_arguments:
...# some code
return arguments
Now, how do i communicate between python and c++.
I have been reading and see that cython, boost python are the options but I have a hard time finding the right simple hello world example.
Any suggestions will be appreciated.
Thanks
To execute C++ code in python, you could effectively use boost python, here is a tutorial:
http://www.boost.org/doc/libs/1_59_0/libs/python/doc/index.html
You write a kind of wrapper outside you C++ code.
If it is C code, python has internal library called ctypes.
In both case, you should compile the C/C++ code into shared library.
How about passing whatever text you generate with Python into the standard input of your C++ program? Basically, you have to use Python's subprocess module to fire up the C++ program and dump the text into its standard output.
In case that your C++ program is required to be running separately in the background, you could try another form of interprocess communication, like unix domain sockets.
Using boost::python is also an option, but it might be a little more difficult to deal with.
A couple of other options besides Boost.python are SIP and SWIG (Simplified Wrapper and Interface Generator). Like Boost, SIP and SWIG are open source.
SWIG is particularly powerful, but also a bit hairy. It provides support for interfacing C and C++ with a boatload of other languages including (not a complete list) Python, Perl, Lua, Tcl/Tk, Ocaml, Ruby, Java. One aspect of SWIG is that it parses your C++ headers. This has benefits and pitfalls. A benefit is that it does most of the work of generating the interfaces. A downside is that it doesn't handle some of the dark corners of C++ 2003, and it hasn't stepped up to C++11 at all. Another downside is that compilation of a large project becomes slow. Very, very slow.
Using boost.python sounds like a good solution for me. But depending on your C++ experience this can be quite tricky. A good point to start is here:
http://wiki.python.org/moin/boost.python
Boost.Python enables you to export C++ classes and member functions to Python to be able to use them from there.
I know this question could be in vain, but it's just out of curiosity, and I'm still much a newb^^ Anyways I've been loving python for some time while learning it. My problem is obviously speed issues. I'd like to get into indie game creation, and for the short future, 2d and pygame will work.
But I'd eventually like to branch into the 3d area, and python is really too slow to make anything 3d and professional. So I'm wondering if there has ever been work to create a high-level language able to import and use c libraries? I've looked at Genie and it seems to be able to use certain libraries, but I'm not sure to what extent. Will I be able to use it for openGL programing, or in a c game engine?
I do know some lisp and enjoy it a lot, but there aren't a great many libraries out there for it. Which leads to the problem: I can't stand C syntax, but C has libraries galore that I could need! And game engines like irrlicht. Is there any language that can be used in place of C around C?
Thanks so much guys
Python can call functions in dynamically loaded C libraries (.so in unix, .dll in Windows) using the ctypes module.
There is also cython - a variation of python that compiles to C and can call C libraries directly. You can mix modules written in pure Python and cython.
You may also want to look at the numerous 3D game engines either written specifically for Python or with a python interface. The ones I have heard the most about (but not used) are Blender and Python-Ogre.
Panda3D is an engine which uses Python as it's "game logic" interface. You basically write everything in Python and the Panda3D backend (which I assume is mostly written in C or C++) is responsible for rendering.
Check out the gallery of projects that use Panda3D. It's not going to be AAA the-next-Gears-of-War level graphics, but it's still pretty impressive.
Using swig you can make C imports in various languages: lua, python, php, c# ...
See more info here about supported wrappers.
Python is able to use C libraries via the ctypes module. You'll have to write some Python code to import the C functions, but if the C API is clean and simple you'll have no trouble at all.
You might find these useful:
C functions from Python
Integrating Python, C and C++
I have been using PyOpenGL, it works great. Swig does its job if you want to call C/C++ libraries from Python.
I'm surprised that no-one has yet stated clearly that C++ is what you are looking for. Like you I have a distaste for C syntax, but that's a poor reason for avoiding C++ if you want to get into 3D gaming. Do you want to dive into 3D gaming, or do you want to sit on the edge of the pool crying that the water is too cold ?
I think you'll also find that C++ plays very well with OpenGL, which is probably not true of a lot of the alternatives that have already been suggested
To some extent, Cython might be what you are looking for. It allows you to use Python as a high level language, but then use C for the parts that need to be optimized.
But, at the end of the day, if you want to do 3D, just learning C or C++ may be the way to go. :-)
There are Python wrappers available for major open source game engines (Ogre, Irrlicht, etc.). Particularly Panda3D ought to have nice bindings.
If you'd like to have a look at .Net platform. You have the following solution:
Use C++/CLI to compile your C/C++ code into .Net assembly, the running time of this part would be as the same as your native C/C++ code.
Use any .Net language (C#, F#, IronPython) to develop high-level stuff using the low level library. For pure number crunching, C#/F# is usually 2-4 times slower than native C code, which is still far faster than Python. For non-number crunching tasks, C#/F# could sometimes match the speed of native code.
I'm learning C++ because it's a very flexible language. But for internet things like Twitter, Facebook, Delicious and others, Python seems a much better solution.
Is it possible to integrate C++ and Python in the same project?
Interfacing Python with C/C++ is not an easy task.
Here I copy/paste a previous answer on a previous question for the different methods to write a python extension. Featuring Boost.Python, SWIG, Pybindgen...
You can write an extension yourself in C or C++ with the Python C-API.
In a word: don't do that except for learning how to do it. It's very difficult to do it correctly. You will have to increment and decrement references by hand and write a lot of code just to expose one function, with very few benefits.
Swig:
pro: you can generate bindings for many scripting languages.
cons: I don't like the way the parser works. I don't know if they've made some progress but two years ago the C++ parser was quite limited. Most of the time I had to copy/paste my .h headers to add some % characters and to give extra hints to the swig parser.
I also needed to deal with the Python C-API from time to time for (not so) complicated type conversions.
I'm not using it anymore.
Boost.Python:
pro:
It's a very complete library. It allows you to do almost everything that is possible with the C-API, but in C++. I never had to write a C-API code with this library. I also never encountered a bug due to the library. Code for bindings either works like a charm or refuses to compile.
It's probably one of the best solutions currently available if you already have some C++ library to bind. But if you only have a small C function to rewrite, I would probably try with Cython.
cons: if you don't have a precompiled Boost.Python library you're going to use Bjam (sort of a replacement of make). I really hate Bjam and its syntax.
Python libraries created with B.P tend to become obese. It also takes a lot of time to compile them.
Py++: it's Boost.Python made easy. Py++ uses a C++ parser to read your code and then generates Boost.Python code automatically. You also have a great support from its author (no it's not me ;-) ).
cons: only the problems due to Boost.Python itself.
Edit this project looks discontinued. While probably still working it may be better to consider switching.
Pybindgen:
It generates the code dealing with the C-API. You can either describe functions and classes in a Python file, or let Pybindgen read your headers and generate bindings automatically (for this it uses pygccxml, a python library wrote by the author of Py++).
cons: it's a young project, with a smaller team than Boost.Python. There are still some limitations: you cannot expose your own C++ exceptions, you cannot use multiple inheritance for your C++ classes.
Anyway it's worth trying!
Pyrex and Cython:
Here you don't write real C/C++ but a mix between Python and C. This intermediate code will generate a regular Python module.
Edit Jul 22 2013: Now Py++ looks discontinued, I'm now looking for a good alternative. I'm currently experimenting with Cython for my C++ library. This language is a mix between Python and C. Within a Cython function you can use either Python or C/C++ entities (functions, variables, objects, ...).
Cython is quite easy to learn, has very good performance, and you can even avoid C/C++ completely if you don't have to interface legacy C++ libraries.
However for C++ it comes with some problems. It is less "automagic" than Py++ was, so it's probably better for stable C++ API (which is now the case of my library). The biggest problem I see with Cython is with C++ polymorphism. With Py++/boost:python I was able to define a virtual method in C++, override it in Python, and have the Python version called within C++. With Cython it's still doable but you need to explicitly use the C-Python API.
Edit 2017-10-06:
There is a new one, pybind11, similar to Boost.Python but with some potential advantages. For example it uses C++11 language features to make it simpler to create new bindings. Also it is a header-only library, so there is nothing to compile before using it, and no library to link.
I played with it a little bit and it was indeed quite simple and pleasant to use. My only fear is that like Boot.Python it could lead to long compilation time and large libraries. I haven't done any benchmark yet.
Yes, it is possible, encouraged and documented. I have done it myself and found it to be very easy.
Python/C API Reference Manual - the API used by C and C++ programmers who want to write extension modules or embed Python.
Extending and Embedding the Python Interpreter
describes how to write modules in C or C++ to extend the Python interpreter with new modules. Those modules can define new functions but also new object types and their methods. The document also describes how to embed the Python interpreter in another application, for use as an extension language. Finally, it shows how to compile and link extension modules so that they can be loaded dynamically (at run time) into the interpreter, if the underlying operating system supports this feature.
Try Pyrex. Makes writing C++ extensions for Python easier.
We use swig very successfully in our product.
Basically swig takes your C++ code and generates a python wrapper around it.
I'd recommend looking at how PyTorch does their integration.
See this:
Extending Python with C or C++
"It is quite easy to add new built-in modules to Python, if you know how to program in C. Such extension modules can do two things that can't be done directly in Python: they can implement new built-in object types, and they can call C library functions and system calls.
To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header "Python.h". "
http://www.python.org/doc/2.5.2/ext/intro.html
PS It's spelt "integrate" :)
I've used PyCxx http://cxx.sourceforge.net/ in the past and i found that it was very good.
It wraps the python c API in a very elegant manner and makes it very simple to use.
It is very easy to write python extension in c++. It is provided with clear examples so it is easy to get started.
I've really enjoyed using this library and I do recommend it.
It depends on your portability requirements. I've been struggling with this for a while, and I ended up wrapping my C++ using the python API directly because I need something that works on systems where the admin has only hacked together a mostly-working gcc and python installation.
In theory Boost.Python should be a very good option, since Boost is available (almost) everywhere. Unfortunately, if you end up on a OS with an older default python installation (our collaboration is stuck with 2.4), you'll run into problems if you try to run Boost.Python with a newer version (we all use Python 2.6). Since your admin likely didn't bother to install a version of Boost corresponding to every python version, you'll have to build it yourself.
So if you don't mind possibly requiring some Boost setup on every system your code runs on, use Boost.Python. If you want code that will definitely work on any system with Python and a C++ compiler, use the Python API.
Another interesting way to do is python code generation by running python itself to parse c++ header files. OpenCV team successfully took this approach and now they have done exact same thing to make java wrapper for OpenCV library. I found this created cleaner Python API without limitation caused by a certain library.
You can write Python extensions in C++. Basically Python itself is written in C and you can use that to call into your C code. You have full access to your Python objects. Also check out Boost.Python.