I'm a newbie to boost and one of its libraries which I can't understand it is Boost.Python. Can anyone explain me in details how does this interoperability achieved?In the documentation there only a few words about metaprogramming.
P.S. I tried to look code but because of my lack of C++ knowledge I didn't understand principles.
Thanks in advance
There are two ways to interoperate:
1) from a "Python process", call functions written in C++.
Python already has a system to load dlls, they're called "extension modules". Boost.Python can compile your source to produce one. Basically you write a little wrapper to declare a function callable from Python, and the "metaprogramming" is there to do stuff like detecting what types the C++ function takes and returns, so that it can emit the right code to convert those from/to the equivalent Python types.
2) from a "C++ process", launch and control the Python interpreter.
Python provides a C API to do this, and Boost.Python knows how to use it.
Related
I have a scenario where I have some functions in C++ classes and I want to be able to call them using a python script. Let's say I have a function
void greet(_msg);
std::cout >> _msg >> std::endl;
I want to be able to call it trough a custom Python call and pass arguments to it, for example using
saySomething("Hello")
As a .py file I want it to call the greet function and pass "Hello" as an argument.
I know it's a subject that has been throughly discussed, and I've done a share of research on embedding python in C++, I've managed to read values from a python script using the standard Python/C API and run a function in Python from C++ and pass argument to it, but I can't seem to get my head around how to achieve this specific outcome.
I've had a look at ctypes and various wrappin libraries such as boost:python or swig, but I can't seem to understand to which degree they could help me achieve want I want.
Depending on which version of Python you are interested in, 2.x or 3.x,
read through the Extending and Embedding the Python Interpreter chapter for 2.x or 3.x. You are interested only in extending Python, so section the 1. Extending Python with C or C++ will provide you with complete explanation how to implement what you need in order to be able to call your functions implemented in C++ from Python script.
Certainly, there are numerous libraries and generators which allow you to wrap C/C++ APIs for Python (e.g. Boost.Python or SWIG), but your case sounds simple enough, that for the purpose of learning it is IMO better to get familiar with Python C API. Even if you use these tools, you will frequently have to get down to Python C API anyway or at least understand it.
I recently needed to do this very thing. Boost.Python does what we're looking for (and more) but personally (as much as I love Boost) I find it a little overkill to have to drag in half the Boost library to get one feature. SWIG also wasn't really an option for me as code generation always becomes a pain to maintain while class structures change (Don't get me wrong, these are BRILLIANT solutions!, just not what I was looking for).
So, the only thing left for me was to implement it from first principles (Python/C API). Hense, "ECS:Python" was born. ECS:Python (Embedded C++ Scripting with Python) is a simple C++ Python wrapper library I designed specifically for C++ developers. It allows you to expose objects from a C++ application to an embedded Python interpreter for interactive scripting, and it's very light-weight and easy to use.
Its free (BSD) and open source. If you're interested here it is:
http://sourceforge.net/projects/ecspython
You can use the weave.inline() function, which is part of the scipy package, to compile and execute C/C++ files and get their output from within your python script.
I want to execute a code helloword.cpp which takes in some argument from console parses those arguments and then prints "hello world" in the console.
Now, I want to parse these arguments from a python scripts parsearguments.py
So for example:
def parse_arguments:
...# some code
return arguments
Now, how do i communicate between python and c++.
I have been reading and see that cython, boost python are the options but I have a hard time finding the right simple hello world example.
Any suggestions will be appreciated.
Thanks
To execute C++ code in python, you could effectively use boost python, here is a tutorial:
http://www.boost.org/doc/libs/1_59_0/libs/python/doc/index.html
You write a kind of wrapper outside you C++ code.
If it is C code, python has internal library called ctypes.
In both case, you should compile the C/C++ code into shared library.
How about passing whatever text you generate with Python into the standard input of your C++ program? Basically, you have to use Python's subprocess module to fire up the C++ program and dump the text into its standard output.
In case that your C++ program is required to be running separately in the background, you could try another form of interprocess communication, like unix domain sockets.
Using boost::python is also an option, but it might be a little more difficult to deal with.
A couple of other options besides Boost.python are SIP and SWIG (Simplified Wrapper and Interface Generator). Like Boost, SIP and SWIG are open source.
SWIG is particularly powerful, but also a bit hairy. It provides support for interfacing C and C++ with a boatload of other languages including (not a complete list) Python, Perl, Lua, Tcl/Tk, Ocaml, Ruby, Java. One aspect of SWIG is that it parses your C++ headers. This has benefits and pitfalls. A benefit is that it does most of the work of generating the interfaces. A downside is that it doesn't handle some of the dark corners of C++ 2003, and it hasn't stepped up to C++11 at all. Another downside is that compilation of a large project becomes slow. Very, very slow.
Using boost.python sounds like a good solution for me. But depending on your C++ experience this can be quite tricky. A good point to start is here:
http://wiki.python.org/moin/boost.python
Boost.Python enables you to export C++ classes and member functions to Python to be able to use them from there.
I have a demo application that is written in Python. It uses a lot of existing C++ code (written by me) that relies on OpenCV for image processing. Currently, communication between Python and C++ is being done through file I/O and subprocess calls, which isn't very efficient. What is the best way to wrap the C++ code so that it can be called from Python?
There is too much C++ code to think about porting it to Python, so that's not really an option.
A long time ago, the Python OpenCV wrappers were written in SWIG, but it looks like the most recent version of the wrappers is completely different. Can anyone point me in the right direction?
There are two ways that you can make your python program interact directly with your C/C++ program:
Wrap your C/C++ code in a DLL with C-API only. Then, use ctypes
to call C-function within the DLL. The advantage of this way is that you don't need to include/link any other library.
Extend python by adding new python module. You may use boost python to easily create a python module. The advantage of this way is that you don't need to wrap your code to C-API.
Without knowing code complexity, variety of C++ code and style of it... I would recommend "Extending Python"
It's not an immediate solution (you should change the C++ code, prototype some new functions or add a simple wrapper layer in C). But, if you plan to do a complex project (and are also a bit worried on performance)... it seems the best way to do it.
Porting C++ code to Python seems a step backwards, doing new code in Python is ok (I'm a fan of it) but C++ will (often) be more efficient.
Edit: Take also a look on ctypes module. Maybe it suits your needs. If you are more comfortable doing the wrapping in python language, then it may be better. If you don't mind playing with the C code, then extend Python by doing a module with your existing code.
I have a libfoo.so library built from C++ code (compiled with gcc), and I would like to quickly test some of its exported classes (basically, instantiating a class then calling its methods to check the output).
While I could do that in C/C++ with a main file that links to the library in question and build my tests, but I think it would be much easier if it was possible to simply call Python from the command line and call the methods from there.
I know I can use CDLL from ctypes to load C-style libraries, but is there a similar functionality for C++ libraries and objects?
EDIT : Ideally, I do not want to modify the C++ code, I would need to use it as-is.
Honestly C++ is a bit messy. You could do something like create a pure C function which wraps the C++ functionality (which you then call from python) but at that point you might as well write your tests in C++. Unfortunately the only tool out there for this (that I know of) is SWIG.
It's sad that it's called the "simplified" wrapper and interface generator, because there's nothing simple about it. If you have VERY primitive data types in your signatures (like, JUST ints or maybe char*) it'll be a pretty quick job. Otherwise you have to tell swig how to marshal your data types between languages, and that gets REALLY ugly very quickly. Furthermore, after short time you realize you have to learn the CPython API in order to write your marshalling code.
And by that point you may as well write your own CPython wrapper without involving SWIG. You suddenly realize you've spent a good month learning a new API and feel horribly frustrated. If you're going to be doing this a lot, it's definitely worth your time. However if this is a one-time thing, just write your tests in C / C++.
(I'm speaking from experience here)
I agree with Chris answer. However, I want to point out that Cython supports C++ (with some limitations).
I'm learning C++ because it's a very flexible language. But for internet things like Twitter, Facebook, Delicious and others, Python seems a much better solution.
Is it possible to integrate C++ and Python in the same project?
Interfacing Python with C/C++ is not an easy task.
Here I copy/paste a previous answer on a previous question for the different methods to write a python extension. Featuring Boost.Python, SWIG, Pybindgen...
You can write an extension yourself in C or C++ with the Python C-API.
In a word: don't do that except for learning how to do it. It's very difficult to do it correctly. You will have to increment and decrement references by hand and write a lot of code just to expose one function, with very few benefits.
Swig:
pro: you can generate bindings for many scripting languages.
cons: I don't like the way the parser works. I don't know if they've made some progress but two years ago the C++ parser was quite limited. Most of the time I had to copy/paste my .h headers to add some % characters and to give extra hints to the swig parser.
I also needed to deal with the Python C-API from time to time for (not so) complicated type conversions.
I'm not using it anymore.
Boost.Python:
pro:
It's a very complete library. It allows you to do almost everything that is possible with the C-API, but in C++. I never had to write a C-API code with this library. I also never encountered a bug due to the library. Code for bindings either works like a charm or refuses to compile.
It's probably one of the best solutions currently available if you already have some C++ library to bind. But if you only have a small C function to rewrite, I would probably try with Cython.
cons: if you don't have a precompiled Boost.Python library you're going to use Bjam (sort of a replacement of make). I really hate Bjam and its syntax.
Python libraries created with B.P tend to become obese. It also takes a lot of time to compile them.
Py++: it's Boost.Python made easy. Py++ uses a C++ parser to read your code and then generates Boost.Python code automatically. You also have a great support from its author (no it's not me ;-) ).
cons: only the problems due to Boost.Python itself.
Edit this project looks discontinued. While probably still working it may be better to consider switching.
Pybindgen:
It generates the code dealing with the C-API. You can either describe functions and classes in a Python file, or let Pybindgen read your headers and generate bindings automatically (for this it uses pygccxml, a python library wrote by the author of Py++).
cons: it's a young project, with a smaller team than Boost.Python. There are still some limitations: you cannot expose your own C++ exceptions, you cannot use multiple inheritance for your C++ classes.
Anyway it's worth trying!
Pyrex and Cython:
Here you don't write real C/C++ but a mix between Python and C. This intermediate code will generate a regular Python module.
Edit Jul 22 2013: Now Py++ looks discontinued, I'm now looking for a good alternative. I'm currently experimenting with Cython for my C++ library. This language is a mix between Python and C. Within a Cython function you can use either Python or C/C++ entities (functions, variables, objects, ...).
Cython is quite easy to learn, has very good performance, and you can even avoid C/C++ completely if you don't have to interface legacy C++ libraries.
However for C++ it comes with some problems. It is less "automagic" than Py++ was, so it's probably better for stable C++ API (which is now the case of my library). The biggest problem I see with Cython is with C++ polymorphism. With Py++/boost:python I was able to define a virtual method in C++, override it in Python, and have the Python version called within C++. With Cython it's still doable but you need to explicitly use the C-Python API.
Edit 2017-10-06:
There is a new one, pybind11, similar to Boost.Python but with some potential advantages. For example it uses C++11 language features to make it simpler to create new bindings. Also it is a header-only library, so there is nothing to compile before using it, and no library to link.
I played with it a little bit and it was indeed quite simple and pleasant to use. My only fear is that like Boot.Python it could lead to long compilation time and large libraries. I haven't done any benchmark yet.
Yes, it is possible, encouraged and documented. I have done it myself and found it to be very easy.
Python/C API Reference Manual - the API used by C and C++ programmers who want to write extension modules or embed Python.
Extending and Embedding the Python Interpreter
describes how to write modules in C or C++ to extend the Python interpreter with new modules. Those modules can define new functions but also new object types and their methods. The document also describes how to embed the Python interpreter in another application, for use as an extension language. Finally, it shows how to compile and link extension modules so that they can be loaded dynamically (at run time) into the interpreter, if the underlying operating system supports this feature.
Try Pyrex. Makes writing C++ extensions for Python easier.
We use swig very successfully in our product.
Basically swig takes your C++ code and generates a python wrapper around it.
I'd recommend looking at how PyTorch does their integration.
See this:
Extending Python with C or C++
"It is quite easy to add new built-in modules to Python, if you know how to program in C. Such extension modules can do two things that can't be done directly in Python: they can implement new built-in object types, and they can call C library functions and system calls.
To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header "Python.h". "
http://www.python.org/doc/2.5.2/ext/intro.html
PS It's spelt "integrate" :)
I've used PyCxx http://cxx.sourceforge.net/ in the past and i found that it was very good.
It wraps the python c API in a very elegant manner and makes it very simple to use.
It is very easy to write python extension in c++. It is provided with clear examples so it is easy to get started.
I've really enjoyed using this library and I do recommend it.
It depends on your portability requirements. I've been struggling with this for a while, and I ended up wrapping my C++ using the python API directly because I need something that works on systems where the admin has only hacked together a mostly-working gcc and python installation.
In theory Boost.Python should be a very good option, since Boost is available (almost) everywhere. Unfortunately, if you end up on a OS with an older default python installation (our collaboration is stuck with 2.4), you'll run into problems if you try to run Boost.Python with a newer version (we all use Python 2.6). Since your admin likely didn't bother to install a version of Boost corresponding to every python version, you'll have to build it yourself.
So if you don't mind possibly requiring some Boost setup on every system your code runs on, use Boost.Python. If you want code that will definitely work on any system with Python and a C++ compiler, use the Python API.
Another interesting way to do is python code generation by running python itself to parse c++ header files. OpenCV team successfully took this approach and now they have done exact same thing to make java wrapper for OpenCV library. I found this created cleaner Python API without limitation caused by a certain library.
You can write Python extensions in C++. Basically Python itself is written in C and you can use that to call into your C code. You have full access to your Python objects. Also check out Boost.Python.