I'm learning C++ because it's a very flexible language. But for internet things like Twitter, Facebook, Delicious and others, Python seems a much better solution.
Is it possible to integrate C++ and Python in the same project?
Interfacing Python with C/C++ is not an easy task.
Here I copy/paste a previous answer on a previous question for the different methods to write a python extension. Featuring Boost.Python, SWIG, Pybindgen...
You can write an extension yourself in C or C++ with the Python C-API.
In a word: don't do that except for learning how to do it. It's very difficult to do it correctly. You will have to increment and decrement references by hand and write a lot of code just to expose one function, with very few benefits.
Swig:
pro: you can generate bindings for many scripting languages.
cons: I don't like the way the parser works. I don't know if they've made some progress but two years ago the C++ parser was quite limited. Most of the time I had to copy/paste my .h headers to add some % characters and to give extra hints to the swig parser.
I also needed to deal with the Python C-API from time to time for (not so) complicated type conversions.
I'm not using it anymore.
Boost.Python:
pro:
It's a very complete library. It allows you to do almost everything that is possible with the C-API, but in C++. I never had to write a C-API code with this library. I also never encountered a bug due to the library. Code for bindings either works like a charm or refuses to compile.
It's probably one of the best solutions currently available if you already have some C++ library to bind. But if you only have a small C function to rewrite, I would probably try with Cython.
cons: if you don't have a precompiled Boost.Python library you're going to use Bjam (sort of a replacement of make). I really hate Bjam and its syntax.
Python libraries created with B.P tend to become obese. It also takes a lot of time to compile them.
Py++: it's Boost.Python made easy. Py++ uses a C++ parser to read your code and then generates Boost.Python code automatically. You also have a great support from its author (no it's not me ;-) ).
cons: only the problems due to Boost.Python itself.
Edit this project looks discontinued. While probably still working it may be better to consider switching.
Pybindgen:
It generates the code dealing with the C-API. You can either describe functions and classes in a Python file, or let Pybindgen read your headers and generate bindings automatically (for this it uses pygccxml, a python library wrote by the author of Py++).
cons: it's a young project, with a smaller team than Boost.Python. There are still some limitations: you cannot expose your own C++ exceptions, you cannot use multiple inheritance for your C++ classes.
Anyway it's worth trying!
Pyrex and Cython:
Here you don't write real C/C++ but a mix between Python and C. This intermediate code will generate a regular Python module.
Edit Jul 22 2013: Now Py++ looks discontinued, I'm now looking for a good alternative. I'm currently experimenting with Cython for my C++ library. This language is a mix between Python and C. Within a Cython function you can use either Python or C/C++ entities (functions, variables, objects, ...).
Cython is quite easy to learn, has very good performance, and you can even avoid C/C++ completely if you don't have to interface legacy C++ libraries.
However for C++ it comes with some problems. It is less "automagic" than Py++ was, so it's probably better for stable C++ API (which is now the case of my library). The biggest problem I see with Cython is with C++ polymorphism. With Py++/boost:python I was able to define a virtual method in C++, override it in Python, and have the Python version called within C++. With Cython it's still doable but you need to explicitly use the C-Python API.
Edit 2017-10-06:
There is a new one, pybind11, similar to Boost.Python but with some potential advantages. For example it uses C++11 language features to make it simpler to create new bindings. Also it is a header-only library, so there is nothing to compile before using it, and no library to link.
I played with it a little bit and it was indeed quite simple and pleasant to use. My only fear is that like Boot.Python it could lead to long compilation time and large libraries. I haven't done any benchmark yet.
Yes, it is possible, encouraged and documented. I have done it myself and found it to be very easy.
Python/C API Reference Manual - the API used by C and C++ programmers who want to write extension modules or embed Python.
Extending and Embedding the Python Interpreter
describes how to write modules in C or C++ to extend the Python interpreter with new modules. Those modules can define new functions but also new object types and their methods. The document also describes how to embed the Python interpreter in another application, for use as an extension language. Finally, it shows how to compile and link extension modules so that they can be loaded dynamically (at run time) into the interpreter, if the underlying operating system supports this feature.
Try Pyrex. Makes writing C++ extensions for Python easier.
We use swig very successfully in our product.
Basically swig takes your C++ code and generates a python wrapper around it.
I'd recommend looking at how PyTorch does their integration.
See this:
Extending Python with C or C++
"It is quite easy to add new built-in modules to Python, if you know how to program in C. Such extension modules can do two things that can't be done directly in Python: they can implement new built-in object types, and they can call C library functions and system calls.
To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header "Python.h". "
http://www.python.org/doc/2.5.2/ext/intro.html
PS It's spelt "integrate" :)
I've used PyCxx http://cxx.sourceforge.net/ in the past and i found that it was very good.
It wraps the python c API in a very elegant manner and makes it very simple to use.
It is very easy to write python extension in c++. It is provided with clear examples so it is easy to get started.
I've really enjoyed using this library and I do recommend it.
It depends on your portability requirements. I've been struggling with this for a while, and I ended up wrapping my C++ using the python API directly because I need something that works on systems where the admin has only hacked together a mostly-working gcc and python installation.
In theory Boost.Python should be a very good option, since Boost is available (almost) everywhere. Unfortunately, if you end up on a OS with an older default python installation (our collaboration is stuck with 2.4), you'll run into problems if you try to run Boost.Python with a newer version (we all use Python 2.6). Since your admin likely didn't bother to install a version of Boost corresponding to every python version, you'll have to build it yourself.
So if you don't mind possibly requiring some Boost setup on every system your code runs on, use Boost.Python. If you want code that will definitely work on any system with Python and a C++ compiler, use the Python API.
Another interesting way to do is python code generation by running python itself to parse c++ header files. OpenCV team successfully took this approach and now they have done exact same thing to make java wrapper for OpenCV library. I found this created cleaner Python API without limitation caused by a certain library.
You can write Python extensions in C++. Basically Python itself is written in C and you can use that to call into your C code. You have full access to your Python objects. Also check out Boost.Python.
Related
I'm a Python guy building a Linux-based web service for a client who wants me to interface with a small C++ library that they're currently using with a bunch of Windows based VB applications.
They have assured me that the library is fairly simple (as far as they go I guess), and that they just need to know how best to compile and deliver it to me so that I can use it in Python under Linux.
I've read a bit about the ctypes library and other options (SWIG, etc), but for some reason I haven't really been able to wrap my head around the concept and still don't know how to tell them what I need.
I'm pretty sure having them re-write it with Python.h, etc is out, so I'm hoping there's a way I can simply have them compile it on Linux as a .so and just import it into Python. Is such a thing possible? How does one accomplish this?
No, such a thing is not possible.
Either they have to provide Python bindings, or you do. Either one of you can do this in any of the following ways:
Using <Python.h> directly to write a C extension module.
Using boost::python to make writing the extension module much easier (especially when they're using C++ rather than C).
Using SWIG, SIP, or similar tools to partly automate the writing of the extension module.
Using Cython to write the extension module in a Python-like language, instead of in C or C++.
Using ctypes from within Python.
For very simple cases (especially if they're actually exporting a C interface to their C++ code), ctypes probably is the easiest solution. Otherwise, I'd suggest looking at Cython first. But at any rate, you're going to have to wrap your head around one of the solutions—or convince them to do it instead.
Due to complexities of the C++ ABI (such as name mangling), it's generally difficult and platform-specific to load a C++ library directly from Python using ctypes.
I'd recommend you either create a simple C API which can be easily wrapped with ctypes, or use SWIG to generate wrapper types and a proper extension module for Python.
I have a scenario where I have some functions in C++ classes and I want to be able to call them using a python script. Let's say I have a function
void greet(_msg);
std::cout >> _msg >> std::endl;
I want to be able to call it trough a custom Python call and pass arguments to it, for example using
saySomething("Hello")
As a .py file I want it to call the greet function and pass "Hello" as an argument.
I know it's a subject that has been throughly discussed, and I've done a share of research on embedding python in C++, I've managed to read values from a python script using the standard Python/C API and run a function in Python from C++ and pass argument to it, but I can't seem to get my head around how to achieve this specific outcome.
I've had a look at ctypes and various wrappin libraries such as boost:python or swig, but I can't seem to understand to which degree they could help me achieve want I want.
Depending on which version of Python you are interested in, 2.x or 3.x,
read through the Extending and Embedding the Python Interpreter chapter for 2.x or 3.x. You are interested only in extending Python, so section the 1. Extending Python with C or C++ will provide you with complete explanation how to implement what you need in order to be able to call your functions implemented in C++ from Python script.
Certainly, there are numerous libraries and generators which allow you to wrap C/C++ APIs for Python (e.g. Boost.Python or SWIG), but your case sounds simple enough, that for the purpose of learning it is IMO better to get familiar with Python C API. Even if you use these tools, you will frequently have to get down to Python C API anyway or at least understand it.
I recently needed to do this very thing. Boost.Python does what we're looking for (and more) but personally (as much as I love Boost) I find it a little overkill to have to drag in half the Boost library to get one feature. SWIG also wasn't really an option for me as code generation always becomes a pain to maintain while class structures change (Don't get me wrong, these are BRILLIANT solutions!, just not what I was looking for).
So, the only thing left for me was to implement it from first principles (Python/C API). Hense, "ECS:Python" was born. ECS:Python (Embedded C++ Scripting with Python) is a simple C++ Python wrapper library I designed specifically for C++ developers. It allows you to expose objects from a C++ application to an embedded Python interpreter for interactive scripting, and it's very light-weight and easy to use.
Its free (BSD) and open source. If you're interested here it is:
http://sourceforge.net/projects/ecspython
You can use the weave.inline() function, which is part of the scipy package, to compile and execute C/C++ files and get their output from within your python script.
I have a demo application that is written in Python. It uses a lot of existing C++ code (written by me) that relies on OpenCV for image processing. Currently, communication between Python and C++ is being done through file I/O and subprocess calls, which isn't very efficient. What is the best way to wrap the C++ code so that it can be called from Python?
There is too much C++ code to think about porting it to Python, so that's not really an option.
A long time ago, the Python OpenCV wrappers were written in SWIG, but it looks like the most recent version of the wrappers is completely different. Can anyone point me in the right direction?
There are two ways that you can make your python program interact directly with your C/C++ program:
Wrap your C/C++ code in a DLL with C-API only. Then, use ctypes
to call C-function within the DLL. The advantage of this way is that you don't need to include/link any other library.
Extend python by adding new python module. You may use boost python to easily create a python module. The advantage of this way is that you don't need to wrap your code to C-API.
Without knowing code complexity, variety of C++ code and style of it... I would recommend "Extending Python"
It's not an immediate solution (you should change the C++ code, prototype some new functions or add a simple wrapper layer in C). But, if you plan to do a complex project (and are also a bit worried on performance)... it seems the best way to do it.
Porting C++ code to Python seems a step backwards, doing new code in Python is ok (I'm a fan of it) but C++ will (often) be more efficient.
Edit: Take also a look on ctypes module. Maybe it suits your needs. If you are more comfortable doing the wrapping in python language, then it may be better. If you don't mind playing with the C code, then extend Python by doing a module with your existing code.
I have fairly large C++ library with several sub-libraries that support it, and I need to turn the whole thing into a python extension. I'm using distutils because it needs to be cross-platform, but if there's a better tool I'm open to suggestions.
Is there a way to make distutils first compile the sub-libraries, and link them in when it creates an extension from the main library?
I do just this with a massive C++ library in our product. There are several tools out there that can help you automate the task of writing bindings: the most popular is SWIG, which has been around a while, is used in lots of projects, and generally works very well.
The biggest thing against SWIG (in my opinion) is that the C++ codebase of SWIG itself is really rather crufty to put it mildly. It was written before the STL and has it's own semi-dynamic type system which is just old and creaky now. This won't matter much unless you ever have to get stuck in and make some modifications to the core (I once tried to add doxygen -> docstring conversion) but if you ever do, good luck to you! People also say that SWIG generated code is not that efficient, which may be true but for me I've never found the SWIG calls themselves to be enough of a bottleneck to worry about it.
There are other tools you can use if SWIG doesn't float your boat: boost.python is also popular and could be a good option if you already use boost libraries in your C++ code. The downside is that it is heavy on compile times since it is pretty much all c++ template based.
Both these tools require you to do some work up-front in order to define what will be exposed and quite how it will be done. For SWIG you provide interface files which are like C++ headers but stripped down, and with some extra directives to tell SWIG how to translate complex types etc. Writing these interfaces can be tedious, so you may want to look at something like pygccxml to help you auto-generate them for you.
The author of that package actually wrote another extension which you might like: py++. This package does two things: it can autogenerate binding definitions that can then be fed to boost.python to generate python bindings: basically it is the full solution for most people. You might want to start there if you no particulrly special or difficult requirements to meet.
Some other questions that might prove useful as a reference:
Extending python - to swig or not to swig
SWIG vs CTypes
Extending Python with C/C++
You may also find this comparison of binding generation tools for Python handy. As Alex points out in the comments though, its rather old now but at least gives you some idea of the landscape...
In terms of how to drive the build, you may want to look at a more advanced built tool that distutils: if you want to stick with Python I would highly recommend Waf as a framework (others will tell you SCons is the way to go, but believe me it's slow as hell: I've been there and back already!)...it takes a little learning, but when you get your head around it is extremely powerful. And since it's pure Python it will integrate perfectly with any other Python code you have as part of your build process (say for example you use Py++ in the end)...
I'm currently working on a project were I had to wrap the C++ classes with Python to be able to script the program. So my specific experience also involved embedding the Python interpreter in our program.
The alternatives I tried were:
Boost.Python
I liked the cleaner API produced by Boost.Python, but the fact that it would have required that users install an additional dependency made us switch to SWIG.
SWIG
SWIG's main advantage for us was that it doesn't require end users to install it to use the final program.
What have you used to do this, and what has been your experience with it?
I've used both (for the same project): Boost is better integrated with the STL, and especially C++ exceptions. Also, its memory management mechanism (which tries to bridge C++ memory management and Python GC) is way more flexible than SWIG's. However, SWIG has much better documentation, no external dependencies, and if you get the library wrapped in SWIG for Python you're more than half-way there to getting a Java/Perl/Ruby wrapper as well.
I don't think there's a clear-cut choice: for smaller projects, I'd go with Boost.Python again, for larger long-lived projects, the extra investment in SWIG is worth it.
EDIT - the Robin project is sadly abandoned, and won't be much use today
I've used Robin with great success.
Great integration with C++ types, and creates a single .cpp file to compile and include in your shared object.
I suggest SIP. SIP is better than SWIG due to the following reasons:
For a given set of files, swig generates more duplicate (overhead) code than SIP. SIP manages to generate less duplicate (overhead) code by using a library file which can be statically or dynamically linked. In other words SIP has better scalability.
Execution time of SIP is much less than that of SWIG. Refer Python Wrapper Tools: A Performance Study. Unfortunately link appears broken. I have a personal copy which can be shared on request.
pyrex or cython are also good and easy ways for mixing the two worlds.
Wrapping C++ using these tools is a bit trickier then wrapping C but it can be done. Here is the wiki page about it.
A big plus for Boost::Python is that it allows for tab completion in the ipython shell: You import a C++ class, exposed by Boost directly, or you subclass it, and from then on, it really behaves like a pure Python class.
The downside: It takes so long to install and use Boost that all the Tab-completion time-saving won't ever amortize ;-(
So I prefer Swig: No bells and whistles, but works reliably after a short introductory example.