I started using SWIG on a huge C++ library (made of of several inter-dependent static libs) to expose it to python. This library defines many primitive classes that are used throughout as parameters (images of different types for example). There is extensive use of STL, inheritance and templating in the lib as well.
So far I have a minimal portion of the lib usable from Python but would like to progressively add the remaining 90+%.
For my part, working with SWIG is really no fun: Battling with template instantiations, learning all that SWIG syntax and keywords, etc. I recently played a bit with CTypes and found it so enjoyable that I am now considering writing an extern C interface for the whole library instead of using SWIG.
I would rather be coding in C/C++ and/or Python than learning an obscure set of SWIG commands (that counts for SIP also).
There are quite a few questions out there already asking similar advice so I'll add something new and specific:
I would like for the library's internal images classes to be visible from python as PIL images transparently, not as SWIG-Wrapped native classes. Will I have to resort to the plain-old Python External API to accomplish that?
I welcome any and all advice!
Have you considered looking into using Boost.Python?
Related
Is there a way to directly import C++ file in python file without converting C++ into DLL for e.g.
import CPP_file.cpp
import os
.
.
.
No. C++ code needs to be compiled!
There are different ways for embeddind c++ code in your python but they will all need you C++ code to be compiled in a sort of library!
one technology that you can use for example is SWIG:
http://www.swig.org/Doc1.3/Python.html
another alternative is to use Boost Python Library. Here a short description taken from their web page:
The Boost Python Library is a framework for interfacing Python and
C++. It allows you to quickly and seamlessly expose C++ classes
functions and objects to Python, and vice-versa, using no special
tools -- just your C++ compiler. It is designed to wrap C++ interfaces
non-intrusively, so that you should not have to change the C++ code at
all in order to wrap it, making Boost.Python ideal for exposing
3rd-party libraries to Python. The library's use of advanced
metaprogramming techniques simplifies its syntax for users, so that
wrapping code takes on the look of a kind of declarative interface
definition language (IDL).
I'm a Python guy building a Linux-based web service for a client who wants me to interface with a small C++ library that they're currently using with a bunch of Windows based VB applications.
They have assured me that the library is fairly simple (as far as they go I guess), and that they just need to know how best to compile and deliver it to me so that I can use it in Python under Linux.
I've read a bit about the ctypes library and other options (SWIG, etc), but for some reason I haven't really been able to wrap my head around the concept and still don't know how to tell them what I need.
I'm pretty sure having them re-write it with Python.h, etc is out, so I'm hoping there's a way I can simply have them compile it on Linux as a .so and just import it into Python. Is such a thing possible? How does one accomplish this?
No, such a thing is not possible.
Either they have to provide Python bindings, or you do. Either one of you can do this in any of the following ways:
Using <Python.h> directly to write a C extension module.
Using boost::python to make writing the extension module much easier (especially when they're using C++ rather than C).
Using SWIG, SIP, or similar tools to partly automate the writing of the extension module.
Using Cython to write the extension module in a Python-like language, instead of in C or C++.
Using ctypes from within Python.
For very simple cases (especially if they're actually exporting a C interface to their C++ code), ctypes probably is the easiest solution. Otherwise, I'd suggest looking at Cython first. But at any rate, you're going to have to wrap your head around one of the solutions—or convince them to do it instead.
Due to complexities of the C++ ABI (such as name mangling), it's generally difficult and platform-specific to load a C++ library directly from Python using ctypes.
I'd recommend you either create a simple C API which can be easily wrapped with ctypes, or use SWIG to generate wrapper types and a proper extension module for Python.
Suppose I have two projects that I would like to link together:
A C++ library compiled with Visual C++ to a DLL file.
A C++ executable compiled with C++ Builder that uses the classes in the library.
I realize that there is no standard C++ ABI and that any attempts to directly link these two C++ projects together will fail. What is a good, automated way of creating a compatibility layer that allows me to accomplish this?
For example, conceivably the C++ library could expose itself via a C interface. Then the executable would have some C++ classes that wrap the C interface exposed by the C++ library. Since there is a standard ABI for C, it would work.
The only question is how to automatically create the C interface and C++ wrapper classes - manually maintaining this would not be an option. The SWIG project looks promising, but unfortunately, C++ is not one of the exits of SWIG listed on their web site. Is there a way to do what I want with SWIG? Or is there another project other than SWIG that would help me with this task?
Or am I going about this the wrong way?
Edit: The core C++ library is intended to be cross-platform. The executable, obviously, is Windows-specific. I don't want to pollute the core library to the extent that it becomes impossible to compile it on other platforms.
If it only has to run on Windows, I would expose the classes as COM objects. They'll still be in a DLL and they can be used by any language which understands COM.
The "standard" way of doing this, in Windows, is to use COM objects. So, that is certainly a good option to look at. In Linux systems, the module interaction model (e.g., executable-DLL interaction) is very different, and ABIs exist for C++.
If you would want to do this manually (create your own COM-like library), it can be a lot of work with many little tricky issues to take seriously. You'll need a cross-module RTTI system, you'll need an interface query/definition protocol, some mechanism to manage memory across modules, etc. Beyond that, to "automate" it, you will probably need a combination of MACROs and template meta-functions.
One cross-platform option that I would highly recommend that you consider or at least look at is using Boost.Python and the Python language as the "glue" between your modules. The Boost.Python library basically does the whole "automated exporting / importing of classes", but it exports your C++ classes and functions as Python classes and functions. And, it is completely non-intrusive and cross-platform, so this is really an ideal example of automated exporting. So, you might consider using Python to write your high-level glue-code, or using Python as an intermediate between the C++ modules, or even reworking the Boost.Python library to use only the "automated exporting" mechanisms to export to whatever interface system you design or use.
I'm sure there a plenty other similar libraries out there. But the number one question is, of course, do you really need this? You might be using a bazooka to kill a fly.
Why not just compile the library with C++ builder as well?
Looking around at swig (I knew swig should be able to wrap C++ in C):
SWIG and C++
If core library is cross-platform why not also write the UI as a cross-platform Qt application and build everything in Visual C++ on Windows.
I have fairly large C++ library with several sub-libraries that support it, and I need to turn the whole thing into a python extension. I'm using distutils because it needs to be cross-platform, but if there's a better tool I'm open to suggestions.
Is there a way to make distutils first compile the sub-libraries, and link them in when it creates an extension from the main library?
I do just this with a massive C++ library in our product. There are several tools out there that can help you automate the task of writing bindings: the most popular is SWIG, which has been around a while, is used in lots of projects, and generally works very well.
The biggest thing against SWIG (in my opinion) is that the C++ codebase of SWIG itself is really rather crufty to put it mildly. It was written before the STL and has it's own semi-dynamic type system which is just old and creaky now. This won't matter much unless you ever have to get stuck in and make some modifications to the core (I once tried to add doxygen -> docstring conversion) but if you ever do, good luck to you! People also say that SWIG generated code is not that efficient, which may be true but for me I've never found the SWIG calls themselves to be enough of a bottleneck to worry about it.
There are other tools you can use if SWIG doesn't float your boat: boost.python is also popular and could be a good option if you already use boost libraries in your C++ code. The downside is that it is heavy on compile times since it is pretty much all c++ template based.
Both these tools require you to do some work up-front in order to define what will be exposed and quite how it will be done. For SWIG you provide interface files which are like C++ headers but stripped down, and with some extra directives to tell SWIG how to translate complex types etc. Writing these interfaces can be tedious, so you may want to look at something like pygccxml to help you auto-generate them for you.
The author of that package actually wrote another extension which you might like: py++. This package does two things: it can autogenerate binding definitions that can then be fed to boost.python to generate python bindings: basically it is the full solution for most people. You might want to start there if you no particulrly special or difficult requirements to meet.
Some other questions that might prove useful as a reference:
Extending python - to swig or not to swig
SWIG vs CTypes
Extending Python with C/C++
You may also find this comparison of binding generation tools for Python handy. As Alex points out in the comments though, its rather old now but at least gives you some idea of the landscape...
In terms of how to drive the build, you may want to look at a more advanced built tool that distutils: if you want to stick with Python I would highly recommend Waf as a framework (others will tell you SCons is the way to go, but believe me it's slow as hell: I've been there and back already!)...it takes a little learning, but when you get your head around it is extremely powerful. And since it's pure Python it will integrate perfectly with any other Python code you have as part of your build process (say for example you use Py++ in the end)...
I'm learning C++ because it's a very flexible language. But for internet things like Twitter, Facebook, Delicious and others, Python seems a much better solution.
Is it possible to integrate C++ and Python in the same project?
Interfacing Python with C/C++ is not an easy task.
Here I copy/paste a previous answer on a previous question for the different methods to write a python extension. Featuring Boost.Python, SWIG, Pybindgen...
You can write an extension yourself in C or C++ with the Python C-API.
In a word: don't do that except for learning how to do it. It's very difficult to do it correctly. You will have to increment and decrement references by hand and write a lot of code just to expose one function, with very few benefits.
Swig:
pro: you can generate bindings for many scripting languages.
cons: I don't like the way the parser works. I don't know if they've made some progress but two years ago the C++ parser was quite limited. Most of the time I had to copy/paste my .h headers to add some % characters and to give extra hints to the swig parser.
I also needed to deal with the Python C-API from time to time for (not so) complicated type conversions.
I'm not using it anymore.
Boost.Python:
pro:
It's a very complete library. It allows you to do almost everything that is possible with the C-API, but in C++. I never had to write a C-API code with this library. I also never encountered a bug due to the library. Code for bindings either works like a charm or refuses to compile.
It's probably one of the best solutions currently available if you already have some C++ library to bind. But if you only have a small C function to rewrite, I would probably try with Cython.
cons: if you don't have a precompiled Boost.Python library you're going to use Bjam (sort of a replacement of make). I really hate Bjam and its syntax.
Python libraries created with B.P tend to become obese. It also takes a lot of time to compile them.
Py++: it's Boost.Python made easy. Py++ uses a C++ parser to read your code and then generates Boost.Python code automatically. You also have a great support from its author (no it's not me ;-) ).
cons: only the problems due to Boost.Python itself.
Edit this project looks discontinued. While probably still working it may be better to consider switching.
Pybindgen:
It generates the code dealing with the C-API. You can either describe functions and classes in a Python file, or let Pybindgen read your headers and generate bindings automatically (for this it uses pygccxml, a python library wrote by the author of Py++).
cons: it's a young project, with a smaller team than Boost.Python. There are still some limitations: you cannot expose your own C++ exceptions, you cannot use multiple inheritance for your C++ classes.
Anyway it's worth trying!
Pyrex and Cython:
Here you don't write real C/C++ but a mix between Python and C. This intermediate code will generate a regular Python module.
Edit Jul 22 2013: Now Py++ looks discontinued, I'm now looking for a good alternative. I'm currently experimenting with Cython for my C++ library. This language is a mix between Python and C. Within a Cython function you can use either Python or C/C++ entities (functions, variables, objects, ...).
Cython is quite easy to learn, has very good performance, and you can even avoid C/C++ completely if you don't have to interface legacy C++ libraries.
However for C++ it comes with some problems. It is less "automagic" than Py++ was, so it's probably better for stable C++ API (which is now the case of my library). The biggest problem I see with Cython is with C++ polymorphism. With Py++/boost:python I was able to define a virtual method in C++, override it in Python, and have the Python version called within C++. With Cython it's still doable but you need to explicitly use the C-Python API.
Edit 2017-10-06:
There is a new one, pybind11, similar to Boost.Python but with some potential advantages. For example it uses C++11 language features to make it simpler to create new bindings. Also it is a header-only library, so there is nothing to compile before using it, and no library to link.
I played with it a little bit and it was indeed quite simple and pleasant to use. My only fear is that like Boot.Python it could lead to long compilation time and large libraries. I haven't done any benchmark yet.
Yes, it is possible, encouraged and documented. I have done it myself and found it to be very easy.
Python/C API Reference Manual - the API used by C and C++ programmers who want to write extension modules or embed Python.
Extending and Embedding the Python Interpreter
describes how to write modules in C or C++ to extend the Python interpreter with new modules. Those modules can define new functions but also new object types and their methods. The document also describes how to embed the Python interpreter in another application, for use as an extension language. Finally, it shows how to compile and link extension modules so that they can be loaded dynamically (at run time) into the interpreter, if the underlying operating system supports this feature.
Try Pyrex. Makes writing C++ extensions for Python easier.
We use swig very successfully in our product.
Basically swig takes your C++ code and generates a python wrapper around it.
I'd recommend looking at how PyTorch does their integration.
See this:
Extending Python with C or C++
"It is quite easy to add new built-in modules to Python, if you know how to program in C. Such extension modules can do two things that can't be done directly in Python: they can implement new built-in object types, and they can call C library functions and system calls.
To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header "Python.h". "
http://www.python.org/doc/2.5.2/ext/intro.html
PS It's spelt "integrate" :)
I've used PyCxx http://cxx.sourceforge.net/ in the past and i found that it was very good.
It wraps the python c API in a very elegant manner and makes it very simple to use.
It is very easy to write python extension in c++. It is provided with clear examples so it is easy to get started.
I've really enjoyed using this library and I do recommend it.
It depends on your portability requirements. I've been struggling with this for a while, and I ended up wrapping my C++ using the python API directly because I need something that works on systems where the admin has only hacked together a mostly-working gcc and python installation.
In theory Boost.Python should be a very good option, since Boost is available (almost) everywhere. Unfortunately, if you end up on a OS with an older default python installation (our collaboration is stuck with 2.4), you'll run into problems if you try to run Boost.Python with a newer version (we all use Python 2.6). Since your admin likely didn't bother to install a version of Boost corresponding to every python version, you'll have to build it yourself.
So if you don't mind possibly requiring some Boost setup on every system your code runs on, use Boost.Python. If you want code that will definitely work on any system with Python and a C++ compiler, use the Python API.
Another interesting way to do is python code generation by running python itself to parse c++ header files. OpenCV team successfully took this approach and now they have done exact same thing to make java wrapper for OpenCV library. I found this created cleaner Python API without limitation caused by a certain library.
You can write Python extensions in C++. Basically Python itself is written in C and you can use that to call into your C code. You have full access to your Python objects. Also check out Boost.Python.