I have a project where I would like to load Go plugins inside a C++ application.
After a lot of research, it is not clear for me whether or not Go supports this.
I encountered a lot of discussions pointing out the bad habit of dynamic linking, proning IPC instead. Moreover it is not clear for me if dynamic linking is intended by the language or not (new Go philosophy ?).
cgo provides the ability to call C from Go or Go from C (inside Go), but not from plain old C. Or does it ?
gc doesn't seem to support shared library (even if https://code.google.com/p/go/issues/detail?id=256 mentions it does)
gccgo support Go shared libraries but I couldn't make it work (probably because the main entry point is not in Go ...)
SWIG doesn't seem to help either :(
Apparently something is going on upstream as well (https://codereview.appspot.com/7304104/)
main.c
extern void Print(void) __asm__ ("example.main.Print");
int main() {
Print();
}
print.go
package main
import "fmt"
func Print() {
fmt.Printf("hello, world\n")
}
Makefile :
all: print.o main.c
gcc main.c -L. -lprint -o main
print.o: print.go
gccgo -fno-split-stack -fgo-prefix=example -fPIC -c print.go -o print.o
gccgo -shared print.o -o libprint.so
Output :
/usr/lib/libgo.so.3: undefined reference to `main.main'
/usr/lib/libgo.so.3: undefined reference to `__go_init_main'
Is there a solution for that ? What is the best approach ? forking + IPC ?
References :
cgo - go wiki
callback in cgo
c callbacks and non go threads
cgo - golang
call go function from c
link c code with go code
I don't think you can embed Go into C. However you can embed C into Go and with a little stub C program you can call into C first thing which is the next best thing! Cgo definitely supports linking with shared libraries so maybe this approach will work for you.
Like this
main.go
// Stub go program to call cmain() in C
package main
// extern int cmain(void);
import "C"
func main() {
C.cmain()
}
main.c
#include <stdio.h>
// Defined in Go
extern void Print(void);
// C Main program
int cmain() {
printf("Hello from C\n");
Print();
}
print.go
package main
import "fmt"
import "C"
//export Print
func Print() {
fmt.Printf("Hello from Go\n")
}
Compile with go build, and produces this output when you run it
Hello from C
Hello from Go
AFAIK, you cannot compile a Go package to a shared library witg 'gc' ATM, it may change in the future. There might be some chance with 'gccgo' as 'libgo' (Go runtime, I suppose) is a shared library already. I guess the missing piece is then only to properly initialize the runtime, which normally a Go command handles automatically.
The gccgo expert is I.L. Taylor, he is reachable at the golang-nuts mailing list almost daily. I suggest to ask him directly.
PS: Other problems possibly include the interaction of the Go garbage collector and the (C++) process memory etc. Maybe I'm too optimistic and it is not at all feasible.
I want to use LEAP Motion in D.
Therefore It doesn't have C library and It has only C++ library.
I tried SWIG 2.0.9 below command.
swig -c++ -d -d2 leap.i
This command output Leap.d, Leap_im.d, Leap_wrap.cxx, Leap_wrap.h.
However, I don't know how to use to wrapper in D and I can't find how to use the wrapper.
Link error displays to use it intact.
How use these wrapper in D2?
And can I use without Leap.cpp (source of Leap.dll)?
Update:
Thanks two answers. and sorry for reply late because of busy.
Say first conclusion I could build Leap sample code on Win64 by following the steps below.
Output wrappers by above command.
Create x64 DLL with VC2010 from Leap_wrap.cxx, Leap_wrap.h, and import Leap.lib(x64).
Compile Leap.d and Leap_im.d with dmd -c.
Build LeapTest.d with Leap.obj and Leap_im.obj
all command is below.
swig -c++ -d -d2 leap.i
dmd -c Leap.d Leap_im.d -m64
dmd LeapTest.d Leap.obj Leap_im.obj -m64
execute LeapTest.exe (require x64 Leap.dll and Leap_wrap.dll)
I could run Leap Program.
But program crach onFrame event callback.
I'll try again on x86 and investigate the causes.
Few helpful links (some information may be outdated):
http://klickverbot.at/blog/2010/11/announcing-d-support-in-swig/
http://www.swig.org/Doc2.0/D.html
http://www.swig.org/tutorial.html
I have never used SWIG personally but my guess based on general knowledge about SWIG:
Leap_wrap.cxx is C++ source file that wraps calls to C++ functions from target library in extern(C) calls
Leap_wrap.h is header file with all extern(C) wrappers listed
Leap_im.d is D module based on Leap_wrap.h with same extern(C) function listed
Leap.d is D module that uses Leap_im.d as an implementation and reproduces API similar to original C++ one.
So in your D code you want to import Leap.d module. Than compile Leap_wrap.cxx to an object file with your C++ compiler and provide D object files, Leap_wrap.o and target library at linking stage. That should do the trick.
P.S. Leap.cpp source should not be needed. All stuff links directly from Leap_wrap.cxx to target library binary.
Go to IRC, either FreeNode, or OFTC, channel #D. In order to help you, we have to see what is in those files. My first guess is that you have to compile both D files, and the C++ file into object files, and link them together. I suppose SWIG is going to flatten the C++ API into bunch of C functions, and that is probably what Leap_wrap.cxx does.
If the LEAP API is not complex (ie. just bunch of simple C++ classes), it may be possible to directly interface with it. Read more about it here: http://dlang.org/cpp_interface.html .
I have been running into trouble recently trying to symbolicate a crash log of an iOS app. For some reason the UUID of the dSYM was not indexed in Spotlight. After some manual search and a healthy dose of command line incantations, I managed to symbolicate partially the crash log.
At first I thought the dSYM might be incomplete or something like that, but then I realized that the method calls missing were the ones occurring in C++ code: this project is an Objective-C app that calls into C++ libraries (via Objective-C++) which call back to Objective-C code (again, via Objective-C++ code). The calls that I'm missing are, specifically, the ones that happen in C++ land.
So, my question is: is there some way that the symbolication process can resolve the function calls of C++ code? Which special options do I need to set, if any?
One useful program that comes with the apple sdk is atos (address to symbol). Basically, here's what you want to do:
atos -o myExecutable -arch armv7 0x(address here)
It should print out the name of the symbol at that address.
I'm not well versed in Objective-C, but I'd make sure that the C++ code is being compiled with symbols. Particularly, did you make sure to include -rdynamic and/or -g when compiling the C++ code?
try
dwarfdump --lookup=0xYOUR_ADRESS YOUR_DSYM_FILE
you will have to look up each adress manually ( or write a script to do this ) but if the symbols are ok ( your dSym file is bigger than say 20MB) this will do the job .
I'm trying to write a binding for a vendor C++ library. I've successfully used snippets such as the below to define init functions in the other modules, but in this one it doesn't seem to work: it compiles fine, but throws the ImportError as soon as I try to import it into a test script. What could be wrong here?
#ifndef PyMODINIT_FUNC /* declarations for DLL import/export */
#define PyMODINIT_FUNC void
#endif
PyMODINIT_FUNC initclient(void) {
PyObject* m;
ClientType.tp_new = PyType_GenericNew;
if (PyType_Ready(&ClientType) < 0)
return;
m = Py_InitModule3("client", client_methods, "Client module");
Py_INCREF(&ClientType);
PyModule_AddObject(m, "Client", (PyObject *) &ClientType);
}
This is on 32-bit Linux, with gcc 4.4.4.
I had the same issue. At compile time:
path to the Python header: OK
path to the Python library: OK
link against the Python library: OK
link against needed third parties libraries/object files: OK
I just forgot to compile the C file that defines my module... Sigh...
So yeah, first thing to check: your makefile or your compilation command! :)
Make sure you don't mix Python versions. In Python version 2 the init function was called Init_, while in version 3 this function is called PyInit_
In my case this was happening when SWIG 3.0.2 used Python 3.4 to generate bindings, while my Python IDE called the Python 2.7 interpreter.
You can see the difference in the generated .cxx file:
#if PY_VERSION_HEX >= 0x03000000
# define SWIG_init PyInit__<modulename>
#else
# define SWIG_init init_<modulename>
#endif
On linux you can also use the following command to check your .so exports:
nm -D <modulename> | grep <modulename>
This will give you the name of the init function within your library.
I had the same error message, but it was because I renamed my .c file, and forgot to update the name inside the code. The "initxxx" function and an argument inside it.
Make sure you include your _wrap.cxx. It seems to me it doesn't get compiled into your module.
On linux it can help to run strace in this case. Check that the name of the library python is searching for is the same as the name of the library you built.
The swig documentation mentions here:
This error is almost always caused when a bad name is given to the shared object file. For example, if you created a file example.so instead of _example.so you would get this error.
In the interface file, SWIG suggests using:
#define SWIG_FILE_WITH_INIT
This turned to be unrelated to Python or the compiler, but was an incorrect compiler incantation (have to pay more attention while editing the Makefile).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
The community reviewed whether to reopen this question last year and left it closed:
Original close reason(s) were not resolved
Improve this question
What would be the quickest way to construct a Python binding to a C or C++ library?
(I am using Windows if this matters.)
ctypes module is part of the standard library, and therefore is more stable and widely available than swig, which always tended to give me problems.
With ctypes, you need to satisfy any compile time dependency on python, and your binding will work on any python that has ctypes, not just the one it was compiled against.
Suppose you have a simple C++ example class you want to talk to in a file called foo.cpp:
#include <iostream>
class Foo{
public:
void bar(){
std::cout << "Hello" << std::endl;
}
};
Since ctypes can only talk to C functions, you need to provide those declaring them as extern "C"
extern "C" {
Foo* Foo_new(){ return new Foo(); }
void Foo_bar(Foo* foo){ foo->bar(); }
}
Next you have to compile this to a shared library
g++ -c -fPIC foo.cpp -o foo.o
g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
And finally you have to write your python wrapper (e.g. in fooWrapper.py)
from ctypes import cdll
lib = cdll.LoadLibrary('./libfoo.so')
class Foo(object):
def __init__(self):
self.obj = lib.Foo_new()
def bar(self):
lib.Foo_bar(self.obj)
Once you have that you can call it like
f = Foo()
f.bar() #and you will see "Hello" on the screen
You should have a look at Boost.Python. Here is the short introduction taken from their website:
The Boost Python Library is a framework for interfacing Python and
C++. It allows you to quickly and seamlessly expose C++ classes
functions and objects to Python, and vice-versa, using no special
tools -- just your C++ compiler. It is designed to wrap C++ interfaces
non-intrusively, so that you should not have to change the C++ code at
all in order to wrap it, making Boost.Python ideal for exposing
3rd-party libraries to Python. The library's use of advanced
metaprogramming techniques simplifies its syntax for users, so that
wrapping code takes on the look of a kind of declarative interface
definition language (IDL).
There is also pybind11, which is like a lightweight version of Boost.Python and compatible with all modern C++ compilers:
https://pybind11.readthedocs.io/en/latest/
The quickest way to do this is using SWIG.
Example from SWIG tutorial:
/* File : example.c */
int fact(int n) {
if (n <= 1) return 1;
else return n*fact(n-1);
}
Interface file:
/* example.i */
%module example
%{
/* Put header files here or function declarations like below */
extern int fact(int n);
%}
extern int fact(int n);
Building a Python module on Unix:
swig -python example.i
gcc -fPIC -c example.c example_wrap.c -I/usr/local/include/python2.7
gcc -shared example.o example_wrap.o -o _example.so
Usage:
>>> import example
>>> example.fact(5)
120
Note that you have to have python-dev. Also in some systems python header files will be in /usr/include/python2.7 based on the way you have installed it.
From the tutorial:
SWIG is a fairly complete C++ compiler with support for nearly every language feature. This includes preprocessing, pointers, classes, inheritance, and even C++ templates. SWIG can also be used to package structures and classes into proxy classes in the target language — exposing the underlying functionality in a very natural manner.
I started my journey in the Python <-> C++ binding from this page, with the objective of linking high level data types (multidimensional STL vectors with Python lists) :-)
Having tried the solutions based on both ctypes and boost.python (and not being a software engineer) I have found them complex when high level datatypes binding is required, while I have found SWIG much more simple for such cases.
This example uses therefore SWIG, and it has been tested in Linux (but SWIG is available and is widely used in Windows too).
The objective is to make a C++ function available to Python that takes a matrix in form of a 2D STL vector and returns an average of each row (as a 1D STL vector).
The code in C++ ("code.cpp") is as follow:
#include <vector>
#include "code.h"
using namespace std;
vector<double> average (vector< vector<double> > i_matrix) {
// Compute average of each row..
vector <double> averages;
for (int r = 0; r < i_matrix.size(); r++){
double rsum = 0.0;
double ncols= i_matrix[r].size();
for (int c = 0; c< i_matrix[r].size(); c++){
rsum += i_matrix[r][c];
}
averages.push_back(rsum/ncols);
}
return averages;
}
The equivalent header ("code.h") is:
#ifndef _code
#define _code
#include <vector>
std::vector<double> average (std::vector< std::vector<double> > i_matrix);
#endif
We first compile the C++ code to create an object file:
g++ -c -fPIC code.cpp
We then define a SWIG interface definition file ("code.i") for our C++ functions.
%module code
%{
#include "code.h"
%}
%include "std_vector.i"
namespace std {
/* On a side note, the names VecDouble and VecVecdouble can be changed, but the order of first the inner vector matters! */
%template(VecDouble) vector<double>;
%template(VecVecdouble) vector< vector<double> >;
}
%include "code.h"
Using SWIG, we generate a C++ interface source code from the SWIG interface definition file..
swig -c++ -python code.i
We finally compile the generated C++ interface source file and link everything together to generate a shared library that is directly importable by Python (the "_" matters):
g++ -c -fPIC code_wrap.cxx -I/usr/include/python2.7 -I/usr/lib/python2.7
g++ -shared -Wl,-soname,_code.so -o _code.so code.o code_wrap.o
We can now use the function in Python scripts:
#!/usr/bin/env python
import code
a= [[3,5,7],[8,10,12]]
print a
b = code.average(a)
print "Assignment done"
print a
print b
For modern C++, use cppyy:
http://cppyy.readthedocs.io/en/latest/
It's based on Cling, the C++ interpreter for Clang/LLVM. Bindings are at run-time and no additional intermediate language is necessary. Thanks to Clang, it supports C++17.
Install it using pip:
$ pip install cppyy
For small projects, simply load the relevant library and the headers that you are interested in. E.g. take the code from the ctypes example is this thread, but split in header and code sections:
$ cat foo.h
class Foo {
public:
void bar();
};
$ cat foo.cpp
#include "foo.h"
#include <iostream>
void Foo::bar() { std::cout << "Hello" << std::endl; }
Compile it:
$ g++ -c -fPIC foo.cpp -o foo.o
$ g++ -shared -Wl,-soname,libfoo.so -o libfoo.so foo.o
and use it:
$ python
>>> import cppyy
>>> cppyy.include("foo.h")
>>> cppyy.load_library("foo")
>>> from cppyy.gbl import Foo
>>> f = Foo()
>>> f.bar()
Hello
>>>
Large projects are supported with auto-loading of prepared reflection information and the cmake fragments to create them, so that users of installed packages can simply run:
$ python
>>> import cppyy
>>> f = cppyy.gbl.Foo()
>>> f.bar()
Hello
>>>
Thanks to LLVM, advanced features are possible, such as automatic template instantiation. To continue the example:
>>> v = cppyy.gbl.std.vector[cppyy.gbl.Foo]()
>>> v.push_back(f)
>>> len(v)
1
>>> v[0].bar()
Hello
>>>
Note: I'm the author of cppyy.
pybind11 minimal runnable example
pybind11 was previously mentioned at https://stackoverflow.com/a/38542539/895245 but I would like to give here a concrete usage example and some further discussion about implementation.
All and all, I highly recommend pybind11 because it is really easy to use: you just include a header and then pybind11 uses template magic to inspect the C++ class you want to expose to Python and does that transparently.
The downside of this template magic is that it slows down compilation immediately adding a few seconds to any file that uses pybind11, see for example the investigation done on this issue. PyTorch agrees. A proposal to remediate this problem has been made at: https://github.com/pybind/pybind11/pull/2445
Here is a minimal runnable example to give you a feel of how awesome pybind11 is:
class_test.cpp
#include <string>
#include <pybind11/pybind11.h>
struct ClassTest {
ClassTest(const std::string &name, int i) : name(name), i(i) { }
void setName(const std::string &name_) { name = name_; }
const std::string getName() const { return name + "z"; }
void setI(const int i) { this->i = i; }
const int getI() const { return i + 1; }
std::string name;
int i;
};
namespace py = pybind11;
PYBIND11_PLUGIN(class_test) {
py::module m("my_module", "pybind11 example plugin");
py::class_<ClassTest>(m, "ClassTest")
.def(py::init<const std::string &, int>())
.def("setName", &ClassTest::setName)
.def("getName", &ClassTest::getName)
.def_readwrite("name", &ClassTest::name)
.def("setI", &ClassTest::setI)
.def("getI", &ClassTest::getI)
.def_readwrite("i", &ClassTest::i);
return m.ptr();
}
class_test_main.py
#!/usr/bin/env python3
import class_test
my_class_test = class_test.ClassTest("abc", 1);
print(my_class_test.getName())
print(my_class_test.getI())
my_class_test.setName("012")
my_class_test.setI(2)
print(my_class_test.getName())
print(my_class_test.getI())
assert(my_class_test.getName() == "012z")
assert(my_class_test.getI() == 3)
Compile and run:
#!/usr/bin/env bash
set -eux
sudo apt install pybind11-dev
g++ `python3-config --cflags` -shared -std=c++11 -fPIC class_test.cpp \
-o class_test`python3-config --extension-suffix` `python3-config --libs`
./class_test_main.py
Stdout output:
abcz
2
012z
3
If we tried to use a wrong type as in:
my_class_test.setI("abc")
it blows up as expected:
Traceback (most recent call last):
File "/home/ciro/test/./class_test_main.py", line 9, in <module>
my_class_test.setI("abc")
TypeError: setI(): incompatible function arguments. The following argument types are supported:
1. (self: my_module.ClassTest, arg0: int) -> None
Invoked with: <my_module.ClassTest object at 0x7f2980254fb0>, 'abc'
This example shows how pybind11 allows you to effortlessly expose the ClassTest C++ class to Python!
Notably, Pybind11 automatically understands from the C++ code that name is an std::string, and therefore should be mapped to a Python str object.
Compilation produces a file named class_test.cpython-36m-x86_64-linux-gnu.so which class_test_main.py automatically picks up as the definition point for the class_test natively defined module.
Perhaps the realization of how awesome this is only sinks in if you try to do the same thing by hand with the native Python API, see for example this example of doing that, which has about 10x more code: https://github.com/cirosantilli/python-cheat/blob/4f676f62e87810582ad53b2fb426b74eae52aad5/py_from_c/pure.c On that example you can see how the C code has to painfully and explicitly define the Python class bit by bit with all the information it contains (members, methods, further metadata...). See also:
Can python-C++ extension get a C++ object and call its member function?
Exposing a C++ class instance to a python embedded interpreter
A full and minimal example for a class (not method) with Python C Extension?
Embedding Python in C++ and calling methods from the C++ code with Boost.Python
Inheritance in Python C++ extension
pybind11 claims to be similar to Boost.Python which was mentioned at https://stackoverflow.com/a/145436/895245 but more minimal because it is freed from the bloat of being inside the Boost project:
pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code. Its goals and syntax are similar to the excellent Boost.Python library by David Abrahams: to minimize boilerplate code in traditional extension modules by inferring type information using compile-time introspection.
The main issue with Boost.Python—and the reason for creating such a similar project—is Boost. Boost is an enormously large and complex suite of utility libraries that works with almost every C++ compiler in existence. This compatibility has its cost: arcane template tricks and workarounds are necessary to support the oldest and buggiest of compiler specimens. Now that C++11-compatible compilers are widely available, this heavy machinery has become an excessively large and unnecessary dependency.
Think of this library as a tiny self-contained version of Boost.Python with everything stripped away that isn't relevant for binding generation. Without comments, the core header files only require ~4K lines of code and depend on Python (2.7 or 3.x, or PyPy2.7 >= 5.7) and the C++ standard library. This compact implementation was possible thanks to some of the new C++11 language features (specifically: tuples, lambda functions and variadic templates). Since its creation, this library has grown beyond Boost.Python in many ways, leading to dramatically simpler binding code in many common situations.
pybind11 is also the only non-native alternative hightlighted by the current Microsoft Python C binding documentation at: https://learn.microsoft.com/en-us/visualstudio/python/working-with-c-cpp-python-in-visual-studio?view=vs-2019 (archive).
Tested on Ubuntu 18.04, pybind11 2.0.1, Python 3.6.8, GCC 7.4.0.
I think cffi for python can be an option.
The goal is to call C code from Python. You should be able to do so
without learning a 3rd language: every alternative requires you to
learn their own language (Cython, SWIG) or API (ctypes). So we tried
to assume that you know Python and C and minimize the extra bits of
API that you need to learn.
http://cffi.readthedocs.org/en/release-0.7/
I love cppyy, it makes it very easy to extend Python with C++ code, dramatically increasing performance when needed.
It is powerful and frankly very simple to use,
here it is an example of how you can create a numpy array and pass it to a class member function in C++.
cppyy_test.py
import cppyy
import numpy as np
cppyy.include('Buffer.h')
s = cppyy.gbl.Buffer()
numpy_array = np.empty(32000, np.float64)
s.get_numpy_array(numpy_array.data, numpy_array.size)
print(numpy_array[:20])
Buffer.h
struct Buffer {
void get_numpy_array(double *ad, int size) {
for( long i=0; i < size; i++)
ad[i]=i;
}
};
You can also create a Python module very easily (with CMake), this way you will avoid recompile the C++ code all the times.
The question is how to call a C function from Python, if I understood correctly. Then the best bet are Ctypes (BTW portable across all variants of Python).
>>> from ctypes import *
>>> libc = cdll.msvcrt
>>> print libc.time(None)
1438069008
>>> printf = libc.printf
>>> printf("Hello, %s\n", "World!")
Hello, World!
14
>>> printf("%d bottles of beer\n", 42)
42 bottles of beer
19
For a detailed guide you may want to refer to my blog article.
Cython is definitely the way to go, unless you anticipate writing Java wrappers, in which case SWIG may be preferable.
I recommend using the runcython command line utility, it makes the process of using Cython extremely easy. If you need to pass structured data to C++, take a look at Google's protobuf library, it's very convenient.
Here is a minimal examples I made that uses both tools:
https://github.com/nicodjimenez/python2cpp
Hope it can be a useful starting point.
First you should decide what is your particular purpose. The official Python documentation on extending and embedding the Python interpreter was mentioned above, I can add a good overview of binary extensions. The use cases can be divided into 3 categories:
accelerator modules: to run faster than the equivalent pure Python code runs in CPython.
wrapper modules: to expose existing C interfaces to Python code.
low level system access: to access lower level features of the CPython runtime, the operating system, or the underlying hardware.
In order to give some broader perspective for other interested and since your initial question is a bit vague ("to a C or C++ library") I think this information might be interesting to you. On the link above you can read on disadvantages of using binary extensions and its alternatives.
Apart from the other answers suggested, if you want an accelerator module, you can try Numba. It works "by generating optimized machine code using the LLVM compiler infrastructure at import time, runtime, or statically (using the included pycc tool)".