link cython module in a c++ program - c++

Question

Is it possible to build a cython module with some cdef functions and link the resulting shared library into a C++ program?
I tried a proof of concept:
cymod.pyx:
# distutils: language=c++
from libcpp.string cimport string
cdef public string simple_echo(string test_string):
return test_string
cpp_test.cpp:
#define PyMODINIT_FUNC void
#include <iostream>
#include "cymod.h"
int main(int argc, char const *argv[])
{
std::cout << simple_echo("test") << std::endl;
return 0;
}
setup.py:
from setuptools import setup, Extension
from Cython.Build import cythonize
setup(
name='cymod',
ext_modules=cythonize(
Extension(
"cymod", ["cymod.pyx"],
),
)
)
The cython module builds fine, but when I try to build the c++ code that will use the cython function I get:
$ g++ -L. -l:cymod.so cpp_test.cpp -o cpp_test
/tmp/cc48Vc2z.o: In function `main':
cpp_test.cpp:(.text+0x51): undefined reference to `simple_echo'
collect2: error: ld returned 1 exit status
Which is odd. The generated header file has it:
cymod.h:
/* Generated by Cython 0.29.1 */
#ifndef __PYX_HAVE__cymod
#define __PYX_HAVE__cymod
#ifndef __PYX_HAVE_API__cymod
#ifndef __PYX_EXTERN_C
#ifdef __cplusplus
#define __PYX_EXTERN_C extern "C"
#else
#define __PYX_EXTERN_C extern
#endif
#endif
#ifndef DL_IMPORT
#define DL_IMPORT(_T) _T
#endif
__PYX_EXTERN_C std::string simple_echo(std::string);
#endif /* !__PYX_HAVE_API__cymod */
/* WARNING: the interface of the module init function changed in CPython 3.5. */
/* It now returns a PyModuleDef instance instead of a PyModule instance. */
#if PY_MAJOR_VERSION < 3
PyMODINIT_FUNC initcymod(void);
#else
PyMODINIT_FUNC PyInit_cymod(void);
#endif
#endif /* !__PYX_HAVE__cymod */
and I see my function in cymod.so:
nm cymod.so| grep simple_echo
0000000000001e50 T simple_echo
NOTE: I realize that to actually get this working I'll need to link against the python libraries and initialize the interpreter etc. I left that out to make this a tad shorter and I get the same error either way.

Answer 1

The short answer is that I was putting the -l argument too early in the compilation command. It is also important to handle the library lookup path. The simplest way is to use rpath. I set the rpath to the directory that the executable is in, i.e., .
Additionally, it is necessary to link against the python libraries and set the include and library paths. These can be determined at compile time by using the output of the python-config utility. Here is the compilation command that ultimately did the trick:
g++ cpp_test.cpp -o cpp_test -L. -l:cymod.so $(python-config --libs) $(python-config --includes) $(python-config --cflags) -Wl,-rpath,"\$ORIGIN"
I also updated the c++ file to #include "Python.h" and added calls to Py_Initialize(), Py_Finalize(), and initcymod():
#include <iostream>
#include "Python.h"
#include "cymod.h"
int main(int argc, char *argv[])
{
Py_Initialize();
initcymod();
std::cout << simple_echo("test") << std::endl;
Py_Finalize();
return 0;
}
NOTE: the call to initcymod() is necessary, but python2 specific. On python3 you should call PyImport_AppendInittab("cymod", PyInit_cymod); prior to Py_Initialize(). The cymod part is the module name, substitute your module name.
Thanks to #ead for the informative link to the docs on this topic https://cython.readthedocs.io/en/latest/src/userguide/external_C_code.html#using-cython-declarations-from-c and his answer to a related question https://stackoverflow.com/a/45424720/2069572
While reading the linked docs, I came across this:
Note On some operating systems like Linux, it is also possible to first build the Cython extension in the usual way and then link against the resulting .so file like a dynamic library. Beware that this is not portable, so it should be avoided.
So it turns out that you should not do what I was trying to do.
Instead, what I should have done was run:
cython --cplus cymod.pyx
And then compiled cpp_test.cpp with the generated cymod.cpp file.
No need to link the cython shared library, and it turns out that it is not a good idea to do so.