Project organization with Cython and C++ - c++

Question

I want to provide my C++ project with a Python interface. Technically, I have decided to use Cython for wrapping the C++ code. Over time, the entire project is meant to become a Python extension module, but at first, this is highly experimental. Gradually, C++ classes need to be exposed to Python.
My question is how to best organize files and build configurations so that Cython-generated and human-written C++ code do not get mixed and the Python extension module is cleanly built seperate from the other targets.
I imagine a directory structure like this for the source files, and some build directory for Cython.
Project/
src/
*.h
*.cpp
cython/
Project.pyx
setup.py

Answer 1

Basically I have 3 folders :
CPROJECT, The C++ library : producing a libcproject.so shared object
CYPROJECT, The cythonized Python extension : producing the cyproject.so using Cython
DEPENDENCIES, The dependencies : where I copy external requirements for both projects
In 1. I build the C++ extension (compiled with gcc - -shared, -fPIC compile options) that will be exposed to python and that the CYPROJECT relies on to expose features to Python. As a post processing command, the resulting .so is copied into DEPENDENCIES/libcproject/ (as well as the include files). This way the library is, of course, usable independently in a pure C++ project as well.
In 2. I make use of 3 sub-folders :
adapters : which mainly contains C++ additional classes (often classes derived from the ones provided by libcproject.so). Those are usually classes that are enhanced with functionalities specific to Cython requirements (such as storing the PyObject * C version of a targeted Python version - inherited from object - of a given class and the reference counting management, via Py_XINCREF and Py_DECREF, ...).
pyext : where are stored all the Cython hand written .pyx files.
setup : containing the setup.sh script (for setting up the dependencies paths and calling the python setup.py build_ext --inplace for generating the final cyproject.so (to be added to the PYTHONPATH) and cyproject.pyx.
So what's in the setup sub-folder ?
Here is a sample code for setup.sh :
export PYTHONPATH=$PYTHONPATH:../../../DEPENDENCIES/Cython-0.18
export PATH=$PATH:../../../DEPENDENCIES/libcproject:../../../DEPENDENCIES/Cython-0.18/bin
# Note the `../../../DEPENDENCIES/libcproject`...
CC="gcc" \
CXX="g++" \
python setup.py build_ext --inplace
And here an example of setup.py (mainly to demonstrate how the additional adapters are compiled):
import sys
import os
import shutil
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
# Cleaning
for root, dirs, files in os.walk(".", topdown=False):
for name in files:
if (name.startswith("cyproject") and not(name.endswith(".pyx"))):
os.remove(os.path.join(root, name))
for name in dirs:
if (name == "build"):
shutil.rmtree(name)
# Building
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [
Extension("cyproject",
sources=["cyproject.pyx", \
"adapter/ALabSimulatorBase.cpp", \
"adapter/ALabSimulatorTime.cpp", \
"adapter/ALabNetBinding.cpp", \
"adapter/AValueArg.cpp", \
"adapter/ALabSiteSetsManager.cpp", \
"adapter/ALabSite.cpp", \
],
libraries=["cproject"],
language="c++",
extra_compile_args=["-I../inc", "-I../../../DEPENDENCIES/python2.7/inc", "-I../../../DEPENDENCIES/gsl-1.8/include"],
extra_link_args=["-L../lib"]
extra_compile_args=["-fopenmp", "-O3"],
extra_link_args=[]
)
]
)
And finally, the main .pyx, that links all the hand written .pyxs of the cython part together [cyproject.pyx] :
include "pyext/Utils.pyx"
include "pyext/TCLAP.pyx"
include "pyext/LabSimulatorBase.pyx"
include "pyext/LabBinding.pyx"
include "pyext/LabSimulatorTime.pyx"
...
Note : All the files generated by Cython remains in this setup folder, well separated from the hand written stuffs (adapters and pyext), as expected.
In 3. Using a separated DEPENDENCIES folder allows to keep things well separated (in case I would move the CYPROJECT - and its dependencies - in some other environment).
All of this to give you an overview (a pertinent one, I hope) on how one can organize that sort of project.