Pass numpy's include dir to Cmake from Setuptools - c++

I have a C++ library which I have successfully exposed to python using Pybind11.
In the CmakeLists.txt file, I have added the numpy include like this:
include_directories("C:\\Python37\\Lib\\site-packages\\numpy\\core\\include")
This works, but is undesirable. I would like to pass the numpy include directory from my setup.py file.
My setup.py file looks very much like this one:
import os
import re
import sys
import sysconfig
import platform
import subprocess
from distutils.version import LooseVersion
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
class CMakeExtension(Extension):
def __init__(self, name, sourcedir=''):
Extension.__init__(self, name, sources=[])
self.sourcedir = os.path.abspath(sourcedir)
class CMakeBuild(build_ext):
def run(self):
try:
out = subprocess.check_output(['cmake', '--version'])
except OSError:
raise RuntimeError(
"CMake must be installed to build the following extensions: " +
", ".join(e.name for e in self.extensions))
if platform.system() == "Windows":
cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)',
out.decode()).group(1))
if cmake_version < '3.1.0':
raise RuntimeError("CMake >= 3.1.0 is required on Windows")
for ext in self.extensions:
self.build_extension(ext)
def build_extension(self, ext):
extdir = os.path.abspath(
os.path.dirname(self.get_ext_fullpath(ext.name)))
cmake_args = ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + extdir,
'-DPYTHON_EXECUTABLE=' + sys.executable]
cfg = 'Debug' if self.debug else 'Release'
build_args = ['--config', cfg]
if platform.system() == "Windows":
cmake_args += ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}'.format(
cfg.upper(),
extdir)]
if sys.maxsize > 2**32:
cmake_args += ['-A', 'x64']
build_args += ['--', '/m']
else:
cmake_args += ['-DCMAKE_BUILD_TYPE=' + cfg]
build_args += ['--', '-j2']
env = os.environ.copy()
env['CXXFLAGS'] = '{} -DVERSION_INFO=\\"{}\\"'.format(
env.get('CXXFLAGS', ''),
self.distribution.get_version())
if not os.path.exists(self.build_temp):
os.makedirs(self.build_temp)
subprocess.check_call(['cmake', ext.sourcedir] + cmake_args,
cwd=self.build_temp, env=env)
subprocess.check_call(['cmake', '--build', '.'] + build_args,
cwd=self.build_temp)
print() # Add an empty line for cleaner output
setup(
name='python_cpp_example',
version='0.1',
author='Benjamin Jack',
author_email='benjamin.r.jack#gmail.com',
description='A hybrid Python/C++ test project',
long_description='',
# add extension module
ext_modules=[CMakeExtension('python_cpp_example')],
# add custom build_ext command
cmdclass=dict(build_ext=CMakeBuild),
zip_safe=False,
)
After having a look at some SO questions like this, I know that you can get the numpy include directory with numpy.get_include().
However, adding the include directory to the path inside the function build_extension with this line ext.include_dirs.append(numpy.get_include()) seems to have no effect.
I'd like to know how to pass the include directory properly.

Is your cmake build using the wrong numpy path or not finding numpy at all? If it's the wrong path, you could try prepending instead of appending numpy.get_include():
ext.include_dirs.insert(0,numpy.get_include())

Related

python-for-android, Cython, C++, CythonRecipe: Operation only allowed in c++

I have this setup.py for my Cython project:
from setuptools import setup
from Cython.Build import cythonize
setup(
name = 'phase-engine',
version = '0.1',
ext_modules = cythonize(["phase_engine.pyx"] + ['music-synthesizer-for-android/src/' + p for p in [
'fm_core.cc', 'dx7note.cc', 'env.cc', 'exp2.cc', 'fm_core.cc', 'fm_op_kernel.cc', 'freqlut.cc', 'lfo.cc', 'log2.cc', 'patch.cc', 'pitchenv.cc', 'resofilter.cc', 'ringbuffer.cc', 'sawtooth.cc', 'sin.cc', 'synth_unit.cc'
]],
include_path = ['music-synthesizer-for-android/src/'],
language = 'c++',
)
)
when I run buildozer, it gets angry about some Cython features only being available in C++ mode:
def __dealloc__(self):
del self.p_synth_unit
^
------------------------------------------------------------
phase_engine.pyx:74:8: Operation only allowed in c++
from which I understand it's ignoring my setup.py and doing its own somehow. How do I give it all these parameters?
CythonRecipe doesn't work well for Cython code that imports C/C++ code. Try CompiledComponentsPythonRecipe, or if you're having issues with #include <ios> or some other thing from the C++ STL, CppCompiledComponentsPythonRecipe:
from pythonforandroid.recipe import IncludedFilesBehaviour, CppCompiledComponentsPythonRecipe
import os
import sys
class MyRecipe(IncludedFilesBehaviour, CppCompiledComponentsPythonRecipe):
version = 'stable'
src_filename = "../../../phase-engine"
name = 'phase-engine'
depends = ['setuptools']
call_hostpython_via_targetpython = False
install_in_hostpython = True
def get_recipe_env(self, arch):
env = super().get_recipe_env(arch)
env['LDFLAGS'] += ' -lc++_shared'
return env
recipe = MyRecipe()
The dependency on setuptools is essential because of some weird stuff, otherwise you get an error no module named setuptools. The two other flags were also related to that error, the internet said they're relevant so I tried value combinations until one worked.
The LDFLAGS thing fixes an issue I had later (see buildozer + Cython + C++ library: dlopen failed: cannot locate symbol symbol-name referenced by module.so).

Carla Python/C++ extension - import error: undefined symbol Ipopt

I implemented a MPC controller within CARLA 0.9.8 trafficmanager. That MPC controller is dependent on IPOPT.
Carla make commands (make rebuild; make PythonAPI) work fine.
However import carla in the python script spawn_npc.py throws the following error:
ImportError: /home/DNDE.EMEA.DENSO/cha/.cache/Python-Eggs/carla-0.9.8-py2.7-linux-x86_64.egg-tmp/carla/libcarla.so: undefined symbol: _ZN5Ipopt16IpoptApplicationC1Ebb
After googling for hours I assume that I have to modify the file setup.py in order to include and link the ipopt library, which is in /usr/local/lib.
But my trial & error procedure was not successful
The setup.py file is:
#!/usr/bin/env python
# Copyright (c) 2019 Computer Vision Center (CVC) at the Universitat Autonoma de
# Barcelona (UAB).
#
# This work is licensed under the terms of the MIT license.
# For a copy, see <https://opensource.org/licenses/MIT>.
from setuptools import setup, Extension
import fnmatch
import os
import platform
import sys
def get_libcarla_extensions():
include_dirs = ['dependencies/include']
library_dirs = ['dependencies/lib']
libraries = []
sources = ['source/libcarla/libcarla.cpp']
def walk(folder, file_filter='*'):
for root, _, filenames in os.walk(folder):
for filename in fnmatch.filter(filenames, file_filter):
yield os.path.join(root, filename)
if os.name == "posix":
# #todo Replace deprecated method.
linux_distro = platform.dist()[0] # pylint: disable=W1505
if linux_distro.lower() in ["ubuntu", "debian", "deepin"]:
pwd = os.path.dirname(os.path.realpath(__file__))
pylib = "libboost_python%d%d.a" % (sys.version_info.major,
sys.version_info.minor)
extra_link_args = [
os.path.join(pwd, 'dependencies/lib/libcarla_client.a'),
os.path.join(pwd, 'dependencies/lib/librpc.a'),
os.path.join(pwd, 'dependencies/lib/libboost_filesystem.a'),
os.path.join(pwd, 'dependencies/lib/libRecast.a'),
os.path.join(pwd, 'dependencies/lib/libDetour.a'),
os.path.join(pwd, 'dependencies/lib/libDetourCrowd.a'),
os.path.join(pwd, 'dependencies/lib', pylib)]
extra_compile_args = [
'-isystem', 'dependencies/include/system', '-fPIC', '-std=c++14',
'-Werror', '-Wall', '-Wextra', '-Wpedantic', '-Wno-self-assign-overloaded',
'-Wdeprecated', '-Wno-shadow', '-Wuninitialized', '-Wunreachable-code',
'-Wpessimizing-move', '-Wold-style-cast', '-Wnull-dereference',
'-Wduplicate-enum', '-Wnon-virtual-dtor', '-Wheader-hygiene',
'-Wconversion', '-Wfloat-overflow-conversion',
'-DBOOST_ERROR_CODE_HEADER_ONLY', '-DLIBCARLA_WITH_PYTHON_SUPPORT'
]
if 'BUILD_RSS_VARIANT' in os.environ and os.environ['BUILD_RSS_VARIANT'] == 'true':
print('Building AD RSS variant.')
extra_compile_args += ['-DLIBCARLA_RSS_ENABLED']
extra_link_args += [os.path.join(pwd, 'dependencies/lib/libad-rss.a')]
if 'TRAVIS' in os.environ and os.environ['TRAVIS'] == 'true':
print('Travis CI build detected: disabling PNG support.')
extra_link_args += ['-ljpeg', '-ltiff']
extra_compile_args += ['-DLIBCARLA_IMAGE_WITH_PNG_SUPPORT=false']
else:
extra_link_args += ['-lpng', '-ljpeg', '-ltiff']
extra_compile_args += ['-DLIBCARLA_IMAGE_WITH_PNG_SUPPORT=true']
# #todo Why would we need this?
include_dirs += ['/usr/lib/gcc/x86_64-linux-gnu/7/include']
library_dirs += ['/usr/lib/gcc/x86_64-linux-gnu/7']
extra_link_args += ['/usr/lib/gcc/x86_64-linux-gnu/7/libstdc++.a']
else:
raise NotImplementedError
elif os.name == "nt":
sources += [x for x in walk('dependencies/include/carla', '*.cpp')]
pwd = os.path.dirname(os.path.realpath(__file__))
pylib = 'libboost_python%d%d' % (
sys.version_info.major,
sys.version_info.minor)
extra_link_args = ['shlwapi.lib' ]
required_libs = [
pylib, 'libboost_filesystem',
'rpc.lib', 'carla_client.lib',
'libpng.lib', 'zlib.lib',
'Recast.lib', 'Detour.lib', 'DetourCrowd.lib']
# Search for files in 'PythonAPI\carla\dependencies\lib' that contains
# the names listed in required_libs in it's file name
libs = [x for x in os.listdir('dependencies/lib') if any(d in x for d in required_libs)]
for lib in libs:
extra_link_args.append(os.path.join(pwd, 'dependencies/lib', lib))
# https://learn.microsoft.com/es-es/cpp/porting/modifying-winver-and-win32-winnt
extra_compile_args = [
'/experimental:external', '/external:I', 'dependencies/include/system',
'/DBOOST_ALL_NO_LIB', '/DBOOST_PYTHON_STATIC_LIB',
'/DBOOST_ERROR_CODE_HEADER_ONLY', '/D_WIN32_WINNT=0x0600', '/DHAVE_SNPRINTF',
'/DLIBCARLA_WITH_PYTHON_SUPPORT', '-DLIBCARLA_IMAGE_WITH_PNG_SUPPORT=true']
else:
raise NotImplementedError
depends = [x for x in walk('source/libcarla')]
depends += [x for x in walk('dependencies')]
def make_extension(name, sources):
return Extension(
name,
sources=sources,
include_dirs=include_dirs,
library_dirs=library_dirs,
libraries=libraries,
extra_compile_args=extra_compile_args,
extra_link_args=extra_link_args,
language='c++14',
depends=depends)
print('compiling:\n - %s' % '\n - '.join(sources))
return [make_extension('carla.libcarla', sources)]
setup(
name='carla',
version='0.9.8',
package_dir={'': 'source'},
packages=['carla'],
ext_modules=get_libcarla_extensions(),
license='MIT License',
description='Python API for communicating with the CARLA server.',
url='https://github.com/carla-simulator/carla',
author='The CARLA team',
author_email='carla.simulator#gmail.com',
include_package_data=True)
includes in MPC.cpp:
#include "carla/trafficmanager/MPC.h"
#include <cppad/cppad.hpp>
#include <cppad/ipopt/solve.hpp>
#include <Eigen-3.3/Eigen/Dense>
includes in MPC.h:
#include <vector>
#include <Eigen-3.3/Eigen/Dense>
#include "carla/trafficmanager/PIDController.h"
#include "carla/client/Vehicle.h"
#include "carla/geom/Math.h"
Can anyone help?
The setup script calls the compiler to generate a shared library. Just modify the extra_link_args to link your library as you would do if you were calling the compiler directly.
Assuming you're using Linux, if you have your library installed in your machine add a -lipopt, in the same way is done with -lpng for libpng. If you have it somewhere not on your system libraries path or want to link statically pass the full path, just as it's done for libcarla_client.a for instance.
Carla uses clang as compiler on Linux, you can check its command-line ref if you want more details.
The recent version of carla was throwing the same error for me after researching for some time ,I have tried this method hope it works for you. First go to the location of the spawn.py folder using command prompt and type the following commands.
py -3.7 spawn.py
It supports only 3.7 version of python .You can refer youtube to understand how to run keep multiple version of python in windows.

difficulties with cx_freeze setup for kivy with Python 2.7

I am trying to build a package for just a simple Python 2.7 + kivy 1.9.1 script using cx_freeze. Unfortunately I get the following error when executing the DoScroll.exe file:
File "C:\Utils\Pyhton27\lib\site-packages\kivy\lang.py", line 1829, in load_file
with open(filename, 'r') as fd:
IOError: [Errno 2] No such file or directory: 'C:\Projects\Testing\build\exe.win32-2.7\library.zip\kivy\data\style.kv'
The zip file does not contain the kivy\data folder. So my idea was to add the kivy package explicitly to the setup.py file. But this does not help.
How could I resolve this issue?
The setup.py file I am using:
import sys
from cx_Freeze import setup, Executable
base = None
if sys.platform == "win32":
base = "Win32GUI"
options = {
'build_exe': {
'packages': ['kivy']
}
}
executables = [
Executable('DoScroll.py', base=base)
]
setup(name='scroller',
version='0.1',
description='demo scroller',
executables = executables,
options=options
)

Unzip zip files in folders and subfolders with python 2.7.8

Continusly to Unzip zip files in folders and subfolders with python this code work with python 3:
#!/usr/bin/env python3
import logging
from pathlib import Path
from shutil import unpack_archive
zip_files = Path(r"C:\Project\layers").rglob("*.zip")
while True:
try:
path = next(zip_files)
except StopIteration:
break # no more files
except PermissionError:
logging.exception("permission error")
else:
extract_dir = path.with_name(path.stem)
unpack_archive(str(path), str(extract_dir), 'zip')
i work with python 2.7.8 and can't change the version of python because it affect other important programs. When i run the code i get an error:
ImportError: No module named pathlib
How can i change the code so it will work?

Speeding up build process with distutils

I am programming a C++ extension for Python and I am using distutils to compile the project. As the project grows, rebuilding it takes longer and longer. Is there a way to speed up the build process?
I read that parallel builds (as with make -j) are not possible with distutils. Are there any good alternatives to distutils which might be faster?
I also noticed that it's recompiling all object files every time I call python setup.py build, even when I only changed one source file. Should this be the case or might I be doing something wrong here?
In case it helps, here are some of the files which I try to compile: https://gist.github.com/2923577
Thanks!
Try building with environment variable CC="ccache gcc", that will speed up build significantly when the source has not changed. (strangely, distutils uses CC also for c++ source files). Install the ccache package, of course.
Since you have a single extension which is assembled from multiple compiled object files, you can monkey-patch distutils to compile those in parallel (they are independent) - put this into your setup.py (adjust the N=2 as you wish):
# monkey-patch for parallel compilation
def parallelCCompile(self, sources, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None):
# those lines are copied from distutils.ccompiler.CCompiler directly
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(output_dir, macros, include_dirs, sources, depends, extra_postargs)
cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
# parallel code
N=2 # number of parallel compilations
import multiprocessing.pool
def _single_compile(obj):
try: src, ext = build[obj]
except KeyError: return
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
# convert to list, imap is evaluated on-demand
list(multiprocessing.pool.ThreadPool(N).imap(_single_compile,objects))
return objects
import distutils.ccompiler
distutils.ccompiler.CCompiler.compile=parallelCCompile
For the sake of completeness, if you have multiple extensions, you can use the following solution:
import os
import multiprocessing
try:
from concurrent.futures import ThreadPoolExecutor as Pool
except ImportError:
from multiprocessing.pool import ThreadPool as LegacyPool
# To ensure the with statement works. Required for some older 2.7.x releases
class Pool(LegacyPool):
def __enter__(self):
return self
def __exit__(self, *args):
self.close()
self.join()
def build_extensions(self):
"""Function to monkey-patch
distutils.command.build_ext.build_ext.build_extensions
"""
self.check_extensions_list(self.extensions)
try:
num_jobs = os.cpu_count()
except AttributeError:
num_jobs = multiprocessing.cpu_count()
with Pool(num_jobs) as pool:
pool.map(self.build_extension, self.extensions)
def compile(
self, sources, output_dir=None, macros=None, include_dirs=None,
debug=0, extra_preargs=None, extra_postargs=None, depends=None,
):
"""Function to monkey-patch distutils.ccompiler.CCompiler"""
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
output_dir, macros, include_dirs, sources, depends, extra_postargs
)
cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
for obj in objects:
try:
src, ext = build[obj]
except KeyError:
continue
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
# Return *all* object filenames, not just the ones we just built.
return objects
from distutils.ccompiler import CCompiler
from distutils.command.build_ext import build_ext
build_ext.build_extensions = build_extensions
CCompiler.compile = compile
I've got this working on Windows with clcache, derived from eudoxos's answer:
# Python modules
import datetime
import distutils
import distutils.ccompiler
import distutils.sysconfig
import multiprocessing
import multiprocessing.pool
import os
import sys
from distutils.core import setup
from distutils.core import Extension
from distutils.errors import CompileError
from distutils.errors import DistutilsExecError
now = datetime.datetime.now
ON_LINUX = "linux" in sys.platform
N_JOBS = 4
#------------------------------------------------------------------------------
# Enable ccache to speed up builds
if ON_LINUX:
os.environ['CC'] = 'ccache gcc'
# Windows
else:
# Using clcache.exe, see: https://github.com/frerich/clcache
# Insert path to clcache.exe into the path.
prefix = os.path.dirname(os.path.abspath(__file__))
path = os.path.join(prefix, "bin")
print "Adding %s to the system path." % path
os.environ['PATH'] = '%s;%s' % (path, os.environ['PATH'])
clcache_exe = os.path.join(path, "clcache.exe")
#------------------------------------------------------------------------------
# Parallel Compile
#
# Reference:
#
# http://stackoverflow.com/questions/11013851/speeding-up-build-process-with-distutils
#
def linux_parallel_cpp_compile(
self,
sources,
output_dir=None,
macros=None,
include_dirs=None,
debug=0,
extra_preargs=None,
extra_postargs=None,
depends=None):
# Copied from distutils.ccompiler.CCompiler
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
output_dir, macros, include_dirs, sources, depends, extra_postargs)
cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
def _single_compile(obj):
try:
src, ext = build[obj]
except KeyError:
return
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
# convert to list, imap is evaluated on-demand
list(multiprocessing.pool.ThreadPool(N_JOBS).imap(
_single_compile, objects))
return objects
def windows_parallel_cpp_compile(
self,
sources,
output_dir=None,
macros=None,
include_dirs=None,
debug=0,
extra_preargs=None,
extra_postargs=None,
depends=None):
# Copied from distutils.msvc9compiler.MSVCCompiler
if not self.initialized:
self.initialize()
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
output_dir, macros, include_dirs, sources, depends, extra_postargs)
compile_opts = extra_preargs or []
compile_opts.append('/c')
if debug:
compile_opts.extend(self.compile_options_debug)
else:
compile_opts.extend(self.compile_options)
def _single_compile(obj):
try:
src, ext = build[obj]
except KeyError:
return
input_opt = "/Tp" + src
output_opt = "/Fo" + obj
try:
self.spawn(
[clcache_exe]
+ compile_opts
+ pp_opts
+ [input_opt, output_opt]
+ extra_postargs)
except DistutilsExecError, msg:
raise CompileError(msg)
# convert to list, imap is evaluated on-demand
list(multiprocessing.pool.ThreadPool(N_JOBS).imap(
_single_compile, objects))
return objects
#------------------------------------------------------------------------------
# Only enable parallel compile on 2.7 Python
if sys.version_info[1] == 7:
if ON_LINUX:
distutils.ccompiler.CCompiler.compile = linux_parallel_cpp_compile
else:
import distutils.msvccompiler
import distutils.msvc9compiler
distutils.msvccompiler.MSVCCompiler.compile = windows_parallel_cpp_compile
distutils.msvc9compiler.MSVCCompiler.compile = windows_parallel_cpp_compile
# ... call setup() as usual
You can do this easily if you have Numpy 1.10 available. Just add:
try:
from numpy.distutils.ccompiler import CCompiler_compile
import distutils.ccompiler
distutils.ccompiler.CCompiler.compile = CCompiler_compile
except ImportError:
print("Numpy not found, parallel compile not available")
Use -j N or set NPY_NUM_BUILD_JOBS.
In the limited examples you provided in the link, it seems fairly obvious that you have some misunderstanding on what some of the features of the language are. For example, the gsminterface.h has a whole lot of namespace level statics, which is probably unintended. Every translation unit that includes that header will compile it's own version for everyone of the symbols declared in that header. Side effects of this are not only the compile time but also code bloat (larger binaries) and link time as the linker needs to process all those symbols.
There are still many questions that affect the build process that you have not answered, for example, whether you clean every time before you recompile. If you are doing that, then you might want to consider ccache, which is a tool that caches the result of the build process, so that if you run make clean; make target only the preprocessor will be run for any translation unit that has not changed. Note that as long as you keep maintaining most code in headers, this will not offer much of an advantage, as a change in a header modifies all translation units that include it. (I don't know your build system, so I cannot tell you whether python setup.py build will clean or not)
The project does not seem large otherwise, so I would be surprised if it took more than a few seconds to compile.