I implemented a MPC controller within CARLA 0.9.8 trafficmanager. That MPC controller is dependent on IPOPT.
Carla make commands (make rebuild; make PythonAPI) work fine.
However import carla in the python script spawn_npc.py throws the following error:
ImportError: /home/DNDE.EMEA.DENSO/cha/.cache/Python-Eggs/carla-0.9.8-py2.7-linux-x86_64.egg-tmp/carla/libcarla.so: undefined symbol: _ZN5Ipopt16IpoptApplicationC1Ebb
After googling for hours I assume that I have to modify the file setup.py in order to include and link the ipopt library, which is in /usr/local/lib.
But my trial & error procedure was not successful
The setup.py file is:
#!/usr/bin/env python
# Copyright (c) 2019 Computer Vision Center (CVC) at the Universitat Autonoma de
# Barcelona (UAB).
#
# This work is licensed under the terms of the MIT license.
# For a copy, see <https://opensource.org/licenses/MIT>.
from setuptools import setup, Extension
import fnmatch
import os
import platform
import sys
def get_libcarla_extensions():
include_dirs = ['dependencies/include']
library_dirs = ['dependencies/lib']
libraries = []
sources = ['source/libcarla/libcarla.cpp']
def walk(folder, file_filter='*'):
for root, _, filenames in os.walk(folder):
for filename in fnmatch.filter(filenames, file_filter):
yield os.path.join(root, filename)
if os.name == "posix":
# #todo Replace deprecated method.
linux_distro = platform.dist()[0] # pylint: disable=W1505
if linux_distro.lower() in ["ubuntu", "debian", "deepin"]:
pwd = os.path.dirname(os.path.realpath(__file__))
pylib = "libboost_python%d%d.a" % (sys.version_info.major,
sys.version_info.minor)
extra_link_args = [
os.path.join(pwd, 'dependencies/lib/libcarla_client.a'),
os.path.join(pwd, 'dependencies/lib/librpc.a'),
os.path.join(pwd, 'dependencies/lib/libboost_filesystem.a'),
os.path.join(pwd, 'dependencies/lib/libRecast.a'),
os.path.join(pwd, 'dependencies/lib/libDetour.a'),
os.path.join(pwd, 'dependencies/lib/libDetourCrowd.a'),
os.path.join(pwd, 'dependencies/lib', pylib)]
extra_compile_args = [
'-isystem', 'dependencies/include/system', '-fPIC', '-std=c++14',
'-Werror', '-Wall', '-Wextra', '-Wpedantic', '-Wno-self-assign-overloaded',
'-Wdeprecated', '-Wno-shadow', '-Wuninitialized', '-Wunreachable-code',
'-Wpessimizing-move', '-Wold-style-cast', '-Wnull-dereference',
'-Wduplicate-enum', '-Wnon-virtual-dtor', '-Wheader-hygiene',
'-Wconversion', '-Wfloat-overflow-conversion',
'-DBOOST_ERROR_CODE_HEADER_ONLY', '-DLIBCARLA_WITH_PYTHON_SUPPORT'
]
if 'BUILD_RSS_VARIANT' in os.environ and os.environ['BUILD_RSS_VARIANT'] == 'true':
print('Building AD RSS variant.')
extra_compile_args += ['-DLIBCARLA_RSS_ENABLED']
extra_link_args += [os.path.join(pwd, 'dependencies/lib/libad-rss.a')]
if 'TRAVIS' in os.environ and os.environ['TRAVIS'] == 'true':
print('Travis CI build detected: disabling PNG support.')
extra_link_args += ['-ljpeg', '-ltiff']
extra_compile_args += ['-DLIBCARLA_IMAGE_WITH_PNG_SUPPORT=false']
else:
extra_link_args += ['-lpng', '-ljpeg', '-ltiff']
extra_compile_args += ['-DLIBCARLA_IMAGE_WITH_PNG_SUPPORT=true']
# #todo Why would we need this?
include_dirs += ['/usr/lib/gcc/x86_64-linux-gnu/7/include']
library_dirs += ['/usr/lib/gcc/x86_64-linux-gnu/7']
extra_link_args += ['/usr/lib/gcc/x86_64-linux-gnu/7/libstdc++.a']
else:
raise NotImplementedError
elif os.name == "nt":
sources += [x for x in walk('dependencies/include/carla', '*.cpp')]
pwd = os.path.dirname(os.path.realpath(__file__))
pylib = 'libboost_python%d%d' % (
sys.version_info.major,
sys.version_info.minor)
extra_link_args = ['shlwapi.lib' ]
required_libs = [
pylib, 'libboost_filesystem',
'rpc.lib', 'carla_client.lib',
'libpng.lib', 'zlib.lib',
'Recast.lib', 'Detour.lib', 'DetourCrowd.lib']
# Search for files in 'PythonAPI\carla\dependencies\lib' that contains
# the names listed in required_libs in it's file name
libs = [x for x in os.listdir('dependencies/lib') if any(d in x for d in required_libs)]
for lib in libs:
extra_link_args.append(os.path.join(pwd, 'dependencies/lib', lib))
# https://learn.microsoft.com/es-es/cpp/porting/modifying-winver-and-win32-winnt
extra_compile_args = [
'/experimental:external', '/external:I', 'dependencies/include/system',
'/DBOOST_ALL_NO_LIB', '/DBOOST_PYTHON_STATIC_LIB',
'/DBOOST_ERROR_CODE_HEADER_ONLY', '/D_WIN32_WINNT=0x0600', '/DHAVE_SNPRINTF',
'/DLIBCARLA_WITH_PYTHON_SUPPORT', '-DLIBCARLA_IMAGE_WITH_PNG_SUPPORT=true']
else:
raise NotImplementedError
depends = [x for x in walk('source/libcarla')]
depends += [x for x in walk('dependencies')]
def make_extension(name, sources):
return Extension(
name,
sources=sources,
include_dirs=include_dirs,
library_dirs=library_dirs,
libraries=libraries,
extra_compile_args=extra_compile_args,
extra_link_args=extra_link_args,
language='c++14',
depends=depends)
print('compiling:\n - %s' % '\n - '.join(sources))
return [make_extension('carla.libcarla', sources)]
setup(
name='carla',
version='0.9.8',
package_dir={'': 'source'},
packages=['carla'],
ext_modules=get_libcarla_extensions(),
license='MIT License',
description='Python API for communicating with the CARLA server.',
url='https://github.com/carla-simulator/carla',
author='The CARLA team',
author_email='carla.simulator#gmail.com',
include_package_data=True)
includes in MPC.cpp:
#include "carla/trafficmanager/MPC.h"
#include <cppad/cppad.hpp>
#include <cppad/ipopt/solve.hpp>
#include <Eigen-3.3/Eigen/Dense>
includes in MPC.h:
#include <vector>
#include <Eigen-3.3/Eigen/Dense>
#include "carla/trafficmanager/PIDController.h"
#include "carla/client/Vehicle.h"
#include "carla/geom/Math.h"
Can anyone help?
The setup script calls the compiler to generate a shared library. Just modify the extra_link_args to link your library as you would do if you were calling the compiler directly.
Assuming you're using Linux, if you have your library installed in your machine add a -lipopt, in the same way is done with -lpng for libpng. If you have it somewhere not on your system libraries path or want to link statically pass the full path, just as it's done for libcarla_client.a for instance.
Carla uses clang as compiler on Linux, you can check its command-line ref if you want more details.
The recent version of carla was throwing the same error for me after researching for some time ,I have tried this method hope it works for you. First go to the location of the spawn.py folder using command prompt and type the following commands.
py -3.7 spawn.py
It supports only 3.7 version of python .You can refer youtube to understand how to run keep multiple version of python in windows.
Related
I have this setup.py for my Cython project:
from setuptools import setup
from Cython.Build import cythonize
setup(
name = 'phase-engine',
version = '0.1',
ext_modules = cythonize(["phase_engine.pyx"] + ['music-synthesizer-for-android/src/' + p for p in [
'fm_core.cc', 'dx7note.cc', 'env.cc', 'exp2.cc', 'fm_core.cc', 'fm_op_kernel.cc', 'freqlut.cc', 'lfo.cc', 'log2.cc', 'patch.cc', 'pitchenv.cc', 'resofilter.cc', 'ringbuffer.cc', 'sawtooth.cc', 'sin.cc', 'synth_unit.cc'
]],
include_path = ['music-synthesizer-for-android/src/'],
language = 'c++',
)
)
when I run buildozer, it gets angry about some Cython features only being available in C++ mode:
def __dealloc__(self):
del self.p_synth_unit
^
------------------------------------------------------------
phase_engine.pyx:74:8: Operation only allowed in c++
from which I understand it's ignoring my setup.py and doing its own somehow. How do I give it all these parameters?
CythonRecipe doesn't work well for Cython code that imports C/C++ code. Try CompiledComponentsPythonRecipe, or if you're having issues with #include <ios> or some other thing from the C++ STL, CppCompiledComponentsPythonRecipe:
from pythonforandroid.recipe import IncludedFilesBehaviour, CppCompiledComponentsPythonRecipe
import os
import sys
class MyRecipe(IncludedFilesBehaviour, CppCompiledComponentsPythonRecipe):
version = 'stable'
src_filename = "../../../phase-engine"
name = 'phase-engine'
depends = ['setuptools']
call_hostpython_via_targetpython = False
install_in_hostpython = True
def get_recipe_env(self, arch):
env = super().get_recipe_env(arch)
env['LDFLAGS'] += ' -lc++_shared'
return env
recipe = MyRecipe()
The dependency on setuptools is essential because of some weird stuff, otherwise you get an error no module named setuptools. The two other flags were also related to that error, the internet said they're relevant so I tried value combinations until one worked.
The LDFLAGS thing fixes an issue I had later (see buildozer + Cython + C++ library: dlopen failed: cannot locate symbol symbol-name referenced by module.so).
I have a C++ library which I have successfully exposed to python using Pybind11.
In the CmakeLists.txt file, I have added the numpy include like this:
include_directories("C:\\Python37\\Lib\\site-packages\\numpy\\core\\include")
This works, but is undesirable. I would like to pass the numpy include directory from my setup.py file.
My setup.py file looks very much like this one:
import os
import re
import sys
import sysconfig
import platform
import subprocess
from distutils.version import LooseVersion
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext
class CMakeExtension(Extension):
def __init__(self, name, sourcedir=''):
Extension.__init__(self, name, sources=[])
self.sourcedir = os.path.abspath(sourcedir)
class CMakeBuild(build_ext):
def run(self):
try:
out = subprocess.check_output(['cmake', '--version'])
except OSError:
raise RuntimeError(
"CMake must be installed to build the following extensions: " +
", ".join(e.name for e in self.extensions))
if platform.system() == "Windows":
cmake_version = LooseVersion(re.search(r'version\s*([\d.]+)',
out.decode()).group(1))
if cmake_version < '3.1.0':
raise RuntimeError("CMake >= 3.1.0 is required on Windows")
for ext in self.extensions:
self.build_extension(ext)
def build_extension(self, ext):
extdir = os.path.abspath(
os.path.dirname(self.get_ext_fullpath(ext.name)))
cmake_args = ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=' + extdir,
'-DPYTHON_EXECUTABLE=' + sys.executable]
cfg = 'Debug' if self.debug else 'Release'
build_args = ['--config', cfg]
if platform.system() == "Windows":
cmake_args += ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_{}={}'.format(
cfg.upper(),
extdir)]
if sys.maxsize > 2**32:
cmake_args += ['-A', 'x64']
build_args += ['--', '/m']
else:
cmake_args += ['-DCMAKE_BUILD_TYPE=' + cfg]
build_args += ['--', '-j2']
env = os.environ.copy()
env['CXXFLAGS'] = '{} -DVERSION_INFO=\\"{}\\"'.format(
env.get('CXXFLAGS', ''),
self.distribution.get_version())
if not os.path.exists(self.build_temp):
os.makedirs(self.build_temp)
subprocess.check_call(['cmake', ext.sourcedir] + cmake_args,
cwd=self.build_temp, env=env)
subprocess.check_call(['cmake', '--build', '.'] + build_args,
cwd=self.build_temp)
print() # Add an empty line for cleaner output
setup(
name='python_cpp_example',
version='0.1',
author='Benjamin Jack',
author_email='benjamin.r.jack#gmail.com',
description='A hybrid Python/C++ test project',
long_description='',
# add extension module
ext_modules=[CMakeExtension('python_cpp_example')],
# add custom build_ext command
cmdclass=dict(build_ext=CMakeBuild),
zip_safe=False,
)
After having a look at some SO questions like this, I know that you can get the numpy include directory with numpy.get_include().
However, adding the include directory to the path inside the function build_extension with this line ext.include_dirs.append(numpy.get_include()) seems to have no effect.
I'd like to know how to pass the include directory properly.
Is your cmake build using the wrong numpy path or not finding numpy at all? If it's the wrong path, you could try prepending instead of appending numpy.get_include():
ext.include_dirs.insert(0,numpy.get_include())
Good afternoon.
I have a build script in Scons:
EnsureSConsVersion(0,14);
import string
import os
import os.path
import glob
import sys
import methods
methods.update_version()
# scan possible build platforms
platform_list = [] # list of platforms
platform_opts = {} # options for each platform
platform_flags = {} # flags for each platform
active_platforms=[]
active_platform_ids=[]
platform_exporters=[]
global_defaults=[]
for x in glob.glob("platform/*"):
if (not os.path.isdir(x)):
continue
tmppath="./"+x
sys.path.append(tmppath)
import detect
if (os.path.exists(x+"/export/export.cpp")):
platform_exporters.append(x[9:])
if (os.path.exists(x+"/globals/global_defaults.cpp")):
global_defaults.append(x[9:])
if (detect.is_active()):
active_platforms.append( detect.get_name() )
active_platform_ids.append(x);
if (detect.can_build()):
x=x.replace("platform/","") # rest of world
x=x.replace("platform\\","") # win32
platform_list+=[x]
platform_opts[x]=detect.get_opts()
platform_flags[x]=detect.get_flags()
sys.path.remove(tmppath)
sys.modules.pop('detect')
module_list=methods.detect_modules()
print "Detected Platforms: "+str(platform_list)
print("Detected Modules: "+str(module_list))
methods.save_active_platforms(active_platforms,active_platform_ids)
custom_tools=['default']
if (os.name=="posix"):
pass
elif (os.name=="nt"):
if (os.getenv("VSINSTALLDIR")==None):
custom_tools=['mingw']
env_base=Environment(tools=custom_tools,ENV = {'PATH' : os.environ['PATH']});
#env_base=Environment(tools=custom_tools);
env_base.global_defaults=global_defaults
env_base.android_source_modules=[]
env_base.android_source_files=[]
env_base.android_module_libraries=[]
env_base.android_manifest_chunk=""
env_base.disabled_modules=[]
env_base.__class__.android_module_source = methods.android_module_source
env_base.__class__.android_module_library = methods.android_module_library
env_base.__class__.android_module_file = methods.android_module_file
env_base.__class__.android_module_manifest = methods.android_module_manifest
env_base.__class__.disable_module = methods.disable_module
env_base.__class__.add_source_files = methods.add_source_files
customs = ['custom.py']
profile = ARGUMENTS.get("profile", False)
if profile:
import os.path
if os.path.isfile(profile):
customs.append(profile)
elif os.path.isfile(profile+".py"):
customs.append(profile+".py")
opts=Options(customs, ARGUMENTS)
opts.Add('target', 'Compile Target (debug/profile/release).', "debug")
opts.Add('platform','Platform: '+str(platform_list)+'(sfml).',"")
opts.Add('python','Build Python Support: (yes/no)','no')
opts.Add('squirrel','Build Squirrel Support: (yes/no)','no')
opts.Add('tools','Build Tools (Including Editor): (yes/no)','yes')
opts.Add('lua','Build Lua Support: (yes/no)','no')
opts.Add('rfd','Remote Filesystem Driver: (yes/no)','no')
opts.Add('gdscript','Build GDSCript support: (yes/no)','yes')
opts.Add('vorbis','Build Ogg Vorbis Support: (yes/no)','yes')
opts.Add('minizip','Build Minizip Archive Support: (yes/no)','yes')
opts.Add('opengl', 'Build OpenGL Support: (yes/no)', 'yes')
opts.Add('game', 'Game (custom) Code Directory', "")
opts.Add('squish','Squish BC Texture Compression (yes/no)','yes')
opts.Add('theora','Theora Video (yes/no)','yes')
opts.Add('freetype','Freetype support in editor','yes')
opts.Add('speex','Speex Audio (yes/no)','yes')
opts.Add('xml','XML Save/Load support (yes/no)','yes')
opts.Add('png','PNG Image loader support (yes/no)','yes')
opts.Add('jpg','JPG Image loader support (yes/no)','yes')
opts.Add('webp','WEBP Image loader support (yes/no)','yes')
opts.Add('dds','DDS Texture loader support (yes/no)','yes')
opts.Add('pvr','PVR (PowerVR) Texture loader support (yes/no)','yes')
opts.Add('builtin_zlib','Use built-in zlib (yes/no)','yes')
opts.Add('musepack','Musepack Audio (yes/no)','yes')
opts.Add('default_gui_theme','Default GUI theme (yes/no)','yes')
opts.Add("CXX", "Compiler");
opts.Add("nedmalloc", "Add nedmalloc support", 'yes');
opts.Add("CCFLAGS", "Custom flags for the C++ compiler");
opts.Add("CFLAGS", "Custom flags for the C compiler");
opts.Add("LINKFLAGS", "Custom flags for the linker");
opts.Add('disable_3d', 'Disable 3D nodes for smaller executable (yes/no)', "no")
opts.Add('disable_advanced_gui', 'Disable advance 3D gui nodes and behaviors (yes/no)', "no")
opts.Add('old_scenes', 'Compatibility with old-style scenes', "yes")
# add platform specific options
for k in platform_opts.keys():
opt_list = platform_opts[k]
for o in opt_list:
opts.Add(o[0],o[1],o[2])
for x in module_list:
opts.Add('module_'+x+'_enabled', "Enable module '"+x+"'.", "yes")
opts.Update(env_base) # update environment
Help(opts.GenerateHelpText(env_base)) # generate help
# add default include paths
env_base.Append(CPPPATH=['#core','#core/math','#tools','#drivers','#'])
# configure ENV for platform
env_base.detect_python=True
env_base.platform_exporters=platform_exporters
"""
sys.path.append("./platform/"+env_base["platform"])
import detect
detect.configure(env_base)
sys.path.remove("./platform/"+env_base["platform"])
sys.modules.pop('detect')
"""
if (env_base['target']=='debug'):
env_base.Append(CPPFLAGS=['-DDEBUG_MEMORY_ALLOC']);
env_base.Append(CPPFLAGS=['-DSCI_NAMESPACE'])
env_base.platforms = {}
for p in platform_list:
sys.path.append("./platform/"+p)
import detect
if "create" in dir(detect):
env = detect.create(env_base)
else:
env = env_base.Clone()
CCFLAGS = env.get('CCFLAGS', '')
env['CCFLAGS'] = ''
env.Append(CCFLAGS=string.split(str(CCFLAGS)))
detect.configure(env)
env['platform'] = p
sys.path.remove("./platform/"+p)
sys.modules.pop('detect')
flag_list = platform_flags[p]
for f in flag_list:
env[f[0]] = f[1]
env.module_list=[]
for x in module_list:
if env['module_'+x+'_enabled'] != "yes":
continue
tmppath="./modules/"+x
sys.path.append(tmppath)
env.current_module=x
import config
if (config.can_build(p)):
config.configure(env)
env.module_list.append(x)
sys.path.remove(tmppath)
sys.modules.pop('config')
if (env['musepack']=='yes'):
env.Append(CPPFLAGS=['-DMUSEPACK_ENABLED']);
if (env["old_scenes"]=='yes'):
env.Append(CPPFLAGS=['-DOLD_SCENE_FORMAT_ENABLED'])
if (env["rfd"]=='yes'):
env.Append(CPPFLAGS=['-DRFD_ENABLED'])
if (env["builtin_zlib"]=='yes'):
env.Append(CPPPATH=['#drivers/builtin_zlib/zlib'])
if (env['squirrel']=='yes'):
env.Append(CPPFLAGS=['-DSQUIRREL_ENABLED'])
env.Append(CPPPATH=['#script/squirrel/src'])
# to test 64 bits compiltion
# env.Append(CPPFLAGS=['-m64'])
if (env['lua']=='yes'):
env.Append(CPPFLAGS=['-DLUA_ENABLED'])
env.Append(CPPPATH=['#script/lua/src'])
if (env_base['squish']=='yes'):
env.Append(CPPFLAGS=['-DSQUISH_ENABLED']);
if (env['vorbis']=='yes'):
env.Append(CPPFLAGS=['-DVORBIS_ENABLED']);
if (env['theora']=='yes'):
env.Append(CPPFLAGS=['-DTHEORA_ENABLED']);
if (env['png']=='yes'):
env.Append(CPPFLAGS=['-DPNG_ENABLED']);
if (env['dds']=='yes'):
env.Append(CPPFLAGS=['-DDDS_ENABLED']);
if (env['pvr']=='yes'):
env.Append(CPPFLAGS=['-DPVR_ENABLED']);
if (env['jpg']=='yes'):
env.Append(CPPFLAGS=['-DJPG_ENABLED']);
if (env['webp']=='yes'):
env.Append(CPPFLAGS=['-DWEBP_ENABLED']);
if (env['speex']=='yes'):
env.Append(CPPFLAGS=['-DSPEEX_ENABLED']);
if (env['tools']=='yes'):
env.Append(CPPFLAGS=['-DTOOLS_ENABLED'])
if (env['disable_3d']=='yes'):
env.Append(CPPFLAGS=['-D_3D_DISABLED'])
if (env['gdscript']=='yes'):
env.Append(CPPFLAGS=['-DGDSCRIPT_ENABLED'])
if (env['disable_advanced_gui']=='yes'):
env.Append(CPPFLAGS=['-DADVANCED_GUI_DISABLED'])
if (env['minizip'] == 'yes'):
env.Append(CPPFLAGS=['-DMINIZIP_ENABLED'])
if (env['xml']=='yes'):
env.Append(CPPFLAGS=['-DXML_ENABLED'])
if (env['default_gui_theme']=='no'):
env.Append(CPPFLAGS=['-DDEFAULT_THEME_DISABLED'])
if (env["python"]=='yes'):
detected=False;
if (env.detect_python):
print("Python 3.0 Prefix:");
pycfg_exec="python3-config"
errorval=os.system(pycfg_exec+" --prefix")
prefix=""
if (not errorval):
#gah, why can't it get both at the same time like pkg-config, sdl-config, etc?
env.ParseConfig(pycfg_exec+" --cflags")
env.ParseConfig(pycfg_exec+" --libs")
detected=True
if (detected):
env.Append(CPPFLAGS=['-DPYTHON_ENABLED'])
#remove annoying warnings
if ('-Wstrict-prototypes' in env["CCFLAGS"]):
env["CCFLAGS"].remove('-Wstrict-prototypes');
if ('-fwrapv' in env["CCFLAGS"]):
env["CCFLAGS"].remove('-fwrapv');
else:
print("Python 3.0 not detected ("+pycfg_exec+") support disabled.");
#if env['nedmalloc'] == 'yes':
# env.Append(CPPFLAGS = ['-DNEDMALLOC_ENABLED'])
Export('env')
#build subdirs, the build order is dependent on link order.
SConscript("core/SCsub")
SConscript("servers/SCsub")
SConscript("scene/SCsub")
SConscript("tools/SCsub")
SConscript("script/SCsub");
SConscript("drivers/SCsub")
SConscript("bin/SCsub")
if env['game']:
SConscript(env['game']+'/SCsub')
SConscript("modules/SCsub")
SConscript("main/SCsub")
SConscript("platform/"+p+"/SCsub"); # build selected platform
This script collects game engine Godot (http://www.godotengine.org)
I want to convert this script in the project file of Visual Studio 2010.
In Scons have corresponding command env.MSVSProject().
Description:
Builds a Microsoft Visual Studio project file, and by default builds a solution file as well.Example usage:
barsrcs = ['bar.cpp'], barincs = ['bar.h'], barlocalincs =
['StdAfx.h'] barresources = ['bar.rc','resource.h'] barmisc =
['bar_readme.txt']
dll = env.SharedLibrary(target = 'bar.dll',
source = barsrcs)
env.MSVSProject(target = 'Bar' + env['MSVSPROJECTSUFFIX'],
srcs = barsrcs,
incs = barincs,
localincs = barlocalincs,
resources = barresources,
misc = barmisc,
buildtarget = dll,
variant = 'Release')
More - http://www.scons.org/doc/1.2.0/HTML/scons-user/a8304.html
But I do not understand how to pass the appropriate parameters, as they choose from a script.
Tell me, please.
The MSVSProject builder can create a Visual Studio project but it's only a project for you library with your files in them. You can't convert a SCons' configuration file to a VS project.
The builder only serve for people that wanted to work in Visual Studio IDE. But you need to continue compile with the scons command line tool.
There's a way to achieve what you want to achieve by properly configuring your project.
In the project's property page, under 'NMake' settings make sure that these parameters reflect the scons build command
Build command line: (eg: scons -C ....\ debug=1)
Rebuild All command line: (eg: scons -C ....\ -c && scons -C ....\ debug=1)
Clean command line: the scons command line to clean the project
These values should also be set to their proper values:
Output: (i.e. executable name)
In the project's property page, under 'General' settings make sure that these parameters correspond to:
Configuration Type: Makefile
You can find a complete description of the above at http://grbd.github.io/posts/2016/07/27/scons-builds-with-visual-studio/
I have a running project made in qt . For building purpose I m using waf build tool. To get the same project up and running from waf I need to add
#include "file.moc"
at the end of some files to avoid undefined reference. But if these includes are not commented in qt I get can not find file errors. How do you make qt ignore certain file includes. I thought something like this should have done the trick
#ifndef Q_MOC_RUN
#include "file.moc"
#endif
Due to the limited information provided, all I can show is what waf can do:
You either can include the moc files or the unprocessed files.
Examples are included in the distributed source at https://code.google.com/p/waf/downloads/detail?name=waf-1.6.11.tar.bz2
Subdirectories:
playground/slow_qt/
demos/qt4/
For the sake of completness, simplified examples:
default includes
def options(opt):
opt.load('compiler_cxx qt4')
def configure(conf):
conf.load('compiler_cxx qt4')
conf.load('slow_qt4')
def build(bld):
bld(
features = 'qt4 cxx cxxprogram',
uselib = 'QTCORE QTGUI QTOPENGL QTSVG',
source = 'some.cpp files.cpp',
includes = '.',
target = 'dummy',
)
moc cpp
def options(opt):
opt.load('compiler_cxx qt4')
def configure(conf):
conf.load('compiler_cxx qt4')
def build(bld):
bld(
features = 'qt4 cxx cxxprogram',
uselib = 'QTCORE QTGUI QTOPENGL QTSVG',
source = 'some.cpp files.cpp',
target = 'dummy',
includes = '.')
)
I am programming a C++ extension for Python and I am using distutils to compile the project. As the project grows, rebuilding it takes longer and longer. Is there a way to speed up the build process?
I read that parallel builds (as with make -j) are not possible with distutils. Are there any good alternatives to distutils which might be faster?
I also noticed that it's recompiling all object files every time I call python setup.py build, even when I only changed one source file. Should this be the case or might I be doing something wrong here?
In case it helps, here are some of the files which I try to compile: https://gist.github.com/2923577
Thanks!
Try building with environment variable CC="ccache gcc", that will speed up build significantly when the source has not changed. (strangely, distutils uses CC also for c++ source files). Install the ccache package, of course.
Since you have a single extension which is assembled from multiple compiled object files, you can monkey-patch distutils to compile those in parallel (they are independent) - put this into your setup.py (adjust the N=2 as you wish):
# monkey-patch for parallel compilation
def parallelCCompile(self, sources, output_dir=None, macros=None, include_dirs=None, debug=0, extra_preargs=None, extra_postargs=None, depends=None):
# those lines are copied from distutils.ccompiler.CCompiler directly
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(output_dir, macros, include_dirs, sources, depends, extra_postargs)
cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
# parallel code
N=2 # number of parallel compilations
import multiprocessing.pool
def _single_compile(obj):
try: src, ext = build[obj]
except KeyError: return
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
# convert to list, imap is evaluated on-demand
list(multiprocessing.pool.ThreadPool(N).imap(_single_compile,objects))
return objects
import distutils.ccompiler
distutils.ccompiler.CCompiler.compile=parallelCCompile
For the sake of completeness, if you have multiple extensions, you can use the following solution:
import os
import multiprocessing
try:
from concurrent.futures import ThreadPoolExecutor as Pool
except ImportError:
from multiprocessing.pool import ThreadPool as LegacyPool
# To ensure the with statement works. Required for some older 2.7.x releases
class Pool(LegacyPool):
def __enter__(self):
return self
def __exit__(self, *args):
self.close()
self.join()
def build_extensions(self):
"""Function to monkey-patch
distutils.command.build_ext.build_ext.build_extensions
"""
self.check_extensions_list(self.extensions)
try:
num_jobs = os.cpu_count()
except AttributeError:
num_jobs = multiprocessing.cpu_count()
with Pool(num_jobs) as pool:
pool.map(self.build_extension, self.extensions)
def compile(
self, sources, output_dir=None, macros=None, include_dirs=None,
debug=0, extra_preargs=None, extra_postargs=None, depends=None,
):
"""Function to monkey-patch distutils.ccompiler.CCompiler"""
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
output_dir, macros, include_dirs, sources, depends, extra_postargs
)
cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
for obj in objects:
try:
src, ext = build[obj]
except KeyError:
continue
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
# Return *all* object filenames, not just the ones we just built.
return objects
from distutils.ccompiler import CCompiler
from distutils.command.build_ext import build_ext
build_ext.build_extensions = build_extensions
CCompiler.compile = compile
I've got this working on Windows with clcache, derived from eudoxos's answer:
# Python modules
import datetime
import distutils
import distutils.ccompiler
import distutils.sysconfig
import multiprocessing
import multiprocessing.pool
import os
import sys
from distutils.core import setup
from distutils.core import Extension
from distutils.errors import CompileError
from distutils.errors import DistutilsExecError
now = datetime.datetime.now
ON_LINUX = "linux" in sys.platform
N_JOBS = 4
#------------------------------------------------------------------------------
# Enable ccache to speed up builds
if ON_LINUX:
os.environ['CC'] = 'ccache gcc'
# Windows
else:
# Using clcache.exe, see: https://github.com/frerich/clcache
# Insert path to clcache.exe into the path.
prefix = os.path.dirname(os.path.abspath(__file__))
path = os.path.join(prefix, "bin")
print "Adding %s to the system path." % path
os.environ['PATH'] = '%s;%s' % (path, os.environ['PATH'])
clcache_exe = os.path.join(path, "clcache.exe")
#------------------------------------------------------------------------------
# Parallel Compile
#
# Reference:
#
# http://stackoverflow.com/questions/11013851/speeding-up-build-process-with-distutils
#
def linux_parallel_cpp_compile(
self,
sources,
output_dir=None,
macros=None,
include_dirs=None,
debug=0,
extra_preargs=None,
extra_postargs=None,
depends=None):
# Copied from distutils.ccompiler.CCompiler
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
output_dir, macros, include_dirs, sources, depends, extra_postargs)
cc_args = self._get_cc_args(pp_opts, debug, extra_preargs)
def _single_compile(obj):
try:
src, ext = build[obj]
except KeyError:
return
self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
# convert to list, imap is evaluated on-demand
list(multiprocessing.pool.ThreadPool(N_JOBS).imap(
_single_compile, objects))
return objects
def windows_parallel_cpp_compile(
self,
sources,
output_dir=None,
macros=None,
include_dirs=None,
debug=0,
extra_preargs=None,
extra_postargs=None,
depends=None):
# Copied from distutils.msvc9compiler.MSVCCompiler
if not self.initialized:
self.initialize()
macros, objects, extra_postargs, pp_opts, build = self._setup_compile(
output_dir, macros, include_dirs, sources, depends, extra_postargs)
compile_opts = extra_preargs or []
compile_opts.append('/c')
if debug:
compile_opts.extend(self.compile_options_debug)
else:
compile_opts.extend(self.compile_options)
def _single_compile(obj):
try:
src, ext = build[obj]
except KeyError:
return
input_opt = "/Tp" + src
output_opt = "/Fo" + obj
try:
self.spawn(
[clcache_exe]
+ compile_opts
+ pp_opts
+ [input_opt, output_opt]
+ extra_postargs)
except DistutilsExecError, msg:
raise CompileError(msg)
# convert to list, imap is evaluated on-demand
list(multiprocessing.pool.ThreadPool(N_JOBS).imap(
_single_compile, objects))
return objects
#------------------------------------------------------------------------------
# Only enable parallel compile on 2.7 Python
if sys.version_info[1] == 7:
if ON_LINUX:
distutils.ccompiler.CCompiler.compile = linux_parallel_cpp_compile
else:
import distutils.msvccompiler
import distutils.msvc9compiler
distutils.msvccompiler.MSVCCompiler.compile = windows_parallel_cpp_compile
distutils.msvc9compiler.MSVCCompiler.compile = windows_parallel_cpp_compile
# ... call setup() as usual
You can do this easily if you have Numpy 1.10 available. Just add:
try:
from numpy.distutils.ccompiler import CCompiler_compile
import distutils.ccompiler
distutils.ccompiler.CCompiler.compile = CCompiler_compile
except ImportError:
print("Numpy not found, parallel compile not available")
Use -j N or set NPY_NUM_BUILD_JOBS.
In the limited examples you provided in the link, it seems fairly obvious that you have some misunderstanding on what some of the features of the language are. For example, the gsminterface.h has a whole lot of namespace level statics, which is probably unintended. Every translation unit that includes that header will compile it's own version for everyone of the symbols declared in that header. Side effects of this are not only the compile time but also code bloat (larger binaries) and link time as the linker needs to process all those symbols.
There are still many questions that affect the build process that you have not answered, for example, whether you clean every time before you recompile. If you are doing that, then you might want to consider ccache, which is a tool that caches the result of the build process, so that if you run make clean; make target only the preprocessor will be run for any translation unit that has not changed. Note that as long as you keep maintaining most code in headers, this will not offer much of an advantage, as a change in a header modifies all translation units that include it. (I don't know your build system, so I cannot tell you whether python setup.py build will clean or not)
The project does not seem large otherwise, so I would be surprised if it took more than a few seconds to compile.