I have an scons project that force includes several header files as a compiler flag.
# Forced include files passed directly to the compiler
env.Append(CCFLAGS = ['/FIinclude.h'])
These files are not included by any files in the project. I need to add an explicit dependency for them.
forced_include_headers = ['include.h']
# Trying to add an explicit dependency for each build target
for object in BUILD_TARGETS:
env.Depends(object, forced_include_headers)
The problem I'm running into is that BUILD_TARGETS list is empty. It seems to only contain things passed from COMMAND_LINE_TARGETS or DEFAULT_TARGETS. All of the targets in our project are built implicitly. We do not make use of env.Default, etc. Is there a way to get the implicit target list, or do I have to manually build it? I noticed that TARGETS is reserved and does not seem to contain what I want either.
I can add an env.Depends(target, forced_include_headers) for all of the targets in their respective SConscript files, but the project is quite large.
I'm not sure that this is the best way to solve the problem (in fact, I think the idea of creating a pseudo-builder works better), but this code will return a list of Object targets:
# Typical SConstruct / SConscript
env = Environment()
env.Program('foo.c')
lib = env.SharedLibrary('bar.c')
env.Program('foo2.c', LIBS=[lib])
SConscript('subdir/SConscript', exports={'env':env})
# Get a list of all Object targets
def get_all_targets(env, node='.'):
def get_all_targets_iter(env, node):
if node.has_builder() and node.get_builder().get_name(env) in ('Object', 'SharedObject'):
yield node
for kid in node.all_children():
for kid in get_all_targets(env, kid):
yield kid
node = env.arg2nodes(node, env.fs.Entry)[0]
return list(get_all_targets_iter(env, node))
# Force them all to depend upon some header
env.Depends(get_all_targets(env), 'site_wide.h')
Related
I'm using Conan as a dependency manager for C++ and I want to create a package which requires a compiled file from another, already created, Conan-package.
I'm currently trying to create a package for the OpenStreetMap-Library OSM-binary (https://github.com/scrosby/OSM-binary.git).
The Makefile for this project (which can be found at ./OSM-binary/src/Makefile) requires a file called protoc from the protobuf-project (https://github.com/google/protobuf). This protoc-file can be found after compiling the protobuf-project in ./protobuf/src.
Without this file compiling the OSM-sources will fail with an error: make: ../protoc: Command not found
The Problem
As conan's documentation suggests to copy my needed files to folders in my package, e.g header-files to ./include, libs to ./lib, etc.
According to this, after building the protobuf-project via make, I'm copying the mentioned file via
def package(self):
self.copy("*.so", dst="lib", keep_path=False)
self.copy("protoc", dst="scripts", src="./protobuf/src")
to an folder called "scripts".
But at this point the black magic starts.
My first question is, how can I access any of these packed files (e. g. the *.so files or any other files (here the protoc-file) which are present in an package) from another package?
For me, even after reading conan's documentation, it's not clear how conan stores it's packed files and how to access these or any other files packed in the previous step.
Now back at the OSM-Project my approach would be setting the correct path manually in the Makefile via the tools.replace command.
First I declared the protobuf-packaged as an requirement
requires = "protobuf/2.5.0#test/testing"
and replaced the corresponding lines (in version 1.3.3, line 7) in the osm-Makefile with the correct path to the protoc-file.
tools.replace_in_file("OSM-binary/src/Makefile",
"PROTOC ?= protoc",
"PROTOC ?= <path-to-file>/protoc")
Now this leads us to my actual question: How do I get the path to the protoc-file which can be found in the protobuf-package in a folder called scripts, or is there any other way to do it?
Thanks,
Chris
There are different ways to access files from your dependencies:
If you want to directly run some file from your dependencies, you could use the self.run(...., run_environment=True), that will automatically set the PATH, LD_LIBRARY_PATH, etc so the binaries are found in the place where the package is installed. Find more information here
You can directly import the files you want from your dependencies, doing a copy (which is done before the build() method) of such files into the build folder, so they can be directly used there. The path you can use in your script is the current one, or self.build_folder. The imported files will be automatically removed after build, so they are not accidentally repackaged. Check imports docs
You can obtain information from your dependencies from the self.deps_cpp_info attribute. Check the reference here. That means you can get the paths to your protobuf dependency with something like
def build(self):
# Get the directory where protobuf package is installed
protoc_root = self.deps_cpp_info["protobuf"].rootpath
# Note this is a list
protoc_bin_paths = self.deps_cpp_info["protobuf"].bin_paths
my autotools project has a couple of unit-tests.
one of these tests (filereader) needs to read a file (data/test1.bin)
Here's my filesystem layout:
- libfoo/tests/filereader.c
- libfoo/tests/data/test1.bin
and my libfoo/tests/Makefile.am:
AUTOMAKE_OPTIONS = foreign
AM_CPPFLAGS = -I$(top_srcdir)/foo
LDADD = $(top_builddir)/src/libfoo.la
EXTRA_DIST = data/file1.bin
TESTS = filereader
check_PROGRAMS= filereader
filereader_SOURCES = filereader.c
this works great, as long as i do in-tree builds.
However, when running the test-suite out-of-tree (e.g. make distcheck), the filereader test cannot find the input file anymore.
This is obviously because only the source tree contains the input file, but not the build tree.
i wonder what is the canonical way to fix this problem?
compile the directory of the test-file into the unittest (AM_CPPFLAGS+=-DSRCDIR=$(srcdir))
pass the qualified input file as a cmdline argument to the test? (e.g. $(builddir)/filereader $(srcdir)/data/file1.bin)
copy the input file from the source tree to the build tree? (cp $(srcdir)/data/file1.bin $(builddir)/data/file1.bin? how would a proper make-rule look like??)
Canonically, the solution would be to define the path to your file into the unittest, so the first option you laid out. The second one is also possible but it requires using an in-between driver script.
I would suggest avoiding the third one, but if you do want to go down that route, use $(LN_S) rather than cp; this way you reduce the I/O load of the test.
There is a way to do this with autoconf. From the netcdf-c configure.ac:
##
# Some files need to exist in build directories
# that do not correspond to their source directory, or
# the test program makes an assumption about where files
# live. AC_CONFIG_LINKS provides a mechanism to link/copy files
# if an out-of-source build is happening.
##
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat1.nc:nc_test4/ref_hdf5_compat1.nc])
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat2.nc:nc_test4/ref_hdf5_compat2.nc])
AC_CONFIG_LINKS([nc_test4/ref_hdf5_compat3.nc:nc_test4/ref_hdf5_compat3.nc])
AC_CONFIG_LINKS([nc_test4/ref_chunked.hdf4:nc_test4/ref_chunked.hdf4])
AC_CONFIG_LINKS([nc_test4/ref_contiguous.hdf4:nc_test4/ref_contiguous.hdf4])
I'd like to use Boost.Test for Test Driven Development.
I asked scons to create two executables, the main one, and the test one.
All my main project files are in ./src/, and all my test dedicated files are in ./test/
The problem is:
the main project object files are put in ./build/src/
the test project object files are put in ./build/test/
and in such a configuration my executable Test cannot link since all the main project object files (of the classes on which I perform my tests) are not in the same directory.
Do you have an idea how I could tweak my scons file so as the linking of the executable Test can use the object files in ./src./ ?
Below is my main.scons file:
import os
env=Environment(CPPPATH=['/usr/local/boost/boost_1_52_0/boost/','./src/'],
CPPDEFINES=[],
LIBPATH=['/usr/local/boost/boost_1_52_0/boost/libs/','.'],
LIBS=['boost_regex'],
CXXFLAGS="-std=c++0x")
env['ENV']['TERM'] = os.environ['TERM']
env.Program('Main', Glob('src/*.cpp'))
#
testEnv = env.Clone()
testEnv['CPPPATH'].append('./test/')
testEnv['LIBS'].append('boost_unit_test_framework')
testEnv.Program('Test', Glob('test/*.cpp'))
While the "duplicate object lists" approach is fine for simple projects, you may run into limitations in which your test code doesn't need to link against the entire object space of your main program. For example, to stub out a database layer that's not the focus of a particular unit test.
As an alternative, you can create (static) libraries of common code that you link against your primary executable and your test framework.
common_sources = ['src/foo.cpp', 'src/bar.cpp'] # or use Glob and then filter
env.Library("common", common_sources)
program_sources = ['src/main.cpp']
env.Program("my_program", program_sources, LIBS=['common'])
...
testEnv['LIBPATH'] = ['.'] # or wherever you build the library
testEnv.Program("unit_test", test_sources, LIBS=['common'])
This also avoids the duplicate main() problem that you mention because only the program_sources and test_sources lists should contain the appropriate file with the main routine.
I have continued searching, and found This post on the web which intrigued me, using the scons env.Object. Indeed this object contains the list of all target object files.
And with slight modifications I have the scons file that does what I wanted (though now I have a problem of dupplicated main function but that's another problem):
import os
env=Environment(CPPPATH=['/usr/local/boost/boost_1_52_0/boost/','./src/'],
CPPDEFINES=[],
LIBPATH=['/usr/local/boost/boost_1_52_0/boost/libs/','.'],
LIBS=['boost_regex'],
CXXFLAGS="-std=c++0x")
env['ENV']['TERM'] = os.environ['TERM']
# here I keep track of the main project object files
mainObjectFiles = env.Object( Glob('src/*.cpp'))
env.Program('PostgresCpp', mainObjectFiles)
#
testEnv = env.Clone()
testEnv['CPPPATH'].append('./test/')
testEnv['LIBS'].append('boost_unit_test_framework')
# here I append all needed object files
testObjectFiles = Glob('test/*.cpp')
testedObjectFiles = Glob('src/*.cpp')
allObjectFilesExceptMain = [x for x in mainObjectFiles if x != 'src/main.o']
allObjectFilesExceptMain.append(testObjectFiles)
testEnv.Program('Test',allObjectFiles)
I have my project targets (binaries and libraries) specified in json files.
I can create an environment for the specified target without any problem. Now I'm trying to support a specific build directory.
My knowledge about scons is still very basic but it seems that the right way to do that is using a SConscript together with VariantDir. But I already have my targets specified in json and creating a SConscript file for each target would be redundant (plus a cost in maintenance).
So my question is: is it possible to create a SConscript object dynamically, at run time?
Thanks in advance.
VariantDir doesn't work with SConscturct file (may be i wrong, but i don't found any way to do it).
Just create SConscript file with variant dir and doing what you need.
#SConsruct
env = CreateEnvironment()
SConscript('SConscript', variant_dir = 'mybuilddir', exports = 'env', duplicate = 0)
# Do all work in SConscript
Import('env')
env.Program(...)
env.SharedLibrary(...)
...
Also, you can split your process into 2 states.
State 1 - generated SConscript files.
State 2 - run generated SConscript files.
if 'generate' in COMMAND_LINE_TARGETS:
# your code to generated SConscript from json
Exit(0)
sconscriptFiles = getSconscriptFiles() # some code to get your sconscript, by mask for example
if len(sconscriptFiles) < 1:
print "You need to generate files at first: scons generate"
Exit(1)
for file in sconscriptFiles :
SConscript(file, variant_dir = 'build' + file, duplicate = 0)
You can specify the build directory with the VariantDir() function, or as part of the SConscript() call. All of the different options are discussed here. Considering you dont want to use several SConstruct files, you should just use the VariantDir() function as described in more detail here.
Here is a simple example:
env = Environment()
# It may be as simple as setting src_dir='.', but set accordingly
# duplicate=0 tells SCons NOT to copy source files to variantDir, set accordingly
# VariantDir() can be called multiple times so as to change dirs per builder call
VariantDir(variant_dir = 'pathToBuildDir', src_dir = 'pathToSource', duplicate=0)
# Now call the builders here
Its still not clear why you want to mix json with SCons. Unless you have some very compelling reasons to do so, I would suggest keeping it all in SCons, which is Python.
EDIT: I just realized you asked about creating a SConscript object, not a file.
I looked through the SCons programming APIs and didnt find anything that allows you to create a SConscript object. Actually, I dont think the concept of a SConscript object exists, since it just treats calls to the SConscript() function as files that need to be opened and processed, and they are almost treated as an extension to the SConstruct.
So, to summarize: You'll either have to create subsidiary SConscript files, or work with calls to VariantDir(). Depending on your project directory structure, it may not be necessary to create SConscript files. You could just do everything from the root SConstruct. The SConscript files arent necessary, they just help organize the build scripts better.
I've recently picked up scons to implement a multi-platform build framework for a medium sized C++ project. The build generates a bunch of unit-tests which should be invoked at the end of it all. How does one achieve that sort of thing?
For example in my top level sconstruct, I have
subdirs=['list', 'of', 'my', 'subprojects']
for subdir in subdirs:
SConscript(dirs=subdir, exports='env', name='sconscript',
variant_dir=subdir+os.sep+'build'+os.sep+mode, duplicate=0)
Each of the subdir has its unit-tests, however, since there are dependencies between the dlls and executables built inside them - i want to hold the running of tests until all the subdirs have been built and installed (I mean, using env.Install).
Where should I write the loop to iterate through the built tests and execute them? I tried putting it just after this loop - but since scons doesn't let you control the order of execution - it gets executed well before I want it to.
Please help a scons newbie. :)
thanks,
SCons, like Make, uses a declarative method to solving the build problem. You don't want to tell SCons how to do its job. You want to document all the dependencies and then let SCons solve how it builds everything.
If something is being executed before something else, you need to create and hook up the dependencies.
If you want to create dmy touch files, you can create a custom builder like:
import time
def action(target, source, env):
os.system('echo here I am running other build')
dmy_fh = open('dmy_file','w')
dmy_fh.write( 'Dummy dependency file created at %4d.%02d.%02d %02dh%02dm%02ds\n'%time.localtime()[0:6])
dmy_fh.close()
bldr = Builder(action=action)
env.Append( BUILDERS = {'SubBuild' : bldr } )
env.SubBuild(srcs,tgts)
It is very important to put the timestamp into the dummy file, because scons uses md5 hashes. If you have an empty file, the md5 will always be the same and it may decide to not do subsequent build steps. If you need to generate different tweaks on a basic command, you can use function factories to modify a template. e.g.
def gen_a_echo_cmd_func(echo_str):
def cmd_func(target,source,env):
cmd = 'echo %s'%echo_str
print cmd
os.system(cmd)
return cmd_fun
bldr = Builder(action = gen_a_echo_cmd_func('hi'))
env.Append(BUILDERS = {'Hi': bldr})
env.Hi(srcs,tgts)
bldr = Builder(action = gen_a_echo_cmd_func('bye'))
env.Append(BUILDERS = {'Bye': bldr})
env.Bye(srcs,tgts)
If you have something that you want to automatically inject into the scons build flow ( e.g. something that compresses all your build log files after everything else has run ), see my question here.
The solution should be as simple as this.
Make the result of the Test builders depend on the result of the Install builder
In pseudo:
test = Test(dlls)
result = Install(dlls)
Depends(test,result)
The best way would be if the Test builder actually worked out the dll dependencies for you, but there may be all kinds of reasons it doesn't do that.
In terms of dependencies, what you want is for all the test actions to depend on all the program-built actions. A way of doing this is to create and export a dummy-target to all the subdirectories' sconscript files, and in the sconscript files, make the dummy-target Depends on the main targets, and have the test targets Depends on the dummy-target.
I'm having a bit of trouble figuring out how to set up the dummy target, but this basically works:
(in top-level SConstruct)
dummy = env.Command('.all_built', 'SConstruct', 'echo Targets built. > $TARGET')
Export('dummy')
(in each sub-directory's SConscript)
Import('dummy')
for target in target_list:
Depends(dummy, targe)
for test in test_list:
Depends(test, dummy)
I'm sure further refinements are possible, but maybe this'll get you started.
EDIT: also worth pointing out this page on the subject.
Just have each SConscript return a value on which you will build dependencies.
SConscript file:
test = debug_environment.Program('myTest', src_files)
Return('test')
SConstruct file:
dep1 = SConscript([...])
dep2 = SConscript([...])
Depends(dep1, dep2)
Now dep1 build will complete after dep2 build has completed.