I'm trying to create a build system for my project which will be based on scons.
The project consist of several directories, each holding part of the whole project.
Each directory hold the sources of either a shared library, a program or a process (daemon).
One directory (bin) holds all shared libraries and all executables.
Most of the directories already contain a sconscript file (named Sconstruct) which builds the directory's module (lib / executable) and put it in the bin directory.
Now I want to create one more sconscript - to rule them all..
In the parent directory I want a sconscript that builds all libraries and executables of the project, so that after I change a few sources here and there I can run scons from the parent directory and all affected modules will be re-built.
I tried a few ways, and they all failed.
I'm quite a noob in the scons business, and I susspect this is the root cause of my failures, but I'm sure this problem was solved many times by other, more experienced, developers - as the situation I described is quite common.
So, any suggestions are welcomed!
EDIT:
My current Sconstruct in the parent directory looks like this:
import os
env = os.environ
Export('env', 'os')
SConscript([
'Server1_Dir/Sconstruct',
'Server2_Dir/Sconstruct',
'Server3_Dir/Sconstruct'
])
The Sconstructs in the sub-directories (I know they should be called Sconscript) start with:
import os
Import('env')
home=env.get('HOME')
So it seems to me that I'm using the same environment for all scripts, though I'm getting a lot of:
scons: warning: Two different environments were specified for target....
warnings, which are unclear for me.
It's also worth noting that not all server-combinations in the main script yield these warnings - some may leave together peacefully, but not all.
It does seem to me that this is the right way to go, but I can't find a way to get rid of those warnings (and their root cause).
Thanks.
What you want to do is called a hierarchical build as described here. In SCons, there can only be one SConstruct per project.
The way to create a hierarchical build in SCons is to make the root SCons script be the SConstruct, and the subdirectory SCons scripts be called SConscript. The root SConstruct script "loads" the subdirectory SConscripts using the SConscript() function described in the above link.
Here's an example assuming you have a root SConstruct and a lib and bin subdir:
SConstruct
env = Environment()
SConscript('lib/SConscript', exports='env')
SConscript('bin/SConscript', exports='env')
lib/SConscript
Import('env')
env.Library('yourlib', ['yourLibSource.cc'])
bin/SConscript
Import('env')
env.Program('yourbin', ['yourBinSource.cc'])
My scripts, before I tried to unify them, looked like the following (and worked properly):
Librarry building-script Stracture:
Environment and Definitions:
~~~~~~~~~~~~~~~~~~~~~~
import os
env = os.environ
home = env.get('HOME')
projects = []
third_parties_projects = []
my_rpath = []
Preparing lists of paths and files:
~~~~~~~~~~~~~~~~~~~~~~~~~
all_user_implemented_libs = os.path.join(home,"Project/lib/")
my_rpath.append("/path/to/lib/dir/")
projects.append(os.path.join(home,"Project/library_1/"))
projects.append(os.path.join(home,"Project/library_2/"))
...
third_parties_projects.append("/opt/some_product/include/")
...
so_files = Split("""
source_1.cpp
source_2.cpp
...
""")
Controlling the build:
~~~~~~~~~~~~~~~~
myprog = Environment(CXXFLAGS = flags, ENV = env, CPPPATH = projects + third_parties_projects + ['.'], RPATH = my_rpath)
myprog.Decider('MD5')
mylib = myprog.SharedLibrary(all_user_implemented_libs + 'lib_name', so_files)
myprog.Default(mylib)
Process building-script Stracture:
Environment and Definitions:
~~~~~~~~~~~~~~~~~~~~~~
import os
env = os.environ
home = env.get('HOME')
projects = []
third_parties_projects = []
user_libpaths = []
my_rpath = []
Preparing lists of paths and files:
~~~~~~~~~~~~~~~~~~~~~~~~~
projects.append(os.path.join(home,"Project/library_1/"))
projects.append(os.path.join(home,"Project/library_2/"))
....
my_rpath.append("/path/to/lib/dir/")
third_parties_projects.append("/opt/some_product/include/")
...
all_user_implemented_bins = os.path.join(home,"Project/bin/")
all_user_implemented_libs = os.path.join(home,"Project/lib/")
flags = ''
libs = Split("""
lib_1
lib_2
...
""")
third_party_libpaths.append("/opt/some_product/")
...
source_ext_files = Split("""
ext_source_1.cpp
...
""")
source_files_for_executable = Split("""
source_1.cpp
sosurce_2.cpp
...
""")
user_libpaths.append(all_user_implemented_libs)
Building the underlaying libs:
~~~~~~~~~~~~~~~~~~~~~~
for project in projects:
SConscript(project+'/Sconstruct')
Controlling the build:
~~~~~~~~~~~~~~~~
myprog = Environment(CXXFLAGS = flags, ENV = env, CPPPATH = projects + third_parties_projects + ['.'], RPATH = my_rpath)
myprog.Decider('MD5')
my_prog = myprog.Program(all_user_implemented_bins + 'process_name', source_files_for_executable + source_ext_files, LIBS = libs, LIBPATH = third_party_libpaths + user_libpaths)
myprog.Default(my_prog)
These scripts are scattered each in its corresponding lib/exec directory, and work fine when executed separately.
Now I want to add one more script (like the one in the original question) in the parent directory which will build the whole product - all executables and all underlaying libraries - as necessary.
Thanks.
Related
I've got the following directory structure:
root/
SConstruct
item0/
SConstruct
unit0/
unit0.h
private/
unit0.c
test/
test_unit0.c
SConstruct
SConscript
unit1/
(repeat unit0 structure)
item1/
(repeat item0 structure)
I want to be able to run scons from root to build all things below it, and from root/item0 to build all things below it, etc.
The top-level SConstruct looks like:
SConscript('item0/SConstruct', duplicate=0)
SConscript('item1/SConstruct', duplicate=0)
The item-level SConstruct:
SConscript('unit0/SConstruct', duplicate=0)
SConscript('unit1/SConstruct', duplicate=0)
The unit-level SConstruct:
import os
proj_root = os.path.abspath('../../../../..')
proj_shared_path = os.path.join(proj_root,
os.path.normpath('trunk/sys/shared'))
env = Environment()
SConscript(
'SConscript',
variant_dir='output',
exports=['env', 'proj_shared_path'],
duplicate=0)
The unit-level SConscript (important stuff only):
Import('*')
# Setup Repositories to access out-of-current-directory-tree files.
lib1_path = os.path.join(proj_shared_path, 'lib1')
env.Repository('#/../private', lib1_path)
# Set general compilation and link flags.
env.Append(
CFLAGS = '-Wall -Wextra -std=c99 -pedantic',
CPPPATH = ['#/../..', proj_shared_path])
# These are the files we don't need to run gcovr on.
non_cov_srcs = [
'testit.c',
'lib1.c'
]
# Create a 'coverage' environment to build files that we will run gcovr on.
cov = env.Clone()
cov.Append(CFLAGS = '--coverage -O0')
cov_srcs = cov.Object('unit0.c')
SideEffect('unit0.gcno', cov_srcs)
# Our unit test program combines all the sources we've listed so far.
test_exe = env.Program(
'test_unit0',
[cov_srcs, non_cov_srcs],
LIBS='gcov', LINKFLAGS='--coverage')
Now, this all works fine when I run scons from root/item0/unit0/test. However, when I run from root or root/item0, I get various types of errors, like below (run from root/item0):
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [unit0\test\output\unit0.obj] Source 'unit0\test\unit0.c' not found, needed by target 'unit0\test\output\unit0.obj'.
scons: building terminated because of errors.
Clearly unit0\test\unit0.c won't be found because it doesn't exist, but I'm not sure why scons thinks it has to be there since the Repository is defined as '#/../private' and there is no private shown in the error message.
edit I've figured out that it's probably looking for unit0\test\unit0.c because unit0\test is the "root" directory for SConscript. I'm also guessing that the #/../private Repository location is only correct when scons is run from unit0\test. But, when I try to specify the Repository in any other way (including an absolute path built with os.path.abspath (and verified to be correct)), scons still doesn't seem to want to look there for unit0.c. Why is that?
With further help from #bdbaddog in the #scons IRC channel, I was able to determine the following:
What I was actually trying to achieve was ensuring the build artifacts for each source ended up in my variant_dir
With SConstruct and SConscript in the same directory, Repository ended up achieving this when building from the unit0/test directory. But, I didn't know why.
To properly achieve (1), Repository is not the right tool. Instead, SConscript should look like this:
Import('*')
# Set general compilation and link flags.
env.Append(
CFLAGS = '-Wall -Wextra -std=c99 -pedantic',
CPPPATH = ['#/item0', '#/lib1'])
# These are the files we don't need to run gcovr on.
non_cov_objs = [
Object(target='testit', source='testit.c'),
Object(target='lib1', source='#/lib1/lib1.c'
]
# Create a 'coverage' environment to build files that we will run gcovr on.
cov = env.Clone()
cov.Append(CFLAGS = '--coverage -O0')
cov_objs = cov.Object(
target='unit0',
source='#/item0/unit0/private/unit0.c')
SideEffect('unit0.gcno', cov_srcs)
# Our unit test program combines all the sources we've listed so far.
test_exe = env.Program(
'test_unit0',
[cov_objs, non_cov_objs],
LIBS='gcov', LINKFLAGS='--coverage')
By specifying target and source for each Object, I can achieve what I want in terms of the object files (and other side-effects) being generated in the variant_dir.
I've gone through the example scons builds here and have found them wanting in providing a solution that fits my project.
The structure is as follows:
root/
Module_A/
include/
foo.h
bar.h
src/
foo.cpp
bar.cpp
Module_.../
Every module follows the same structure, an include folder for all the .h's and a src file for the cpps. Each module builds into a shared object. There is no executable.
Modules have cross dependencies. For instance Module_A is the logging mechanism and it is used in modules B, C, D, etc. Likewise, Module_B is the Configuration loader, which is used in several other modules. And Module_C would be the IPC module, used in almost each module listed. Lastly, Module_D is the command center and links against EVERY other module (literally).
I am interested in replacing the current setup we have of using recursive make to build the project. I am trying to build the sconstruct and SConscripts necessary to do so, but I am very new to even make, let alone scons.
I am interested in turning each Module's .cpp and .h into a .so and to have its dependencies resolved automagically as is done with make now.
In the SConscript, I currently use glob to get the *.cpps and then include the module's './include' in the CPPPATH.
I have used
env.SharedLibrary(CPPATH='./include', source= (list of the cpps))
But since this depends on other Modules, it will not work, stating the other module's functions that are used are "not declared".
How do I go about getting this kind of complex structure to build using a hierarchical scons setup?
This should be quite easy to do with SCons. You'll probably want a SConscript script at the root of every module. These will all be invoked by a SConstruct script located at the root of the entire project.
If I understand the question correctly, the problem of the dependencies between the modules can be solved by correctly specifying the include paths of all of the modules. This can be done once in an Environment created in the SConstruct, which should then be passed to the module SConscript scripts.
Here's a brief example:
Sconstruct
env = Environment()
# Notice that the '#' in paths makes the path relative to the root SConstruct
includePaths = [
'#/Module_A/include',
'#/Module_B/include',
'#/Module_N/include',
]
env.Append(CPPPATH=includePaths)
SConscript('Module_A/SConscript', exports='env', duplicate=0)
SConscript('Module_B/SConscript', exports='env', duplicate=0)
SConscript('Module_N/SConscript', exports='env', duplicate=0)
Module_A/SConscript
Import('env')
# Notice the CPPPATH's have already been set on the env created in the SConstruct
env.SharedLibrary(target = 'moduleA', source = ModuleA_SourceFiles)
Module_B/SConscript
Import('env')
# Notice the CPPPATH's have already been set on the env created in the SConstruct
env.SharedLibrary(target = 'moduleB', source = ModuleB_SourceFiles)
Module_N/SConscript
Import('env')
# Notice the CPPPATH's have already been set on the env created in the SConstruct
env.SharedLibrary(target = 'moduleN', source = ModuleN_SourceFiles)
I'm trying to use scons for my project. I have the following so far:
SConstruct
path = ['/usr/local/bin', '/usr/bin', '/bin']
libpath = ['/usr/local/lib', '/usr/lib', '/lib']
env = Environment()
env.Append(ENV = {'PATH' : path})
env.Append(DFLAGS = ['-version=placeholder', '-g'])
env.Append(LIBPATH = libpath)
env.Append(DPATH = '#/build_debug/')
Export('env')
env.SConscript('src/SConscript', variant_dir = 'build_debug',
exports = {'DFLAGS' : '-debug'})
src/SConscript
Import('env')
env.Program(target = 'a.out',
source = ['stubs.d'] + Glob('*/*.d'))
When trying to build, though, it gives:
build_debug/foo/foo.d:9: Error: module bar is in file 'bar/bar.d' which cannot be read
Checking ls build_debug shows that not all the files have been copied over.
I think that I had this working earlier; but I must've messed something up since then.
EDIT:
All my source code sits in project/src/ and I would like for the build to happen in project/build/; my SConstruct file sits in project/.
Out of source builds are not supported by SCons, so dont be surprised by unexpected behavior.
They are talking about adding this feature on the SCons dev email lists soon, but they havent said when.
Ive noticed a few things that could be improved in your scripts:
In the call to SConscript() in the SConstruct, the exports is the DFLAGS variable, but in the SConscript, you are only importing the 'env', so the DFLAGS variable wont be visible. You should either import the DFLAGS, or just set the DFLAGS in the env as a different variable, and just get it from the env in the SConscript.
Also, Im not sure the library paths will work correctly as is. You have this:
libpath = ['/usr/local/lib', '/usr/lib', '/lib']
env.Append(LINKFLAGS = libpath)
But as is, SCons wont prepend the -L to each libpath. Instead of using LINKFLAGS use LIBPATH, then SCons will prepend the -L.
You can try to set duplicate parameter to 0
env.SConscript('src/SConscript', variant_dir = 'build_debug',
exports = {'DFLAGS' : '-debug'}, duplicate=0)
I have a hierarchical build system based on SCons. I have a root SConstruct that calls into a SConscript that builds a shared library and then into a different SConscript that builds an executable that depends on the shared library.
So here's my question: my understanding of shared libraries on linux is that when you want to do the final ld link for the executable that will be using the shared lib, the shared lib has to be included on the executable's ld command line as a source to reference it (unless it's in a standard location in which case the -l option works).
So here's something like what my SCons files look like:
=== rootdir/SConstruct
env=DefaultEnvironment()
shared_lib = SConscript('foolib/SConscript')
env.Append( LIBS=[shared_lib] )
executable = SConscript('barexec/SConscript')
=== rootdir/foolib/SConscript
env=DefaultEnvironment()
env.Append(CPPPATH=Glob('inc'))
penv = env.Clone()
penv.Append(CPPPATH=Glob('internal/inc'))
lib = penv.SharedLibrary( 'foo', source=['foo.c', 'morefoo.c']
Return("lib")
=== rootdir/barexec/SConscript
env=DefaultEnvironment()
exe = env.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] )
Return("exe")
So the hitch here is this line:
env.Append( LIBS=[shared_lib] )
This would be a great way to add generated libraries to the command line for any other libs that need them, EXCEPT that because SCons is doing a two-pass run through the SConscripts (first to generate it's dependency tree, then to do the work), rootdir/foolib/libfoo.so winds up on the command line for ALL products, EVEN libfoo.so itself:
gcc -g -Wall -Werror -o libfoo.so foo.o morefoo.o libfoo.so
So how is this best done with SCons? For now I've resorted to this hack:
=== rootdir/SConstruct
env=DefaultEnvironment()
shared_lib = SConscript('foolib/SConscript')
env['shared_lib'] = shared_lib
executable = SConscript('barexec/SConscript')
...
=== rootdir/barexec/SConscript
env=DefaultEnvironment()
exe = env.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] + env['shared_lib'] )
Return("exe")
Is there a more SCons-y way of doing this?
You should allow the shared libraries to be found by the build.
Look for the LIBPATH and RPATH variables in the SCons documentation; these are the "Scons-y" way to set up search paths so that any generated -l options find libraries properly.
Having mentioned the above, here's what you should see gcc do based on the setup of SCons (and if it doesn't, you may have to do it manually).
The -l option always finds shared libraries provided that you also give the compiler the location of the library. There are two times this is needed: at compile time (-L option) and at runtime (-rpath generated linker option).
The LIBPATH SCons setup should generate something that looks like -L/some/directory/path for the compile-time search path.
The RPATH SCons setup should generate a linker option to embed a search path; e.g. -Wl,-rpath -Wl,\$ORIGIN/../lib would embed a search path that searches relative to the executable so that executables placed in bin search in the parallel lib directory of the installation.
Here is a better way to organize your SConsctruct/SConscript files. Usually with Hierarchical builds you should share the env with the rest of the sub-directories. Notice that I cloned the main env in the barexec directory as well, so that the foolib is only used to link that binary.
=== rootdir/SConstruct
import os
env=DefaultEnvironment()
subdirs = [
'foolib',
'barexec'
]
# The exports attribute allows you to pass variables to the subdir SConscripts
for dir in subdirs:
SConscript( os.path.join(dir, 'SConscript'), exports = ['env'])
=== rootdir/foolib/SConscript
# inports the env created in the root SConstruct
#
# Any changes made to 'env' here will be reflected in
# the root/SConstruct and in the barexec/SConscript
#
Import('env')
# Adding this 'inc' dir to the include path for all users of this 'env'
env.Append(CPPPATH=Glob('inc'))
penv = env.Clone()
# Adding this include only for targets built with penv
penv.Append(CPPPATH=Glob('internal/inc'))
penv.SharedLibrary( 'foo', source=['foo.c', 'morefoo.c'])
=== rootdir/barexec/SConscript
Import('env')
clonedEnv = env.Clone()
# The foo lib will only be used for targets compiled with the clonedEnv env
# Notice that specifying '#' in a path means relative to the root SConstruct
# for each [item] in LIBS, you will get -llib on the compilation line
# for each [item] in LIBPATH, you will get -Lpath on the compilation line
clonedEnv.Append(LIBS=['foo'], LIBPATH=['#foolib'])
clonedEnv.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] )
Additional to Brady decision i use static/global variables to store targets name and path. It's allow me more control over build.
# site_scons/project.py
class Project:
APP1_NAME = "app1_name"
APP2_NAME = "app2_name"
MYLIB1_NAME = "mylib1_name"
# etc
APP_PATH = "#build/$BuildMode/bin" # BuildMode - commonly in my projects debug or release, `#` - root of project dir
LIB_PATH = "#build/$BuildMode/lib"
#staticmethod
def appPath(name) :
return os.path.join(APP_PATH, name)
#staticmethod
def libPath(name) :
return os.path.join(LIB_PATH, name)
Define targets:
from project import Project
...
env.SharedLibrary(Project.libPath(Project.MYLIB1_NAME), source=['foo.c', 'morefoo.c'])
Application:
from project import Project
...
env.Append(LIBPATH = [Project.LIB_PATH])
env.Append(LIBS = [Project.MYLIB1_NAME])
env.Program(Project.appPath(Project.MYAPP1_NAME), source=[...])
In my projects it works fine, scons automatically find depends of library without any additional commands. And if i want to change name of library i just change my Project class.
One issue that Brady's answer doesn't address is how to get correct library paths when building out-of-source using variant dirs. Here's a very similar approach that builds two different variants:
SConstruct
# Common environment for all build modes.
common = Environment(CCFLAGS=["-Wall"], CPPPATH=["#foolib/inc"])
# Build-mode specific environments.
debug = common.Clone()
debug.Append(CCFLAGS=["-O0"])
release = common.Clone()
release.Append(CCFLAGS=["-O"], CPPDEFINES=["NDEBUG"])
# Run all builds.
SConscript("SConscript", exports={"env": debug}, variant_dir="debug")
SConscript("SConscript", exports={"env": release}, variant_dir="release")
The # in the value for CPPPATH makes the include path relative to the project root instead of the variant dir.
SConscript
Import("env")
subdirs=["barexec", "foolib"]
senv = env.Clone(FOOLIBDIR=Dir("foolib"))
SConscript(dirs=subdirs, exports={"env": senv})
This root-level SConscript is required to build the subdirectories in each variant_dir.
By using the function Dir() when setting FOOLIBDIR, the library's variant build directory is resolved relative to this file rather than where it's used.
foolib/SConscript
Import("env")
penv = env.Clone()
penv.Append(CPPPATH=["internal/inc"])
penv.SharedLibrary("foo", source=["foo.c", "morefoo.c"])
It's important to clone the environment before making any changes to avoid affecting other directories.
barexec/SConscript
Import("env")
clonedEnv = env.Clone()
clonedEnv.Append(LIBPATH=["$FOOLIBDIR"], LIBS=["foo"])
clonedEnv.Program("bar", source=["main.c", "bar.c", "rod.c"])
The library's variant build dir is added to LIBPATH so both SCons and the linker can find the correct library.
Adding "foo" to LIBS informs SCons that barexec depends on foolib which must be built first, and adds the library to the linker command line.
$FOOLIBDIR should only be added to LIBPATH when "foo" is also added to LIBS – if not, barexec might be built before foolib, resulting in linker errors because the specified library path does not (yet) exist.
I'm trying to use SCons for building a piece of software that depends on a library that is available in sources that are installed in system. For example in /usr/share/somewhere/src. *.cpp in that directory should be built into static library and linked with my own code. Library sources have no SConscript among them.
Since library is in system directory I have no rights and don't want to put build artefacts somewhere under /usr. /tmp or .build in current working directory is OK. I suspect this can be done easily but I've got entangled by all these SConscripts and VariantDirs.
env = Environment()
my_things = env.SConscript('src/SConsctipt', variant_dir='.build/my_things')
sys_lib = env.SConscript(????)
result = env.Program('result', [my_things, sys_lib])
What is intended way to solve the problem with SCons?
You could use a Repository to do this. For example, in your SConstruct you could write:
sys_lib = env.SConscript("external.scons", variant_dir=".build/external")
Then in the external.scons file (which is in your source tree), you add the path to the external source tree and how to build the library therein.
env = Environment()
env.Repository("/usr/share/somewhere/src")
lib = env.Library("library_name", Glob("*.cpp"))
Return("lib")