I have a hierarchical build system based on SCons. I have a root SConstruct that calls into a SConscript that builds a shared library and then into a different SConscript that builds an executable that depends on the shared library.
So here's my question: my understanding of shared libraries on linux is that when you want to do the final ld link for the executable that will be using the shared lib, the shared lib has to be included on the executable's ld command line as a source to reference it (unless it's in a standard location in which case the -l option works).
So here's something like what my SCons files look like:
=== rootdir/SConstruct
env=DefaultEnvironment()
shared_lib = SConscript('foolib/SConscript')
env.Append( LIBS=[shared_lib] )
executable = SConscript('barexec/SConscript')
=== rootdir/foolib/SConscript
env=DefaultEnvironment()
env.Append(CPPPATH=Glob('inc'))
penv = env.Clone()
penv.Append(CPPPATH=Glob('internal/inc'))
lib = penv.SharedLibrary( 'foo', source=['foo.c', 'morefoo.c']
Return("lib")
=== rootdir/barexec/SConscript
env=DefaultEnvironment()
exe = env.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] )
Return("exe")
So the hitch here is this line:
env.Append( LIBS=[shared_lib] )
This would be a great way to add generated libraries to the command line for any other libs that need them, EXCEPT that because SCons is doing a two-pass run through the SConscripts (first to generate it's dependency tree, then to do the work), rootdir/foolib/libfoo.so winds up on the command line for ALL products, EVEN libfoo.so itself:
gcc -g -Wall -Werror -o libfoo.so foo.o morefoo.o libfoo.so
So how is this best done with SCons? For now I've resorted to this hack:
=== rootdir/SConstruct
env=DefaultEnvironment()
shared_lib = SConscript('foolib/SConscript')
env['shared_lib'] = shared_lib
executable = SConscript('barexec/SConscript')
...
=== rootdir/barexec/SConscript
env=DefaultEnvironment()
exe = env.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] + env['shared_lib'] )
Return("exe")
Is there a more SCons-y way of doing this?
You should allow the shared libraries to be found by the build.
Look for the LIBPATH and RPATH variables in the SCons documentation; these are the "Scons-y" way to set up search paths so that any generated -l options find libraries properly.
Having mentioned the above, here's what you should see gcc do based on the setup of SCons (and if it doesn't, you may have to do it manually).
The -l option always finds shared libraries provided that you also give the compiler the location of the library. There are two times this is needed: at compile time (-L option) and at runtime (-rpath generated linker option).
The LIBPATH SCons setup should generate something that looks like -L/some/directory/path for the compile-time search path.
The RPATH SCons setup should generate a linker option to embed a search path; e.g. -Wl,-rpath -Wl,\$ORIGIN/../lib would embed a search path that searches relative to the executable so that executables placed in bin search in the parallel lib directory of the installation.
Here is a better way to organize your SConsctruct/SConscript files. Usually with Hierarchical builds you should share the env with the rest of the sub-directories. Notice that I cloned the main env in the barexec directory as well, so that the foolib is only used to link that binary.
=== rootdir/SConstruct
import os
env=DefaultEnvironment()
subdirs = [
'foolib',
'barexec'
]
# The exports attribute allows you to pass variables to the subdir SConscripts
for dir in subdirs:
SConscript( os.path.join(dir, 'SConscript'), exports = ['env'])
=== rootdir/foolib/SConscript
# inports the env created in the root SConstruct
#
# Any changes made to 'env' here will be reflected in
# the root/SConstruct and in the barexec/SConscript
#
Import('env')
# Adding this 'inc' dir to the include path for all users of this 'env'
env.Append(CPPPATH=Glob('inc'))
penv = env.Clone()
# Adding this include only for targets built with penv
penv.Append(CPPPATH=Glob('internal/inc'))
penv.SharedLibrary( 'foo', source=['foo.c', 'morefoo.c'])
=== rootdir/barexec/SConscript
Import('env')
clonedEnv = env.Clone()
# The foo lib will only be used for targets compiled with the clonedEnv env
# Notice that specifying '#' in a path means relative to the root SConstruct
# for each [item] in LIBS, you will get -llib on the compilation line
# for each [item] in LIBPATH, you will get -Lpath on the compilation line
clonedEnv.Append(LIBS=['foo'], LIBPATH=['#foolib'])
clonedEnv.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] )
Additional to Brady decision i use static/global variables to store targets name and path. It's allow me more control over build.
# site_scons/project.py
class Project:
APP1_NAME = "app1_name"
APP2_NAME = "app2_name"
MYLIB1_NAME = "mylib1_name"
# etc
APP_PATH = "#build/$BuildMode/bin" # BuildMode - commonly in my projects debug or release, `#` - root of project dir
LIB_PATH = "#build/$BuildMode/lib"
#staticmethod
def appPath(name) :
return os.path.join(APP_PATH, name)
#staticmethod
def libPath(name) :
return os.path.join(LIB_PATH, name)
Define targets:
from project import Project
...
env.SharedLibrary(Project.libPath(Project.MYLIB1_NAME), source=['foo.c', 'morefoo.c'])
Application:
from project import Project
...
env.Append(LIBPATH = [Project.LIB_PATH])
env.Append(LIBS = [Project.MYLIB1_NAME])
env.Program(Project.appPath(Project.MYAPP1_NAME), source=[...])
In my projects it works fine, scons automatically find depends of library without any additional commands. And if i want to change name of library i just change my Project class.
One issue that Brady's answer doesn't address is how to get correct library paths when building out-of-source using variant dirs. Here's a very similar approach that builds two different variants:
SConstruct
# Common environment for all build modes.
common = Environment(CCFLAGS=["-Wall"], CPPPATH=["#foolib/inc"])
# Build-mode specific environments.
debug = common.Clone()
debug.Append(CCFLAGS=["-O0"])
release = common.Clone()
release.Append(CCFLAGS=["-O"], CPPDEFINES=["NDEBUG"])
# Run all builds.
SConscript("SConscript", exports={"env": debug}, variant_dir="debug")
SConscript("SConscript", exports={"env": release}, variant_dir="release")
The # in the value for CPPPATH makes the include path relative to the project root instead of the variant dir.
SConscript
Import("env")
subdirs=["barexec", "foolib"]
senv = env.Clone(FOOLIBDIR=Dir("foolib"))
SConscript(dirs=subdirs, exports={"env": senv})
This root-level SConscript is required to build the subdirectories in each variant_dir.
By using the function Dir() when setting FOOLIBDIR, the library's variant build directory is resolved relative to this file rather than where it's used.
foolib/SConscript
Import("env")
penv = env.Clone()
penv.Append(CPPPATH=["internal/inc"])
penv.SharedLibrary("foo", source=["foo.c", "morefoo.c"])
It's important to clone the environment before making any changes to avoid affecting other directories.
barexec/SConscript
Import("env")
clonedEnv = env.Clone()
clonedEnv.Append(LIBPATH=["$FOOLIBDIR"], LIBS=["foo"])
clonedEnv.Program("bar", source=["main.c", "bar.c", "rod.c"])
The library's variant build dir is added to LIBPATH so both SCons and the linker can find the correct library.
Adding "foo" to LIBS informs SCons that barexec depends on foolib which must be built first, and adds the library to the linker command line.
$FOOLIBDIR should only be added to LIBPATH when "foo" is also added to LIBS – if not, barexec might be built before foolib, resulting in linker errors because the specified library path does not (yet) exist.
Related
Assume that a prec-compiled dependency is supplied by a vendor:
$ ls /opt/vendor/protobuf/lib
libprotobuf.so.3 -> libprotobuf.so.3.0.0
libprotobuf.so.3.0.0
To use this with Bazel, the following target can be created:
cc_import(
name = "protobuf",
shared_library = "lib/libprotobuf.3.0.0",
)
This way, a bazel-built application can link to the library, however, it fails to start:
error while loading shared libraries: libprotobuf.so.3: cannot open shared object file: no such file or directory
The root cause is that the actual so file has a custom SONAME field:
objdump -x libprotobuf.so.3.0.0|grep SONAME
SONAME libprotobuf.so.3
The loader will look for libprotobuf.so.3 (instead of 3.0.0), but will not find it in the sandbox, as we never told bazel about the symlink. The symlink is relative, specifying it in the cc_import target will yield to a similar error.
Is it possible to create a runnable binary with bazel that links to such a shared library that is supposed to be found via a symlink?
Setting the RPATH can be a workaround. The cc_import need to be wrapped by a cc_library:
cc_library(
name = "protobuf",
deps = [":protobuf_impl"],
linkopts = ["-Wl,-rpath=/opt/vendor/protobuf/lib"],
)
cc_import(
name = "protobuf_impl",
shared_library = "lib/libprotobuf.3.0.0",
)
This will make the binary run, but assumes that "/opt/vendor/protobuf/lib" is present on every system (incl. remote execution), and the loader during runtime still escapes the sandbox. A clearer solution would be nice.
I build my shared library:
env.SharedLibrary(target,Split(sources))
Documentation says
"On Windows systems, the SharedLibrary builder method will always build an import (.lib) library in addition to the shared (.dll) library, adding a .lib library with the same basename". That is right but I need another directory for it, so my question:
Is it possible to set another target directory for import library?
I want .dll and .lib in different directories:
bin/target.dll
lib/target.lib
It is possible to do it in VS projects but I also need a decision for Scons.
Thanks.
UPD:
We have the following structure
/project
/bin
/lib
/include
/source
SConstruct
/library
lib.cpp
SConscript
/app
SConscript
main.cpp
app depends on library.
The following scripts are very simplified.
SConstruct
g_env = Environment()
...
g_target = 'Library_' + g_arch
if g_debug: g_target += 'd'
SConscript('library/SConscript')
SConscript('app/SConscript')
library/SConscript
sources = [ .. ]
env_lib = g_env.Clone()
...
env_lib.SharedLibrary('#../lib/' + g_target,sources)
app/SConscript
sources = [ .. ]
app_env = g_env.Clone()
app_env.Append(LIBPATH = Split('#../lib'))
app_env.Append(LIBS = Split(g_target))
app_env.Program('app',sources)
If I go to app dir and run
scons -u
I get all I need:
lib/Library.dll
lib/Library.lib
source/app/app.exe
But if I want just to rebuild Library running
scons -u
from library directory - just builds me .obj files, there is no final shared library.
I have no idea why it works so, I'm not quite familiar with it. But now we need to get final libraries in different directories (.lib in lib, .dll in bin) as I mentioned above.
The standard way of doing this, would be to use the Install() method (see chap 11 "Installing files in other directories" of our UserGuide):
Install('lib','bin/target.lib')
You should set no_import_lib in your call to SharedLibrary()
env_lib.SharedLibrary('#../lib/' + g_target,sources,no_import_lib=True)
Also, are you outputing a .exp file?
Just list the name of the lib in the list of target files.
env.SharedLibrary([target, 'lib/anyname.lib'], Split(sources))
SCons will recognize the target .lib file based on its suffix (LIBSUFFIX), and it will adapt the /IMPLIB argument of the linker automatically.
How to compile and link third party libraries with automake?
My file structure is:
program/
|
+--src/
| |
| +--Makefile.am
| +--main.cpp
|
+--lib/
| |
| +--Makefile.am
| +--library.cpp
|
+--Makefile.am
+--configure.ac
+--README
Contents of automake files are pretty generic:
# src/Makefile.am
bin_PROGRAMS = program
program_SOURCES = main.cpp
# Makefile.am
SUBDIRS = src lib
dist_doc_DATA = README
# configure.ac
AC_INIT([program], [1.0])
AM_INIT_AUTOMAKE([-Wall])
AC_PROG_CXX
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_FILES([Makefile src/Makefile lib/Makefile])
AC_OUTPUT
What should be the contents of lib/Makefile.am?
(Not sure why you said "third-party" when you appear to have control of the library code yourself... For more info related to creating and working with libraries using Automake, I refer you to the GNU Automake manual's section on libraries)
lib/Makefile.am
lib_LIBRARIES = libYOURLIB.a
libYOURLIB_a_SOURCES = library.cpp
You can use noinst_lib_LIBRARIES if you don't want to install the library itself. Note that I'm assuming you want to build a static library only. See the Building A Shared Library section of the GNU Automake manual for integrating with Libtool to produce a shared library. You can do it manually of course, but it's a lot easier with Libtool as it takes care of various platform differences.
To link your library to program, you'd add the following lines insrc/Makefile.am:
program_DEPENDENCIES = $(top_builddir)/lib/libYOURLIB.a
program_LDADD = $(top_builddir)/lib/libYOURLIB.a
The _DEPENDENCIES line simply tells Automake that program relies on lib/libYOURLIB.a being built first, and the _LDADD line simply adds the library to the linker command.
The above assumes that you have a rule to build the library already. Since you're using SUBDIRS, you received a "no rule to make target XXXXXX" build failure, which indicates that you don't (at least from the perspective of the Makefile in the src subdirectory). To remedy this, you can try the following in src/Makefile.am (taken from "Re: library dependency" on the GNU Automake mailing list archives):
FORCE:
$(top_builddir)/lib/libYOURLIB.a: FORCE
<TAB>(cd $(top_builddir)/lib && $(MAKE) $(AM_MAKEFLAGS) libYOURLIB.a)
You can also simply make lib a subdirectory of src as your comment indicated of course and make it simpler.
Alternatively, you can stop using a recursive build setup and use what is perhaps a simpler non-recursive build setup. See GNU Automake Manual §7.3: An Alternative Approach to Subdirectories and Non-recursive Automake for some information on that, but the general idea would be to alter things to allow for :
configure.ac
AM_INIT_AUTOMAKE([-Wall subdir-objects])
...
AC_CONFIG_FILES([Makefile])
Makefile.am
# Instead of using the SUBDIRS variable.
include src/Makefile.am.inc
include lib/Makefile.am.inc
dist_doc_DATA = README
lib/Makefile.am renamed to lib/Makefile.am.inc
# Full path relative to the top directory.
lib_LIBRARIES = lib/libYOURLIB.a
lib_libYOURLIB_a_SOURCES = lib/library.cpp
src/Makefile.am renamed to src/Makefile.am.inc
# Full path relative to the top directory.
bin_PROGRAMS = bin/program
bin_program_SOURCES = src/main.cpp
bin_program_DEPENDENCIES = lib/libYOURLIB.a
bin_program_LDADD = lib/libYOURLIB.a
Renaming the files is optional (you could always just include src/Makefile.am), but it helps to denote that it isn't meant to be a standalone Automake source file.
Also, supposing that lib/library.cpp and src/main.cpp both #include "library.hpp", and it's in another directory, you might also want to use AM_CPPFLAGS = -I $(top_srcdir)/include for all files or obj_program_CPPFLAGS = -I include for all source files that are used in building bin/program, assuming library.hpp is in program/include. I'm not sure if $(top_srcdir) is right when another project includes your entire program source directory in its own SUBDIRS variable, but $(srcdir) will always refer to the top-level program directory in the case of a non-recursive automake, making it perhaps more useful in larger projects that include this package as a component.
I'm trying to create a build system for my project which will be based on scons.
The project consist of several directories, each holding part of the whole project.
Each directory hold the sources of either a shared library, a program or a process (daemon).
One directory (bin) holds all shared libraries and all executables.
Most of the directories already contain a sconscript file (named Sconstruct) which builds the directory's module (lib / executable) and put it in the bin directory.
Now I want to create one more sconscript - to rule them all..
In the parent directory I want a sconscript that builds all libraries and executables of the project, so that after I change a few sources here and there I can run scons from the parent directory and all affected modules will be re-built.
I tried a few ways, and they all failed.
I'm quite a noob in the scons business, and I susspect this is the root cause of my failures, but I'm sure this problem was solved many times by other, more experienced, developers - as the situation I described is quite common.
So, any suggestions are welcomed!
EDIT:
My current Sconstruct in the parent directory looks like this:
import os
env = os.environ
Export('env', 'os')
SConscript([
'Server1_Dir/Sconstruct',
'Server2_Dir/Sconstruct',
'Server3_Dir/Sconstruct'
])
The Sconstructs in the sub-directories (I know they should be called Sconscript) start with:
import os
Import('env')
home=env.get('HOME')
So it seems to me that I'm using the same environment for all scripts, though I'm getting a lot of:
scons: warning: Two different environments were specified for target....
warnings, which are unclear for me.
It's also worth noting that not all server-combinations in the main script yield these warnings - some may leave together peacefully, but not all.
It does seem to me that this is the right way to go, but I can't find a way to get rid of those warnings (and their root cause).
Thanks.
What you want to do is called a hierarchical build as described here. In SCons, there can only be one SConstruct per project.
The way to create a hierarchical build in SCons is to make the root SCons script be the SConstruct, and the subdirectory SCons scripts be called SConscript. The root SConstruct script "loads" the subdirectory SConscripts using the SConscript() function described in the above link.
Here's an example assuming you have a root SConstruct and a lib and bin subdir:
SConstruct
env = Environment()
SConscript('lib/SConscript', exports='env')
SConscript('bin/SConscript', exports='env')
lib/SConscript
Import('env')
env.Library('yourlib', ['yourLibSource.cc'])
bin/SConscript
Import('env')
env.Program('yourbin', ['yourBinSource.cc'])
My scripts, before I tried to unify them, looked like the following (and worked properly):
Librarry building-script Stracture:
Environment and Definitions:
~~~~~~~~~~~~~~~~~~~~~~
import os
env = os.environ
home = env.get('HOME')
projects = []
third_parties_projects = []
my_rpath = []
Preparing lists of paths and files:
~~~~~~~~~~~~~~~~~~~~~~~~~
all_user_implemented_libs = os.path.join(home,"Project/lib/")
my_rpath.append("/path/to/lib/dir/")
projects.append(os.path.join(home,"Project/library_1/"))
projects.append(os.path.join(home,"Project/library_2/"))
...
third_parties_projects.append("/opt/some_product/include/")
...
so_files = Split("""
source_1.cpp
source_2.cpp
...
""")
Controlling the build:
~~~~~~~~~~~~~~~~
myprog = Environment(CXXFLAGS = flags, ENV = env, CPPPATH = projects + third_parties_projects + ['.'], RPATH = my_rpath)
myprog.Decider('MD5')
mylib = myprog.SharedLibrary(all_user_implemented_libs + 'lib_name', so_files)
myprog.Default(mylib)
Process building-script Stracture:
Environment and Definitions:
~~~~~~~~~~~~~~~~~~~~~~
import os
env = os.environ
home = env.get('HOME')
projects = []
third_parties_projects = []
user_libpaths = []
my_rpath = []
Preparing lists of paths and files:
~~~~~~~~~~~~~~~~~~~~~~~~~
projects.append(os.path.join(home,"Project/library_1/"))
projects.append(os.path.join(home,"Project/library_2/"))
....
my_rpath.append("/path/to/lib/dir/")
third_parties_projects.append("/opt/some_product/include/")
...
all_user_implemented_bins = os.path.join(home,"Project/bin/")
all_user_implemented_libs = os.path.join(home,"Project/lib/")
flags = ''
libs = Split("""
lib_1
lib_2
...
""")
third_party_libpaths.append("/opt/some_product/")
...
source_ext_files = Split("""
ext_source_1.cpp
...
""")
source_files_for_executable = Split("""
source_1.cpp
sosurce_2.cpp
...
""")
user_libpaths.append(all_user_implemented_libs)
Building the underlaying libs:
~~~~~~~~~~~~~~~~~~~~~~
for project in projects:
SConscript(project+'/Sconstruct')
Controlling the build:
~~~~~~~~~~~~~~~~
myprog = Environment(CXXFLAGS = flags, ENV = env, CPPPATH = projects + third_parties_projects + ['.'], RPATH = my_rpath)
myprog.Decider('MD5')
my_prog = myprog.Program(all_user_implemented_bins + 'process_name', source_files_for_executable + source_ext_files, LIBS = libs, LIBPATH = third_party_libpaths + user_libpaths)
myprog.Default(my_prog)
These scripts are scattered each in its corresponding lib/exec directory, and work fine when executed separately.
Now I want to add one more script (like the one in the original question) in the parent directory which will build the whole product - all executables and all underlaying libraries - as necessary.
Thanks.
I have the following SCons configuration:
current_directory
|-<.cpp files>
|-<.h files>
|-SConstruct
|-SConscript
|-bin
|-<empty>
I want to build my source files and put the executable and the object files into the bin directory.
This is what I have in my SConstruct file:
SConscript('SConscript', variant_dir='bin', duplicate=0)
While in the SConsript file I have:
debug_environment.Program(target = 'SsaTest', src_files, LIBS=libraries, LIBPATH=libraries_path)
When I build using scons command I get the SsaTest executable in the bin directory (as desired), but the object files are left in the current directory.
How can I have the .o files be built in the bin directory as well?
Many thanks.
EDIT: Complete SConscript file (forgive me for the xxxs)
import os
# This is correctly Exported()
Import('debug_flags')
# Paths to header files
headers_paths = ['#/../../xxx/include/',
'#/../../../xxx/include/',
'#/../../xxx/include/',
'#/../../xxx/include/',
'#/../../xxx/include/']
# Path to source files
src_folder = '#./'
# Source files list
src_files = ['xxx.cpp',
'xxx.cpp',
'xxx.cpp',
'xxx.cpp',
'xxx.cpp']
# Prepend the relative path to each source file name
src_files = [src_folder + filename for filename in src_files]
libraries = ['xxx', 'xxx', 'xxx', 'xxx', 'xxx', 'xxx', 'xxx', 'xxx']
libraries_path = ['#/../../xxx/lib',
'#/../../../xxx/bin',
'#/../lib',
'#/../../xxx/lib',
'#/../../xxx/lib',
'#/../../xxx/lib']
# Debug environment
debug_environment = Environment(CC = 'g++', CCFLAGS=debug_flags, ENV = os.environ, CPPPATH=headers_paths);
# Executable build command
debug_environment.Program(target = 'SsaTest', src_files, LIBS=libraries, LIBPATH=libraries_path)
Using '#' with source files not recommended, because you have your situation, scons can't correctly process it with variant dirs and how result create object files in directory where sources placed.
So, i tryed to build your example with same configuration and have no troubles:
#SConsctruct
SConscript('SConscript', variant_dir='bin', duplicate=0)
#SConscript
src_files = Glob('*.cpp')
debug_environment = Environment()
debug_environment.Program('SsaTest', src_files)
So, all object files are generated in bin directory.
Finally, you have no troubles with relation dirs with sources files then using variant dirs. But include dirs are depended from variant dirs.
Configuration for example :
build
app
--SConscript
--src
----*.h
----*.cpp
SConstruct
#SConstruct
rootEnv = Environment()
Export('rootEnv')
SConscript('app/SConscript', variant_dir='build', duplicate=0)
You SConscript will be looking like it:
Import('rootEnv')
env = rootEnv.Clone()
env.Append(CPPPATH = ['#app/src'])
env.Program('app', Glob('src/*.cpp'))
'#app/src' - where # is very important when using variant dir, because if would be app/src, build command will be looking: '-Ibuild/app/src' (adding variant dir before include path). But adding '#' command will be looking correctly : '-Iapp/src'.
One thing that sticks out in your SConscript is how you are prepending the path to each source file with #./.
# Path to source files
src_folder = '#./'
Why do you use the dot in that path, its not necessary? Try with the following #/ like you do with the rest of the paths, like this:
# Path to source files
src_folder = '#/'
Another option would be to put the source files and the respective SConscript in its own subdirectory. Its not real clear why you have a SConstruct, SConscript, and the source files all in one directory. Either create a subdir, or consider removing the SConscript if its not necessary.
In the SConscript() function call in the SConstruct, refer to the variant_dir as "#bin" and not "bin". Not sure if this will help, but its better practice.
Ive seen this behaviour before using the Repository() SCons function to reference the source files as mentioned here.
Also, this is off-topic, but if your include and library paths (headers_paths and libraries_path variables) are outside of the project directory structure, you may consider using absolute paths instead. Personally I find it rather ugly to use relative paths with several ../ paths.