Scons Hierarchical Build - build

I've gone through the example scons builds here and have found them wanting in providing a solution that fits my project.
The structure is as follows:
root/
Module_A/
include/
foo.h
bar.h
src/
foo.cpp
bar.cpp
Module_.../
Every module follows the same structure, an include folder for all the .h's and a src file for the cpps. Each module builds into a shared object. There is no executable.
Modules have cross dependencies. For instance Module_A is the logging mechanism and it is used in modules B, C, D, etc. Likewise, Module_B is the Configuration loader, which is used in several other modules. And Module_C would be the IPC module, used in almost each module listed. Lastly, Module_D is the command center and links against EVERY other module (literally).
I am interested in replacing the current setup we have of using recursive make to build the project. I am trying to build the sconstruct and SConscripts necessary to do so, but I am very new to even make, let alone scons.
I am interested in turning each Module's .cpp and .h into a .so and to have its dependencies resolved automagically as is done with make now.
In the SConscript, I currently use glob to get the *.cpps and then include the module's './include' in the CPPPATH.
I have used
env.SharedLibrary(CPPATH='./include', source= (list of the cpps))
But since this depends on other Modules, it will not work, stating the other module's functions that are used are "not declared".
How do I go about getting this kind of complex structure to build using a hierarchical scons setup?

This should be quite easy to do with SCons. You'll probably want a SConscript script at the root of every module. These will all be invoked by a SConstruct script located at the root of the entire project.
If I understand the question correctly, the problem of the dependencies between the modules can be solved by correctly specifying the include paths of all of the modules. This can be done once in an Environment created in the SConstruct, which should then be passed to the module SConscript scripts.
Here's a brief example:
Sconstruct
env = Environment()
# Notice that the '#' in paths makes the path relative to the root SConstruct
includePaths = [
'#/Module_A/include',
'#/Module_B/include',
'#/Module_N/include',
]
env.Append(CPPPATH=includePaths)
SConscript('Module_A/SConscript', exports='env', duplicate=0)
SConscript('Module_B/SConscript', exports='env', duplicate=0)
SConscript('Module_N/SConscript', exports='env', duplicate=0)
Module_A/SConscript
Import('env')
# Notice the CPPPATH's have already been set on the env created in the SConstruct
env.SharedLibrary(target = 'moduleA', source = ModuleA_SourceFiles)
Module_B/SConscript
Import('env')
# Notice the CPPPATH's have already been set on the env created in the SConstruct
env.SharedLibrary(target = 'moduleB', source = ModuleB_SourceFiles)
Module_N/SConscript
Import('env')
# Notice the CPPPATH's have already been set on the env created in the SConstruct
env.SharedLibrary(target = 'moduleN', source = ModuleN_SourceFiles)

Related

CMake with 3rd party libraries that need to be built along with the project

I am confused on the right way to get an external library integrated into my own Cmake project (This external project needs to be built along with my project, it's not installed separately, so we can't use find_library, or so I think)
Let's assume we have a project structure like this (simplified for this post):
my_proj/
--CMakeLists.txt
--src/
+---CMakeLists.txt
+---my_server.cpp
That is, we have a master CMakeLists.txt that basically sits at root and invokes CMakeLists for sub directories. Obviously, in this example, because its simplified, I'm not showing all the other files/directories.
I now want to include another C++ GitHub project in my build, which happens to be this C++ bycrypt implementation: https://github.com/trusch/libbcrypt
My goal:
While building my_server.cpp via its make process, I'd like to include the header files for bcrypt and link with its library.
What I've done so far:
- I added a git module for this external library at my project root:
[submodule "third_party/bcrypt"]
path = third_party/bcrypt
url = https://github.com/trusch/libbcrypt
So now, when I checkout my project and do a submodule update, it pulls down bcrypt to ${PROJ_ROOT}/third_party
Next up, I added this to my ROOT CMakeLists.txt
# Process subdirectories
add_subdirectory(third_party/bcrypt)
add_subdirectory(src/)
Great. I know see when I invoke cmake from root, it builds bcrypt inside third_party. And then it builds my src/ directory. The reason I do this is I assume this is the best way to make sure the bcrypt library is ready before my src directory is built.
Questions:
a) Now how do I correctly get the include header path and the library location of this built library into the CMakeLists.txt file inside src/ ? Should I be hardcoding #include "../third_party/bcrypt/include/bcrypt/bcrypt.h" into my_server.cpp and -L ../third_party/libcrypt.so into src/CMakeLists.txt or is there a better way? This is what I've done today and it works, but it looks odd
I have, in src/CMakeLists.txt
set(BCRYPT_LIB,"../third_party/bcrypt/libbcrypt.so")
target_link_libraries(my app ${MY_OTHERLIBS} ${BCRYPT_LIB})
b) Is my approach of relying on sequence of add_directory correct?
Thank you.
The best approach depends on what the bcrypt CMake files are providing you, but it sounds like you want to use find_package, rather than hard-coding the paths. Check out this answer, but there are a few different configurations for find_package: MODULE and CONFIG mode.
If bcrypt builds, and one of the following files gets created for you:
FindBcrypt.cmake
bcrypt-config.cmake
BcryptConfig.cmake
that might give you an idea for which find_package configuration to use. I suggest you check out the documentation for find_package, and look closely at how the search procedure is set up to determine how CMake is searching for bcrypt.

File path issue with Meson and Eigen

I cannot make local include paths work in the Meson build system.
This C++ inclusion works correctly:
#include </cygdrive/c/Users/user/project/Third-Party/eigen/Eigen/Dense>
This one does not:
#include "Third-Party/eigen/Eigen/Dense"
fatal error: Eigen/Dense: No such file or directory
In the Meson build file, I tried to add Eigen's path, without success:
# '.' will refer to current build directory
include_dirs = include_directories('include', '.', '../project/Third-Party/eigen')
This is the project tree structure:
project
meson.build
src
meson.build
example.h
example.cpp
Third-Party
eigen (headers only lib)
Eigen
Note: with CMake I do not have this issue.
For dependency management, meson allows you to manually declare include_directories() in your build files. However, there is another way do handle dependencies: using dependency() command.
dependency() is a much better way to handle dependencies, because meson will build it if necessary (if dependency is a shared or a static library) and safely allows you to use includes. That means that you don't have to know where includes for dependency are located physically or care about their paths ever after. The only downside is that this kind of dependency needs it's own meson.build file.
Using dependency() command:
To actually use it, you have to write a wrap file for dependency. Or, if you are lucky enough, there is already a wrap file for you in the Wrap DB -- a community-driven database for meson wrap files. Wrap file is a config of some kind that declare where you can get a dependency and in what form. Wrap file can wrap around zip archives and git repositories.
For your given dependency, there is wrap file in Wrap DB: eigen. All you have to do is download it and place it in the subprojects directory near your meson.build. For example:
$ cd project
$ mkdir subprojects
$ wget "https://wrapdb.mesonbuild.com/v1/projects/eigen/3.3.4/1/get_wrap" \
-O subprojects/eigen.wrap
Now, not every project builds with meson. For the ones that don't, wrap file also specify a patch. Patch is used to just copy appropriate meson.build file into dependency directory (as well as any other files that would be needed for building that particular dependency with meson). Eigen wrap file contains a patch.
To find out how any particular dependency declare itself as a dependency (using declare_dependency() command), you need to investigate meson.build file in dependency source directory (although it's often just name of the dependency plus _dep, e.g. "eigen_dep"). For me, eigen directory was subprojects/eigen-eigen-5a0156e40feb. So, you search for the declare_dependency() command:
$ grep declare_dependency subprojects/eigen-eigen-5a0156e40feb/meson.build
eigen_dep = declare_dependency(
As you can see, eigen declare dependency as eigen_dep. If you want to know what exactly is declared, just scroll down the dependency meson.build file.
Now, to use that eigen_dep in your project, create a dependency object with a dependency() command. Here is a sample project that I used to compile "A simple first program" from Eigen: Getting Started:
project('example', 'cpp')
eigen_dependency = dependency('eigen', fallback: ['eigen', 'eigen_dep'])
executable('example', 'example.cpp', dependencies: eigen_dependency)
Notice arguments for the dependency() command. The first one is system-wide dependency that meson is searching for. If there is no eigen for development installed in your system, then meson uses fallback: first item in fallback is basename of the wrap file, second item is a name of declared dependency.
Then use eigen_dependency variable in whatever you build, passing it to the dependencies argument.
Using include_directories() command:
If you want to just include some files from external directory (such as your "Third-Party" directory) using include_directories() command, that directory has to be relative to the meson.build file where you use it.
To use manually declared includes, you need to call include_directories() command to get the include_directories object. Pass that object to include_directories argument in whatever you build.
Given your example, I assume that root meson.build file is a project build file. Then in that root meson.build, for example, you can write:
# File: project/meson.build
project('example', 'cpp')
eigen_includes = include_directories('Third-Parties/eigen')
executable('example', 'example.cpp', include_directories: eigen_includes)
But if you want to get eigen includes from src/meson.build, then you need to change include_directories to:
# File: project/src/meson.build
eigen_includes = include_directories('../Third-Parties/eigen')
...

How do I build from different locations in my tree with multiple SConstruct files that use Repositories?

I've got the following directory structure:
root/
SConstruct
item0/
SConstruct
unit0/
unit0.h
private/
unit0.c
test/
test_unit0.c
SConstruct
SConscript
unit1/
(repeat unit0 structure)
item1/
(repeat item0 structure)
I want to be able to run scons from root to build all things below it, and from root/item0 to build all things below it, etc.
The top-level SConstruct looks like:
SConscript('item0/SConstruct', duplicate=0)
SConscript('item1/SConstruct', duplicate=0)
The item-level SConstruct:
SConscript('unit0/SConstruct', duplicate=0)
SConscript('unit1/SConstruct', duplicate=0)
The unit-level SConstruct:
import os
proj_root = os.path.abspath('../../../../..')
proj_shared_path = os.path.join(proj_root,
os.path.normpath('trunk/sys/shared'))
env = Environment()
SConscript(
'SConscript',
variant_dir='output',
exports=['env', 'proj_shared_path'],
duplicate=0)
The unit-level SConscript (important stuff only):
Import('*')
# Setup Repositories to access out-of-current-directory-tree files.
lib1_path = os.path.join(proj_shared_path, 'lib1')
env.Repository('#/../private', lib1_path)
# Set general compilation and link flags.
env.Append(
CFLAGS = '-Wall -Wextra -std=c99 -pedantic',
CPPPATH = ['#/../..', proj_shared_path])
# These are the files we don't need to run gcovr on.
non_cov_srcs = [
'testit.c',
'lib1.c'
]
# Create a 'coverage' environment to build files that we will run gcovr on.
cov = env.Clone()
cov.Append(CFLAGS = '--coverage -O0')
cov_srcs = cov.Object('unit0.c')
SideEffect('unit0.gcno', cov_srcs)
# Our unit test program combines all the sources we've listed so far.
test_exe = env.Program(
'test_unit0',
[cov_srcs, non_cov_srcs],
LIBS='gcov', LINKFLAGS='--coverage')
Now, this all works fine when I run scons from root/item0/unit0/test. However, when I run from root or root/item0, I get various types of errors, like below (run from root/item0):
scons: done reading SConscript files.
scons: Building targets ...
scons: *** [unit0\test\output\unit0.obj] Source 'unit0\test\unit0.c' not found, needed by target 'unit0\test\output\unit0.obj'.
scons: building terminated because of errors.
Clearly unit0\test\unit0.c won't be found because it doesn't exist, but I'm not sure why scons thinks it has to be there since the Repository is defined as '#/../private' and there is no private shown in the error message.
edit I've figured out that it's probably looking for unit0\test\unit0.c because unit0\test is the "root" directory for SConscript. I'm also guessing that the #/../private Repository location is only correct when scons is run from unit0\test. But, when I try to specify the Repository in any other way (including an absolute path built with os.path.abspath (and verified to be correct)), scons still doesn't seem to want to look there for unit0.c. Why is that?
With further help from #bdbaddog in the #scons IRC channel, I was able to determine the following:
What I was actually trying to achieve was ensuring the build artifacts for each source ended up in my variant_dir
With SConstruct and SConscript in the same directory, Repository ended up achieving this when building from the unit0/test directory. But, I didn't know why.
To properly achieve (1), Repository is not the right tool. Instead, SConscript should look like this:
Import('*')
# Set general compilation and link flags.
env.Append(
CFLAGS = '-Wall -Wextra -std=c99 -pedantic',
CPPPATH = ['#/item0', '#/lib1'])
# These are the files we don't need to run gcovr on.
non_cov_objs = [
Object(target='testit', source='testit.c'),
Object(target='lib1', source='#/lib1/lib1.c'
]
# Create a 'coverage' environment to build files that we will run gcovr on.
cov = env.Clone()
cov.Append(CFLAGS = '--coverage -O0')
cov_objs = cov.Object(
target='unit0',
source='#/item0/unit0/private/unit0.c')
SideEffect('unit0.gcno', cov_srcs)
# Our unit test program combines all the sources we've listed so far.
test_exe = env.Program(
'test_unit0',
[cov_objs, non_cov_objs],
LIBS='gcov', LINKFLAGS='--coverage')
By specifying target and source for each Object, I can achieve what I want in terms of the object files (and other side-effects) being generated in the variant_dir.

SCons out of source build only works for executable program, not for object files

I have the following SCons configuration:
current_directory
|-<.cpp files>
|-<.h files>
|-SConstruct
|-SConscript
|-bin
|-<empty>
I want to build my source files and put the executable and the object files into the bin directory.
This is what I have in my SConstruct file:
SConscript('SConscript', variant_dir='bin', duplicate=0)
While in the SConsript file I have:
debug_environment.Program(target = 'SsaTest', src_files, LIBS=libraries, LIBPATH=libraries_path)
When I build using scons command I get the SsaTest executable in the bin directory (as desired), but the object files are left in the current directory.
How can I have the .o files be built in the bin directory as well?
Many thanks.
EDIT: Complete SConscript file (forgive me for the xxxs)
import os
# This is correctly Exported()
Import('debug_flags')
# Paths to header files
headers_paths = ['#/../../xxx/include/',
'#/../../../xxx/include/',
'#/../../xxx/include/',
'#/../../xxx/include/',
'#/../../xxx/include/']
# Path to source files
src_folder = '#./'
# Source files list
src_files = ['xxx.cpp',
'xxx.cpp',
'xxx.cpp',
'xxx.cpp',
'xxx.cpp']
# Prepend the relative path to each source file name
src_files = [src_folder + filename for filename in src_files]
libraries = ['xxx', 'xxx', 'xxx', 'xxx', 'xxx', 'xxx', 'xxx', 'xxx']
libraries_path = ['#/../../xxx/lib',
'#/../../../xxx/bin',
'#/../lib',
'#/../../xxx/lib',
'#/../../xxx/lib',
'#/../../xxx/lib']
# Debug environment
debug_environment = Environment(CC = 'g++', CCFLAGS=debug_flags, ENV = os.environ, CPPPATH=headers_paths);
# Executable build command
debug_environment.Program(target = 'SsaTest', src_files, LIBS=libraries, LIBPATH=libraries_path)
Using '#' with source files not recommended, because you have your situation, scons can't correctly process it with variant dirs and how result create object files in directory where sources placed.
So, i tryed to build your example with same configuration and have no troubles:
#SConsctruct
SConscript('SConscript', variant_dir='bin', duplicate=0)
#SConscript
src_files = Glob('*.cpp')
debug_environment = Environment()
debug_environment.Program('SsaTest', src_files)
So, all object files are generated in bin directory.
Finally, you have no troubles with relation dirs with sources files then using variant dirs. But include dirs are depended from variant dirs.
Configuration for example :
build
app
--SConscript
--src
----*.h
----*.cpp
SConstruct
#SConstruct
rootEnv = Environment()
Export('rootEnv')
SConscript('app/SConscript', variant_dir='build', duplicate=0)
You SConscript will be looking like it:
Import('rootEnv')
env = rootEnv.Clone()
env.Append(CPPPATH = ['#app/src'])
env.Program('app', Glob('src/*.cpp'))
'#app/src' - where # is very important when using variant dir, because if would be app/src, build command will be looking: '-Ibuild/app/src' (adding variant dir before include path). But adding '#' command will be looking correctly : '-Iapp/src'.
One thing that sticks out in your SConscript is how you are prepending the path to each source file with #./.
# Path to source files
src_folder = '#./'
Why do you use the dot in that path, its not necessary? Try with the following #/ like you do with the rest of the paths, like this:
# Path to source files
src_folder = '#/'
Another option would be to put the source files and the respective SConscript in its own subdirectory. Its not real clear why you have a SConstruct, SConscript, and the source files all in one directory. Either create a subdir, or consider removing the SConscript if its not necessary.
In the SConscript() function call in the SConstruct, refer to the variant_dir as "#bin" and not "bin". Not sure if this will help, but its better practice.
Ive seen this behaviour before using the Repository() SCons function to reference the source files as mentioned here.
Also, this is off-topic, but if your include and library paths (headers_paths and libraries_path variables) are outside of the project directory structure, you may consider using absolute paths instead. Personally I find it rather ugly to use relative paths with several ../ paths.

Building external code trees with SCons

I'm trying to use SCons for building a piece of software that depends on a library that is available in sources that are installed in system. For example in /usr/share/somewhere/src. *.cpp in that directory should be built into static library and linked with my own code. Library sources have no SConscript among them.
Since library is in system directory I have no rights and don't want to put build artefacts somewhere under /usr. /tmp or .build in current working directory is OK. I suspect this can be done easily but I've got entangled by all these SConscripts and VariantDirs.
env = Environment()
my_things = env.SConscript('src/SConsctipt', variant_dir='.build/my_things')
sys_lib = env.SConscript(????)
result = env.Program('result', [my_things, sys_lib])
What is intended way to solve the problem with SCons?
You could use a Repository to do this. For example, in your SConstruct you could write:
sys_lib = env.SConscript("external.scons", variant_dir=".build/external")
Then in the external.scons file (which is in your source tree), you add the path to the external source tree and how to build the library therein.
env = Environment()
env.Repository("/usr/share/somewhere/src")
lib = env.Library("library_name", Glob("*.cpp"))
Return("lib")