I am migrating large legacy makefiles project to Bazel. Project used to copy all sources and headers into single "build dir" before build, and because of this all source and header files use single level includes, without any prefix (#include "1.hpp").
Bazel requires that modules (libraries) use relative path to header starting at WORKSPACE file, however my goal is to introduce Bazel build files, which require 0 modifications of a source code.
I use bazelrc to globally set paths to includes as if structure was flat:
.bazelrc:
build --copt=-Ia/b/c
/a/b/BUILD
cc_library(
name = "lib",
srcs = ["c/1.cpp"],
hdrs = ["c/1.hpp"],
visibility = ["//visibility:public"]
)
When I build this target, I see my -I flag in compiler invocation, but compilation fails because bazel can not find header 1.hpp:
$ bazel build -s //a/b:lib
...
a/b/c/1.cpp:13:10: fatal error: 1.hpp: No such file or directory
13 | #include "1.hpp"
|
Interestingly enough, it prints me gcc command that it invokes during build and if I run this command, compiler is able to find 1.hpp and 1.cpp compiles.
How to make bazel "see" this includes? Do I really need to additionally specify copts for every target in addition to global -I flags?
Bazel use sandboxing: for each action (compile a C++ file, link a library) the specific build directory is prepared. That directory contains only files (using symlinks and other Linux sorcery), which are explicitly defined as dependency/source/header for given target.
That trick with --copt=-Ia/b/c is a bad idea, because that option will work only for targets, which depend on //a/b:lib.
Use includes or strip_include_prefix attribute instead:
cc_library(
name = "lib",
srcs = ["c/1.cpp"],
hdrs = ["c/1.hpp"],
strip_include_prefix = "c",
visibility = ["//visibility:public"]
)
and add the lib as a dependency of every target, which need to access these headers:
cc_binary(
name = "some bin",
srcs = ["foo.cpp"],
deps = ["//a/b:lib"],
)
Related
I have a cc_library that is header-only. Whenever I try to compile such library by itself, it won't actually compile anything. I purposely put some errors to try to get such errors on compilation, but bazel doesn't actually compile anything. Here's a small example.
// test.h
This should not compile fdsafdsafdsa
int foo() { return 1; }
# BUILD
cc_library(
name = 'test',
hdrs = ['test.h']
)
// bazel build :test
INFO: Analyzed target //:test (2 packages loaded, 3 targets configured).
INFO: Found 1 target...
Target //:test up-to-date (nothing to build)
INFO: Elapsed time: 0.083s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
Is this behavior intended?
I also ran the same experiment but splitting the .h and .cc files, and in that case, I got the error when I compiled.
cc_library (other rules as well incl. pkg_tar for instance) does not have to have any sources. This is also valid:
cc_library(
name = "empty",
srcs = [],
)
And it is actually quite useful too. You may have configurable attributes such as deps (or srcs) where actual content is only applicable for certain conditions:
cc_binary(
name = "mybinary",
srcs = ["main.c"],
deps = select({
":platform1": [":some_plat1_only_lib"],
":platform2": [":empty"], # as defined in the above snippet
}),
)
Or (since above you could have just as well used [] for :platform2 deps) where you have a larger tree and you expect developers to just depend on //somelib:somelib, you could use this empty library through an alias to give them a single label without having to worry about all the platform specific details and how providing certain function is handled where:
# somelib/BUILD:
alias(
name = "somelib",
actual = select({
":platform1": [":some_plat1_only_lib"],
":platform2": [":empty"], # as defined in the above snippet
}),
visibility = ["//visibility:public"], # set appropriately
)
And mybinary or any other target could now say:
cc_binary(
name = "mybinary",
srcs = ["main.c"],
deps = ["//somelib"],
)
And of course, as the other answer here states, there are header only libraries.
Also in the example you've used in your question. (bazel or not) you would normally not (and it would not be very useful either) compile a header file on its own. You would only use its content and only then see the compiler fails as you attempt to build a source the header is #included from. That is for bazel build to fail, another target would have to depend on test and #include "test.h".
A header only library means that you don't need to build anything. Simply include the headers you need in your program and use them.
I have built a lib.so using make rule in bazel. How do I link this external lib.so to a regular cc_library rule. I tried adding it in deps, but the guide suggests that deps can have cc_library or objc_library targets.
Also, do I need to pass any specific linking options, and how can I read more about them?
In the BUILD file, create a cc_library target that imports the built lib.so for other cc_library targets to depend on:
cc_library(
name = "lib",
srcs = ["lib.so"],
linkopts = ["...", "..."],
)
See the documentation on C++ use cases for more information.
I have a genrule that takes "in" a config file, and spits "out" a large number of built files (*.so*s and *.h files):
genrule(
name = "ros_auto",
local = True,
srcs = [":package_list"],
outs = ["ros_built.tgz"],
tools = [":build_packages.bash"],
cmd = "$(location :build_packages.bash) $< $#"
)
Next up I need to take all of the files that are output from the above genrule, and create a cc_library from them like below (at the moment the only manageable way I can find to register the output files in the genrule is tarballing them & declaring the tar):
cc_library(
name = "ros",
srcs = glob(["lib/*.so*"]),
hdrs = glob([
"include/**/*.h",
"include/**/*.hpp",
]),
strip_include_prefix = "include",
visibility = ["//visibility:public"],
)
No matter where I look I seem to continue to find deadend after deadend (http_archive uses a download_and_extract method which assumes the *tgz is remote, the cc_library implementation is inaccessible / unextendible Java, etc.).
I would've thought the problem of "I have node A that generates a tonne of outfiles, which node B depends on" would be extremely common and have a simple solution. Am I missing something obvious?
Context:
I have this working with a local repository rule that takes in the local directory, and uses the cc_library rule above as the build_file parameter (but that means building the first step of this process has to be done completely outside of the Bazel build process which is not how this should be done):
new_local_repository(
name = "ros",
path = "/tmp/ros_debug",
build_file = "//ros:BUILD.bazel",
licenses = ["https://docs.ros.org/diamondback/api/licenses.html"]
)
A basic bazel philosophie is that a build should depend on unambigious inputs/dependencies and outputs strictly defined files in order to guarantee reproducable builds. Having a genrule generating a "random" number of files that a target should depend on is against this philosophy. Thats why you cant find a good way to achieve this.
A way to work around this is to have a single outs file in your genrule where you write down the names of the generated files (or a log of your script or whatever). Then you can define an additional filegroup containing all the generated files, using glob. Then add your genrule as dependency to the rule of node B and the filegroup to the srcs. Therefore it is guaranteed, that the files are generated before building the node B
In bazel documentation (https://docs.bazel.build/versions/master/cpp-use-cases.html) there's an example like this:
cc_library(
name = "build-all-the-files",
srcs = glob(["*.cc"])
hdrs = glob(["*.h"]),
)
How incremental it is? I.e. if I change only one of the *.cc files, will it rebuild the whole target or only what's required?
It will just recompile the modified file. Bazel will then link the library if the object file changes (so if you just change a comment, it may skip the link step).
You still have doubts?
Add the flag -s when you build and you will see what Bazel actually runs.
I have a hierarchical build system based on SCons. I have a root SConstruct that calls into a SConscript that builds a shared library and then into a different SConscript that builds an executable that depends on the shared library.
So here's my question: my understanding of shared libraries on linux is that when you want to do the final ld link for the executable that will be using the shared lib, the shared lib has to be included on the executable's ld command line as a source to reference it (unless it's in a standard location in which case the -l option works).
So here's something like what my SCons files look like:
=== rootdir/SConstruct
env=DefaultEnvironment()
shared_lib = SConscript('foolib/SConscript')
env.Append( LIBS=[shared_lib] )
executable = SConscript('barexec/SConscript')
=== rootdir/foolib/SConscript
env=DefaultEnvironment()
env.Append(CPPPATH=Glob('inc'))
penv = env.Clone()
penv.Append(CPPPATH=Glob('internal/inc'))
lib = penv.SharedLibrary( 'foo', source=['foo.c', 'morefoo.c']
Return("lib")
=== rootdir/barexec/SConscript
env=DefaultEnvironment()
exe = env.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] )
Return("exe")
So the hitch here is this line:
env.Append( LIBS=[shared_lib] )
This would be a great way to add generated libraries to the command line for any other libs that need them, EXCEPT that because SCons is doing a two-pass run through the SConscripts (first to generate it's dependency tree, then to do the work), rootdir/foolib/libfoo.so winds up on the command line for ALL products, EVEN libfoo.so itself:
gcc -g -Wall -Werror -o libfoo.so foo.o morefoo.o libfoo.so
So how is this best done with SCons? For now I've resorted to this hack:
=== rootdir/SConstruct
env=DefaultEnvironment()
shared_lib = SConscript('foolib/SConscript')
env['shared_lib'] = shared_lib
executable = SConscript('barexec/SConscript')
...
=== rootdir/barexec/SConscript
env=DefaultEnvironment()
exe = env.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] + env['shared_lib'] )
Return("exe")
Is there a more SCons-y way of doing this?
You should allow the shared libraries to be found by the build.
Look for the LIBPATH and RPATH variables in the SCons documentation; these are the "Scons-y" way to set up search paths so that any generated -l options find libraries properly.
Having mentioned the above, here's what you should see gcc do based on the setup of SCons (and if it doesn't, you may have to do it manually).
The -l option always finds shared libraries provided that you also give the compiler the location of the library. There are two times this is needed: at compile time (-L option) and at runtime (-rpath generated linker option).
The LIBPATH SCons setup should generate something that looks like -L/some/directory/path for the compile-time search path.
The RPATH SCons setup should generate a linker option to embed a search path; e.g. -Wl,-rpath -Wl,\$ORIGIN/../lib would embed a search path that searches relative to the executable so that executables placed in bin search in the parallel lib directory of the installation.
Here is a better way to organize your SConsctruct/SConscript files. Usually with Hierarchical builds you should share the env with the rest of the sub-directories. Notice that I cloned the main env in the barexec directory as well, so that the foolib is only used to link that binary.
=== rootdir/SConstruct
import os
env=DefaultEnvironment()
subdirs = [
'foolib',
'barexec'
]
# The exports attribute allows you to pass variables to the subdir SConscripts
for dir in subdirs:
SConscript( os.path.join(dir, 'SConscript'), exports = ['env'])
=== rootdir/foolib/SConscript
# inports the env created in the root SConstruct
#
# Any changes made to 'env' here will be reflected in
# the root/SConstruct and in the barexec/SConscript
#
Import('env')
# Adding this 'inc' dir to the include path for all users of this 'env'
env.Append(CPPPATH=Glob('inc'))
penv = env.Clone()
# Adding this include only for targets built with penv
penv.Append(CPPPATH=Glob('internal/inc'))
penv.SharedLibrary( 'foo', source=['foo.c', 'morefoo.c'])
=== rootdir/barexec/SConscript
Import('env')
clonedEnv = env.Clone()
# The foo lib will only be used for targets compiled with the clonedEnv env
# Notice that specifying '#' in a path means relative to the root SConstruct
# for each [item] in LIBS, you will get -llib on the compilation line
# for each [item] in LIBPATH, you will get -Lpath on the compilation line
clonedEnv.Append(LIBS=['foo'], LIBPATH=['#foolib'])
clonedEnv.Program( 'bar', source=['main.c', 'bar.c', 'rod.c'] )
Additional to Brady decision i use static/global variables to store targets name and path. It's allow me more control over build.
# site_scons/project.py
class Project:
APP1_NAME = "app1_name"
APP2_NAME = "app2_name"
MYLIB1_NAME = "mylib1_name"
# etc
APP_PATH = "#build/$BuildMode/bin" # BuildMode - commonly in my projects debug or release, `#` - root of project dir
LIB_PATH = "#build/$BuildMode/lib"
#staticmethod
def appPath(name) :
return os.path.join(APP_PATH, name)
#staticmethod
def libPath(name) :
return os.path.join(LIB_PATH, name)
Define targets:
from project import Project
...
env.SharedLibrary(Project.libPath(Project.MYLIB1_NAME), source=['foo.c', 'morefoo.c'])
Application:
from project import Project
...
env.Append(LIBPATH = [Project.LIB_PATH])
env.Append(LIBS = [Project.MYLIB1_NAME])
env.Program(Project.appPath(Project.MYAPP1_NAME), source=[...])
In my projects it works fine, scons automatically find depends of library without any additional commands. And if i want to change name of library i just change my Project class.
One issue that Brady's answer doesn't address is how to get correct library paths when building out-of-source using variant dirs. Here's a very similar approach that builds two different variants:
SConstruct
# Common environment for all build modes.
common = Environment(CCFLAGS=["-Wall"], CPPPATH=["#foolib/inc"])
# Build-mode specific environments.
debug = common.Clone()
debug.Append(CCFLAGS=["-O0"])
release = common.Clone()
release.Append(CCFLAGS=["-O"], CPPDEFINES=["NDEBUG"])
# Run all builds.
SConscript("SConscript", exports={"env": debug}, variant_dir="debug")
SConscript("SConscript", exports={"env": release}, variant_dir="release")
The # in the value for CPPPATH makes the include path relative to the project root instead of the variant dir.
SConscript
Import("env")
subdirs=["barexec", "foolib"]
senv = env.Clone(FOOLIBDIR=Dir("foolib"))
SConscript(dirs=subdirs, exports={"env": senv})
This root-level SConscript is required to build the subdirectories in each variant_dir.
By using the function Dir() when setting FOOLIBDIR, the library's variant build directory is resolved relative to this file rather than where it's used.
foolib/SConscript
Import("env")
penv = env.Clone()
penv.Append(CPPPATH=["internal/inc"])
penv.SharedLibrary("foo", source=["foo.c", "morefoo.c"])
It's important to clone the environment before making any changes to avoid affecting other directories.
barexec/SConscript
Import("env")
clonedEnv = env.Clone()
clonedEnv.Append(LIBPATH=["$FOOLIBDIR"], LIBS=["foo"])
clonedEnv.Program("bar", source=["main.c", "bar.c", "rod.c"])
The library's variant build dir is added to LIBPATH so both SCons and the linker can find the correct library.
Adding "foo" to LIBS informs SCons that barexec depends on foolib which must be built first, and adds the library to the linker command line.
$FOOLIBDIR should only be added to LIBPATH when "foo" is also added to LIBS – if not, barexec might be built before foolib, resulting in linker errors because the specified library path does not (yet) exist.