I want bazel to fetch an external dependency from a URL. The file is an rpm file.
I added this in the WORKSPACE file in root dir:
http_archive(
name = "mylib",
url = "someURL/somefile.rpm",
build_file = "example.BUILD"
)
When I try:
bazel fetch #mylib//...
It says:
"com.google.devtools.build.lib.syntax.EvalException: Expected a file with a .zip, .jar, .war, .tar, .tar.gz, .tgz, .tar.xz, .txz, or .tar.bz2 suffix"
Basically it is not allowing to download rpm deps. How can I achieve this ?
How to download external dependency of type rpm.
You're trying to fetch an .rpm file, but it seems that bazel only accepts archive files with the suffixes .zip, .jar, .war, .tar, .tar.gz, .tgz, .tar.xz, .txz, or .tar.bz2.
You certainly figured it out by now.
Like #sebastian-nowak commented, you can do something like:
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_file")
http_file(
name = "mylib",
url = "someURL/somefile.rpm"
)
and in your example.BUILD, reference it with #mylib//file
(Source: https://docs.bazel.build/versions/master/repo/http.html#http_file)
Related
I have a project that builds via CMake and requires a lot of manual installations of additional deps. I want to migrate this project to Bazel and make these libs automatically downloadable. I found a solution for Boost, but I can't understand how to add icu4c and other libs which builds via other tools.
There are many ways to make use of third-party libraries using Bazel. The chosen approach depends on different properties of the third-party library, e.g.: Does the third-party library already support Bazel? Is the library available only as a pre-build package? Does the library use code generators, or any other tools, or transitive dependencies?
Given the example of {fmt} which uses CMake as build system you can proceed as the following:
First approach: Inject a BUILD file
In your WORKSPACE file you can do something like:
maybe(
new_git_repository,
name = "fmt",
branch = "master",
remote = "https://github.com/fmtlib/fmt",
build_file = "//third_party:fmt.BUILD",
)
The corresponding fmt.BUILD file can look like this:
cc_library(
name = "fmt",
srcs = [
#"src/fmt.cc", # No C++ module support
"src/format.cc",
"src/os.cc",
],
hdrs = [
"include/fmt/args.h",
"include/fmt/chrono.h",
"include/fmt/color.h",
"include/fmt/compile.h",
"include/fmt/core.h",
"include/fmt/format.h",
"include/fmt/format-inl.h",
"include/fmt/locale.h",
"include/fmt/os.h",
"include/fmt/ostream.h",
"include/fmt/printf.h",
"include/fmt/ranges.h",
"include/fmt/xchar.h",
],
includes = [
"include",
"src",
],
strip_include_prefix = "include",
visibility = ["//visibility:public"],
)
Advantages:
fmt-8.01 does not have out-of-the-box support for {fmt}. This way Bazel can make use of {fmt} without the need that {fmt} knows anything about Bazel
fmt-8.0.1 needs not to be modified
Disadvantages:
Reinvent the wheel: Every Bazel project that wants to use {fmt} has to reinvent this fmt.BUILD file.
Maintenance costs: If different Bazel projects want to adapt to future versions of {fmt} every single project has to do this maintenance by its own. Maybe new files will be introduced.
Missing Knowledge: Maybe for some reason, it makes sense to define some special defines upfront, etc. It also takes some time and knowledge of {fmt} to set up such a BUILD file. What is the best practice to build this lib?
Second approach: Bazelize {fmt}
Add a WORKSPACE file and BUILD file to the {fmt} repository.
This way {fmt} gets bazelized and can be used in your Bazel builds.
You could use it then this way:
Example
Create a WORKSPACE.bazel file with the following content:
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
# Fetch bazelized fmt
git_repository(
name = "fmt",
branch = "bazel-support", # A copy of master where BUILD.bazel, WORKSPACE.bazel, .bazelrc and .bazelversion are moved to root
remote = "https://github.com/<user_or_organisation>/fmt", # replace <user_or_organisation> by a valid account
)
Create a BUILD.bazel file and add a dependency to {fmt} (wit the content of fmt.BUILD).
In favor of keeping the {fmt} project directory clean, those files were not added to the project root directory (see here for details).
Third approach: Using the {fmt} repository with Bazel
Even though the {fmt} repository does not contain a WORKSPACE file in its root directory, there is an easy approach to use the {fmt} repository with Bazel out of the box. This is demonstrated in the following example.
Add to your WORKSPACE file:
load("#bazel_tools//tools/build_defs/repo:git.bzl", "new_git_repository")
# Fetch all files from fmt including the BUILD file `support/bazel/BUILD.bazel`
new_git_repository(
name = "fmt_workaround",
branch = "master",
remote = "https://github.com/fmtlib/fmt/",
build_file_content = "# Empty build file on purpose"
)
# Now the BUILD file `support/bazel/BUILD.bazel` can be used:
new_git_repository(
name = "fmt",
branch = "master",
remote = "https://github.com/fmtlib/fmt/",
build_file = "#fmt_workaround//:support/bazel/BUILD.bazel"
)
Create a BUILD.bazel file and add a dependency to {fmt}:
cc_binary( # Build a binary
name = "Demo", # Name of the binary
srcs = ["main.cpp"], # List of files - we only have main.cpp
deps = ["#fmt//:fmt"], # Depend on fmt
)
Make use of {fmt} in main.cpp:
#include "fmt/core.h"C
int main() {
fmt::print("The answer is {}.\n", 42);
}
The expected output of this example is The answer is 42.
Forth approach: Make use of patch_cmd
load("#bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
git_repository(
name = "fmt",
branch = "master",
patch_cmds = [
"mv support/bazel/.bazelrc .bazelrc",
"mv support/bazel/.bazelversion .bazelversion",
"mv support/bazel/BUILD.bazel BUILD.bazel",
"mv support/bazel/WORKSPACE.bazel WORKSPACE.bazel",
],
# Windows related patch commands are only needed in the case MSYS2 is not installed
patch_cmds_win = [
"Move-Item -Path support/bazel/.bazelrc -Destination .bazelrc",
"Move-Item -Path support/bazel/.bazelversion -Destination .bazelversion",
"Move-Item -Path support/bazel/BUILD.bazel -Destination BUILD.bazel",
"Move-Item -Path support/bazel/WORKSPACE.bazel -Destination WORKSPACE.bazel",
],
remote = "https://github.com/fmtlib/fmt",
)
More details here.
Other libraries
I have written a few blog posts about Bazelizing different libs:
Bazel: Bazelizing Qt5 for macOS
Bazel: Bazelizing Embree 3.13.0
Bazel: Bazelizing Qt5 & Qt6
Bazel: Handling external dependencies in OpenEXR
Bazel: Offical support for OpenEXR
Bazel: Bazelizing OpenEXR
Bazel: Bazelizing Embree 3.12.1
In the code base I am working with we use the oracle instant client library as a third party dependency in Bazel as follows:
cc_library(
name = "instant_client_basiclite",
srcs = glob(["*.so*"]),
visibility = ["//visibility:public"],
)
The library looks as this:
$ bazel query 'deps(#instant_client_basiclite//:instant_client_basiclite)'
#instant_client_basiclite//:instant_client_basiclite
#instant_client_basiclite//:liboramysql.so
#instant_client_basiclite//:libociicus.so
#instant_client_basiclite//:libocci.so.21.1
#instant_client_basiclite//:libocci.so
#instant_client_basiclite//:libnnz21.so
#instant_client_basiclite//:libclntshcore.so
...
It works as far as linking is concerned, but it seems that the path to the library is still needed because otherwise I get a run time error (oracle error 1804). The error can be solved by setting any of the environment variables ORACLE_HOME or LD_LIBRARY_PATH. In fact for the IBM WebSphere MQ there is the same need (character encoding table files need to be found).
ldd on a binary points to .../bazel-bin/app/../../../_solib_k8/_U#instant_Uclient_Ubasiclite_S_S_Cinstant_Uclient_Ubasiclite___U/libocci.so.21.1
How can I set those needed path variables so that bazel test, bazel run and Bazel container image rules work?
One possibility is to add the following command line option:
--test_env=ORACLE_HOME="$(bazel info output_base)/external/instant_client_basiclite"
It is a pity that it cannot be put in .bazelrc.
I am just starting to learn how to use bazel following this tutorial
One thing I am unsure how to do is how to use a submodule, from my own repo for example. where I do not use bazel. The repo just has some c/h files that I need to pull in. To make things work locally I added a BUILD file in folder pulled in from submodules. However after I commit and push my changes the BUILD file is obviously not there. How do I add the c/h files from the submodule folder to my build. Bazel seems to be looking for a BUILD folder in that directory and there will be none, if for example someone else clones this repo.
currently my "main" Directory has this build file:
cc_library(
name = "uCShell-main",
srcs = ["main.c"],
deps = ["//uCShell:uCShell-lib" ],
)
cc_binary(
name = "uCShell-bin",
deps = [":uCShell-main" , "//uCShell:uCShell-lib" ]
)
and the folder with the pulled in submodule has this locally added BUILD file:
cc_library(
name = "uCShell-lib",
srcs = glob(["*.c"]),
hdrs = glob(["*.h"]),
visibility = ["//visibility:public"],
)
This works and compiles just fine. However do correct any issues or misuse of Bazel you see here, I'd like to learn.
Ultimately to reiterate the issue, when I push this project the locally added BUILD file will not be in the project because it is not in the original submodule. So how can I inlcude the c/h files in that submodule in order to build my main. And I would like to leave it as a submodule. Ultimately I can just add a BUILD file for the submodule's repo, but would like to find a better way, for example what if this was not my repo where I can just add a BUILD file.
If you use new_local_repository with a relative path to import the submodule, you can set build_file or build_file_content to add the BUILD file. You should be able to use that same BUILD file as-is.
Because it will be in a different external repository, you'll need to access it via the corresponding label.
For example, if you put the BUILD file at build/BUILD.my_lib.bazel and this in WORKSPACE:
new_local_repository(
name = "my_lib",
path = "submodules/my_lib",
build_file = "#//build:BUILD.my_lib.bazel",
)
then you can put #my_lib//:uCShell-lib in deps of any other target and it will all work.
Assume that a prec-compiled dependency is supplied by a vendor:
$ ls /opt/vendor/protobuf/lib
libprotobuf.so.3 -> libprotobuf.so.3.0.0
libprotobuf.so.3.0.0
To use this with Bazel, the following target can be created:
cc_import(
name = "protobuf",
shared_library = "lib/libprotobuf.3.0.0",
)
This way, a bazel-built application can link to the library, however, it fails to start:
error while loading shared libraries: libprotobuf.so.3: cannot open shared object file: no such file or directory
The root cause is that the actual so file has a custom SONAME field:
objdump -x libprotobuf.so.3.0.0|grep SONAME
SONAME libprotobuf.so.3
The loader will look for libprotobuf.so.3 (instead of 3.0.0), but will not find it in the sandbox, as we never told bazel about the symlink. The symlink is relative, specifying it in the cc_import target will yield to a similar error.
Is it possible to create a runnable binary with bazel that links to such a shared library that is supposed to be found via a symlink?
Setting the RPATH can be a workaround. The cc_import need to be wrapped by a cc_library:
cc_library(
name = "protobuf",
deps = [":protobuf_impl"],
linkopts = ["-Wl,-rpath=/opt/vendor/protobuf/lib"],
)
cc_import(
name = "protobuf_impl",
shared_library = "lib/libprotobuf.3.0.0",
)
This will make the binary run, but assumes that "/opt/vendor/protobuf/lib" is present on every system (incl. remote execution), and the loader during runtime still escapes the sandbox. A clearer solution would be nice.
I'm building a program with gRPC library using bazel. My WORKSPACE file:
http_archive(
name = "com_github_grpc_grpc",
urls = ["https://github.com/grpc/grpc/archive/v1.8.3.zip"],
sha256 = "57a2c67abe789ce9e80d49f473515c7479ae494e87dba84463b10bbd0990ad62",
strip_prefix = "grpc-1.8.3",
)
load("#com_github_grpc_grpc//bazel:grpc_deps.bzl", "grpc_deps")
grpc_deps()
BUILD file:
proto_library(
name = "test_proto",
srcs = ["test.proto"],
)
cc_proto_library(
name = "test_cc_proto",
deps = [":test_proto"],
)
cc_binary(
name = "hello",
srcs = ["hello.cc"],
deps = [
":test_cc_proto",
"#com_github_grpc_grpc//:grpc++",
],
)
Compiling this throws error:
every rule of type proto_library implicitly depends upon the target '#com_google_protobuf_cc//:cc_toolchain', but this target could not be found because of: no such package '#com_google_protobuf_cc//': The repository could not be resolved.
If I include com_google_protobuf_cc repository manually, the version doesn't match and I get error saying test.pb.h was generated using a newer version of protoc.
How do I make gRPC load right version of com_google_protobuf_cc?
How are you including Protobuf manually, what did you put in your BUILD and/or WORKSPACE file/s to achieve this? It's hard to comment on what could be wrong without knowing exactly what you have tried.
As far as I know you can include it by downloading the version you require then adding something like the following to your WORKSPACE file:
local_repository(
name = "com_google_protobuf",
path = "../protobuf-3.4.1",
)
local_repository(
name = "com_google_protobuf_cc",
path = "../protobuf-3.4.1",
)
Of course change the paths to match the version and location of your downloaded copy of Protobuf. Alternatively you can probably use http_archive to point it directly to where it should be downloaded, in the same way as you have done for gRPC.