How to properly link the QT library during bazel build compilation? - c++

I am working on extending a project using bazel to build. However, one of my thrid_party dependency relying on a dynamic linked QT library. And I had a hard time linking it.
My project base is envpool, and I am using procgen as my third-party dependency. However, procgen relies on a series of QT library.
My approachs so far:
In the WorkSpace file, I specify the local directory of the qt library. (I am working on a new EC2 instance of Ubuntu 20.04 LTS on amazon cloud, and install qt5 and other base tools)
// download the github project
maybe(
http_archive,
name = "procgen",
sha256 = "8d443b7b8fba44ef051b182e9a87abfa4e05292568e476ca1e5f08f9666a1b72",
strip_prefix = "procgen-0.10.7/procgen/src/",
urls = [
"https://github.com/openai/procgen/archive/refs/tags/0.10.7.zip"
]
build_file = "//third_party/procgen:procgen.BUILD",
)
new_local_repository(
name = "qt",
path = "/usr/include/x86_64-linux-gnu/qt5", // I check indeed the header files are there
build_file = "BUILD.qt"
)
And the BUILD.qt file is
cc_library(
name = "qt_core",
hdrs = glob(["QtCore/**"]),
includes = ["."],
linkopts = [
"-lQt5Core",
],
visibility = ["//visibility:public"],
)
cc_library(
name = "qt_widgets",
hdrs = glob(["QtWidgets/**"]),
includes = ["."],
deps = [":qt_core"],
linkopts = [
"-lQt5Widgets",
],
visibility = ["//visibility:public"],
)
cc_library(
name = "qt_gui",
hdrs = glob(["QtGui/**"]),
includes = ["."],
deps = [":qt_core"],
linkopts = [
"-lQt5Gui",
],
visibility = ["//visibility:public"],
)
And the BUILD file for the procgen is
package(default_visibility = ["//visibility:public"])
cc_library(
name = "procgen",
srcs = glob(["*.cpp", "games/*.cpp"]),
hdrs = glob(["*.h"]),
deps = [
"#qt//:qt_widgets",
"#qt//:qt_gui",
"#qt//:qt_core",
]
)
However, when I use Bazel to build the project, it gives back the error that
Use --sandbox_debug to see verbose messages from the sandbox
In file included from external/procgen/games/starpilot.cpp:2:
external/procgen/games/../assetgen.h:10:10: fatal error: QColor: No such file or directory
10 | #include <QColor>
| ^~~~~~~~
compilation terminated.
I know I probably mess up the path or header file's include somewhere. For example, I am basically follow this post to include the QT library in the project, but I notice that guys use the full path "#include <qt/QTWidgets/xxxx>" instead. But I cannot change the code for "include" in the procgen.
This is the first time I use bazel, and on such a big project. Really appreciate if someone could help me out.
I put my current whole project package at here for reproducibility. You can use short-cut "sudo make bazel-build" to build it.
Best,
YJ

Related

Include a header file from a third party package to a C++ code in my project

I am trying to include a header file from a third party package to a C++ code in my project. This is how I structured it:
main_repo/
- third_party/some_lib/
- BUILD
- header_file.h
- extensions/my_project/
- BUILD
- my_app.cpp
- my_app.hpp
main_repo/extensions/my_project/BUILD
cc_library(
name = "my_app",
srcs = ["my_app.cpp"],
hdrs = ["my_app.hpp"],
visibility = ["//visibility:public"],
deps = [
"//third_party/some_lib:some_lib",
],
)
main_repo/third_party/some_lib/BUILD
cc_library(
name = "some_lib",
hdrs = glob(["*.h"]),
includes = ["."],
visibility = ["//visibility:public"],
)
main_repo/extensions/my_project/my_app.hpp
// some code here
#include "header_file.h"
// some code here
I see
header_file.h: No such file or directory
error when I try building it. Do you see a problem with my approach?

How to use c++ external library in bazel

I consider this a fairly fundamental requirement for a build system, but somehow it is not that straightforward - I've found 2 pieces of information, but without any luck yet:
The so-called official doc: https://docs.bazel.build/versions/main/cpp-use-cases.html#adding-dependencies-on-precompiled-libraries, it is really not much information
The cc_import reference:https://docs.bazel.build/versions/main/be/c-cpp.html#cc_import, which claimed for "allows users to import precompiled C/C++ libraries"
But for both cases, I don't see a way to specify the path/to/precompiled/lib, then how suppose it should work? I tried to ln the library folder into the Bazel workspace, but also got no luck.
Would really appreciate if you could shed lights on how to use an external dependency in Bazel, or point me to a real, working example.
Take a look at https://github.com/justbuchanan/bazel_rules_qt to see how a precompiled Qt5 version can be used with Bazel. The problem with precompiled libs is that you need them for every OS, compiler(-settings), platform. If you only target a specific OS + host compiler and target platform then precompiled libs are fine.
Think also about bazelizing your third-party library. I did this for instance for OpenEXR and oneTBB. I also hacked together some scripting to convert Visual Studio Project files to Bazel BUILD files.
Here is an example of how to use a precompiled OpenCV version on Windows:
WORKSPACE.bazel:
workspace(name = "OpenCVDemo")
load("#bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
# OpenCV
http_archive(
name = "opencv",
build_file = "//third_party:opencv.BUILD",
strip_prefix = "opencv/build",
# Release
url = "https://github.com/opencv/opencv/releases/download/4.3.0/opencv-4.3.0-dldt-2020.2-vc16-avx2.zip",
)
opencv.BUILD:
package(default_visibility = ["//visibility:public"])
MAIN_MODULES = [
"core",
"imgproc",
"imgcodecs",
"videoio",
"highgui",
"video",
"calib3d",
"features2d",
"objdetect",
"dnn",
"ml",
"flann",
"photo",
"stitching",
"gapi",
]
# https://stackoverflow.com/questions/56108940/how-to-specify-the-compiler-flags-to-be-used-in-opt-compilation-mode-by-my-own
config_setting(
name = "fastbuild_mode",
values = {"compilation_mode": "fastbuild"},
)
config_setting(
name = "dbg_mode",
values = {"compilation_mode": "dbg"},
)
cc_import(
name = "tbb",
shared_library = select({
":fastbuild_mode": "bin/tbb.dll",
":dbg_mode": "bin/tbb_debug.dll",
"//conditions:default": "bin/tbb.dll",
}),
)
[
(
cc_import(
name = module,
interface_library = select({
":fastbuild_mode": "lib/opencv_{}430.lib".format(module),
":dbg_mode": "lib/opencv_{}430d.lib".format(module),
"//conditions:default": "lib/opencv_{}430.lib".format(module),
}),
shared_library = select({
":fastbuild_mode": "bin/opencv_{}430.dll".format(module),
":dbg_mode": "bin/opencv_{}430d.dll".format(module),
"//conditions:default": "bin/opencv_{}430.dll".format(module),
}),
)
)
for module in MAIN_MODULES
]
cc_library(
name = "opencv",
hdrs = [
"include/opencv2/calib3d.hpp",
"include/opencv2/calib3d/calib3d.hpp",
"include/opencv2/calib3d/calib3d_c.h",
"include/opencv2/core.hpp",
"include/opencv2/core/hal/interface.h",
"include/opencv2/cvconfig.h",
"include/opencv2/dnn.hpp",
"include/opencv2/features2d.hpp",
"include/opencv2/flann.hpp",
"include/opencv2/flann/config.h",
"include/opencv2/flann/defines.h",
"include/opencv2/flann/miniflann.hpp",
"include/opencv2/highgui.hpp",
"include/opencv2/highgui/highgui.hpp",
"include/opencv2/highgui/highgui_c.h",
"include/opencv2/imgcodecs.hpp",
"include/opencv2/imgproc.hpp",
"include/opencv2/ml.hpp",
"include/opencv2/ml/ml.inl.hpp",
"include/opencv2/objdetect.hpp",
"include/opencv2/opencv.hpp",
"include/opencv2/opencv_modules.hpp",
"include/opencv2/photo.hpp",
"include/opencv2/stitching.hpp",
"include/opencv2/video.hpp",
"include/opencv2/video/background_segm.hpp",
"include/opencv2/video/tracking.hpp",
"include/opencv2/videoio.hpp",
"include/opencv2/videoio/videoio_c.h",
],
includes = ["include"],
deps = MAIN_MODULES + [
"tbb",
],
)
BUILD.bazel:
cc_binary(
name = "OpenCVDemo",
srcs = ["main.cpp"],
deps = ["#opencv"],
)
Should work similar for other libraries

The hdrs of cc_library seems not working of Bazel

I'm working on a project, which depends on a third party project, let's say project E, which is a precompiled libraries. I've set a new repository_rule and invoked it in the workspace.bzl, and in the BUILD of the project E, the corresponding library rule has been set like following:
cc_library(
name = "e_lib",
srcs = [
"#e//:e_lib1",
"#e//:e_lib2",
],
hdrs = ["#e//:e_headers"],
visibility = ["//visibility:public"],
)
And the target e_lib1, e_lib2 and e_headers are defined in the file e.BUILD, and it is correctly symblo linked to the downloaded contents folder in the last part of implementation function of repository_rule.
The e.BUILD is something like:
filegroup(
name = "e_headers",
srcs = glob(["include/*"]),
visibility = ["//visibility:public"],
)
filegroup(
name = "e_lib1",
srcs = if_darwin(["lib/lib1.dylib"])
+ if_linux_x86_64(["lib/lib1.so"]),
visibility = ["//visibility:public"],
)
filegroup(
name = "e_lib2",
srcs = if_darwin(["lib/lib2.dylib"])
+ if_linux_x86_64(["lib/lib2.so"]),
visibility = ["//visibility:public"],
)
Let's say there is a header baz.h in the directory include of the project E.But during the building phase, an error, baz.h file not found, occurred when bazel tried to compile a file core.cc. The pseudocodes are something like:
// core.cc
#include "core.h"
...
// core.h
#include "im.h"
...
// im.h
#include "baz.h"
// This file doesn't have the corresponding
// .cc file since all the implementations
// are put in this header.
The dependency of these files is core.cc -> core.h -> im.h -> baz.h, and the building rule of core.cc is:
cc_library(
name = "core",
srcs = [
"core.cc",
],
hdrs = "core.h",
deps = [
"//third_party/E:e_lib"],
alwayslink = 1,
)
So is there something wrong with my BUILD file? I guess that maybe the core.cc doesn't directly depend on the e_lib, so the deps of it is actually useless. Should I build a cc_library for the im.h?

Bazel build with OpenCV 3.3 dependencies

I'm trying to use Bazel to compile and distribute an OpenCV based C++ code and I'm facing an issue I can't resolve.
I build and install OpenCV 3.3 from sources, on an Ubuntu 16.04 LTS, with CUDA support (CUDA 8). I install it in the standard directory /usr/local.
Given it, I created my project with this WORKSPACE file :
new_local_repository(
name = "opencv",
path = "/usr/local",
build_file = "opencv.BUILD",
)
The opencv.BUILD contains :
cc_library(
name = "opencv",
srcs = glob(["lib/*.so*"]),
hdrs = glob(["include/**/*.hpp"]),
includes = ["include"],
visibility = ["//visibility:public"],
linkstatic = 1,
)
And I can use it in my own code using :
cc_binary(
name = "main",
srcs = ["main.cc"],
deps = [
"#opencv//:opencv"
],
)
but some source files in OpenCV, as :
/usr/local/include/opencv2/flann/flann_base.hpp
includes headers file from the same directory, like :
#include "general.h"
And when I build with Bazel, I get this error :
ERROR: /home/damien/main/BUILD:1:1: C++ compilation of rule '//main:main' failed (Exit 1)
In file included from external/opencv/include/opencv2/flann.hpp:48:0,
from external/opencv/include/opencv2/opencv.hpp:62,
from main/main.cc:1:
external/opencv/include/opencv2/flann/flann_base.hpp:38:21: fatal error: general.h: No such file or directory
(general.h is in the same directory as flann_base.hpp).
If I rewrite the #include directive as :
#include "opencv2/flann/general.h"
It compiles well. But this is not a convenient solution.
So my question is : is there a way to tell Bazel to look for headers in the same directory as the "current" file in this library ? I look upon every C++ directives of Bazel, but I don't see anything to achieve it.
Thank you in advance.
Ok, shame on me. I have to import *.h :
cc_library(
name = "opencv",
srcs = glob(["lib/*.so*"]),
hdrs = glob(["include/**/*.hpp", "include/**/*.h"]),
includes = ["include"],
visibility = ["//visibility:public"],
linkstatic = 1,
)
In my case using opencv4 and Damien setup I was getting the error while including highgui.hpp:
external/opencv/include/opencv4/opencv2/highgui.hpp:46:10: fatal
error: opencv2/core.hpp: No such file or directory #include
"opencv2/core.hpp"
I could fix it adjusting the includes adding opencv4:
cc_library(
name = "opencv",
srcs = glob(["lib/*.so*"]),
hdrs = glob(["include/**/*.hpp", "include/**/*.h"]),
includes = ["include/opencv4"],
visibility = ["//visibility:public"],
linkstatic = 1,
)

Bazel Link .so libraries located in a completely different, very remote folder

Folks,
I am trying to link .h and static libraries into my tensorflow program. My headers are in
/usr/local/include/lcm
And static/shared libraries (.so, etc.) in
/usr/local/lib
But Bazel complains they don't exist, or that it cannot find them. Here is my code on my BUILD file:
package(default_visibility = ["//tensorflow:internal"])
licenses(["notice"]) # Apache 2.0
exports_files(["LICENSE"])
# LCM shared libraries path
cc_library(
name = "lcm_lib",
srcs = glob([
"*.so",
]),
copts = ["-L/usr/local/lib"],
linkopts = ["-pthread", "-shared"],
visibility = ["//visibility:public"],
)
# LCM header libraries path
cc_library(
name = "lcm_headers",
hdrs = glob([
"include/**/*.h",
]),
copts = ["-L/usr/local/include"],
linkopts = ["-pthread"],
visibility = ["//visibility:public"],
)
cc_binary(
name = "myProject",
srcs = [
"main.cc",
],
linkopts = ["-lm"],
deps = [
"//tensorflow/cc:cc_ops",
"//tensorflow/core:framework_internal",
"//tensorflow/core:tensorflow",
],
)
filegroup(
name = "all_files",
srcs = glob(
["**/*"],
exclude = [
"**/METADATA",
"**/OWNERS",
"bin/**",
"gen/**",
],
),
visibility = ["//tensorflow:__subpackages__"],
)
If I remove LCM related code (from both BUILD and main.cc), then my program builds and runs. When I include LCM, then I get errors saying that lcm::LCM::~LCM() is undefined, and that it cannot find the liblcm.so.
Now, my non-tensorflow code (or the majority of my project) is running cmake, and LCM and the rest of the libraries I need (openCV, etc.) can be found. I use commands in my CMakeList.txt like:
# search path for LCM header files
set(LCM_IncludeSearchPaths
/usr/include/
/usr/local/include/
/opt/local/include
)
# search path for LCM static/dynamic libraries
set(LCM_LibrarySearchPaths
/usr/lib/
/usr/local/lib/
/opt/local/lib/
)
find_path(LCM_INCLUDE_DIR
NAMES lcm/lcm.h
HINTS ${LCM_IncludeSearchPaths}
)
FIND_LIBRARY(LCM_LIBS
NAMES lcm
HINTS ${LCM_LibrarySearchPaths}
PATH_SUFFIXES lib
)
And it all works. But it does not work for tensorflow and Bazel
Here are my build and WORKSPACE files, located in the same directory:
This is my touched WORKSPACE file:
# LCM static libraries
new_local_repository(
name = "lcm_libs",
# pkg-config --variable=libdir x11
path = "/usr/local/lib",
build_file_content = """
cc_library(
name = "liblcm",
srcs = ["liblcm.so"],
visibility = ["//visibility:public"],
)
""",
)
# LCM header files
new_local_repository(
name = "lcm_headers",
# pkg-config --variable=libdir x11
path = "/usr/local/include",
build_file_content = """
cc_library(
name = "lcm",
hdrs = glob([
"lcm/*.h", "lcm/*.hpp",
]),
visibility = ["//visibility:public"],
)
""",
)
# bind to a name to avoid using the "actual" format
#bind(
# name = "liblcm",
# actual = "#lcm_libs//:liblcm",
#)
#bind(
# name = "lcm",
# actual = "#lcm_headers//:lcm",
#)
#
And my BUILD:
# Description:
# Tensorflow C++ inference example for labeling images.
package(default_visibility = ["//tensorflow:internal"])
licenses(["notice"]) # Apache 2.0
exports_files(["LICENSE"])
cc_binary(
name = "facialFatigue",
srcs = [
"main.cc",
],
linkopts = ["-lm"],
deps = [
"//tensorflow/cc:cc_ops",
"//tensorflow/core:framework_internal",
"//tensorflow/core:tensorflow",
"#lcm_libs//:liblcm",
"#lcm_headers//:lcm",
],
)
filegroup(
name = "all_files",
srcs = glob(
["**/*"],
exclude = [
"**/METADATA",
"**/OWNERS",
"bin/**",
"gen/**",
],
),
visibility = ["//tensorflow:__subpackages__"],
)
Sorry for the long question :-(
Bazel executes action (in your case C++ compile action) in a sandbox, to ensure the hermeticity. This is needed to be correct when inputs of the action change. Therefore you have to tell Bazel about all the inputs, including the system ones.
But of course you can depend on system libraries, take a look at local_repository rule. You might also find this example in bazel-discuss# thread helpful.