I want to know is it possible to configure Gradle to use one global build directory for all projects? I need this because I store all my projects in Google Drive folder (for backup and fast access from other devices) and when I build project Google Drive is loading CPU while syncing project build directory. So I want to move build directories outside of Google Drive.
Also this option will be useful for those who want to build projects using RAM disk as storage for build directory.
Thanks to Peter Niederwieser. I decided to post my own answer because of additional details.
To setup global build directory you need to add these lines to ~/.gradle/init.gradle:
gradle.projectsLoaded {
rootProject.allprojects {
buildDir = "/path/to/build/${rootProject.name}/${project.name}"
}
}
Nice option to have global build directory on the RAM disk.
If you are macOS (OS X) user you can do this with:
diskutil erasevolume HFS+ 'RAMDiskName' `hdiutil attach -nomount ram://XXXXXX`
where XXXXXX is a count of 512-byte blocks (for example, it's 262114 for 128MB RAM disk)
And buildDir line will be:
buildDir = "/Volumes/RAMDiskName/${rootProject.name}/${project.name}"
Also you can extend configuration to:
get global build path from environment variable or fallback to build directory inside project directory;
include project group in path, this makes path more unique especially if you have several projects with the same name.
~/.gradle/init.gradle:
def configProject(p, buildDir) {
p.buildDir = "${buildDir}/${p.name}"
p.subprojects { s ->
configProject(s, "${p.buildDir}")
}
}
def buildDir = System.env.GRADLE_GLOBAL_BUILD_PATH
if (!buildDir?.trim()) {
buildDir = "build"
}
gradle.projectsLoaded {
if (ext.has("group")) {
buildDir += "/${ext.group}"
}
configProject(rootProject, buildDir)
}
And in project settings.gradle:
gradle.ext.group = 'com.example.yourproject'
Also you can use this setting in project build.gradle to set project group but this is optional:
allprojects {
...
group gradle.ext.group
}
build.gradle:
def baseDir = "/global/build/dir/$project.name"
buildDir = baseDir + "/root"
subprojects {
buildDir = baseDir + project.path.replaceFirst(":", "/").replace(":", ".")
}
You could also go the opposite way. Replace the build folder with a global folder.
Remove project's build folder
mklink /d ...disk\path\to\project\build ...RAM\path\to\project (or ln in Unix)
Build fast
You can automate this in the clean task, so a gradlew clean cleans the folder contents and recreates the link.
With some quick testing it seems that Google Drive doesn't recognize directory in junction links on Windows. It'll sync the folders, but not the contents. Weirdly if I upload files to Drive into those folders, they're synced into the local copy. But, if I delete the synced file from the disk, the deletion is not synced back. It seems links cause Drive to be one way.
Related
I'm currently messing around with Google's Mediapipe, which uses Bazel as a build tool. The folder has the following structure:
mediapipe
├ mediapipe
| └ examples
| └ desktop
| └ hand_tracking
| └ BUILD
├ calculators
| └ tensor
| └ tensor_to_landmarks_calculator.cc
| └ BUILD
└ WORKSPACE
There are a bunch of other files in there as well, but they are rather irrelevant to this problem. They can be found in the git repo linked above if you need them.
I'm at a stage where I can build and run the hand_tracking example without any problems. Now, I want to include the cereal library to the build, so that I can use #include <cereal/archives/binary.hpp> from within tensors_to_landmarks_calculator.cc. The cereal library is located at C:\\cereal, but can be moved to other locations if it simplifies the process.
Basically, I'm looking for the Bazel equivalent of adding a path to Additional Include Directories in Visual Studio.
How would I need to modify the WORKSPACE and BUILD files in order to include the library in my project, assuming they are in a default state?
Unfortunately, this official doc page only covers one-file libraries, and other implementations kept giving me File could not be found errors at build time.
Thanks in advance!
First you have to tell Bazel about the code living "outside" the
workspace area. It needs to know how to find it, how to build it, and
what to call it, etc. These are known as remote repositories. They
can be local to your disk (outside the Bazel workspace area), or
actually remote on another machine or server, like github. The
important thing is it must be described to Bazel with enough
information that it can use.
As most third party code does not come with BUILD.bazel files, you may
need to provide one yourself and tell Bazel "use this as if it was a
build file found in that code."
For a local directory outside your bazel project
Add a repository rule like this to your WORKSPACE file:
# This could go in your WORKSPACE file
# (But prefer the http_archive solution below)
new_local_repository(
name = "cereal",
build_file = "//third_party:cereal.BUILD.bazel",
path = "<path-to-directory>",
)
("new_local_repository" is built-in to bazel)
Somewhere under your Bazel WORKSPACE area you'll also need to make a
cereal.BUILD.bazel file and export it from the package. I choose a directory called //third_party, but you can put it anywhere
else, and name it anything else, as long as the repository rule
provides a proper bazel label for it.) The contents might look like
this:
# contents of //third_party/cereal.BUILD.bazel
cc_library(
name = "cereal-lib",
srcs = glob(["**/*.hpp"]),
includes = ["include"],
visibility = ["//visibility:public"],
)
Bazel will pretend this was the BUILD file that "came with" the remote
repository, even though it's actually local to your repo. When Bazel fetches this remote repostiory code it copies it, and the BUILD file you provide, into its external area for caching, building, etc.
To make //third_party:cereal.BUILD.bazel a valid target in your directory, add a BUILD.bazel file to that directory:
# contents of //third_party/BUILD.bazel
exports_files = [
"cereal.BUILD.bazel",
]
Without exporting it, you won't be able to refer to the buildfile from your repository rule.
Local disk repositories aren't very portable since people may have
different versions installed and it's not very hermetic (making it
hard to share caches of builds with others), and it requires they put
them in the same place, and that kind of setup can be problematic. It
also will fail when you mix operating systems, etc, if you refer to it as "C:..."
Downloading a tarball of the library from github, for example
A better way is to download a fixed version from github, for example,
and let Bazel manage it for you in its external area:
http_archive(
name = "cereal",
sha256 = "329ea3e3130b026c03a4acc50e168e7daff4e6e661bc6a7dfec0d77b570851d5",
urls =
["https://github.com/USCiLab/cereal/archive/refs/tags/v1.3.0.tar.gz"],
build_file = "//third_party:cereal.BUILD.bazel",
)
The sha256 is important, since it downloads and computes it, compares to what you specified, and can cache it. In the future, it won't re-download it if the local file's sha matches.
Notice, it again says build_file = //third_party:cereal.BUILD.bazel., all
the same things from new_local_repository above apply here. Make sure you provide the build file for it to use, and export it from where you put it.
*To test that the remote repository is setup ok
on the command line issue
bazel fetch #cereal//:cereal-lib
I sometimes have to clear it out to make it try again, if my rule isn't quite right, but the "bad" version sticks around.
bazel clean --expunge
will remove it, but might be overkill.
Finally
We have:
defined a remote repository called #cereal
defined a target in it called cereal-lib
the target is thus #cereal//:cereal-lib
To use it
Go to the package where you would like to include cereal, and add a
dependency on this repository to the rule that builds the c++ code that would like to use cereal. That is, in your case, the BUILD rule that causes tensor_to_landmarks_calculator.cc to get built, add:
deps = [
"#cereal//:cereal-lib",
...
]
And then in your c++ code:
#include "cereal/cereal.hpp"
That should do it.
In an iOS c++/Qt application, I need to ship a few files and to keep them in their directory structure.
For the Android version, we bundle a zip which we unzip on the target before creating the QApplication.
On iOS, it seems that CMake is not capable of bundling files in a tree:
https://cmake.org/cmake/help/latest/prop_tgt/RESOURCE.html#prop_tgt:RESOURCE
https://cmake.org/cmake/help/latest/prop_sf/MACOSX_PACKAGE_LOCATION.html
I am not sure if this is a limitation of cmake or if this is a global limitation on iOS.
From the docs about iOS bundles:
It uses a relatively flat structure with few extraneous directories in an effort to save disk space and simplify access to the files.
What would be the preferred approach?
Is there a solution to ship the files from CMake directly?
If not, how can I achieve this so that they are available before the QApplication is created?
The xcode command
Thanks to #Cy-4AH, I added the folder in Xcode and could get the command to do this:
CpResource _PATH_TO_DIRECTORY_ _APP_BUNDLE_DIRECTORY_/_RESOURCE_DIR_NAME_
cd /Users/denis/opt/qfield/ios/QField
export PATH="....."
builtin-copy -exclude .DS_Store -exclude CVS -exclude .svn -exclude .git -exclude .hg -strip-debug-symbols -strip-tool /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/strip -resolve-src-symlinks _PATH_TO_DIRECTORY_ _APP_BUNDLE_DIRECTORY_
But how can I create this from cmake? builtin-copyis an xcode command.
Simple system copy command
From an old (2008) discussion, we could use simple cp commands.
This works up to signing, but then I get an error unsealed contents present in the bundle root.
From this answer, it seems related that I cannot simply add folders in the resource directory. From the docs anatomy of framework bundles: Nonlocalized resources reside at the top level of the Resources directory
(Disclaimer: I'm not a CMake user, and there may be a more CMake-ey way to do this)
If you can set up post-build action, the following terminal script can efficiently sync files into your bundle from another location. I use it in my game engine because it only copies updated or new files upon subsequent builds, and preserves directory structure:
mkdir -p PATHTO/ORIGINFOLDERNAME
mkdir -p PATHTOBUILDFOLDER/PROJECTNAME.app/Contents/Resources/DESTINATIONFOLDERNAME
rsync -avu --delete --exclude=".*" PATHTO/ORIGINFOLDERNAME/ PATHTOBUILDFOLDER/PROJECTNAME.app/Contents/Resources/DESTINATIONFOLDERNAME
The mkdir commands are only to ensure that the folders are generated, if they were deleted.
So apparently the CMake method also works for directories.
target_sources(${QT_IOS_TARGET} PRIVATE ${_resource})
set_source_files_properties(${_resource} PROPERTIES MACOSX_PACKAGE_LOCATION Resources)
It will just be added at the root directory of the bundle and not within the Resources.
If the embedded file is not too big, you might consider :
in your source tree, generating a C++ file embedding that file as a constant array. For example, if your file contains just hello, world with a new line, you could have something like
/// file contents.cc
const char file_contents[] = "hello, world\n";
and at the beginning of your program (perhaps in your main function, before your QApplication) call a C++ function which writes such a file (perhaps in /tmp/).
in your build automation (e.g your Makefile or your qmake things), have something which generates the C++ contents.cc file from the genuine source
This is with a POSIX/Linux point of view, adapt my answer to your iOS.
I have the following directory structure:
my_dir
|
--> src
| |
| --> foo.cc
| --> BUILD
|
--> WORKSPACE
|
--> bazel-out/ (symlink)
|
| ...
src/BUILD contains the following code:
cc_binary(
name = "foo",
srcs = ["foo.cc"]
)
The file foo.cc creates a file named bar.txt using the regular way with <fstream> utilities.
However, when I invoke Bazel with bazel run //src:foo the file bar.txt is created and placed in bazel-out/darwin-fastbuild/bin/src/foo.runfiles/foo/bar.txt instead of my_dir/src/bar.txt, where the original source is.
I tried adding an outs field to the foo rule, but Bazel complained that outs is not a recognized attribute for cc_binary.
I also thought of creating a filegroup rule, but there is no deps field where I can declare foo as a dependency for those files.
How can I make sure that the files generated by running the cc_binary rule are placed in my_dir/src/bar.txt instead of bazel-out/...?
Bazel doesn't allow you to modify the state of your workspace, by design.
The short answer is that you don't want the results of the past builds to modify the state of your workspace, hence potentially modifying the results of the future builds. It'll violate reproducibility if running Bazel multiple times on the same workspace results in different outputs.
Given your example: imagine calling bazel run //src:foo which inserts
#define true false
#define false true
at the top of the src/foo.cc. What happens if you call bazel run //src:foo again?
The long answer: https://docs.bazel.build/versions/master/rule-challenges.html#assumption-aim-for-correctness-throughput-ease-of-use-latency
Here's more information on the output directory: https://docs.bazel.build/versions/master/output_directories.html#documentation-of-the-current-bazel-output-directory-layout
There could be a workaround to use genrule. Below is an example that I use genrule to copy a file to the .git folder.
genrule(
name = "precommit",
srcs = glob(["git/**"]),
outs = ["precommit.txt"],
# folder contain this BUILD.bazel file is tool which will be symbol linked, we use cd -P to get to the physical path
cmd = "echo 'setup pre-commit.sh' > $(OUTS) && cd -P tools && ./path/to/your-script.sh",
local = 1, # required
)
If you're passing the name of the output file in when running, you can simply use absolute paths. To make this easier, you can use the realpath utility if you're in linux. If you're on a mac, it is included in brew install coreutils. Then running it looks something like:
bazel run my_app_dir:binary_target -- --output_file=`realpath relative/path/to.output
This has been discussed and explained in a Bazel issue. Recommendation is to use a tool external to Bazel:
As I understand the use-case, this is out-of-scope for building and in the scope of, perhaps, workspace configuration. What I'm sure of is that an external tool would be both easier and safer to write for this purpose, than to introduce such a deep design change to Bazel.
The tool would copy the files from the output tree into the source tree, and update a manifest file (also in the source tree) that lists the path-digest pairs. The sources and the manifest file would all be versioned. A genrule or a sh_test would depend on the file-generating genrules, as well as on this manifest file, and compare the file-generating genrules' outputs' digests (in the output tree) to those in the manifest file, and would fail if there's a mismatch. In that case the user would need to run the external tool, thus update the source tree and the manifest, then rerun the build, which is the same workflow as you described, except you'd run this tool instead of bazel regenerate-autogenerated-sources.
Having used CMake, I've become used to out-of-source builds, which are encouraged with CMake. How can out-of-source builds be done with Cargo?
Using in-source-builds again feels like a step backwards:
Development tools need to be configured to ignore paths. Sometimes multiple plugins and development tools - especially using VIM or Emacs!
Some tools can't be configured to easily hide build files. While dotfiles are typically hidden, they will still show Cargo.lock and target/, worse still, recursively exposing their contents.
Deleting un-tracked files to remove everything outside of version control, typically to cleanup editor temp files or some test output, can backfire if you forgot to add a new file to version control and don't manually check the file list properly before deleting them.
Dependencies are downloaded into your source code path, sometimes adding *.rs files in the target directory as part of building indirect deps, so operating on all *.rs files may accidentally pickup other files which aren't in a hidden directory, so might not be ignored even after development tools have been configured.
While it's possible to work around all these issues, I'd rather just have an external build path and keep the source directory pristine.
You can specify the directory of the target/ folder either via configuration file (key build.target-dir) or environment variable (CARGO_TARGET_DIR). Here is an example using a configuration file:
Suppose you want to have a directory ~/work/ in which you want to save the Cargo project (~/work/foo/) and next to it the target directory (~/work/my-target/).
$ cd ~/work
$ cargo new --bin foo
$ mkdir .cargo
$ $EDITOR .cargo/config
Then insert the following into the configuration file:
[build]
target-dir = "./my-target"
If you then build in your normal Cargo project directory:
$ cd foo
$ cargo build
You will notice that there is no target/ dir, but everything is in ~/work/my-target/.
However, the Cargo.lock is still saved inside the Cargo project directory, but that kinda makes sense. For executables, you should check the Cargo.lock file into your git! For libraries, you shouldn't. I guess having to ignore one file is better than having to ignore an entire folder.
Lastly, there are a few caveats to changing the target-dir, which are listed in the PR which introduced the feature.
While useful manually setting this up isn't all that convenient, I wanted to be able to build multiple crates within a source tree, having all of them out-of-source, something that ../target-dir configuration option wouldn't achieve.
Helper utility for convenient out-of-source builds
Using the environment variable I've written a small utility to wrap cargo, so it automatically builds out-of-source, supporting crates both at the top-level, on in a subdirectory of the source tree.
Thanks to Lukas for pointing out CARGO_TARGET_DIR and target-dir configuration option.
What I really wanted was a dynamic CARGO_TARGET_DIR that changes relative to where I am.
This bash alias puts all builds in a mirrored directory structure, e.g. instead of putting target into ~/mydir/myproj it puts in into ~/rustbuild/mydir/myproj
alias cargo='CARGO_TARGET_DIR=$(echo $PWD | sed "s|$HOME|$HOME/rustbuild|g") cargo'
You could also make your rustbuild directory hidden.
When I try run make from cmd-console on Windows, it runs Turbo Delphi's make.exe but I need MSYS's make.exe. There is no mention about Turbo Delphi in %path% variable, maybe I can change it to MSYS in registry?
The path is in the registry but usually you edit through this interface:
Go to Control Panel -> System -> System settings -> Environment Variables.
Scroll down in system variables until you find PATH.
Click edit and change accordingly.
BE SURE to include a semicolon at the end of the previous as that is the delimiter, i.e. c:\path;c:\path2
Launch a new console for the settings to take effect.
Here I'm providing solution to setup Terraform environment variable in windows for beginners.
Download the terraform ZIP file from Terraform site.
Extract the .exe from the ZIP file to a folder eg C:\Apps\Terraform
copy this path location like C:\Apps\terraform\
Add the folder location to your PATH variable, eg: Control Panel -> System -> System settings -> Environment Variables
In System Variables, select Path > edit > new > Enter the location of the Terraform .exe, eg C:\Apps\Terraform then click OK
Open a new CMD/PowerShell and the Terraform command should work
Or you can just run this PowerShell command to append extra folder to the existing path:
$env:Path += ";C:\temp\terraform"
To add a PERSISTENT path (eg one that's permanent), you can do this one-liner in PowerShell (adjust the last c:\apps\terraform part)
Set-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH -Value (((Get-ItemProperty -Path 'Registry::HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment' -Name PATH).path) + ";c:\apps\terraform" )
Alternatively, you can jump directly to the Environment Variables dialog by RUNning/CMD/PowerShell this:
rundll32.exe sysdm.cpl,EditEnvironmentVariables
I had issues for a whilst not getting Terraform commands to run unless I was in the directory of the exe, even though I set the path correctly.
For anyone else finding this issue, I fixed it by moving the environment variable higher than others!
Why don't you create a bat file makedos.bat containing the following line?
c:\DOS\make.exe %1 %2 %5
and put it in C:\DOS (or C:\Windowsè or make sure that it is in your %path%)
You can run from cmd, SET and it displays all environment variables, including PATH.
In registry you can find environment variables under:
HKEY_CURRENT_USER\Environment
HKEY_CURRENT_USER\Volatile Environment
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\Session Manager\Environment
just copy it to system32 call make1 or whatever if the name conflicts.