I am using GoCD to build a project with a large number of modules and I have modeled it as a pipeline with two stages (stage 1 is to build the code, stage 2 is to run the tests). The directory structure after a successful build stage looks about like this:
myproject/
|-- myproject-module1
| |-- build <-- created by stage 1, required by stage 2
| `-- src
|-- myproject-module2
| |-- build <-- created by stage 1, required by stage 2
| `-- src
|-- myproject-module3
| |-- build <-- created by stage 1, required by stage 2
| `-- src
`-- ... many more modules ...
In stage 1 I have configured a Build Artifact with source */build and in stage 2 I'm trying to fetch all the build folders again with source * , with the intention that they would end up in the correct location next to the src folder inside each of the project modules.
Unfortunately, I have found no way to achieve this yet. GoCD seems to create a separate ZIP file of all the *\build folders and during the fetch, the file *.zip cannot be found (I assume that it really looks for a file with that exact name, instead of using wildcards). Of course I could hard-code all the module names and individually fetch myproject-module[1:n], but that's exactly what I want to avoid.
Does anyone have some advice on how this could be achieved?
In this discussion from 2014, it is claimed that wildcards cannot be used to fetch artifacts. Is that really still the case?!
I don't know if it's possible to do using built-in features of GoCD, but it should be definitely be possible using REST API.
Knowing current pipeline name, you can get all available stages of it and calculate previous one. Next, using a possibility to download an artifact directory as zip archive, you can get what you want.
So you can add this as a script for the second stage, which will get the zipped artifact and after that you can continue with testing.
To do that, I can recommend my implementation of GoCD API - yagocd. This python library would let you to program aforementioned logic in a natural way.
Some tips:
you can get current pipeline name from GO_PIPELINE_NAME (there are a lot environment variables at your service)
to find pipeline by name you can use PipelineManager: go.pipelines.get($GO_PIPELINE_NAME, $GO_PIPELINE_COUNTER)
having pipeline instance, you can iterate stages by pipeline_instance.stages object
having a stage, you can get it job and download directory by some path using go.artifacts.directory_wait method
If you would have questions about the implementation I can try to help you.
You can select artifacts by wildcard, you just have to give the artifact a destination (attribute dest in the XML config). You can do that multiple times inside the same job, just use a different dest each time:
<artifact src="myproject/myproject-module1/build/*" dest="module1/" />
<artifact src="myproject/myproject-module2/build/*" dest="module2/" />
<artifact src="myproject/myproject-module3/build/*" dest="module3/" />
The corresponding <fetchartifact ...> tags then need to use srcdir="module1" etc.
Related
I'm currently messing around with Google's Mediapipe, which uses Bazel as a build tool. The folder has the following structure:
mediapipe
├ mediapipe
| └ examples
| └ desktop
| └ hand_tracking
| └ BUILD
├ calculators
| └ tensor
| └ tensor_to_landmarks_calculator.cc
| └ BUILD
└ WORKSPACE
There are a bunch of other files in there as well, but they are rather irrelevant to this problem. They can be found in the git repo linked above if you need them.
I'm at a stage where I can build and run the hand_tracking example without any problems. Now, I want to include the cereal library to the build, so that I can use #include <cereal/archives/binary.hpp> from within tensors_to_landmarks_calculator.cc. The cereal library is located at C:\\cereal, but can be moved to other locations if it simplifies the process.
Basically, I'm looking for the Bazel equivalent of adding a path to Additional Include Directories in Visual Studio.
How would I need to modify the WORKSPACE and BUILD files in order to include the library in my project, assuming they are in a default state?
Unfortunately, this official doc page only covers one-file libraries, and other implementations kept giving me File could not be found errors at build time.
Thanks in advance!
First you have to tell Bazel about the code living "outside" the
workspace area. It needs to know how to find it, how to build it, and
what to call it, etc. These are known as remote repositories. They
can be local to your disk (outside the Bazel workspace area), or
actually remote on another machine or server, like github. The
important thing is it must be described to Bazel with enough
information that it can use.
As most third party code does not come with BUILD.bazel files, you may
need to provide one yourself and tell Bazel "use this as if it was a
build file found in that code."
For a local directory outside your bazel project
Add a repository rule like this to your WORKSPACE file:
# This could go in your WORKSPACE file
# (But prefer the http_archive solution below)
new_local_repository(
name = "cereal",
build_file = "//third_party:cereal.BUILD.bazel",
path = "<path-to-directory>",
)
("new_local_repository" is built-in to bazel)
Somewhere under your Bazel WORKSPACE area you'll also need to make a
cereal.BUILD.bazel file and export it from the package. I choose a directory called //third_party, but you can put it anywhere
else, and name it anything else, as long as the repository rule
provides a proper bazel label for it.) The contents might look like
this:
# contents of //third_party/cereal.BUILD.bazel
cc_library(
name = "cereal-lib",
srcs = glob(["**/*.hpp"]),
includes = ["include"],
visibility = ["//visibility:public"],
)
Bazel will pretend this was the BUILD file that "came with" the remote
repository, even though it's actually local to your repo. When Bazel fetches this remote repostiory code it copies it, and the BUILD file you provide, into its external area for caching, building, etc.
To make //third_party:cereal.BUILD.bazel a valid target in your directory, add a BUILD.bazel file to that directory:
# contents of //third_party/BUILD.bazel
exports_files = [
"cereal.BUILD.bazel",
]
Without exporting it, you won't be able to refer to the buildfile from your repository rule.
Local disk repositories aren't very portable since people may have
different versions installed and it's not very hermetic (making it
hard to share caches of builds with others), and it requires they put
them in the same place, and that kind of setup can be problematic. It
also will fail when you mix operating systems, etc, if you refer to it as "C:..."
Downloading a tarball of the library from github, for example
A better way is to download a fixed version from github, for example,
and let Bazel manage it for you in its external area:
http_archive(
name = "cereal",
sha256 = "329ea3e3130b026c03a4acc50e168e7daff4e6e661bc6a7dfec0d77b570851d5",
urls =
["https://github.com/USCiLab/cereal/archive/refs/tags/v1.3.0.tar.gz"],
build_file = "//third_party:cereal.BUILD.bazel",
)
The sha256 is important, since it downloads and computes it, compares to what you specified, and can cache it. In the future, it won't re-download it if the local file's sha matches.
Notice, it again says build_file = //third_party:cereal.BUILD.bazel., all
the same things from new_local_repository above apply here. Make sure you provide the build file for it to use, and export it from where you put it.
*To test that the remote repository is setup ok
on the command line issue
bazel fetch #cereal//:cereal-lib
I sometimes have to clear it out to make it try again, if my rule isn't quite right, but the "bad" version sticks around.
bazel clean --expunge
will remove it, but might be overkill.
Finally
We have:
defined a remote repository called #cereal
defined a target in it called cereal-lib
the target is thus #cereal//:cereal-lib
To use it
Go to the package where you would like to include cereal, and add a
dependency on this repository to the rule that builds the c++ code that would like to use cereal. That is, in your case, the BUILD rule that causes tensor_to_landmarks_calculator.cc to get built, add:
deps = [
"#cereal//:cereal-lib",
...
]
And then in your c++ code:
#include "cereal/cereal.hpp"
That should do it.
I have the following directory structure:
my_dir
|
--> src
| |
| --> foo.cc
| --> BUILD
|
--> WORKSPACE
|
--> bazel-out/ (symlink)
|
| ...
src/BUILD contains the following code:
cc_binary(
name = "foo",
srcs = ["foo.cc"]
)
The file foo.cc creates a file named bar.txt using the regular way with <fstream> utilities.
However, when I invoke Bazel with bazel run //src:foo the file bar.txt is created and placed in bazel-out/darwin-fastbuild/bin/src/foo.runfiles/foo/bar.txt instead of my_dir/src/bar.txt, where the original source is.
I tried adding an outs field to the foo rule, but Bazel complained that outs is not a recognized attribute for cc_binary.
I also thought of creating a filegroup rule, but there is no deps field where I can declare foo as a dependency for those files.
How can I make sure that the files generated by running the cc_binary rule are placed in my_dir/src/bar.txt instead of bazel-out/...?
Bazel doesn't allow you to modify the state of your workspace, by design.
The short answer is that you don't want the results of the past builds to modify the state of your workspace, hence potentially modifying the results of the future builds. It'll violate reproducibility if running Bazel multiple times on the same workspace results in different outputs.
Given your example: imagine calling bazel run //src:foo which inserts
#define true false
#define false true
at the top of the src/foo.cc. What happens if you call bazel run //src:foo again?
The long answer: https://docs.bazel.build/versions/master/rule-challenges.html#assumption-aim-for-correctness-throughput-ease-of-use-latency
Here's more information on the output directory: https://docs.bazel.build/versions/master/output_directories.html#documentation-of-the-current-bazel-output-directory-layout
There could be a workaround to use genrule. Below is an example that I use genrule to copy a file to the .git folder.
genrule(
name = "precommit",
srcs = glob(["git/**"]),
outs = ["precommit.txt"],
# folder contain this BUILD.bazel file is tool which will be symbol linked, we use cd -P to get to the physical path
cmd = "echo 'setup pre-commit.sh' > $(OUTS) && cd -P tools && ./path/to/your-script.sh",
local = 1, # required
)
If you're passing the name of the output file in when running, you can simply use absolute paths. To make this easier, you can use the realpath utility if you're in linux. If you're on a mac, it is included in brew install coreutils. Then running it looks something like:
bazel run my_app_dir:binary_target -- --output_file=`realpath relative/path/to.output
This has been discussed and explained in a Bazel issue. Recommendation is to use a tool external to Bazel:
As I understand the use-case, this is out-of-scope for building and in the scope of, perhaps, workspace configuration. What I'm sure of is that an external tool would be both easier and safer to write for this purpose, than to introduce such a deep design change to Bazel.
The tool would copy the files from the output tree into the source tree, and update a manifest file (also in the source tree) that lists the path-digest pairs. The sources and the manifest file would all be versioned. A genrule or a sh_test would depend on the file-generating genrules, as well as on this manifest file, and compare the file-generating genrules' outputs' digests (in the output tree) to those in the manifest file, and would fail if there's a mismatch. In that case the user would need to run the external tool, thus update the source tree and the manifest, then rerun the build, which is the same workflow as you described, except you'd run this tool instead of bazel regenerate-autogenerated-sources.
I've cloned the OpenLayers 3 repo and merged the latest from master. There exists a recently merged pull request that I'm interested in exploring, but I'm not sure how to create a regular old comprehensive, non-minified build.
Does anyone know how to create a non-minified, kitchen sink (everything included) build for OpenLayers?
(similar to ol-debug.js).
You can use the ol-debug.json config to concatenate all sources for the library without any minification.
node tasks/build.js config/ol-debug.json ol-debug.js
Where the ol-debug.json looks like this:
{
"exports": ["*"],
"umd": true
}
The build.js task generates builds of the library given a JSON config files. The custom build tutorial describes how this can be used to create minified profiles of the library. For a debug build, you can simply omit the compile member of the build config. This is described in the task readme:
If the compile object is not provided, the build task will generate a "debug" build of the library without any variable naming or other minification. This is suitable for development or debugging purposes, but should not be used in production.
So I am running into an issue when I go to build my projects using tfs build controller using the Output location "AsConfigred" it will not detect my unit tests. Let me give a little info on my setup.
TFS 2013 Update 2, Default Process Template
Here is a few screenshots that can hopefully help fill in what I can't in typing. I am copying my build out to a file share on our network so that we can use other utilities use the output. I don't want to use "PerProject" or "SingleFolder" because they mess up the file structure we have configured (These both will run the tests). So i have the files copy to folder names "SingleOutputFolder" which is a child of the DropLocation. I would like to be able to run from the drop folder or run from the bin folder for each of my tests (I don't care which). However it doesn't seem to detect/run ANY of the tests. Any help would be greatly appreciated. Please let me know if you need any additional information.
I have tried using ***test*.dll, Install\SingleFolderOutput**.test.dll, and $(TF_BUILD_DROPLOCATION)\Install\SingleFolderOutput*test*.dll
But I am not sure what variables are available and understand where the scope of its execution is.
Given that you're using Build Output location set to AsConfigured you have to change the default values of the Test sources spec setting to allow build to find the test libraries in the bin folders. Here's an example.
If the full path to the unit test libraries is:
E:\Builds\7\<TFS Team Project>\<Build Definition>\src\<Unit Test Project>\bin\Release\*test*.dll
use
..\src\*UnitTest*\bin\*\*test*.dll;
This question was asked on MSDN forums here.
MSDN Forums Suggested Workaround
The suggested workaround in the accepted answer (as of 8 a.m. on June 20) is to specify the full path to the test projects' binary folders: For example:
C:\Builds\{agentId}\{teamProjectName}\{buildDefinitionName}\src\{solutionName}\{testProjectName}\bin*\Debug\*test*.dll*
which really should have been shown as
{agentWorkingFolder}\src\{relativePathToTestProjectBinariesFolder}\*test*.dll
However this approach is very brittle, for the following reasons:
Any new test projects you add to the solution will not be executed until you add them to the build definition's list of test sources:
It will break under any of the following circumstances:
the build definition is renamed
the working folder in build agent properties is modified
you have multiple build agents, and a different agent than the one you specified in {id} runs the build
Improved Workaround
My workaround mitigates the issues listed in #2 (can't do anything about #1).
In the path specified above, replace the initial part:
{agentWorkingFolder}
with
..
so you have
..\src\{relativePathToTestProjectBinariesFolder}\*test*.dll
This works because the internal working directory is apparently the \binaries\ folder that is a sibling of the \src\ folder. Navigating up to the parent folder (whatever it is named, we don't care) and back in to \src\ before specifying the path to the test projects binaries does the trick.
Note: If you have multiple test projects, you add additional entries, separated with semicolons:
..\src\{relativePathToTestProjectONEBinariesFolder}\*test*.dll;..\src\{relativePathToTestProjectTWOBinariesFolder}\*test*.dll;..\src\{relativePathToTestProjectTHREEBinariesFolder}\*test*.dll;
What I ended up doing was adding a post build event to copy all of the test.dll into the staging location folder in the specific build that is basically equivalent to where it would go on a SingleFolder build and do that on each test project.
if "$(TeamBuildOutDir)" == "" (
echo "Building Interactively not in TFS"
) else (
echo "Building in TFS"
xcopy "$(TargetDir)*.*" "$(TeamBuildBinaries)\" /Y /E /S
)
MSBUILD parameter in the build def that told it to basically drop in the folder that TFS looks for them.
/p:TeamBuildBinaries="$(TF_BUILD_BINARIESDIRECTORY)"
Kept the default Test assembly file specification:
**\*test*.dll
View this link for the information on the variable that I used and what relative path it exists at.
Another solution is to do the reverse.
Leave all of the files in the root so that all of the built in functionality works. There is more than just test execution in there. What about static code analysis, impact analysis..among others. You would have to do something custom for them all.
Instead use a pre-drop powershell script to create your Install arrangement from the root files.
If it is an application then you can use the _ApplicationFolder Nuget package to create an _PublishApplications folder same as you get for web applications.
I'm moving from StarTeam to SVN, and I've decided to take snapshots of each of our releases. However, I'm experiencing a problem when dealing with files which existed in revision 1 were deleted for revision 2.
How do I commit the snapshots when files are missing?
I've tried fully deleting the trunk/src/ folder and then replacing it with the new /trunk/src/ folder, but that seems to cause conflicts with missing files. When I
svn add" everything, TortoiseSVN seems to detect that the files were missing, and when I commit it seems to attempt to delete the missing files, but it seems to fail. Presumably this is because it's trying to delete directories after deleting the files in that directory?
I'm getting the following error:
deleting C:\trunk\src\myfile.h // this one's okay
deleting C:\trunk\src\res
Commit failed (details follow):
Directory 'C:\trunk\src\res' is out of date
Item '/trunk/src/res' is out of date
You have to update your working copy first.
What is a solution to this problem? Surely I'm not the first to run into this issue, but I cannot seem to find anything on google or stackoverflow. Some people suggest running a script to do it, but I'm still confused about the process. Do I need to delete my old trunk folder so that the missing files will get removed locally? Or should I diff and delete with a script?
Thanks!
Old (unclear) post: Migration to SVN, confused about deleting old files
Edit:
This is going from one snapshot to the next. I'm migrating from a different repository (StarTeam), so I didn't have anything in trunk. I just want to check in all the different snapshots and have the files which were deleted be removed as well. Isn't it a bad idea to tag if I don't have anything in trunk?
There's actually a Subversion script that does this for you called svn-merge-repos.pl
I'm not 100% sure you're understanding the concept of how Subversion works. Did you go through the Subversion book?
In Subversion, there are no real tags/labels or branches metadata like you find in many version control systems. Instead, you put tags and branches in their own directories. To create a branch or tag, you copy what you want to branch or tag into the directory:
# Creating a branch for 2.0 development from trunk
$ cp http://server/svn/module/trunk http://server/svn/module/branches/2.0
# Tagging my 2.0 development as 2.0.1
$ cp http://server/svn/module/branches/2.0 http://server/svn/module/tags/2.0.1
In theory, you can simply create a new branch or tag directory for each release and branch you're working on without a need for merging repositories. That's what I did when I did a StarTeam to Subversion conversion. The problem is that you lose the relationship between say revision 2.0.1 and 2.0.2 since they don't share a common history. 99% of the time, that's not really a problem, and you can always go back to the original StarTeam archive if you need anything. In a few months, no one will care.
However, if you know the relationship between branches and tags, and want to keep that information, you'll have to do the two step script I described above.
For example, you have a 2.0 branch that comes from trunk, a 2.0.1 tag, a 2.0.2 tag, and a 2.0.3 tag, you might want to do this:
Put branch 2.0.1 release onto trunk.
Copy trunk to branches/2.0.
Put the next branch on trunk and copy it to its branch (use the svn-merge-repos.pl script)
Finally, put the current trunk.
Now go to that branches/2.0 directory and copy that to tags/2.0.1. Using the svn-merge-repos.pl script, create the 2.0.1 release on branches/2.0 and copy that to tags/2.0.2. Keep going until you get to the tip of the 2.0 branch.
That takes a lot longer to do, but it's feasible. Last time I did that, it took me about a week and a half to do the entire conversion. Fortunately, I did the trunk first and then the active release which I could do in a day. Then, worked my way back to the less active stuff.
I'm certainly not an expert on SVN, but I believe that you've got to issue the svn delete command for the items you want to delete upon the next commit. It's described here.
You should create a tag instead.
EDIT:
svn status list all changes
you can print all deleted files like this:
svn status | grep -E '^!' | awk '{print $2}' > /tmp/changes
You can do a svn delete on each file with a batch ;)
EDIT: (SVN Migration How-To)
Let's assume you have a project named projectA, and a blank SVN repository.
First, you should create a Folder structure like this:
projectA/
|-- branches
|-- tags
|-- trunk
Then import your project files.
projectA/
|-- branches
|-- tags
`-- trunk
|-- README
`-- src
After the import you can create a tag to "mark" the initial release:
projectA/
|-- branches
|-- tags
| `-- release-1
`-- trunk
|-- README
`-- src
After this point, you should edit your files as you like.
When you think all your edits are "Good", then commit.
After some commits, if you want to release, create a tag.
Trunk should always contain the up-to-date version of your code.
I highly recommend you to use TortoiseSVN on Windows if you are a rookie ;)
You should use svn-load-dirs.pl:
"This Perl script is designed to load a number of directories into
Subversion. This is useful if you have a number of .zip's or
tar.{Z,gz,bz2}'s for a particular package and want to load them into
Subversion."
Basically what you do is:
export all you tags chronologically
create an empty repository
use svn_load_dirs.pl to "stack" tag after tag in your subversion.
svn_load_dirs.pl creates a single revision for each tag and you can as well create a (subversion-)tag after each import. It will keep track on all deleted and added files and will perform the appropriate svn actions. This means you can explicitly start with an empty trunk
From your question, it looks like that you are completing misusing SVN. There is no point in deleting the entire trunk/src folder and replacing it with a new one except in case of a huge change in the code.
Instead, create a tag for the release.