Is there a way to have the artifacts of a TeamCity build referenced (but not copied) as a dependency for another TeamCity build?
Some background: I've been trying to reduce the build times in a couple of our TeamCity configurations. It's a c++ program that depends on several 3rd party libraries, which our Sys Admin has been loathe to install on the build machine.
Our first run had the libraries zipped up and uncompressed / compiled as a build step within the configurations. This takes a while, so the Sys Admin auggested moving the 3rd party lib decompression / compilation into a separate configuration and setting the artifacts of that build as a dependency for the build I'm trying to speed up.
Things are worse under this build configuration, however. The size of the expanded / compiled 3rd party libs (over 1GB) actually makes the original configuration speedier by over 10 minutes. If there was a way to just reference the artifact directory without copying stuff over, that would be fantastic.
Do not use artefact dependencies.
Instead create two or more build configurations (one for your main application, one or more for the 3rd party libraries) then create snapshot dependencies between them, configuring it to Run build on the same agent.
Doing this will ensure the binaries from your 3rd party libraries are always available on the local file system and always up-to-date (yet without being constantly rebuilt - assuming no source changes).
You should be able to locate the 3rd party binaries easy enough in the checkout directory.
The reason artefacts are slow is they get uploaded to the central central server, then downloaded by agents. Obviously not a good fit for a 1GB of 3rd party libraries.
As far as I know there is no way to prevent artifact copy from server to agent: it will be impossible for the compiler / linker to find dependencies...
In my opinion you can take the best of both configurations by publishing zipped artifacts (just postpone a ".zip" to the destination path) and fetching them from "last successful build".
This way you will trigger the lib recompile only on respective source code changes (decreasing overall build time) and artifacts will be transferred as a compressed archive (decreased transfer time).
Maybe you can optimize further by building each lib separately from others: only libs with pending changes will be recompiled.
Related
Currently building tensorflow from source need to connect to internet to download some dependencies. Every time when I rebuild it, bazel will delete what have been downloaded and re-download them.
I wonder if there is any possible way to avoid this by pre-download all the dependencies and just build it without internet?
It is possible to pre-download 3rd party dependencies, as explained here.
In tensorflow v0.11.0, they are listed in "tensorflow/workspace.bzl". After downloading files you need, replace links like this:
url = "http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz",
to
url = "file:////mnt/a/usr/bzip2-1.0.6.tar.gz",
Note, that there's about 20 dependencies to download.
It seems to be a case not well supported by TensorFlow, apparently one way is to build Docker or other VM environment, and distribute that --
https://github.com/tensorflow/tensorflow/issues/3194#issuecomment-231326381
Bazel automatically caches the external dependencies it downloads. Is it possible you are:
Moving the tensorflow source around
Changing the BUILD files it uses for external repositories
Building different targets (that might require other dependencies) each time?
If none of those seem likely, can you add the output for running identical bazel builds twice in a row where you're seeing re-downloading behavior, using --explain?
So, we are in this process of migrating XAML Builds to vNext (2015) Builds on TFS, and we are trying to "do things as clean as possible", since we had many, many customizations on the XAML builds that could be avoided and actually gave us problems along the way.
One major issue we are facing is with paths and "global files". Let me explain:
There are some files that, for commodity reasons, we have on a single place and every SLN file on that Collection refers them. Those files are such ones as Code Analysis RuleSets, Signing Files (SNK), etc. So the change is made in one place only and it affects every build.
Well, in XAML Builds we have a Build that runs with CI that downloads (Gets) those files, and since we hammered-in the same exact pathing for TFS and Machine (with a environment variable for the beginning of the path), the path is the same on the Developers and Build machines. However, this creates dependencies between builds and workspace issues.
My question here is, is there a configuration that I am missing that allows referring to files in other branches other than the build one? Since I’m trying to keep the build machines as “disposable” as possible, it’s running with an Agent Config Out of the Box: No custom paths, no hardwiring.
I already tried referring the files directly with their source control path, for example. The only options I’m seeing are either creating a PowerShell/CMD Script that downloads those files right into the same folder as the SLN or keeping it “as it is” and use relative paths putting a “Build” Build Step before the actual Build Step so it downloads the files to the server.
Isn’t there an “Elegant” way of doing this? Or is our methodology wrong from the get go?
You can add a Copy Files step to copy the files that the build needs:
I have a project that depends on many external libraries like GLFW3, GLEW, GLM, FreeType2, zlib etc. It would be best to store/share installed dependencies between jobs so it wouldn't have to download/install them all the time which takes about half of the time. I can see couple ideas how to handle it:
a) for each job for each build download dependencies and install them
b) put dependencies (sources) inside my repo and have little speedup becouse i will no longer have to download them from outside servers (still have to compile and install them)
c) compile them by hand, put on some server and just download right package for each build
a) it leaves least work for me to update dependencies for building and testing, allows to use newest versions to build my project but it takes most time (both compiling and downloading)
b) bloats repository with extra code (not mine), gives little speedup (downloading is not that slow usually), adds manual work to update dependencies, i guess worse then a)
c) fastest but requires most work from me to constantly keep built dependencies up to date and upload them on fast server (also diffrent per each build task (compiler etc)), allows for fastest builds (just download & copy/install).
So, how are you managing your external dependencies and keep them up to date for your travis builds?
Note that i use free version of Travis and kinda need sudo for updating cmake, gcc etc. and installing dependencies... Could somehow trick CMake to use local versions of dependencies instead of /usr/... but this somehow bloats CMake which i believe should be very simple and clear.
Let's call the entire set of dependencies your build requires at some point in time the dependency "lineup".
I'd separate the use of (a certain version of) the lineup in your builds from the task of updating the lineup version (when a new version of a dependency is needed) - mixing them unnecessarily complicates the picture IMHO (I'm assuming that many of your builds would use the same dependency lineup version).
Let's look at the use of the lineup first.
From your description the installation of the (already built) dependencies is common to all 3 options and executed at every build, so let's put that aside for now.
The difference between your approaches remains only in how you obtain the built dependencies:
a - download them from outside servers - at every build
b - build them from repo sources - at every build
c - download them from a local fast server - at every build
It seems a and c are fundamentally the same except c would be faster due to local access. Most likely b would be the slowest.
So it looks like c is the winner from the build speed prospective in this context.
Now the question would be if/how you can get your built dependencies lineup on the local server faster for the c approach. I see 2 options (unsure if they're both possible in your context):
download dependencies already built (as you would in your a
approach) from outside servers (effectively just caching them on the
local servers)
building dependencies locally from source, except you don't have to neither place those sources in the same repo as your project nor build them for every project build (as in your b approach) - you only need to do this when you update your lineup (and only for new versions of the respective dependencies).
You could also look into mixing of the 2 options: some dependencies using option 1, others using option 2, as you see fit.
Now, if you're using VM (or docker or similar) images for your build machines and you have control over such images it might be possible to significantly speedup your builds by customizing these VM images - have the dependency lineup installed on them, making them immediately available to any build on a machine running such customized image.
Finally, when time comes to update your dependency lineup (which, BTW, should be present in your a and b approaches, too, not only in the c one) you'd have to just download the new dependencies versions, build them if needed, store the built dependencies on the local fast server and, if the customized VM image solution works for you, update/re-create the customized VM image with the installation of the new dependency lineup.
I want to make it easy for others to work on my repository. However, since some of the compiled dependencies are over 100mb in size, I cannot include them into the repository. Github rejects those files.
What is the best way to handle large binaries of dependencies? Building the libraries from source is not easy under Windows and takes hours. I don't want every developer to struggle with this process.
I've recently been working on using Ivy (http://ant.apache.org/ivy/) with C++ binaries. The basic idea is that you build the binaries for every build combination. You will then zip each build combination into a file with a name like mypackage-windows-vs12-x86-debug.zip. In your ivy.xml, you will associate each zip file with exactly one configuration (ex: windows-vs12-x86-debug). Then you publish this package of multiple zip files to an Ivy repo. You can either host the repo yourself or you can try to upload to an existing Ivy repo. You would create a package of zip files for each dependency, and the ivy.xml files will describe the dependency chain among all the packages.
Then, your developers must set up Ivy. In their ivy.xml files, they will list your package as a dependency, along with the configuration they need (ex: windows-vs12-x86-debug). They will also need to add an ivy resolve/retrieve step to their build. Ivy will download the zip files for your package and everything that your package depends on. Then they will need to set up unzip & move tasks in their builds to extract the binaries you are providing, and put them in places their build is expecting.
Ivy's a cool tool but it is definitely streamlined for Java and not for C++. When it's all set up, it's pretty great. However, in my experience as a person who is not really familiar with DevOps at all, integrating it into a C++ build has been challenging. I found that it was easiest to create simple ant tasks that do the required ivy actions, then use my "regular" build system (make) to call those ant tasks when needed.
So I should also mention that the reason I looked into using Ivy was that I was implementing this in a corporate environment where I couldn't change system files. If you and your developers can do that, you may be better off with a RPM/APT system. You'd set up a repo and get your developers to add your repo to the appropriate RPM/APT config file. Then they would run commands like sudo apt-get install mypackage and apt-get would do all the work of downloading and installing the right files in the right places. I don't know how this would work on Windows, maybe someone has created a windows RPM/APT client.
If your project requirements for a large application with many 3rd party dependencies included:
1) Maintain a configuration
management system capable of
reproducing from source bit-for-bit
identical copies of any build for 25
years after the original build was run and
2) Use Maven2 as a build
tool to compile the build and to
resolve dependencies
What process would need to be followed to meet those requirements?
25 years? Let's see, I think I have my old Commodore 64 sitting around here somewhere...
Seriously though - if this is a real question, then you have to consider the possibility that the maven central repository will at some point in the future go away. Maven is heavily reliant on the maven central repository.
You will also need to archive any tools (besides maven) used to create the build. An ideal build process will create an identical binary file at any time, whether it is next week or in 25 years. In practice, there are a lot of things that can prevent you from being able to reliably reproduce your builds.
1) Use a maven repository manager to host all dependencies, and back up the contents of the maven repository.
2) Archive any tools used to create the build. Basically maven and the jdk, but if you are using any other maven plugins like NSIS or Ant, then you need to archive those as well. If you are creating any platform specific binaries (like using NSIS), then you need to archive those tools, and probably the OS used to run the tools.
3) Archive your source code repository and make sure the software needed to run it is also archived as well.