I pulled the engine's source repository and completed the build, which took over an hour; then I deleted the Intermediate folder and rebuilt, again taking over an hour. I replaced the Intermediate folder of the first build with the Intermediate folder of the second build, and built again, which only took 9 minutes. This means that the Intermediate folder does hold the build cache.
I backed up the Intermediate folder, re-pulled the engine source code, and put Intermediate into the source repository, and the build still took over an hour. Why is this happening?
Are there other files besides the Intermediate folder that determine whether this build will use Cache?
Every time I pull a new engine source code repository, I will run Setup.bat first, so please suddenly it affects this problem
I use an onsite build agent to perform my VSTS builds. This is working fine, sort of. I have 2 build definitions, one of which is a clone of the other and the only difference between the 2 is the solution that is built, all other parameters are exactly the same.
One of my builds completes without error and creates build artifacts and compiled code zip files in the 'build/1/a' artifacts folder. My other build completes without error BUT there are no build artifacts and compiled zip files created, my 'build/3/a' directory for this build is empty and I cannot see anywhere in the logs where the tasks to create this was executed, if at all. This did used to work before I cloned the build definition though. These are the MSBuild arguments that I have defined for both build definitions;
/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\" /p:PackageTempRootDir="D:\Build\SiteManagerDev"
The only difference between them is the last parameter '/p:PackageTempRootDir'.
I have tried switching between the 2 directories for both build definitions to make sure it is not a permission error and the definition that finishes correctly works against either directory. I am starting to tear my hair out now and I have even tried creating a completely new build definition for the solution that creates an incomplete build and it is the same result, it is almost as if it is the solution itself that is causing the issue?
Any help would be greatly appreciated.
UPDATE: 05/02/2017
I think I finally understand what is going on! Question, if a build is manually triggered, not by a check-in trigger, if there have been no changes made to the code, and even though the build is executed, does this prevent the build artifacts from being created again because nothing has changed? The reason I ask is because I have found a strange in-house housekeeping routine that goes and deletes the contents of the 'D:\Build\1\a' directory on our build machine on a regular basis (I have no idea why!) and this results in there being nothing to publish UNTIL there is a code change checked in and then they are generated again! What a waste of everyone time this has been, my apologies and thank you for your help.
I'm trying to automate our main project build (C++) via Team Build system (TFS 2013).
However, I see that a couple of projects are always built, even if no code change has occurred, while this does not happen using VS2013 on my development machine. This would cause some headache since binaries would always be generated and sent to test team even if not really modified.
Enabling "diagnostic" verbosity in build output, I see that the two project exhibit different behavior.
In the first project the log says that all .cpp files are rebuilt because the .PCH file has been modified (although no change happened). I could try disabling the PCH but would really avoid it if possible. Besides, not going to the root cause of the error would leave an open door to the error representing again and again.
In the second project, we have a pre build step that generates a .h file. However, prebuild steps should not run if no change in the code has been detected (https://msdn.microsoft.com/en-us/library/42x5kfw4.aspx), so happens indeed on my development machine. On the build machine, instead, the prebuild step is executed, the .h is generated and this forces a complete rebuild.
In the team build settings I have "Clean workspace=false", "Clean build=false". I also tried "/p:IncrementalBuild=True" in MSBuild settings, but this did not fix the issue.
Note - I already looked at Visual Studio Rebuilds unmodified projects and VS2010 always rebuilds solution? before posting.
According to your info about project2, the pre build step executed on the build agent and not executed on your local machine. There must be something different with your local build environment and build agent environment. This may be the root cause.
Suggest you to double check it and make sure the environment is the same with each other. And try it again.
I am using Jenkins CI as the build server on a project that I am working on, I am also using Klocwork as a static analysis tool to identify deviations from our coding standards.
At present Jenkins has two builds (being performed in separate directories), a full build on a nightly basis that wipes out the workspace and performs a fresh checkout and full rebuild of everything.
In addition to the overnight build I also have an incremental build happening within 15 mins of a check in. Both builds are using the Klocwork analysis tool.
Klockwork works by displaying a list of potential issues which can then be fixed or chosen to be ignored if they are not applicable to the project, when issues are being ignored Klocwork uses the build file paths to remember where the issues that have been ignored reside. This means that when in Klocwork once I have ignored a warning in the full build and an incremental build is triggered the warning once again returns as the build path is different.
The most sensible solution I can see to this is for Jenkins to perform its full build on a nightly basis but for the incremental build to do an update in the full build location and to then do an incremental build - in the same way that an IDE on a PC functions.
The problem is that I have Jenkins running the full build and the incremental build as two separate jobs which causes them to check out into different locations and I cannot find a way of having the two jobs share a common directory.
Also I cannot find a way of having a single job that performs a nightly full checkout and rebuild, and an incremental build with an update on check in at the same time.
Is anyone familiar with a way of making Jenkins use a common source directory across multiple jobs?
Many thanks,
Pete.
Here's what I did.
Used one job to only check out source code.
In other jobs configuration settings', I set an environment variable that pointed to the workspace directory tree that contains the first job's source code (command line access to the Jenkins server is helpful here to figure out where it is, but not necessary). Then in my config scripting in Jenkins in the regular jobs, I 'cd' to that location and use the environment variable as path to all files so these other jobs would use the first job's checked out code.
Used locks, so regular jobs would not be running at same time as the check-out code job.
Since some results files (because of the tests being run) were generated and created in the source code tree because of the specifics of some of these jobs, in the config post-action script, I copied/moved the desired results back to the workspace of the job that should have them so I could process these results in the right job.
Worked for me.
You can easily make the two builds share the same build-area,
simply by extracting the files in both build-jobs to a shared location.
I strongly advise NOT to do this, as you can quickly get to a situation where
the nightly-build is cleaning the build-area while the incremental build is still running
(or that the incremental-build is checking-out sources while the nightly is still running).
Suggest you connect Klockwork only to one of the build-jobs (the nightly, probably)
so to avoid duplicate warnings.
We recently moved to an automatic build system (something internal, not Hudson or Teamcity, yet).
Our version is stored in a header file and is included by some cpp and resource files. It is also used by the installer.
Its format is A.B.C.D where:
A didn't change in years.
B changes rarely (major version).
C changes with minor versions.
D changes when a new minor version (bug fix) is delivered to QA.
Up until now, the one incharge of building a new version, incremented C/D by hand (D being the more common) before starting the build, checked in the change and then started the build. The version stayed the same until that person built the app successfully.
Naturally with the move to an automatic build system I'd like to get rid of the manual step of changing the version number.
How should this be approached?
Do I increment D whenever a new build is made, whether it's a QA build or an internal-test build (i.e. I'm working on some feature and I'd like to test I haven't broke anything)?
Is the increment step a task in the automatic build system?
After incrementing, should I commit the version file?
How do I avoid having a lot of noise in my version control? I don't want tons of "version incremented" commits.
What do I do if the build failed? Still increment the version and commit?
Do I increment D whenever a new build is made, whether it's a QA build or an internal-test build (i.e. I'm working on some feature and I'd like to test I haven't broke anything)?
The Eclipse Foundation adds an E element, the date and time of the build. I think that's a good idea for the internal-test builds. It's up to you if you want to use E for the QA builds.
Is the increment step a task in the automatic build system?
Seems logical, but you have to have some way of telling that task what kind of build you're doing.
How do I avoid having a lot of noise in my version control? I don't want tons of "version incremented" commits.
Commit the version control file with the source code.
Basically, your development build process should proceed in the following order.
Build the product from the development source code.
If the build succeeds, increment the version number.
Commit the source code and the version control file.
Build the product again from your version control system.
If the build fails, back out the source code and version control file commit.
This tests your build and your build process. The second build should never fail, but if it does, there's a problem in the process.
Your production build process would start at the 2nd step, skipping the 3rd step.
What do I do if the build failed? Still increment the version and commit?
My E is auto-incrementing. :-) I'd say no for the other elements A, B, C, or D.
Do I increment D whenever a new build is made, whether it's a QA build or an internal-test build (i.e. I'm working on some feature and I'd like to test I haven't broke anything)?
Yes, change your process so that D increments with every build (successful or not) rather than with every delivery to QA.
It can be quite frustrating having several builds, some working some not and not being able to tell them apart because the failed build is the same id as the good one, well eventually.
Then you don't even have to consider if it was on the same day or in the same hour.
Is the increment step a task in the automatic build system?
I'd have the build system auto increment the build number (D) only.
After incrementing, should I commit the version file?
How do I avoid having a lot of noise in my version control? I don't want tons of "version incremented" commits.
The version control storage is all about recording the detailed noise.
I'd have the version update checked in, this can make a reasonable tag visible in SVN of what build the previous changes where included in, have the build system ignore checkins by the build system, or those identified as the version update checkin.
Then to view the version history you should have an appropriate tool that allows you to filter the history to show you the view you need, in some cases excluding the version commit tags.
If you choose not to commit the version number for each build, then it might be a good idea to maintain the version number in a separate file to avoid accidental updates.
What do I do if the build failed? Still increment the version and commit?
Still increment the version number, I wouldn't commit the version number unless it was a successful build. You can have a variety of failures outside of source change in version control that don't need to be recorded - build server out of disk, server crash, compiler got all wobbly in the knees building 32 and 64 bit, debug and release aix, linux and windows builds at the same time...
You could consider to use the convention for .NET assemblies, as described in the documentation for class System.Version. Quote:
Build [your C]: A difference in build number represents a recompilation of the same source. Different build numbers might be used when the processor, platform, or compiler changes.
Revision [your D]: Assemblies with the same name, major, and minor version numbers but different revisions are intended to be fully interchangeable. A higher revision number might be used in a build that fixes a security hole in a previously released assembly.
How are you going to automate this? I mean, what system would know that "this build is the release build!". It would seem to me that all your digits in a version is relevant. If the next release (D + 1) requires two builds, then would A.B.C.D+2 be the next version? Sounds fishy to me. I would rather add the build number on top of the version instead, if it's really necessary to have this information on your DLLs and EXEs.
I don't think the build number is a relevant piece of information to have attached to the binary, unless you distribute files of version A.B.C.D from different builds (which you shouldn't do anyway!)
I would setup the build server to store the artifacts (DLLs, EXEs, MSIs, PDBs, etc) in a directory, whose name includes the build number and version, and then burn DVD/whatever from there. If you ever need to back track from a version to a specific build, you can use this information, provided that you keep an archive of your releases (recommended!).
I would recommend the use of autorevision.
You could still keep the A.B.C.D format for your tags and use the script to create header files that are generated at build time that have the needed info in them.