Build systems like make use timestamp check if a dependency is changed during two build. Here are a few common issue I run into with timestamp
Open a file, make some change, but later, I decided it is not good. I revert the change, for example, git checkout -- file if I am using git for the project.
Open a file, I just accidentally hit keyboard shortcut for save of the editor
Either way, the file's timestamp is changed. If now I want to build the project, everything depending on that file need to be rebuilt. This often means the whole project.
Is there anyway around these issues? For example, a build system using a version control for checking file change, preferably git. Or any other solutions to the above issues are welcome.
Many thanks in advance.
SCons uses checksums, not timestamps by default. However, checksumming requires reading all the contents of all the source files on disk, and that is much slower than simply reading directory entries, which is why most build systems use timestamps.
Software Build Systems gives a good overview of these issues.
Related
I think that it would be a good idea to have a track of the build file of a project. But, I've read that build files should not be public, since they can contain "sensible" information.
Is recommendable to have the build file in the same repository of the source code? Even if the repo is public? If it is not recommendable, would I have to have an independent repo for each project only for the build file?
Edit: the projects would be plugins for Moodle. And, I've never used build systems before, as I guess you have already deduced.
You didn't mention what kind of build system you're using. Nevertheless, having the build file along the project source has advantages:
The build file is versioned the same way the source code is
You don't have to keep track of which build file belongs to which project
You shouldn't have any sensible information within a build file - most systems allow you do have encrypted authentication information in a build file. In the worst case, you'll have to keep the information stored elsewhere.
I'm wondering if there's a macro or a simple way to let the compiler increment either major, minor or revision of my code each time when I compile?
By the way I'm using the ARM compiler and uVision from Keil.
To set the version is not a compiler topic. This should be done in connection with a source code control / version control system like cvs/svn/git or others. Your build id should be connected to the content of your source code database to get reproducible builds from checkouts from your version control system. Or, if your code is not already committed to your database a dirty-tag should be provided and compiled in to give the user of the software a chance to see that this is not a controlled version.
Simply counting a value in a variable can be done by a Makefile or in pre- and post-build instructions which depends on the used IDE. Sorry, for keil I have no experience...
Define Post-Build Event to run small external program. The program has to modify specific .h file. In the header file define macros like VER_MAJOR, VER_MINOR, VER_BUILD. Date/time string can also be updated. I use this method and can control the version numbers as I wish.
IMO, you do not need to do this, especially increase a number every time you compile your code.
Set the major and minor revision manually in a header file; you should not have to do this often.
Build number should only be related to the source control revision number (i.e. you should be able to build and rebuild any revision under source control).
Imagine if you are a team of 5 developers, and everyone build and rebuild on their side what is the actual build number?
do they all update a header file ?
who is responsible of owning that header file ?
Some compilers do support features like "post build", which runs a program of your choice after compiling, but that would be tricky if your program is built from multiple source files. Not all compilers do though.
For that reason, I wouldn't actually do this sort of thing via the compiler. I'd do it in the build script (e.g. makefile) or by configuring the build settings in your IDE.
Assuming you're using make or similar, add a target something like setversion (you pick the name) that runs a program which modifies a header file which specifies the components of your version number. So typing make setversion will update your version numbers.
Optionally, that target can also - after updating the version numbers - do a make clean (i.e. delete all object files and executables) and make all (to recompile and link everything).
I also suggest avoiding changing version numbers after each recompile. Imagine you're busily testing and debugging code, and go through several rebuild cycles. Do you really want the version number updated every time you recompile even one source file? It can be done that way if you choose, but will make every rebuild take longer (and, in projects with multiple source files) you will need to take care if you want to preserve capability for incremental builds.
We have the Boost library in our side. It consists of a huge number of files which never change and only a tiny portion of it is used. We swap the whole boost directory if we are changing versions. Currently we have the Boost sources in our SVN, file by file which makes the checkout operations very slow, especially on Windows.
It would be nice if there were a notation / plugin to address C++ files inside ZIP files, something like:
// #ZIPFS ASSIGN 'boost' 'boost.zip/boost'
#include <boost/smart_ptr/shared_ptr.hpp>
Are there any support for compiler hooks in g++? Are there any effort regarding ZIP support? Other ideas?
I assume that make or a similar buildsystem is involved in the process of building your software. I'd put the zip file in the repository, and add a rule to the Makefile to extract it before the actual build starts.
For example, suppose your zip file is in the source tree at "external/boost.zip", and it shall be extracted to "external/boost", and it contains at its toplevel a file "boost_version.h".
# external/Makefile
unpack_boost: boost/boost_version.h
boost/boost_version.h: boost.zip
unzip $<
I don't know the exact syntax of the unzip call, ask your manpage about this.
Then in other Makefiles, you can let your source files depend on the unpack_boost target in order to have make unpack Boost before a source file is compiled.
# src/Makefile (excerpt)
unpack_boost:
make -C ../external unpack_boost
source_file.cpp: unpack_boost
If you're using a Makefile generator (or an entirely different buildsystem), please check the documentation for these programs for how to create something like the custom target unpack_boost. For example, in CMake, you can use the add_custom_command directive.
The fine print: The boost/boost_version.h file is not strictly necessary for the Makefile to work. You could just put the unzip command into the unpack_boost target, but then the target would effectively be phony, that is: it would be executed during each build. The file inbetween (which of course you need to replace by a file which is actually present in the zip archive) ensures that unzip only runs if necessary.
A year ago I was in the same position as you. We kept our source in SVN and, even worse, included boost in the same repository (same branch) as our own code. Trying to work on multiple branches was impossible, as it would take most of a day to check-out a fresh working copy. Moving boost into a separate vendor repository helped, but it would still take hours to check-out.
I switched the team over to git. To give you an idea of how much better it is than SVN, I have just created a repository containing the boost 1.45.0 release, then cloned it over the network. (Cloning copies all of the repository history, which in this case is a single commit, and creates a working copy.)
That clone took six minutes.
In the first six seconds a compressed copy of the repository was copied to my machine. The rest of the time was spent writing all of those tiny files.
I heartily recommend that you try git. The learning curve is steep, but I doubt you'll get much pre-compiler hacking done in the time it would take to clone a copy of boost.
We've been facing similar issues in our company. Managing boost versions in build environments is never going to be easy. With 10+ developers, all coding on their own system(s), you will need some kind of automation.
First, I don't think it's good idea to store copies of big libraries like boost in SVN or any SCM system for that matter, that's not what those systems are designed for, except if you plan to modify code in boost yourself. But let's assume you're not doing that.
Here's how we manage it now, after trying lots of different methods, this works best for us.
For every version of boost that we use, we put the whole tree (unzipped) on a file server and we add extra subdirectories, one for each architecture/compiler-combination, where we put the compiled libraries.
We keep copies of these trees on every build system and in the global system environment we add variables like:
BOOST_1_48=C:\boost\1.48 # Windows environment var
or
BOOST_1_48=/usr/local/boost/1.48 # Linux environment var, e.g. in /etc/profile.d/boost.sh
This directory contains the boost tree (boost/*.hpp) and the added precompiled libs (e.g. lib/win/x64/msvc2010/libboost_system*.lib, ...)
All build configurations (vs solutions, vs property files, gnu makefiles, ...) define an internal variable, importing the environment vars, like:
BOOSTROOT=$(BOOST_1_48) # e.g. in a Makefile, or an included Makefile
and further build rules all use the BOOSTROOT setting for defining include paths and library search paths, e.g.
CXXFLAGS += -I$(BOOSTROOT)
LFLAGS += -L$(BOOSTROOT)/lib/linux/x64/ubuntu/precise
LFLAGS += -lboost_date_time
The reason for keeping local copies of boost is compilation speed. It takes up quite a bit of disk space, especially the compiled libs, but storage is cheap and a developer losing lots of time compiling code is not. Plus, this only needs to be copied once.
The reason for using global environment vars is that build configurations are transferrable from one system to another, and can thus be safely checked in to your SCM system.
To smoothen things a bit, we've developed a little tool that takes care of the copying and setting the global environment. With a CLI, this can even be included in the build process.
Different working environments mean different rules and cultures, but believe me, we've tried lots of things and finally, we decided to define some kind of convention. Maybe ours can inspire you...
This is something you would not do in g++, because any other application that wants to do it would also have to be modified.
Store the files on a compressed filesystem. Then every application gets the benefit automatically.
It should be possible in an OS to allow transparent access to files inside a ZIP file. I know that I put it in the design of my own OS a long time ago (2004 or so) but never got it to a point where it was usable. The downside is that seeking backwards in a file inside a ZIP is slower as it's compressed (and you can't rewind the compressor state, so you have to seek from the start instead). This also makes using a zip-inside-a-zip slow for rewinding and reading. Fortunately, most cases just read a file sequentially.
It should also be retrofittable to current OSes, at least in client space. You can hook the filesystem access functions used (fopen, open, ...) and add a set of virtual file descriptors that your own software would return for a given filename. If it's a real file just pass it on, if it's not open the underlying file (possibly again via this very function) and pass a virtual handle. When accessing the file contents, read directly from the zip file without caching.
On Linux you would use an LD_PRELOAD to inject it into existing software (at usage time), on Windows you can hook the system calls or inject a DLL into the space of software to hook the same functions.
Does anybody know if this already exists? I can't see any clear reason it wouldn't...
Is it possible to combine the following properties, and if so, how?
Store in our version control system some Visual Studio 2008 native C++ (VCPROJ) project files for the developers in our team that use this IDE.
Allow some of those developers to tweak their projects (e.g. using debug version of third-party libraries instead of the usual ones).
Make sure these modifications are done in files that are not versioned.
In other words, I would like to allow developers to tweak some settings in their projects without risking that these changes are committed.
An 'optional VSPROP' file approach seems doomed to fail, as VS2008 refuses to load projects that refer to non-existent VSPROP files...
Any other suggestion? Is this possible with VS2010?
You may not be able to do this but using a solution that generates the vcproj like CMake for example would let you do this. Scripts all your project with CMake and literally conditionally include a config file(if present for example) that developers can change on their setup.
Branches could solve this problem: you create a branch, play with different versions of third-party, merge changes to trunk if results are good.
Well, as a preliminary solution you could put the project file into something like .hgignore or .gitignore after its initial commit.
This way changes to it can't be done accidentally.
At least that's how I handle .hgignore itself.
We use a versionned "common_configuration" folder, and a script which copies project files from this "common_configuration" folder towards the "project" folder.
We have another script to copy the configuration backwards, so the developpers need to make a conscious action to commit their local changes to the global version control system.
It answers partly your needs :
The upside : we have a way to keep a common configuration for everyone, and no accidental committing of local configuration
The downside : blindly copying the files actually crushes local changes. We live with it. We could write some more clever merger tool (using diff, or xml specific manipulations), but don't want to spend to much time on supporting the deployment tools.
We're looking for a way to include some sort of build ID automatically in our builds. This needs to be portable (VC++, g++ on Linux and Mac) and automatic. VC++ is what matters most, since in the other environments we use custom Python build scripts so I can do whatever I want.
We use SVN, so we were looking at using the output of svnversion to write the revision to a header and include it. This has problems : if we put the file in SVN, it will appear as modified every time, but it would be a superfluous commit and in a sense generate an infinite loop of increasing revisions. If we don't put the file in SVN and just create it as a pre-build step, the sources wouldn't be complete, as they'd need the pre-build step or Makefile to generate that file.
We could also use __DATE__ but we can't guarantee the file that uses the __DATE__ (ie writes it to a log file) will be compiled if some other file is modified - except if we "touch" it, but then we'd cause the project to be always out of date. We could touch it as the pre-build step, so it would get touched only if the rest of the project is out of date, thus not causing a spurious compile, but if VC++ computes the dependencies before the pre-build step, this wouldn't work (the file with __DATE__ won't get compiled)
Any interesting ideas?
We're using the output of svnversion, written to a header file and included. We omit the file from the repository and create it in a pre-build step; this has worked quite well for us. (I'm not sure why you object to using a pre-build step?)
We're currently using a Perl script to convert svnversion's output into a header file; I later found out that TortoiseSVN includes a subwcrev command (which has also been ported to Linux) that can do much of the same thing.
If you don't like the idea of an include file not in source control that is required for a build, consider a batch file or other build step that programmatically creates a file/include and call the svnversion within your build process.
basically GENERATE the file so you don't have an unversioned and required file.
EDIT
Josh's subwcrev is probably the best idea.
Before that was implemented I wrote my own hacky tool to do the same thing - do replacement in a template file.
It could be as simple as:
% make -DBUILD_NUMBER=`svnlook youngest /path/to/repo`
I'd look at SvnRev. You can use it as a custom pre-build step in VS, or call it from a makefile, or whatever else you need to do, and it generates a header file that you can include in your other files that will give you what you need. There's good documentation on the site.
SubWCRev is another option, though the Linux port is newer, and I don't know that a Mac version exists. It's very useful on Windows for .NET (which I'm guessing isn't an issue for you, but I'm adding this for future reference), because it allows you to create a template file that can be used to generate, for example, the Properties file for a .NET assembly.
Automatic builds can typically be full, clean builds. In that case, you start in a clean directory and there would be no issue with __DATE__ in any case. Otherwise, see Paul Beckinham's idea.
Why not tie a GUID to it, almost every language has support for generating one, or if your's doesn't there are alot of algorithms for that around.
(Although, if you do use subversion, I personally like Josh's idea better!)