I currently maintain a few boxes that house a loosely related cornucopia of coding projects, databases and repositories (ranging from a homebrew *nix distro to my class notes), maintained by myself and a few equally pasty-skinned nerdy friends (all of said cornucopia is stored in SVN).
The vast majority of our code is in C/C++/assembly (a few utilities are in python/perl/php, we're not big java fans), compiled in gcc. Our build toolchain typically consists of a hodgepodge of make, bash, grep, sed and awk. Recent discovery of a Makefile nearly as long as the program it builds (as well as everyone's general anxiety with my cryptic sed and awking) has motivated me to seek a less painful build system.
Currently, the strongest candidate I've come across is Boost Build/Bjam as a replacement for GNU make and python as a replacement for our build-related bash scripts. Are there any other C/C++/asm build systems out there worth looking into? I've browsed through a number of make alternatives, but I haven't found any that are developed by names I know aside from Boost's.
(I should note that an ability to easily extract information from svn commandline tools such as svnversion is important, as well as enough flexibility to configure for builds of asm projects as easily as c/c++ projects)
We've started using CMake here at work recently. I've been pretty happy with it so far.
scons or cmake.
Both "cross platform" - enable compiling on Windows and Linux.
Cmake now very popular (for example boost uses it). It creates native build files for each platform - Makefiles (for Linux), VC++ projects (for Visual Studio) from custom files (CMakeList.txt). In can also create Eclipse projects, KDevelop projects etc. Since it creates native build files, you can continue using all features you're used for. For example, ccache/colorgcc/distcc for Makefiles or Visual Assist X for Visual Studio.
We use it our project and are happy with it - automatic dependencies, easy syntax, robust builds.
Scons is python bases system, which perform the builds by itself. It's IMHO less popular, and still slow for large project. But for msmall to medium project maybe good alternative.
You could use python-based build system, too -- http://code.google.com/p/waf/
Related
I'm responsible for developing a set of C++ libraries and programs. Currently building on Linux and MacOS, but Windows support is also a requirement. We will need to support VS2010 and VS2012, and in the future will also include VS2013 and maybe also MinGW. We're using cmake for building, so our code should build on all the platforms without issues; my problem is how to manage all the dependencies on Windows in order to be able to build in the first place, and keeping it up-to-date over time. At the moment, we have one virtual machine per visual studio version as a jenkins slave, so parallel builds of all the variants is fairly easy, but managing it is not.
The problem is the number of variants this requires building. If we consider only VS2010 and VS2012, with debug/release and i386/x64 builds, that's already 8 copies of each library; 16 if we include the other compilers. We will need all the libraries our code depends on, which will include at a minimum boost, qt, xerces+xalan, zlib, icu, libpng/tiff/jpeg, hdf5 and more, plus python, and all their dependencies. And as new upstream releases are made, we'll need to keep the entire collection up-to-date and consistent for all the build/arch/compiler variants.
I don't want to do this by hand, since this really needs automating. However, I'm unaware of any good solution for doing this on Windows. The Windows building guides I've seen for other projects often involve hand-building all the dependencies, and only build for a single variant. On Linux, it's already packaged, you don't need separate debug builds, and the arch variants can be catered for with chroots; on MacOS there's homebrew, macports etc., and it's also fairly simple to automate stuff there as well. Is there any equivalent for Windows? I've looked at stuff like chocolatey, but it's entirely unsuited to handling libraries, and is pretty poor as a package manager.
This seems like it should be a common problem for anyone doing C++ development on Windows? Are there any common solutions, tools or methodologies for managing a complex set of libraries and tools for development? How do other developers manage this?
NB. Just for the record, we are not using the visual studio application; we're doing all builds non-interactively via scripts driving the compilers directly with cmake and/or msbuild.
Many thanks,
Roger
I worked on large windows C++ project that delivers X86 Release, x86 Debug, x64 Release and x64 debug. Very similarly I used build system that does parallel builds for all target platforms using custom script.
We manage all third party dependency libraries in organized folders.
For example x86\release\Zlib.dll x86\Debug\zlib.dll x64\release\zlib.dll x64\zlib.dll
Custom script is made to pick all these libraries and project source code from configuration management tool. This allows to automatically build the relevant target binaries as needed.
any third party libraries change is updated in configuration management tool and then later picked up by the script for the next build.
For your question on VS2010 and 2012 support I don't understand importance. Is not one version of VS enough to support for the project?
you may take a look at Link, and their build system https://github.com/gisinternals/buildsystem
It's basicly a set of batch and make files calling each others. You still need keep track of lib update manually.
I am currently working on an opensource C++ project. I don't have much experience in C++ as my daily job mainly involves with Java. I am now planning to release my C++ project and I would like to know how should I should I arrange the packaging of my project. E.g, in Java, all the class files are packaged into jar file. So what is the equivalent approach in C++? Is there any good practise for organizing the source code/binary? My target platform is Linux by the way.
Another question is I am currently using Eclipse CDT plugin for development and building. So is there anyway that I can extract build script from Eclipse project and use it as generic build script? Is there any good reference regarding C++ build/packaging? Thanks in advance.
Edited
To clarify a bit more, I think the release of my project can be considered as an application. It's a command line tool for software configuration management.
I am currently working on an opensource C++ project.
That simplifies many things. You should supply the build scripts for you project and support them for different use cases (learn about Makefiles, there are similar concepts like "target").
I don't have much experience in C++ as my daily job mainly involves with Java.
Most of the things you're used to have (and ask for right now) in Java are invented because they lack in C/C++. Learn at least something about dynamic(shared)/static libraries (.so and .a files to be simple).
I am now planning to release my C++ project and I would like to know how should I should I arrange the packaging of my project.
The "packaging of a C++ project" is something informal. You may supply the sources, build scripts and some project-files for the well-known IDEs.
EDIT: you've specified that you're building the command-line application. I believe all you need is to start from a simple Makfile for that application. The reference Makefile can be automatically generated by Eclipse. If you are planning to deploy your application as a stand-alone software, then you have to earn about packaging (rpm, deb, tgz).
E.g, in Java, all the class files are packaged into jar file.
There are no such thing as a C++ "package" compatible accross compilers (even the "modules" were rejected in the latest C++11 standard)
because there is no binary standard to encode C++ classes (ABI). On linux you're most likely going to use GCC (or Intel's compiler, or LLVM's CLang, or custom build of OpenWatcom or...),
but the version of standard library you are linking to makes the release of binary builds close to useless.
That is the reason for redistibuting source code.
So what is the equivalent approach in C++?
No clear answer here.
Is there any good practise for organizing the source code/binary?
Look at the large-scale projects, see the way they organize their builds. The term "build engineer" as an occupation emphasizes the difficulties of large-scale projects compilation/linking.
My target platform is Linux by the way.
This is also something of an incomplete description. "Linux" is a blurry term. You should speak about the Linux distribution, compiler toolchain and package manager. E.g., Ubuntu 12, amd64, GCC 4.6 toolchain, APT package manager.
There are different "linuxes" built around the same kernel source. There are different compilers. There are at least three major package managers to consider: Debian/Ubuntu(deb,apt), Red Hat(rpm), Slackware(tgz).
Another question is I am currently using Eclipse CDT plugin for development and building. So is there anyway that I can extract build script from Eclipse project
There's a sort of "meta-technique": you write a "description" of your project and then a tool generates the project-file and build scripts for your sources. Have a look at CMake. Since you're on "linux", try looking at the somewhat standard way of autotools (autoconf).
Is there any good reference regarding C++ build/packaging?
You should start by building your application and then move on to the deployment issues. C/C++ is a hard-to-learn legacy with a lot of subtleties which are avoided in Java.
Seems most open source projects in C/C++ only provide the source code,i.e. nginx
Is this a convention that anyone interested in joining the developing team should figure out the .sln/.project files himself to qualify??
most open source projects are coming from the linux side of computing. thus, they are mainly using unix style build tools, as well as open source compilers.
the main build tool is make, which uses a makefile to know how to build a project. on Windows, the main open source compiler is MinGW which is a win32 port of gcc. the use of those tools allows to keep a maximum of common things between unix and windows.
note that .sln files are specific to microsoft compilers which are not free to use (and are rather costly), they are not portable and so are not suitable for multi-platform programming.
Some projects use CMake, which can generate project files for Your Favourite Build System, but as mentioned above, you don't need to use .sln and pro files, even if a project is built with the MSVC compiler, MinGW + makefiles, or scons, or CMake, or any other number of scripty methods to invoke the right commands to compile the program will work just fine.
Don't confuse the IDE with the compiler!
No, most opensource project do not use MSVC solutions as they not portable and very weak in terms of features.
In most cases they use what is called "build-system" like autotools, CMake or SCons.
These build systems include information about:
source code and how to build it
various system checks that should be done (like find various 3rd part libraries)
How to build and run unit-tests
How to install application
They also allow important tasks like cross compilation and packaging for deploy.
These task done via specific build system scripting language that allow you big flexibility.
So, these build systems generally much more powerful then typical "project files" and they
generally work with multiple compilers on different platforms and operating systems.
Some of build systems (like CMake) allow as one of the options to generate MSVC solutions as well as one of optional ways to build application.
What would be the best choice of build system for a more than one million line multi platform project, which produces drivers, libraries, command line tools, GUIs, and OS install packages for all the mainstream OSes, using both the GNU and Microsoft toolchains?
Our source code is mainly C, with Python, C# and GNU makefile, and a little C++ and bash. It resides mainly in one repository, but we push source code to various third parties all of whom have their source code code repositories. There is also some interest in keeping the build fast, which might involve splitting up the project.
Currently we use a mixture of GNU make, bash, python and Microsoft's DDKBUILD. The main problems are that we are maintaining a complex set of scripts on top of make and would prefer to use third party (preferably open source) tools, and that cygwin is not proving to be robust on Windows (e.g. fork isn't always possible), and that our current build system does not build or install the toolchain so is vulnerable to tool chain version changes.
I vote for CMake, as a meta-building tool that really rewrite KDE4 build system from scratch -- and make KDE4 now a cross-platform desktop that even running on WindowsCE!
CMake is the carrier porting KDE4 to any OS on earth -- by generating Makefile( or vcprojs in Windows case) for about 40 OSes with relative toolchains!
JetBrains TeamCity works very well in general, so should be worth having on the eval list.
ThoughtWorks Cruise is also in the same space. While its v1, it comes from a stable that's been around for a while.
There's nothing about Team Foundation Server that would make necessarily count it out for your situation, but out of the box it might be more MS-shop centric that the other two I've mentioned.
As a general comment, with the level of variety you have, you definitely want to trial whatever it is you want to use - just because something is supported as a tick on the box doesnt mean its going to suit what oyu're looking for.
Dickson,
Is your build mostly monolithic or do you want to build some libraries separately and assemble them into the larger application? If inter-project dependencies are a big deal, your choices become limited quickly. AnthillPro does it well, and I think TeamCity has some Ivy integration support. From what you're saying, it sounds like this is not an absolute need, but might be helpful in speeding the build. It's certainly a strategy that we've seen a number of teams execute effectively.
Since you're looking at cross-platform (I assume multiple machine) builds, most of the open source tools other than Hudson are ruled out.
A build server comparison matrix is hosted by our friends at Thoughtworks here: confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
Good luck.
You should have CMake on your list of alternatives to investigate. CMake is a meta-tool, i.e. it generates the input to the build-tool of your choice (GNU make, Visual Studio, etc.). I can recommend it strongly.
You may want to look at Cruise. It is built on Java so it will run on any platform that supports that. You can also have multiple build agents on different machines that can perform the different tasks on the different platforms. Thoughtworks is still building it out so some of the functionality is lacking, but it may be a a good option since you are looking for true cross-platform capabilities.
SCons is a cross-platform build system implemented in Python. We use it to build our code on three platforms. It can automatically detect your build tools but you can also put arbitrary Python code in your build script. It also lets you separate your environment setup from description of your project structure, a great feature for reuse of your buidl scripts in different environments. Besides building your project directly, it can also generate Visual Studio project files.
We maintain a RPM based software distribution at work so that we have a common set of software across all the platforms that we support. As a result we have to build a lot of third party software, and frequently find situations where we need to run autoconf/automake/libtoolize/etc to get it to build on Solaris or another platform.
I've had very mixed results with this. It seems that these tools are fairly brittle and frequently the files only work with the version of autoconf/automake/etc that they were originally written for.
Ideally I'd like to only have to support one version of the GNU autotools, but I get the impression that I'm really going to end up having to have a copy of every version lying around.
Is this unusual, or do other people have the same problems? Is there a subset of the versions of autotools that will cover all cases?
It is true that autotools can be brittle and version specific. But remember that you only need to use these tools on your development machines. Deploying the project to the target machine doesn't require that any of the tools are installed on the target. Even test machines don't need any of the tools. They are really only need to run when the dependencies change, such as adding additional files or libraries to the project.
We have been using these tools for inhouse projects for many years, and haven't come across a better solution. If you are in a Unix world, don't underestimate the benefit of having a system where configure; make; make install just works.
Your experiences are not unusual. Autotools is brittle like that, specially for complex projects. There seems to be no alternative for having a lot of versions of it around, sadly.