Related
A little background, we have a fairly large code base, which builds in to a set of libraries - which are then distributed for internal use in various binaries. At the moment, the build process for this is haphazard and everything is built off the trunk.
We would like to explore whether there is a build system which will allow us to manage releases and automatically pull in dependencies. Such a tool exists for java, Maven. I like it's package, repository and dependency mechanism, and I know that with either the maven-native or maven-nar plugin we could get this. However the problem is that we cannot fix the source trees to the "maven way" - and unfortunately (at least the maven-nar) plugins don't seem to like code that is not structured this way...
So my question is, is there a tool which satisfies the following for C++
build
package (for example libraries with all headers, something like the .nar)
upload package to a "repository"
automatically pull in the required dependencies from said repository, extract headers and include in build, extract libraries and link. The depedencies would be described in the "release" for that binary - so if we were to use CI server to build that "release", the build script has the necessary dependencies listed (like the pom.xml files).
I could roll my own by modifying either make+shell scripts or waf/scons with extra python modules for the packaging and dependency management - however I would have thought that this is a common problem and someone somewhere has a tool for this? Or does everyone roll their own? Or have I missed a significant feature of waf/scons or CMake?
EDIT: I should add, OS is preferred, and non-MS...
Most of the linux distributions, for example, contain dependency tracking for their packages. Of all the things that I've tried to cobble together myself to take on your problem, in the end they all are "not quite perfect". The best thing to do, IMHO, is to create a local yum/deb repository or something (continuing my linux example) and then pull stuff from there as needed.
Many of the source-packages also quickly tell you the minimum components that must be installed to do a self-build (as opposed to installing a binary pre-compiled package).
Unfortunately, these methods are that much easier, though it's better than trying to do it yourself. In the end, to be cross-platform supporting, you need one of these systems per OS as well. Fun!
I am not sure if I understand correctly what you want to du, but I will tell you what we use and hope it helps.
We use cmake for our build. It hat to be noted that cmake is quite powerful. Among other things, you can "make install" in custom directories to collect headers and binaries there to build your release. We combine this with some python scripting to build our releases. YMMV, but some things might just be too specific for a generic tool and a custom script may be the simpler solution.
Our build tool builds releases directly from a svn reposity (checkout, build, ...) which I can really recommend to avoid some local state polluting the release in some unforseen way. It also enforces reproducability.
It depends a lot on the platforms you're targeting. I can only really speak for Linux, but there it also depends on the distributions you're targeting, packages being a distribution-level concept. To make things a bit simpler, there are families of distributions using similar packaging mechanisms and package names, meaning that the same recipe for making a Debian package will probably make an Ubuntu package too.
I'd definitely say that if you're willing to target a subset of all known Linux distros using a manageable set of packaging mechanisms, you will benefit in the long run from not rolling your own and building packages the way the distribution creators intended. These systems allow you to specify run- and build-time dependencies, and automatic CI environments also exist (like OBS for rpm-based distros).
What are the differences between Autotools, Cmake and Scons?
In truth, Autotools' only real 'saving grace' is that it is what all the GNU projects are largely using.
Issues with Autotools:
Truly ARCANE m4 macro syntax combined with verbose, twisted shell scripting for tests for "compatibility", etc.
If you're not paying attention, you will mess up cross-compilation ability (It
should clearly be noted that Nokia came up with Scratchbox/Scratchbox2 to side-step highly broken Autotools build setups for Maemo/Meego.) If you, for any reason, have fixed, static paths in your tests, you're going to break cross-compile support because it won't honor your sysroot specification and it'll pull stuff from out of your host system. If you break cross-compile support, it renders your code unusable for things like
OpenEmbedded and makes it "fun" for distributions trying to build their releases on a cross-compiler instead of on target.
Does a HUGE amount of testing for problems with ancient, broken compilers that NOBODY currently uses with pretty much anything production in this day and age. Unless you're building something like glibc, libstdc++, or GCC on a truly ancient version of Solaris, AIX, or the like, the tests are a waste of time and are a source for many, many potential breakages of things like mentioned above.
It is pretty much a painful experience to get an Autotools setup to build usable code for a Windows system. (While I've little use for Windows, it is a serious concern if you're developing purportedly cross-platform code.)
When it breaks, you're going to spend HOURS chasing your tail trying to sort out the things that whomever wrote the scripting got wrong to sort out your build (In fact, this is what I'm trying to do (or, rather, rip out Autotools completely- I doubt there's enough time in the rest of this month to sort the mess out...) for work right now as I'm typing this. Apache Thrift has one of those BROKEN build systems that won't cross-compile.)
The "normal" users are actually NOT going to just do "./configure; make"- for many things, they're going to be pulling a package provided by someone, like out of a PPA, or their distribution vendor. "Normal" users aren't devs and aren't grabbing tarballs in many cases. That's snobbery on everyone's part for presuming that is going to be the case there. The typical users for tarballs are devs doing things, so they're going to get slammed with the brokenness if it's there.
It works...most of the time...is all you can say about Autotools. It's a system that solves several problems that only really concerns the GNU project...for their base, core toolchain code. (Edit (05/24/2014): It should be noted that this type of concern is a potentially BAD thing to be worrying about- Heartbleed partially stemmed from this thinking and with correct, modern systems, you really don't have any business dealing with much of what Autotools corrects for. GNU probably needs to do a cruft removal of the codebase, in light of what happened with Heartbleed) You can use it to do your project and it might work nicely for a smallish project that you don't expect to work anywhere except Linux or where the GNU toolchain is clearly working correctly on. The statement that it "integrates nicely with Linux" is quite the bold statement and quite incorrect. It integrates with the GNU toolsuite reasonably well and solves problems that IT has with it's goals.
This is not to say that there's no problems with the other options discussed in the thread here.
SCons is more of a replacement for Make/GMake/etc. and looks pretty nice, all things considered However...
It is still really more of a POSIX only tool. You could probably more easily get MinGW to build Windows stuff with this than with Autotools, but it's still really more geared to doing POSIX stuff and you'd need to install Python and SCons to use it.
It has issues doing cross-compilation unless you're using something like Scratchbox2.
Admittedly slower and less stable than CMake from their own comparison. They come up with half-hearted (the POSIX side needs make/gmake to build...) negatives for CMake compared to SCons. (As an aside, if you're needing THAT much extensibility over other solutions, you should be asking yourself whether your project's too complicated...)
The examples given for CMake in this thread are a bit bogus.
However...
You will need to learn a new language.
There's counter-intuitive things if you're used to Make, SCons, or Autotools.
You'll need to install CMake on the system you're building for.
You'll need a solid C++ compiler if you don't have pre-built binaries for it.
In truth, your goals should dictate what you choose here.
Do you need to deal with a LOT of broken toolchains to produce a valid working binary? If yes, you may want to consider Autotools, being aware of the drawbacks I mentioned above. CMake can cope with a lot of this, but it worries less with it than Autotools does. SCons can be extended to worry about it, but it's not an out-of-box answer there.
Do you have a need to worry about Windows targets? If so, Autotools should be quite literally out of the running. If so, SCons may/may not be a good choice. If so, CMake's a solid choice.
Do you have a need to worry about cross-compilation (Universal apps/libraries, things like Google Protobufs, Apache Thrift, etc. SHOULD care about this...)? If so, Autotools might work for you so long as you don't need to worry about Windows, but you're going to spend lots of time maintaining your configuration system as things change on you. SCons is almost a no-go right at the moment unless you're using Scratchbox2- it really doesn't have a handle on cross-compilation and you're going to need to use that extensibility and maintain it much in the same manner as you will with Automake. If so, you may want to consider CMake since it supports cross-compilation without as many of the worries about leaking out of the sandbox and will work with/without something like Scratchbox2 and integrates nicely with things like OpenEmbedded.
There is a reason many, many projects are ditching qmake, Autotools, etc. and moving over to CMake. So far, I can cleanly expect a CMake based project to either drop into a cross-compile situation or onto a VisualStudio setup or only need a small amount of clean up because the project didn't account for Windows-only or OSX-only parts to the codebase. I can't really expect that out of an SCons based project- and I fully expect 1/3rd or more Autotools projects to have gotten SOMETHING wrong that precludes it building right on any context except the host building one or a Scratchbox2 one.
An important distinction must be made between who uses the tools. Cmake is a tool that must be used by the user when building the software. The autotools are used to generate a distribution tarball that can be used to build the software using only the standard tools available on any SuS compliant system. In other words, if you are installing software from a tarball that was built using the autotools, you are not using the autotools. On the other hand, if you are installing software that uses Cmake, then you are using Cmake and must have it installed to build the software.
The great majority of users do not need to have the autotools installed on their box. Historically, much confusion has been caused because many developers distribute malformed tarballs that force the user to run autoconf to regenerate the configure script, and this is a packaging error. More confusion has been caused by the fact that most major linux distributions install multiple versions of the autotools, when they should not be installing any of them by default. Even more confusion is caused by developers attempting to use a version control system (eg cvs, git, svn) to distribute their software rather than building tarballs.
It's not about GNU coding standards.
The current benefits of autotools — specifically when used with automake — is that they integrate very well with building Linux distribution.
With cmake for example, it's always "was it -DCMAKE_CFLAGS or -DCMAKE_C_FLAGS that I need?" No, it's neither, it's "-DCMAKE_C_FLAGS_RELEASE". Or -DCMAKE_C_FLAGS_DEBUG. It's confusing - in autoconf, it's just ./configure CFLAGS="-O0 -ggdb3" and you have it.
In integration with build infrastructures, scons has the problem that you cannot use make %{?_smp_mflags}, _smp_mflags in this case being an RPM macro that roughly expands to (admin may set it) system power. People put things like -jNCPUS here through their environment. With scons that's not working, so the packages using scons may only get serialed built in distros.
What is important to know about the Autotools is that they are not a general build system - they implement the GNU coding standards and nothing else. If you want to make a package that follows all the GNU standards, then Autotools are an excellent tool for the job. If you don't, then you should use Scons or CMake. (For example, see this question.) This common misunderstanding is where most of the frustration with Autotools comes from.
While from a developers point of view, cmake is currently the most easy to use, from a user perspective autotools have one big advantage
autotools generate a single file configure script and all files to generate it are shipped with the distribution. it is easy to understand and fix with help of grep/sed/awk/vi. Compare this to Cmake where a lot of files are found in /usr/share/cmak*/Modules, which can't be fixed by the user unless he has admin access.
So, if something does not quite work, it can usually easily be "fixed" by using Standard Unix tools (grep/sed/awk/vi etc.) in a sledgehammer way without having to understand the buildsystem.
Have you ever digged through your cmake build directory to find out what is wrong? Compared to the simple shellscript which can be read from top to bottom, following the generated Cmake files to find out what is going on is quite difficult. ALso, with CMake, adapting the FindFoo.cmake files requires not only knowledge of the CMake language, but also might require superuser privileges.
I currently maintain a few boxes that house a loosely related cornucopia of coding projects, databases and repositories (ranging from a homebrew *nix distro to my class notes), maintained by myself and a few equally pasty-skinned nerdy friends (all of said cornucopia is stored in SVN).
The vast majority of our code is in C/C++/assembly (a few utilities are in python/perl/php, we're not big java fans), compiled in gcc. Our build toolchain typically consists of a hodgepodge of make, bash, grep, sed and awk. Recent discovery of a Makefile nearly as long as the program it builds (as well as everyone's general anxiety with my cryptic sed and awking) has motivated me to seek a less painful build system.
Currently, the strongest candidate I've come across is Boost Build/Bjam as a replacement for GNU make and python as a replacement for our build-related bash scripts. Are there any other C/C++/asm build systems out there worth looking into? I've browsed through a number of make alternatives, but I haven't found any that are developed by names I know aside from Boost's.
(I should note that an ability to easily extract information from svn commandline tools such as svnversion is important, as well as enough flexibility to configure for builds of asm projects as easily as c/c++ projects)
We've started using CMake here at work recently. I've been pretty happy with it so far.
scons or cmake.
Both "cross platform" - enable compiling on Windows and Linux.
Cmake now very popular (for example boost uses it). It creates native build files for each platform - Makefiles (for Linux), VC++ projects (for Visual Studio) from custom files (CMakeList.txt). In can also create Eclipse projects, KDevelop projects etc. Since it creates native build files, you can continue using all features you're used for. For example, ccache/colorgcc/distcc for Makefiles or Visual Assist X for Visual Studio.
We use it our project and are happy with it - automatic dependencies, easy syntax, robust builds.
Scons is python bases system, which perform the builds by itself. It's IMHO less popular, and still slow for large project. But for msmall to medium project maybe good alternative.
You could use python-based build system, too -- http://code.google.com/p/waf/
We have a handful of developers working on a non-commercial (read: just for fun)
cross-platform C++ project. We've already identified all the cross-platform libraries we'll need. However, some of our developers prefer to use Microsoft Visual C++ 2008, others prefer to code in Emacs on GNU/Linux. We're wondering if it is possible for all of us to work more or less simultaneously out of both environments, from the same code repository. Ultimately we want the project to compile cleanly on both platforms from the start.
Any of our developers are happily willing to switch over to the other environment if this is not possible. We all use both Linux and Windows on a regular basis and enjoy both, so this isn't a question of trying to educate one set devs about the virtues of the other platform. This is about each of us being able to develop in the environment we enjoy most yet still collaborate on a fun project.
Any suggestions or experiences to share?
Use CMake to manage your build files.
This will let you setup a single repository, with one set of text files in it. Each dev can then run the appropriate cmake scripts to build the correct build environment for their system (Visual Studio 2008/2005/GNU C++ build scripts/etc).
There are many advantages here:
Each dev can use their own build environment
Dependencies can be handled very cleanly, including platform specific deps.
Builds can be out of source, which helps prevent accidentally committing inappropriate files
Easy migration to new dev. environments (ie: when VS 2010 is released, some devs can migrate there just by rebuilding their build folder)
I've done it and seen it done without too many problems.
You'll want to try an isolate the code that is different for the different platforms. Also, you'll want to think about your directory structure. Something like
project/src <- .cc and .h files
project/src/linux|win <- code that is specific to one platform or the other
project/linux <- make files and other project related stuff
project/win <- .sln and .csproj files
Basically you just want to be really clear what is specific to each system and what is common.
Also, unit tests are going to be really important since there may be minor difference and you want to make it easy for the windows guys to run some tests to make sure the linux code works as expected and the other way around.
as mentioned in previous posts, Qt is a very easy method to do real simultaneous multi-platform development - independent of your IDE and with many different compilers (even for Symbian, ARM, Windows CE/Mobile...).
In my last and current company I work in mixed Linux- and Windows-developer teams, who work together using Subversion and Qt (which has a very easy and powerful build-system "QMake", which hides all the different platform/compiler specific build environments - you just have to write one build file for all platforms - so easy!).
And: Qt contains nearly everything you need:
simple string handling, with easy translation support and easy string conversion (utf8, utf16, asci...)
simple classes for file i/o, images, networking etc.
comfortable gui classes
graphical designer for the gui layout/design
graphical translation tool to create dynamic translations for your apps (you can run one binary with different selectable languages - even cyrillic, asian fonts...)
integrated testing framework (unit tests)
So Qt is a full featured and very reliable environment - I use it for 10 years.
And: Qt integrates seamlessly into IDEs like VC++, Eclipse or provides its own IDE "QtCreator".
see: http://www.trolltech.com + http://doc.trolltech.com
Best Regards,
Chris
I had such experience working in Acronis. We had the same codebase used to build binaries (fully packaged installers, actually) targetting Win32/64, Linux, and OS X (and a bunch of other more exotic platforms, such as EFI), with everyone working in their IDE of choice. The trick here is to avoid compiler- and IDE-specific solutions, and make your project fully buildable from clean cross-platform make files. Note that you can perfectly well use any make system together with VC++, so it isn't a problem (we used Watcom make for historical reasons, but I wouldn't recommend it).
One other trick you can do is add a make script that automatically produces project files from the input lists in your makefiles, for all IDEs you use (e.g. VS and Eclipse CDT). That way, every developer generates that stuff for himself, and then opens those projects in IDE to edit/build/debug, but the source repository only has makefiles in it.
Ensuring that code is compilable for everyone can be a problem, mostly because VC++ is generally more lax in applying the rules than g++ (for example, it will let you bind an rvalue to a non-const reference, albeit with a warning). If you compile with treat warnings as errors, and highest warning levels (with perhaps a few hand-picked warnings disabled), you will mostly avoid this. Having contiguous rolling build set up is another way to catch those early. One other approach we've used in Acronis is to have the Windows build environment have Cygwin-based cross-compilation tools in it, so any Windows dev could do a build targeting Linux (with the same g++ version and all) from his box, and see if that fails, to verify that his changes will compile in g++.
I personally use cmake/mingw32/Qt4 for all my C++ needs.
QtCreator is a crossplatform IDE which is somewhat optimized for qt4 programming.
We're working on a cross-platform project as well. We're using Emacs to code, SCons to build and Visual Studio 2008 to debug. It's my 1st time using Emacs + SCons and I must say that It's very very nifty once you figure out how does the SConstruct and SConscripts work.
I'm late to this question, and there are a lot of good answers here, but I haven't seen anyone enumerate all the issues I've run into (most of my work in C++ is and has been cross-platform), so:
Cross-platform compilation. Everyone has covered this quite well. CMake, and SCons are useful among others.
Platform differences. These aren't actually limited to Linux v Windows. Different versions of Windows have plenty of subtle issues. If you ever want to jump from Linux to OS X you'll find the same. So do processor architecture differences: 32 v 64 bit doesn't usually matter - until it does and things break on you very badly. This is something C++ programmers face more than in most other languages, whose implementations are working hard to hide this sort of thing from programmers.
Test everything. Alan mentions unit tests but let me emphasize them. The real problem with cross-platform programming is not code that won't compile: it's code that compiles just fine and does the wrong thing in subtle cases. Unit tests matter here much more than they do when you are working on a single platform.
Use portable libraries whenever possible. Getting this right is full of subtle gotchas. It's better to take advantage of someone else's work than to roll your own. Qt is useful here, as are NSPR and APR (the latter two are C, but wrappers exist if you don't want to mess with them directly.) These are the libraries that explicitly target platform abstraction as a goal, but most other libraries you use should be checked for this. Be reasonably wary of cool libraries that don't have a track record of portability. I'm not saying don't use them: just test first. Doing this right saves you enormous effort on testing.
Pavel mentions continual integration. This may not matter if there are only a handful of you. But I've found that the number of platforms you get when you consider all of the OS, OS variant, and processor differences means that without some form of continuous build-test cycle you will always be missing some edge case. If your project gets bigger than a handful of people consider doing this.
Version control. The most obvious issue is handling newlines and filename case-sensitivity but there are others. Most of the well-known open source VCSes do this right by now, but it's notable that it usually takes them several years to find all their bugs related to portability. As an example Mercurial, which has had portability as one of its selling points, has a series of fixes spanning three or four releases dealing with Windows filenames in uncommon situations. Make sure you can check out what the others check in early on.
Scripts. Use Perl/Python/Ruby (or something like them) for your scripting. Trying to keep Linux shell and Windows batch scripts in sync is painfully tedious.
Despite all of the above: overall cross-platform is quite doable, and there's no real reason to avoid it. My experience is that the problems tend to crop up sporadically, but when they do they take more than there fair share of time. If you've been programming a while though you won't find that unusual.
The issue is not really editing the C++ source files, but the build process itself. VS doesn't use the same Makefile architecture as Linux development typically does. A radical suggestion, but both groups could actually use the same C++ IDE and build process - see Code::Blocks for more info.
This is possible (I do that to earn my daily money :-) ).
But you have to keep in mind the differences in the OS supplied libraries which could be very annoying.
Two good options to get around that are BOOST and QT.
Both supply you with platform independent functions of useful stuff that isn't handled by the C lib and the STL.
I've done this in two ways:
Each platform has its own build system. Whenever someone changes something (adding a source file), they either adapt the other platform too, or they mail a notification and someone more familiar with the other platform adapts it accordingly.
Use CMake.
I can't say I really liked CMake, but a cross-platform tool certainly has its advantages.
Generally, using different compilers to compile your code from the very first few lines is always a good idea, as it helps improving code quality.
Another vote for cmake.
One thing to watch out for, filenames are cased but not case sensitive on Windows. They are case sensitive on most unix filesystems.
This is generally a problem with #include
eg. given a file FooBar.h
#include "foobar.h" // works on Windows, fails on Linux.
I do agree with everyone here that has suggested cmake for cross platform development in C++. It is an excellent tool.
Another thing I would suggest is to use eclipse CDT as development environment. It works in any place where you can run Java an gcc and that unifies your development environment.
I think that was Alan Jackson who gave emphasis to the unit test. For doing so yo would need some cross platform unit test libraries. I read this post time ago regarding C++ unit test frameworks. It is a bit outdated but thoughtful. The one missing that also works in both platforms is googletest.
Finally if you want to run those test automatically in both platforms cmake has another tool called ctest that is very good for doing so.
Check out premake... It is pretty similar to CMake but written using lua.
I have used this on a number of development projects and find it easy to learn and integrate into existing company and project structures.
Give it a try!
http://industriousone.com/premake
What would be the best choice of build system for a more than one million line multi platform project, which produces drivers, libraries, command line tools, GUIs, and OS install packages for all the mainstream OSes, using both the GNU and Microsoft toolchains?
Our source code is mainly C, with Python, C# and GNU makefile, and a little C++ and bash. It resides mainly in one repository, but we push source code to various third parties all of whom have their source code code repositories. There is also some interest in keeping the build fast, which might involve splitting up the project.
Currently we use a mixture of GNU make, bash, python and Microsoft's DDKBUILD. The main problems are that we are maintaining a complex set of scripts on top of make and would prefer to use third party (preferably open source) tools, and that cygwin is not proving to be robust on Windows (e.g. fork isn't always possible), and that our current build system does not build or install the toolchain so is vulnerable to tool chain version changes.
I vote for CMake, as a meta-building tool that really rewrite KDE4 build system from scratch -- and make KDE4 now a cross-platform desktop that even running on WindowsCE!
CMake is the carrier porting KDE4 to any OS on earth -- by generating Makefile( or vcprojs in Windows case) for about 40 OSes with relative toolchains!
JetBrains TeamCity works very well in general, so should be worth having on the eval list.
ThoughtWorks Cruise is also in the same space. While its v1, it comes from a stable that's been around for a while.
There's nothing about Team Foundation Server that would make necessarily count it out for your situation, but out of the box it might be more MS-shop centric that the other two I've mentioned.
As a general comment, with the level of variety you have, you definitely want to trial whatever it is you want to use - just because something is supported as a tick on the box doesnt mean its going to suit what oyu're looking for.
Dickson,
Is your build mostly monolithic or do you want to build some libraries separately and assemble them into the larger application? If inter-project dependencies are a big deal, your choices become limited quickly. AnthillPro does it well, and I think TeamCity has some Ivy integration support. From what you're saying, it sounds like this is not an absolute need, but might be helpful in speeding the build. It's certainly a strategy that we've seen a number of teams execute effectively.
Since you're looking at cross-platform (I assume multiple machine) builds, most of the open source tools other than Hudson are ruled out.
A build server comparison matrix is hosted by our friends at Thoughtworks here: confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
Good luck.
You should have CMake on your list of alternatives to investigate. CMake is a meta-tool, i.e. it generates the input to the build-tool of your choice (GNU make, Visual Studio, etc.). I can recommend it strongly.
You may want to look at Cruise. It is built on Java so it will run on any platform that supports that. You can also have multiple build agents on different machines that can perform the different tasks on the different platforms. Thoughtworks is still building it out so some of the functionality is lacking, but it may be a a good option since you are looking for true cross-platform capabilities.
SCons is a cross-platform build system implemented in Python. We use it to build our code on three platforms. It can automatically detect your build tools but you can also put arbitrary Python code in your build script. It also lets you separate your environment setup from description of your project structure, a great feature for reuse of your buidl scripts in different environments. Besides building your project directly, it can also generate Visual Studio project files.