What would be the best choice of build system for a more than one million line multi platform project, which produces drivers, libraries, command line tools, GUIs, and OS install packages for all the mainstream OSes, using both the GNU and Microsoft toolchains?
Our source code is mainly C, with Python, C# and GNU makefile, and a little C++ and bash. It resides mainly in one repository, but we push source code to various third parties all of whom have their source code code repositories. There is also some interest in keeping the build fast, which might involve splitting up the project.
Currently we use a mixture of GNU make, bash, python and Microsoft's DDKBUILD. The main problems are that we are maintaining a complex set of scripts on top of make and would prefer to use third party (preferably open source) tools, and that cygwin is not proving to be robust on Windows (e.g. fork isn't always possible), and that our current build system does not build or install the toolchain so is vulnerable to tool chain version changes.
I vote for CMake, as a meta-building tool that really rewrite KDE4 build system from scratch -- and make KDE4 now a cross-platform desktop that even running on WindowsCE!
CMake is the carrier porting KDE4 to any OS on earth -- by generating Makefile( or vcprojs in Windows case) for about 40 OSes with relative toolchains!
JetBrains TeamCity works very well in general, so should be worth having on the eval list.
ThoughtWorks Cruise is also in the same space. While its v1, it comes from a stable that's been around for a while.
There's nothing about Team Foundation Server that would make necessarily count it out for your situation, but out of the box it might be more MS-shop centric that the other two I've mentioned.
As a general comment, with the level of variety you have, you definitely want to trial whatever it is you want to use - just because something is supported as a tick on the box doesnt mean its going to suit what oyu're looking for.
Dickson,
Is your build mostly monolithic or do you want to build some libraries separately and assemble them into the larger application? If inter-project dependencies are a big deal, your choices become limited quickly. AnthillPro does it well, and I think TeamCity has some Ivy integration support. From what you're saying, it sounds like this is not an absolute need, but might be helpful in speeding the build. It's certainly a strategy that we've seen a number of teams execute effectively.
Since you're looking at cross-platform (I assume multiple machine) builds, most of the open source tools other than Hudson are ruled out.
A build server comparison matrix is hosted by our friends at Thoughtworks here: confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
Good luck.
You should have CMake on your list of alternatives to investigate. CMake is a meta-tool, i.e. it generates the input to the build-tool of your choice (GNU make, Visual Studio, etc.). I can recommend it strongly.
You may want to look at Cruise. It is built on Java so it will run on any platform that supports that. You can also have multiple build agents on different machines that can perform the different tasks on the different platforms. Thoughtworks is still building it out so some of the functionality is lacking, but it may be a a good option since you are looking for true cross-platform capabilities.
SCons is a cross-platform build system implemented in Python. We use it to build our code on three platforms. It can automatically detect your build tools but you can also put arbitrary Python code in your build script. It also lets you separate your environment setup from description of your project structure, a great feature for reuse of your buidl scripts in different environments. Besides building your project directly, it can also generate Visual Studio project files.
Related
I'm a C++ beginner and I'm starting to develop my first cross-platform C++ project. I need to use platform-specific calls (Win32 and POSIX) so I need to compile frequently both in Windows and Linux.
Whit single-platform projects I'm using, until now, KDevelop in Linux and Visual Studio 2012 in Windows.
How can I use two different IDEs in two different Operating Systems with the same project?
Should I use a single, cross-platform, IDE?
Should I learn CMake (or similar) and configure it to work with both IDEs?
Could/Should I host my code in the web and sync automatically with offline projects?
Alternatives?
Thanks in advance to everyone.
EDIT:
Just for clarification, the project will be a simple server for a scholastic protocol. There will be a client asking for upload/retrieve some files to/from the server.
With scholastic I mean that, for example, I have to use pthreads/win32 threads instead of an higher level C++ threads library.
Maybe - really depends on what you feel most comfortable with. As the project is non-graphical, all the IDE gives you is editing of files and compilation. So you can build the project on one machine with the IDE there, and then move the sources to another machine for compiling there.
I personally would just have two makefiles, one for Linux and one for Widnows. Makes life fairly simple [you could have a "outer" makefile that picks the right one based on some clever method].
Yes, you should find a version control system that works for both Windows and Linux (git, mercurial, subversion, bazaar and several others). That way, not only do you have a central repository [you can use either of your machines as "server" for any of these], but it also allows you to keep track of your changes. Definitely worthwile doing!
There are hundreds of different alternatives. But the simpler you keep it, and the less complicated your tools are, the more time you get to spend on actually programming your project, rather than, for example, figure out why CMake isn't building the way you want it to.
Also, make sure that you separate out all your system-specific code to one file per architecture. That way, it's easy to port to another architecture later, and it makes MOST of your code compile on both systems.
Typically, it's easy to adjust the IDE-specific project/build files to added/moved/deleted source files. Therefore, using a cross-platform IDE isn't that important.
You can do that, I think that CMake can also create project files for some IDEs that can then be used to build the project.
Ahem, if you want to host it online or not is your choice. What you should definitely do is to use some kind of version control. A bug-tracking system is also helpful. If you want to open-source the code anyway, using one of the existing hosting facilities is a clear yes.
Not really.
One comment though: You will have much more trouble making the C++ code portable. Building on top of a toolkit like Qt is a tremendous help. If you want to stay closer to standard C++, at least consider using Boost for stuff like threads, smart pointers, filesystem access. Good luck!
My recent experience suggest to take a look at Qt. The IDE (QtCreator) it's very good, and available on all major platforms.
I've used for a fairly simple project, that uses fairly complex components, like OpenCV and ZBar. I develop on Linux, copy the source to Windows, and recompile.
I had some trouble to setup OpenCV on both platforms, so I can't say it's super easy, but it's working. Since you already know KDevelop, you should already know Qt.
I also put much value in recent trend that see Qt5 as the platform for Ubuntu on smartphones. I really hope to see this developing.
HTH
How can I use two different IDEs in two different Operating Systems with the same project?
Should I use a single, cross-platform, IDE?
No. I think this is a case of asking the wrong question. To make a cross-platform project, what matters is your build scripts and the system-neutral nature of your code. Sometimes it might help to have project files for your preferred IDE, but maintaining multiple project files for multiple IDEs will only make things more difficult and complex for you. Instead, you should focus on finding a build system that minimizes the amount of time you spend on project maintenance.
For that, CMake and PreMake seem to be two of the best tools to make that happen.
There are dozens of alternatives (like SCons, Cook, kbuild, Jam and Boost Jam, and many others), but since CMake and PreMake both generate project files and build scripts, they might be the best solutions.
Your mileage will vary.
Could/Should I host my code in the web and sync automatically with offline projects?
You should have robust source control that works everywhere you do. Git and Mercurial seem to work best if you use some kind of "cloud" hosting like Github or BitBucket, but they by no means require it. Depending on your work environment and team size, you may prefer Subversion or PerForce or something else, but that's up to you and your team.
it will help, you will quite likely need to debug on many platforms... Qt Creator, Netbeans and Eclipse come to mind.
Yes. cmake, or qmake for Qt maybe
Not technical question. Just use version control! github and gitorious are easy choices for open source project though.
Qt is a no-brainer choice for cross-platform C++ GUI app, and also decent choice for network app with no GUI.
I currently maintain a few boxes that house a loosely related cornucopia of coding projects, databases and repositories (ranging from a homebrew *nix distro to my class notes), maintained by myself and a few equally pasty-skinned nerdy friends (all of said cornucopia is stored in SVN).
The vast majority of our code is in C/C++/assembly (a few utilities are in python/perl/php, we're not big java fans), compiled in gcc. Our build toolchain typically consists of a hodgepodge of make, bash, grep, sed and awk. Recent discovery of a Makefile nearly as long as the program it builds (as well as everyone's general anxiety with my cryptic sed and awking) has motivated me to seek a less painful build system.
Currently, the strongest candidate I've come across is Boost Build/Bjam as a replacement for GNU make and python as a replacement for our build-related bash scripts. Are there any other C/C++/asm build systems out there worth looking into? I've browsed through a number of make alternatives, but I haven't found any that are developed by names I know aside from Boost's.
(I should note that an ability to easily extract information from svn commandline tools such as svnversion is important, as well as enough flexibility to configure for builds of asm projects as easily as c/c++ projects)
We've started using CMake here at work recently. I've been pretty happy with it so far.
scons or cmake.
Both "cross platform" - enable compiling on Windows and Linux.
Cmake now very popular (for example boost uses it). It creates native build files for each platform - Makefiles (for Linux), VC++ projects (for Visual Studio) from custom files (CMakeList.txt). In can also create Eclipse projects, KDevelop projects etc. Since it creates native build files, you can continue using all features you're used for. For example, ccache/colorgcc/distcc for Makefiles or Visual Assist X for Visual Studio.
We use it our project and are happy with it - automatic dependencies, easy syntax, robust builds.
Scons is python bases system, which perform the builds by itself. It's IMHO less popular, and still slow for large project. But for msmall to medium project maybe good alternative.
You could use python-based build system, too -- http://code.google.com/p/waf/
What we need in our firm is a sort of release management tool for Linux/C++. Our products consist of multiple libraries and config files. Here I will list the basic features we want such system to have:
Ability to track dependencies, easily increase major versions of libraries whose dependencies got their major version increased. It should build some sort of dependency graph internally so it can know who is affected by an update.
Know how to build the products it handle. Either a specific build file or even better - ability to read and understand makefiles.
Work with SVN so it can check for new releases from there and does the build.
Generate some installers - in rpm or tar.gz format. For that purpose it should be able to understand the rpm spec file format.
Currently we are working on such tool which is already pretty usable. However I believe that our task is not unique and there should be some tool out there which does the job.
You should look into using a mix between Hudson, Maven (for build management), Ivy (for dependencies management) and Archiva (for artifacts archival).
Also, if you are looking into cross.compilation, take a look at Make Project Creator (MPC) and Bakefile.
Have fun!!
In the project I'm currently working on we use cmake and other Kitware tools to handle most of this issues for native code (C++). Answering point by point:
The cmake scripts handle the dependencies for our different projects. We have a dependency graph but I don't know if is a home-made script or it is a functionality that cmake provides.
Well cmake generates the makefiles regarding the platform. I generates projects for eclipse cdt and visual studio if it is asked to do so in case of developing.
Cmake has a couple of tools, ctest and cdash that we use to do the daily build and see how the test are doing.
In order to create the installer cmake has cpack. From just one script it can generate tar.gz, deb or rpm files in Linux or an automatically generated NSIS script to generate installers in windows.
For Java code we use maven and hudson that have been already mentioned here.
Take a look at this article from DDJ, in which a more robust build system concept (than make) is presented and implemented. Not sure it will fit well to your requirements, but it's the closest I've ever seen. I was looking for the same thing months ago, and then I discovered the article.
http://www.drdobbs.com/architect/218400678
Maven has a native code plugin. I don't think it'll do everything you want, but it's good at tracking version numbers of dependencies, will build artefacts and it'll work with your VCS.
No idea
cmake/scons: I have used cmake but I don't exactly love it, but I have heard really good things about scons. But scons is python-based, so you need to have python installed on the build/dev machines.
I use Hudson, which has a plugin to fetch from svn. It performs intelligently in general, and in particular builds only if some file has changed in an svn update. Hudson is easy to get started with. Hudson is java-based and is pretty popular with the Java community. This means it is quite cross-platform, but you need to have JRE installed on the build machine.
Probably can call some rpm tool within hudson.
We have a handful of developers working on a non-commercial (read: just for fun)
cross-platform C++ project. We've already identified all the cross-platform libraries we'll need. However, some of our developers prefer to use Microsoft Visual C++ 2008, others prefer to code in Emacs on GNU/Linux. We're wondering if it is possible for all of us to work more or less simultaneously out of both environments, from the same code repository. Ultimately we want the project to compile cleanly on both platforms from the start.
Any of our developers are happily willing to switch over to the other environment if this is not possible. We all use both Linux and Windows on a regular basis and enjoy both, so this isn't a question of trying to educate one set devs about the virtues of the other platform. This is about each of us being able to develop in the environment we enjoy most yet still collaborate on a fun project.
Any suggestions or experiences to share?
Use CMake to manage your build files.
This will let you setup a single repository, with one set of text files in it. Each dev can then run the appropriate cmake scripts to build the correct build environment for their system (Visual Studio 2008/2005/GNU C++ build scripts/etc).
There are many advantages here:
Each dev can use their own build environment
Dependencies can be handled very cleanly, including platform specific deps.
Builds can be out of source, which helps prevent accidentally committing inappropriate files
Easy migration to new dev. environments (ie: when VS 2010 is released, some devs can migrate there just by rebuilding their build folder)
I've done it and seen it done without too many problems.
You'll want to try an isolate the code that is different for the different platforms. Also, you'll want to think about your directory structure. Something like
project/src <- .cc and .h files
project/src/linux|win <- code that is specific to one platform or the other
project/linux <- make files and other project related stuff
project/win <- .sln and .csproj files
Basically you just want to be really clear what is specific to each system and what is common.
Also, unit tests are going to be really important since there may be minor difference and you want to make it easy for the windows guys to run some tests to make sure the linux code works as expected and the other way around.
as mentioned in previous posts, Qt is a very easy method to do real simultaneous multi-platform development - independent of your IDE and with many different compilers (even for Symbian, ARM, Windows CE/Mobile...).
In my last and current company I work in mixed Linux- and Windows-developer teams, who work together using Subversion and Qt (which has a very easy and powerful build-system "QMake", which hides all the different platform/compiler specific build environments - you just have to write one build file for all platforms - so easy!).
And: Qt contains nearly everything you need:
simple string handling, with easy translation support and easy string conversion (utf8, utf16, asci...)
simple classes for file i/o, images, networking etc.
comfortable gui classes
graphical designer for the gui layout/design
graphical translation tool to create dynamic translations for your apps (you can run one binary with different selectable languages - even cyrillic, asian fonts...)
integrated testing framework (unit tests)
So Qt is a full featured and very reliable environment - I use it for 10 years.
And: Qt integrates seamlessly into IDEs like VC++, Eclipse or provides its own IDE "QtCreator".
see: http://www.trolltech.com + http://doc.trolltech.com
Best Regards,
Chris
I had such experience working in Acronis. We had the same codebase used to build binaries (fully packaged installers, actually) targetting Win32/64, Linux, and OS X (and a bunch of other more exotic platforms, such as EFI), with everyone working in their IDE of choice. The trick here is to avoid compiler- and IDE-specific solutions, and make your project fully buildable from clean cross-platform make files. Note that you can perfectly well use any make system together with VC++, so it isn't a problem (we used Watcom make for historical reasons, but I wouldn't recommend it).
One other trick you can do is add a make script that automatically produces project files from the input lists in your makefiles, for all IDEs you use (e.g. VS and Eclipse CDT). That way, every developer generates that stuff for himself, and then opens those projects in IDE to edit/build/debug, but the source repository only has makefiles in it.
Ensuring that code is compilable for everyone can be a problem, mostly because VC++ is generally more lax in applying the rules than g++ (for example, it will let you bind an rvalue to a non-const reference, albeit with a warning). If you compile with treat warnings as errors, and highest warning levels (with perhaps a few hand-picked warnings disabled), you will mostly avoid this. Having contiguous rolling build set up is another way to catch those early. One other approach we've used in Acronis is to have the Windows build environment have Cygwin-based cross-compilation tools in it, so any Windows dev could do a build targeting Linux (with the same g++ version and all) from his box, and see if that fails, to verify that his changes will compile in g++.
I personally use cmake/mingw32/Qt4 for all my C++ needs.
QtCreator is a crossplatform IDE which is somewhat optimized for qt4 programming.
We're working on a cross-platform project as well. We're using Emacs to code, SCons to build and Visual Studio 2008 to debug. It's my 1st time using Emacs + SCons and I must say that It's very very nifty once you figure out how does the SConstruct and SConscripts work.
I'm late to this question, and there are a lot of good answers here, but I haven't seen anyone enumerate all the issues I've run into (most of my work in C++ is and has been cross-platform), so:
Cross-platform compilation. Everyone has covered this quite well. CMake, and SCons are useful among others.
Platform differences. These aren't actually limited to Linux v Windows. Different versions of Windows have plenty of subtle issues. If you ever want to jump from Linux to OS X you'll find the same. So do processor architecture differences: 32 v 64 bit doesn't usually matter - until it does and things break on you very badly. This is something C++ programmers face more than in most other languages, whose implementations are working hard to hide this sort of thing from programmers.
Test everything. Alan mentions unit tests but let me emphasize them. The real problem with cross-platform programming is not code that won't compile: it's code that compiles just fine and does the wrong thing in subtle cases. Unit tests matter here much more than they do when you are working on a single platform.
Use portable libraries whenever possible. Getting this right is full of subtle gotchas. It's better to take advantage of someone else's work than to roll your own. Qt is useful here, as are NSPR and APR (the latter two are C, but wrappers exist if you don't want to mess with them directly.) These are the libraries that explicitly target platform abstraction as a goal, but most other libraries you use should be checked for this. Be reasonably wary of cool libraries that don't have a track record of portability. I'm not saying don't use them: just test first. Doing this right saves you enormous effort on testing.
Pavel mentions continual integration. This may not matter if there are only a handful of you. But I've found that the number of platforms you get when you consider all of the OS, OS variant, and processor differences means that without some form of continuous build-test cycle you will always be missing some edge case. If your project gets bigger than a handful of people consider doing this.
Version control. The most obvious issue is handling newlines and filename case-sensitivity but there are others. Most of the well-known open source VCSes do this right by now, but it's notable that it usually takes them several years to find all their bugs related to portability. As an example Mercurial, which has had portability as one of its selling points, has a series of fixes spanning three or four releases dealing with Windows filenames in uncommon situations. Make sure you can check out what the others check in early on.
Scripts. Use Perl/Python/Ruby (or something like them) for your scripting. Trying to keep Linux shell and Windows batch scripts in sync is painfully tedious.
Despite all of the above: overall cross-platform is quite doable, and there's no real reason to avoid it. My experience is that the problems tend to crop up sporadically, but when they do they take more than there fair share of time. If you've been programming a while though you won't find that unusual.
The issue is not really editing the C++ source files, but the build process itself. VS doesn't use the same Makefile architecture as Linux development typically does. A radical suggestion, but both groups could actually use the same C++ IDE and build process - see Code::Blocks for more info.
This is possible (I do that to earn my daily money :-) ).
But you have to keep in mind the differences in the OS supplied libraries which could be very annoying.
Two good options to get around that are BOOST and QT.
Both supply you with platform independent functions of useful stuff that isn't handled by the C lib and the STL.
I've done this in two ways:
Each platform has its own build system. Whenever someone changes something (adding a source file), they either adapt the other platform too, or they mail a notification and someone more familiar with the other platform adapts it accordingly.
Use CMake.
I can't say I really liked CMake, but a cross-platform tool certainly has its advantages.
Generally, using different compilers to compile your code from the very first few lines is always a good idea, as it helps improving code quality.
Another vote for cmake.
One thing to watch out for, filenames are cased but not case sensitive on Windows. They are case sensitive on most unix filesystems.
This is generally a problem with #include
eg. given a file FooBar.h
#include "foobar.h" // works on Windows, fails on Linux.
I do agree with everyone here that has suggested cmake for cross platform development in C++. It is an excellent tool.
Another thing I would suggest is to use eclipse CDT as development environment. It works in any place where you can run Java an gcc and that unifies your development environment.
I think that was Alan Jackson who gave emphasis to the unit test. For doing so yo would need some cross platform unit test libraries. I read this post time ago regarding C++ unit test frameworks. It is a bit outdated but thoughtful. The one missing that also works in both platforms is googletest.
Finally if you want to run those test automatically in both platforms cmake has another tool called ctest that is very good for doing so.
Check out premake... It is pretty similar to CMake but written using lua.
I have used this on a number of development projects and find it easy to learn and integrate into existing company and project structures.
Give it a try!
http://industriousone.com/premake
We maintain a RPM based software distribution at work so that we have a common set of software across all the platforms that we support. As a result we have to build a lot of third party software, and frequently find situations where we need to run autoconf/automake/libtoolize/etc to get it to build on Solaris or another platform.
I've had very mixed results with this. It seems that these tools are fairly brittle and frequently the files only work with the version of autoconf/automake/etc that they were originally written for.
Ideally I'd like to only have to support one version of the GNU autotools, but I get the impression that I'm really going to end up having to have a copy of every version lying around.
Is this unusual, or do other people have the same problems? Is there a subset of the versions of autotools that will cover all cases?
It is true that autotools can be brittle and version specific. But remember that you only need to use these tools on your development machines. Deploying the project to the target machine doesn't require that any of the tools are installed on the target. Even test machines don't need any of the tools. They are really only need to run when the dependencies change, such as adding additional files or libraries to the project.
We have been using these tools for inhouse projects for many years, and haven't come across a better solution. If you are in a Unix world, don't underestimate the benefit of having a system where configure; make; make install just works.
Your experiences are not unusual. Autotools is brittle like that, specially for complex projects. There seems to be no alternative for having a lot of versions of it around, sadly.