I'm responsible for developing a set of C++ libraries and programs. Currently building on Linux and MacOS, but Windows support is also a requirement. We will need to support VS2010 and VS2012, and in the future will also include VS2013 and maybe also MinGW. We're using cmake for building, so our code should build on all the platforms without issues; my problem is how to manage all the dependencies on Windows in order to be able to build in the first place, and keeping it up-to-date over time. At the moment, we have one virtual machine per visual studio version as a jenkins slave, so parallel builds of all the variants is fairly easy, but managing it is not.
The problem is the number of variants this requires building. If we consider only VS2010 and VS2012, with debug/release and i386/x64 builds, that's already 8 copies of each library; 16 if we include the other compilers. We will need all the libraries our code depends on, which will include at a minimum boost, qt, xerces+xalan, zlib, icu, libpng/tiff/jpeg, hdf5 and more, plus python, and all their dependencies. And as new upstream releases are made, we'll need to keep the entire collection up-to-date and consistent for all the build/arch/compiler variants.
I don't want to do this by hand, since this really needs automating. However, I'm unaware of any good solution for doing this on Windows. The Windows building guides I've seen for other projects often involve hand-building all the dependencies, and only build for a single variant. On Linux, it's already packaged, you don't need separate debug builds, and the arch variants can be catered for with chroots; on MacOS there's homebrew, macports etc., and it's also fairly simple to automate stuff there as well. Is there any equivalent for Windows? I've looked at stuff like chocolatey, but it's entirely unsuited to handling libraries, and is pretty poor as a package manager.
This seems like it should be a common problem for anyone doing C++ development on Windows? Are there any common solutions, tools or methodologies for managing a complex set of libraries and tools for development? How do other developers manage this?
NB. Just for the record, we are not using the visual studio application; we're doing all builds non-interactively via scripts driving the compilers directly with cmake and/or msbuild.
Many thanks,
Roger
I worked on large windows C++ project that delivers X86 Release, x86 Debug, x64 Release and x64 debug. Very similarly I used build system that does parallel builds for all target platforms using custom script.
We manage all third party dependency libraries in organized folders.
For example x86\release\Zlib.dll x86\Debug\zlib.dll x64\release\zlib.dll x64\zlib.dll
Custom script is made to pick all these libraries and project source code from configuration management tool. This allows to automatically build the relevant target binaries as needed.
any third party libraries change is updated in configuration management tool and then later picked up by the script for the next build.
For your question on VS2010 and 2012 support I don't understand importance. Is not one version of VS enough to support for the project?
you may take a look at Link, and their build system https://github.com/gisinternals/buildsystem
It's basicly a set of batch and make files calling each others. You still need keep track of lib update manually.
Related
The problem:
I'm writing an SDK that is primarily C++. The source code will be licensed to developers who pay for it, and the output libraries and include headers will be free for public usage. The SDK will target a plethora of platforms including Windows, Xbox, Playstation, Android, iOS, Mac OS X, and Linux. I'm a kind of guy who mostly likes Visual Studio and usually develops software using Windows machines. In the last few years, Visual Studio has made this quite a lot easier than it used to, where I have a mostly clear path to target all the previously mentioned platforms using a Visual Studio set of project files as the source of truth that brings all my source code files together... except for Mac OS X, unfortunately. Visual Studio is able to build executable code for iOS and Linux by remotely interfacing with a Mac or Linux box respectively for compilation and debugging, which is really quite cool, but for some reason Mac OS X as a target is left out here. Additionally I'm well aware that there are plenty of other professional developers out there that don't write code on Windows machines, nor do they have any interest in buying a commercial license for Visual Studio.
The question:
Since C++ still does not yet have a build system standard, and may not ever, how do I maintain a single source of truth that maintains the build configurations for all my source files targeting so many different platforms while simultaneously minimizing the barrier to entry of supporting software developer clients who need to build, run, and debug my source code?
Possible answers I'm aware of:
A) Visual Studio projects remain the source of truth from which any other C++ project types (such as Android Studio, XCode, and .make files) are derived. I'm aware of tools that can convert VS to .make and the like, but haven't actually tried them yet (my source code base is starting to get somewhat large already). Or I could just bite the bullet and write them by hand and try to keep them all in sync.
B) CMake. Sigh. So frustrating that it's very popular, and seems to exactly solve my issues, but it has its own set of problems that seem to be deal-breakers. For starters, once you go in on CMake, you pretty much can't come back. Using Visual Studio and property sheets, I've been able to tweak my build configurations with properties that are mostly inherited and rarely duplicated across projects and configurations. As far as I can see, CMake doesn't care about respecting such things, and for common properties, it just duplicates them on all vcxproj files. To make matters worse, all file paths it generates in the output projects are absolute, not relative, and to top it all off, it forces anyone else who builds your code to use CMake, disallowing distribution of the project build files it creates without it. Also, does this even work for game consoles? Last I checked I couldn't find a reasonable way of supporting them without hacking the source code.
C) Roll my own script that's similar to CMake, but allows redistribution of its output projects, and supports all the platforms I need. It goes without saying that this would consume a lot of dev time.
Any other options I'm missing here? Your input is greatly appreciated.
I agree with the comments from #Scheff and #arrowd. Use CMake. I have built and deployed software to multiple platforms and CMake is the best, though not perfect, solution I have found for building C++ code.
I have not had to hack the cmake code to get it to work on various platforms.
Do not worry about properties being duplicated in vcxproj files. With CMake the build language is in the CMakeLists.txt file(s). The vcxproj files are generated code. As long as you are do not have redundant cmake logic, you should not care about replicated properties in the generated vcxproj files. Similarly, you should not care that the generated vcxproj files have absolute paths; you do not reuse the vcxproj files you reuse the CMakeLists.txt files and regenerate vcxproj files for each new platform or build.
Use the top-level CMakeLists.txt to define the properties that are common to all targets. Then in individual target(s) CMakeLists.txt files use target properties to tweak builds of specific targets. In my experience the replication of properties in the generated files helps because they make the builds more consistent; I am able to minimize replication in the source CMake logic.
Various IDEs (VS 2017, CLion, QtCreator) can use cmake based projects directly.
There is nothing cmake specific about the generated artifacts. The headers, libraries (and dlls), and executables of our SDK are standalone artifacts. Yes, cmake can make it easier for your SDK users but using CMake in your SDK will not force your customers to use CMake.
Have you tried CMake and not been able to use it? Or are you looking for something better? There are certainly multiple C++ build systems, but despite its shortcomings I believe CMake is the best one available right now.
I made a little app and built a release version. Now I want to upload it to my site. I have never done this before with Qt, so I'm unsure as to what I should include along with the binary.
How do I figure out which DLLs should be included with my app? And where do I get them? I'm running Windows, but I'd also like to know what I should do in case I want to release a Linux version.
For windows:
You can use dependency walker to see what Qt libraries (or others) you should ship. This is the depends.exe executable that is included with Visual Studio, but you can download it separately from: http://www.dependencywalker.com/
Load your app into that and it will list out all the modules it expects at runtime. You might also have to ship a Visual C++ Runtime Redistributable compatible with the compiler that you built the executable with (if it's VC++).
Do note that dependency walker does not account for things like Qt's plugins. An example of this would be the QtAssistant system (for help menu-type functionality), which as of Qt4 relies on Qt's sqlite functionality, which is typically built as a plugin (qtsqlite4.dll if I remember).
For Linux:
This is trickier because of wider disparities in Linux distributions. You can of course use the GNU build system if you want to ship source, but if you're shipping binaries, and want to support a variety of distros, you might do best to build packages for each platform you want to release on.
In my past, a company I worked for switched to using cmake and after setting up all the project and build files, used that to generate builds and packages for different OSes. On Windows, this meant hooking in with Inno Setup, and for Unix-like systems, cmake knows how to generate things like installable shell scripts. Definitely made life much easier.
Our QA department would test our software in virtual machine instances of our supported platforms, completely clean, and see if anything was missing.
If you're talking about DLLs, I assume it is about Windows.
Use Dependency Walker to see the DLL dependencies.
Or... take a clean system, with no dev tools installed, and put your executable, try to run it there, and see what DLLs are reported as necessary and inexistent. Put the DLLs near the executable.
For a Linux version, you can either create platform targeted releases of installers for each Linux fork or you can let people compile from source. If your app is new, the only way you get exposure is supply people with readymade installers, the targeted installers. New users loathe compiling packages from source.
You can try debian (.deb) and redhat (.rpm) packages first. These two are extremely popular lines and will let you have a taste of things.
Seems most open source projects in C/C++ only provide the source code,i.e. nginx
Is this a convention that anyone interested in joining the developing team should figure out the .sln/.project files himself to qualify??
most open source projects are coming from the linux side of computing. thus, they are mainly using unix style build tools, as well as open source compilers.
the main build tool is make, which uses a makefile to know how to build a project. on Windows, the main open source compiler is MinGW which is a win32 port of gcc. the use of those tools allows to keep a maximum of common things between unix and windows.
note that .sln files are specific to microsoft compilers which are not free to use (and are rather costly), they are not portable and so are not suitable for multi-platform programming.
Some projects use CMake, which can generate project files for Your Favourite Build System, but as mentioned above, you don't need to use .sln and pro files, even if a project is built with the MSVC compiler, MinGW + makefiles, or scons, or CMake, or any other number of scripty methods to invoke the right commands to compile the program will work just fine.
Don't confuse the IDE with the compiler!
No, most opensource project do not use MSVC solutions as they not portable and very weak in terms of features.
In most cases they use what is called "build-system" like autotools, CMake or SCons.
These build systems include information about:
source code and how to build it
various system checks that should be done (like find various 3rd part libraries)
How to build and run unit-tests
How to install application
They also allow important tasks like cross compilation and packaging for deploy.
These task done via specific build system scripting language that allow you big flexibility.
So, these build systems generally much more powerful then typical "project files" and they
generally work with multiple compilers on different platforms and operating systems.
Some of build systems (like CMake) allow as one of the options to generate MSVC solutions as well as one of optional ways to build application.
I currently maintain a few boxes that house a loosely related cornucopia of coding projects, databases and repositories (ranging from a homebrew *nix distro to my class notes), maintained by myself and a few equally pasty-skinned nerdy friends (all of said cornucopia is stored in SVN).
The vast majority of our code is in C/C++/assembly (a few utilities are in python/perl/php, we're not big java fans), compiled in gcc. Our build toolchain typically consists of a hodgepodge of make, bash, grep, sed and awk. Recent discovery of a Makefile nearly as long as the program it builds (as well as everyone's general anxiety with my cryptic sed and awking) has motivated me to seek a less painful build system.
Currently, the strongest candidate I've come across is Boost Build/Bjam as a replacement for GNU make and python as a replacement for our build-related bash scripts. Are there any other C/C++/asm build systems out there worth looking into? I've browsed through a number of make alternatives, but I haven't found any that are developed by names I know aside from Boost's.
(I should note that an ability to easily extract information from svn commandline tools such as svnversion is important, as well as enough flexibility to configure for builds of asm projects as easily as c/c++ projects)
We've started using CMake here at work recently. I've been pretty happy with it so far.
scons or cmake.
Both "cross platform" - enable compiling on Windows and Linux.
Cmake now very popular (for example boost uses it). It creates native build files for each platform - Makefiles (for Linux), VC++ projects (for Visual Studio) from custom files (CMakeList.txt). In can also create Eclipse projects, KDevelop projects etc. Since it creates native build files, you can continue using all features you're used for. For example, ccache/colorgcc/distcc for Makefiles or Visual Assist X for Visual Studio.
We use it our project and are happy with it - automatic dependencies, easy syntax, robust builds.
Scons is python bases system, which perform the builds by itself. It's IMHO less popular, and still slow for large project. But for msmall to medium project maybe good alternative.
You could use python-based build system, too -- http://code.google.com/p/waf/
What would be the best choice of build system for a more than one million line multi platform project, which produces drivers, libraries, command line tools, GUIs, and OS install packages for all the mainstream OSes, using both the GNU and Microsoft toolchains?
Our source code is mainly C, with Python, C# and GNU makefile, and a little C++ and bash. It resides mainly in one repository, but we push source code to various third parties all of whom have their source code code repositories. There is also some interest in keeping the build fast, which might involve splitting up the project.
Currently we use a mixture of GNU make, bash, python and Microsoft's DDKBUILD. The main problems are that we are maintaining a complex set of scripts on top of make and would prefer to use third party (preferably open source) tools, and that cygwin is not proving to be robust on Windows (e.g. fork isn't always possible), and that our current build system does not build or install the toolchain so is vulnerable to tool chain version changes.
I vote for CMake, as a meta-building tool that really rewrite KDE4 build system from scratch -- and make KDE4 now a cross-platform desktop that even running on WindowsCE!
CMake is the carrier porting KDE4 to any OS on earth -- by generating Makefile( or vcprojs in Windows case) for about 40 OSes with relative toolchains!
JetBrains TeamCity works very well in general, so should be worth having on the eval list.
ThoughtWorks Cruise is also in the same space. While its v1, it comes from a stable that's been around for a while.
There's nothing about Team Foundation Server that would make necessarily count it out for your situation, but out of the box it might be more MS-shop centric that the other two I've mentioned.
As a general comment, with the level of variety you have, you definitely want to trial whatever it is you want to use - just because something is supported as a tick on the box doesnt mean its going to suit what oyu're looking for.
Dickson,
Is your build mostly monolithic or do you want to build some libraries separately and assemble them into the larger application? If inter-project dependencies are a big deal, your choices become limited quickly. AnthillPro does it well, and I think TeamCity has some Ivy integration support. From what you're saying, it sounds like this is not an absolute need, but might be helpful in speeding the build. It's certainly a strategy that we've seen a number of teams execute effectively.
Since you're looking at cross-platform (I assume multiple machine) builds, most of the open source tools other than Hudson are ruled out.
A build server comparison matrix is hosted by our friends at Thoughtworks here: confluence.public.thoughtworks.org/display/CC/CI+Feature+Matrix
Good luck.
You should have CMake on your list of alternatives to investigate. CMake is a meta-tool, i.e. it generates the input to the build-tool of your choice (GNU make, Visual Studio, etc.). I can recommend it strongly.
You may want to look at Cruise. It is built on Java so it will run on any platform that supports that. You can also have multiple build agents on different machines that can perform the different tasks on the different platforms. Thoughtworks is still building it out so some of the functionality is lacking, but it may be a a good option since you are looking for true cross-platform capabilities.
SCons is a cross-platform build system implemented in Python. We use it to build our code on three platforms. It can automatically detect your build tools but you can also put arbitrary Python code in your build script. It also lets you separate your environment setup from description of your project structure, a great feature for reuse of your buidl scripts in different environments. Besides building your project directly, it can also generate Visual Studio project files.