BACKGROUND
Over the course of my career I have been surprised by how many projects I've seen where it is a real challenge to compile and execute a project in Visual Studio. The source of the problem generally is due to: missing dependencies, lack of documentation, broken project references, etc.
To avoid these headaches I try to automate projects/solutions such that:
the run-time environment is automatically setup when a project is compiled on the developer machine (e.g. use batch scripts to import missing Windows Registry keys)
when compiling a project, the correct dependencies are automatically retrieved (on both the build machine & the developer machines)
THE PROBLEM
To date, I have had a fair amount of success with this approach. However, I have recently been handed a native C++ project that has a dependency on the Microsoft Windows SDK. At compile time, the project makes use of Windows environment variables to locate missing dependencies (e.g. Microsoft Windows SDK).
I understand that using environment variables is how things used to be done. However, by relying on the software developer to configure the development environment:
you are assuming that they will configure the environment properly
the developer is wasting time on configuration when their time could be better spent developing
I do not want to debate the merits of having a developer configure the development environment, but rather, I would like to know:
Given the technology (e.g. TFS) that exists today, what is a reliable and repeatable approach to handling large dependencies (e.g. Windows SDK) for C++ projects in a team environment?
POTENTIAL SOLUTIONS
continue to use environment variables
Adv: once the dependencies are installed, it is very easy for the build machine to compile projects
Dis: you have to spend time documenting to ensure that you can configure the build machine from scratch (e.g. step1: install dependency A, step2: install dependency B, etc.)
Dis: You are relying on the magic environment variables to be pointing at the right target.
Dis: the developer is wasting time configuring when they should be developing
check dependencies into TFS
Adv: everything is kept in one centralized location
Adv: by design, source control keeps a history
Adv: in a sense, source control makes things self-documenting
Dis: Compiling on the build machine now takes considerably longer as the build machine
workspace has to repeatedly retrieve the Windows SDK from TFS
Other?
CONTEXT
Programming Language: unmanaged C++
Source Control: TFS 2012
Dependencies:
Microsoft Windows SDK (~416Mb)
in house libraries
I have limited knowledge of how to administer/configure the TFS build machine.
REFERENCES
Microsoft: Team Development with Visual Studio TFS (Chapter 6)
I remember while working for a security company, the team had a script that usually copies all dependencies for you as soon as you hit compile, to a specific folder for you. its in build properties, for an MFC project, however, it was confusing to me at the time.
the reference seemed very helpful thank you
Related
I'm interested in using Azure Functions in order to build release packages for windows. Mainly centering around electron/node. Specifically, need to be able to do binary compiles of some modules (sqlite3 namely). Which means VC++ Build environment + Windows SDK, Python ^2.7, Node and related...
Has anyone done this via Azure Functions? I know there are similar questions for getting NuGet packages in an azure function, and it does seem that there is an option for the C++ tools, but not sure where to even start.
The main hope is to do this for less than the cost of a full VM for something that is likely to run less than 2-3 times a month.
I'm sorry - this is not a scenario suitable for Azure Functions.
The only way to include "binaries" in a Functions is by referencing assemblies you uploaded upfront. You do not have access to other binaries like VC++ build tools.
If you do not want to pay for a full fledged VM, may be think about using VSTS with its build and release features. With these you basically utilize MS VC++ etc. build tools in a shared and I assume cheaper way. You then are able to trigger those builds from outside with REST / service hooks.
We have a complex C++ codebase targeting multiple mobile platforms. We currently have Windows CE (4.2 to 6.5 both raw CE and Mobile based on those), Android (2.1+), iPhone (4+), almost working Bada (2.0+) and if anything comes of the new C++/CX thing, are likely to add Windows Phone (8+). Plus a testing version on Win32 and service application on Win64 that shares some code. We also already tried to compile unit tests on Linux and questions already came (too small business volume so far, but that might change) to get it working on some other Linux platforms.
We currently compile the code with native tools for each platform. Each of them is pretty complex and has some hacks in or around it to achieve reasonably single-click builds. And for Bada, we didn't solve builds outside of the Samsung-tweaked Eclipse yet, which we'll have to do for production.
So far it works, but is becoming more and more of a maintenance problem. The biggest problem currently is the iPhone build, because unlike Visual Studio project files and plain makefiles, it's not possible to add/remove/rename files in XCode project by hand and only two people have MacOS boxen and any experience with XCode (while everybody has Windows and knows Visual Studio). We also need to create some makefiles for the Bada target (it's plain GNU toolchain, so anything capable of cross-compiling with those should do) and I wouldn't mid getting rid of some of the kludges in Android build: fix for bug in dependency handling when building under cygwin, some hacks on top of the ant build script and shell glue to massage manifest and hold it all together.
So I am looking for advice on ways to unify the build process for this set of diverse platforms.
It absolutely has to handle building the iPhone and Bada executables and the native part of the Android build (because nobody has all three platforms to test separately).
Has to handle large project consisting of several shared libraries, one main binary, one test binary for the same platforms and some auxiliary binaries built only on some platforms (well, currently Win32 only).
Has to be able to generate build configuration header and Java file with version from version control system and some user-defied variables, since we do 18 and counting slightly different builds for various customers.
And of course it has to automatically handle dependencies (header files) and generally be reliable.
For the other things I am prepared to hack around any deficiencies, but I would obviously like to keep the amount of shell duct tape and spit to minimum, so it should:
Be able to integrate with Visual Studio, Eclipse and XCode enough that each environment can trigger build, upload the build product to the respective target and attach debugger to it there.
Be able to build Java and call the custom packaging tools for Android, so we don't have to hack around the ant build script that Google gratuitously and incompatibly changed twice in last two years already (and old SDK can't be downloaded).
Be able to install various data files and call random packagers and random other scripts and tools and stuff, so it does not have to be mixed with too much shell script.
We have so far started trying CMake (didn't have much time, so didn't get far, but will have to do something shortly) and also thought about SCons. However I have already tried building with SCons for Windows CE a few years ago, but since it generates makefile-type projects for Visual Studio and those didn't work for embedded platforms in VS2005, I gave up. CMake can generate native makefiles, but a custom, bit out of date, branch is needed for CE. So I'd like to ask whether there are any other tools we might want to look into or any known stumbling blocks with these tools we should be aware of.
Updates: I've found instructions for android native binaries in several places and pixellight even has cmake script to generate apk by directly calling the packaging tools. Also this shell script shows it. iPhone usage seems to be documented here.
The same CMake build files can be used to generate and build projects across platforms for atleast Win32 (VS2005), Linux(ubuntu), and OS X (10.5.8 Leopard), in my personal experince. There are multiple projects for using CMake in the android app build process, eg http://code.google.com/p/android-cmake/. With all these options, I would consider CMake a good option. Further CMake is easy to pick up, and even easier if you have prior experience with writing configure.ac and Makefile.ac files.
Note: How CMake works:
Developer/Maintainer creates CMakefile.
Builder runs cmake, which uses the CMakefile to generate platform specific build files.(linux makefiles/OSX makefiles/VS project files)
Builder fires off the build command.
CMake is the best choise.
Starting with CMake 2.8.11 Windows CE is supported out of the box. Until the release of that version a ​Nightly Binary of CMake can be used.
I'm looking to implement a build server for Qt-based C++ projects. The server needs to check out the necessary code / assets from Subversion, build the executable files, assemble the artifacts for installation projects, and build the installation media files. The target platforms and (rough) toolchains are:
Windows (32- and 64-bit): qmake, nmake, msbuild, wix toolchain. The end result is an installer EXE and DVD image.
Mac OS X: qmake, make, custom bash scripts to assemble package. The end result is an application bundle within a disk image and a DVD image.
Ubuntu (32- and 64-bit): qmake, make, debuild-based scripts. The end result is a collection of DEB files and a DVD image.
Fedora (32- and 64-bit): qmake, make, rpmbuild-based scripts. The end result is a collection of RPM files and a DVD image.
So that's at least 4 build agents (maybe more if 32- and 64-bit can't be done on the same box) and 7 configurations. Open-source projects are preferred, but that is not an absolute requirement.
Most of the tools I'm seeing seem to be catered to Java (Jenkins, CruiseControl, etc.) or .Net (CruiseControl.net, etc.) Can those be used with a C++ toolchain, or will I constantly be fighting the system? Anything that you have used in the past and found works well with Qt / C++?
I use Jenkins for building and packaging many C++ projects, based on qmake, cmake, and makefiles.
There are plugins for cmake, qmake, and msbuild, but any command line scripts can be run as well.
I have done packaging using Jenkins with no problems, as it is just another command line step in a project.
There are good plugins for monitoring the number of warnings/errors produced by the compiler (I normally use GCC).
It also has matrix builds which allow you to build a project several times with different combinations of compiler flags, pre-processor variables, platform, etc. One project I set up is a matrix build with 5 boolean preprocessor flags on two platforms, which then does 2^6=64 builds. These can take a bit of setting up to get correct.
Here you can read a quick example:
Continuous Integration Server - Hudson
I think that Hudson, jenkins and builbot are worth a try. Wasting a day or two evaluating and trying them with a quick example will help you to choose confidently.
Most of the tools I'm seeing seem to be catered to Java (Jenkins, CruiseControl, etc.) or .Net (CruiseControl.net, etc.) Can those be used with a C++ toolchain, or will I constantly be fighting the system? Anything that you have used in the past and found works well with Qt / C++?
Any reasonably capable CI system will have a piece that will allow you to execute any program you want for your build command.
Here's what I would consider:
Does the CI system run on your system(s) of choice
Does it allow you an easy way to view your logs
Does it integrate with your test runner
Does it integrate with your code coverage reports (e.g. BullseyeCoverage w/C++ & Qt)
Will it publish your files in a manner sensible for your needs
Will at provide an archive/store of files, if necessary (e.g. pdbs & lib*.so.debug)
If the CI system doesn't support feature X, will you have to write it for each supported OS/system
Is the CI system / UI easy for you to use.
I did the above using CruiseControl and most things were pretty easy. I wrote everything in make or qmake and simply called out to the command that I needed executed. For unit test and code coverage integration I output stuff to XML and transformed it to something supported by CruiseControl.
My recommendation, take a look at the recommended CI systems and examine them based on the criteria above.
I'm using buildbot for this. I've been using it for 4 years, and I feel very happy with it.
It is an application written in python, that runs on a server and can manage multiple clients on various OSes. I'm currently using Windows XP, Windows 7, Debian, Ubuntu and CentOS build slaves. My projects are C++, and one of them (the end user GUI) is made in Python. But we've also integrated with other frameworks, for other features than GUI.
What is really good about buildbot is that it works by running command lines on slaves. With this, you can do whatever you want. Even on Windows systems to compile using Visual Studio! From these command lines, you get all the output centralized on the server, and accessible.
You may also find alternatives on this site that references many of them.
Disclaimer: I looked at it 3 years ago, I don't know if it is still accurate.
Hudson or Jenkins is pretty good.
Jenkins is indeed pretty popular for developing such a custom service, even after all these years, considering the question is already 7 years old.
Felgo also offers a Continuous Integration and Delivery (CI/CD) service for Qt. It supports desktop platforms as well as iOS, Android and embedded targets. The full feature set is described in the blog post.
Disclaimer: I am a software developer at Felgo
I am currently researching a good setup for a continuous integration server which would build various C++ applications for several Linux distributions.
My primary question is how other users here have handled the differences in system libraries between Linux distributions?
While it might be relatively easy to build direct dependencies such as UI libraries along with an application, "indirect" dependencies such as glibc look like a big pain if they had to be built alongside the application every time. I am therefore thinking of moving the actual build execution into a separate virtual machine for each distribution, e.g. using rlogin to run the commands. My goal is to prevent binary incompatibilities between build-machine library versions and those deployed in the target distributions.
Does anyone here have any experience with such a process and could tell if the above sounds like a feasible approach?
We use Jenkins (Contiguous Integration) and CMake (build system) for this purpose. Jenkins is similar to Buildbot, i.e. it also has buildmaster and buildslaves. Currently I have setup 8 slaves to build for 4 different platforms (FC8, FC10, FC12 and Windows 7). We build both debug and release binaries, so I dedicated one slave for each platform and build type.
As for the third party libraries like Qt & Boost, I compiled them on each platform and checked them into a separate repository.
#esavard: We use CMake 2.8 to do cross compilation, I have not used minigw but a quick google search indicates that it is possible. Here is a link to a tutorial to cross compile for Windows on Linux using CMake and miniGW.
I have not used Buildbot and cannot comment on its features but thought I should mention an alternative that we are currently using.
Hope this helps.
Buildbot has the notion of buildmasters and buildslaves.
A buildmaster takes care of displaying the web GUI, sending email, triggering builds, and other housekeeping. The buildslaves wait on the buildmaster and when commanded perform builds.
We have buildbot set up to build on a number of different platforms, some of them VMs, and it's working well for us.
Certainly buildbot and many virtual machines is the way to go with this. We have VMWare ESX server hosting many build slaves which overnight compile our application. The application is then tested on another virtual machine (not the build slave and just having a default OS install) to verify that it works and all dependency's are packaged.
The nice thing I would like to do is make the testing run time phase an automated step but I haven't been given the time to do that yet.
I'm involved in C++ project targeted for Windows and Linux (RHEL) platforms. Till now the development was purely done on Visual Studio 2008. For Linux compilation we used 3rd party Visual Studio plugin, which read VS solution/perojects files and remotely compiled on Linux machine.
Recently the decision was to abandon the 3rd party plugin.
Now my big concern is a build system. I was looking around for cross platform build tools. This way I don't need to maintain two set of build files (e.g. vcproj/solution for Windows and make files for Linux).
I found the following candidates:
a. Scons
b. cmake
What do you think about the tools for cross-platfrom development?
Yet another point that bothers me is that Visual Studio (+ Visual Assist) will loose a lot functionality without vcproj files - how you handle the issue with the tools?
Thanks
Dima
PS 1: Something that I like about Scons is that it
(a) uses python and hence it's flexible, while cmake uses propriety language (I understand that it's not a winner feature for a build-system) (b) self contained (no need to generate makefiles on Linux as with cmake).
So why not Scons? Why in your projects the decision was to use cmake?
CMake will allow you to still use Visual Studio solutions and project files. Cmake doesn't build the source code itself, rather it generated build-files for you. For Linux this can be Code::Blocks, KDevelop or plain makefiles or still other more esoteric choices . For Windows it can be among others Visual Studio project files and still others for MacOS.
So Visual Studio solutions and projects are created from your CMakeLists.txt. This works for big projects just fine. E.g. current Ogre3d uses CMake for all platforms (Windows, Linux, MacOS and IPhone) and it works really well.
I don't know much about scons in that regard though, I only used to build one library and only in Linux. So I can't compare these two on fair ground. But for our multi-platform projects CMake is strong enough.
I haven't used Scons before, so can't say how that works, but CMake works pretty well.
It works by producing the build files needed for the platform you're targeting.
When used to target VC++, it produces solution and project files so from VS, it appears as if they were native VS projects. The only difference is, of course, that if you edit the project or solution directly through VS, the changes will be erased the next time you run CMake, as it overwrites your project/solution files.
So any changes have to be made to the CMake files instead.
We have a big number of core libraries and applications based on those libraries. We maintain a Makefile based build system on Linux and on Windows using the Visual Studio solution for each project or library.
We find it works well for our needs, each library or app is developed either on linux or windows with cross compilation in mind (e.g. don't use platform specific api's). We use boost for stuff like file paths, threads and so on. In specific cases we use templates/#defines to select platform specific solution (for example events). When is ready we move to the other system (linux or windows), recompile, fix warnings/errors and test.
Instead of spending time figuring out tools that can cross compile on both platforms we use system that is best for each platform and spend time fixing specific issues and making the software better.
We have GUI apps only on Windows atm. so there's no GUI to cross compile. Most of our development that is shared between Windows and Linux is server side networking (sockets, TCP/IP, UDP ...) and then client side tools on Linux and GUI apps on Windows.
Using with perforce for source code version management we find in quite many cases that the Linux Makefile system is much more flexible for what we need then Windows VS. Especially for using multiple workspaces (views of source code versions) where we need to point to common directories and so on. On Linux this can be done automatically running a script to update environment variables, on Visual Studio referencing environment variables is very inflexible because it's hard to update automatically between views/branches.
Re sync question:
I assume you are asking how to make sure that the two build systems get synchronized between linux and windows. We are actually using Hudson on Linux and CruiseControl on Windows (we had windows first with cruise control, when I went to setup linux version I figured Hudson is better so now we have mixed environment). Our systems are running all the time. When something is updated it is tested and released (either windows or linux version) so you would know right away if it does not work. During testing we make sure all the latest features are there and fully functional. I guess that's it, no dark magic involved.
Oh you mean build scripts ... Each application has it's own solution, in solution you setup up dependencies. On Linux side I have a makefile for each project and a build script in project directory that takes care of all dependencies, this mostly means build core libraries and couple of specific frameworks required for given app. As you can see this is different for each platform, it is easy to add line to build script that changes to directory and makes required project.
It helps to have projects setup in consistent way.
On Windows you open project and add dependency project. Again no magic involved. I see this kind of tasks as development related, for example you added new functionality to a project and have to link in the frameworks and headers. So from my point of view there is no reason to automate these - as they are part of what developers do when they implement features.
Another options is premake. It's like cmake in that it generates solutions from definition files. It's open source and the latest version is very highly customizable using Lua scripting. We were able to add custom platform support without too much trouble. For your situation it has support for both Visual Studio and GNU makefiles standard.
See Premake 4.0 Homepage
CruiseControl is a good choice for continuous integration. We have it running on Linux using Mono with success.
Here is an article about the decision made by KDE developers to choose CMake over SCons. However I've to point that this article is almost three years old, so scons should have improved.
Here is comparison of SCons with other building tools.
Had to do this a lot in the past. What we did is use gnu make for virtually everything including windows at times.
You can use the project files under windows if you prefer and use gnu make for Linux.
There isn't really a nice way to write cross platform makefiles because the target file will
be different among other things (and pathname issues, \ vs / etc). In general, you'll probably be tweaking the code across the various platforms to take subtle differences into account, so a tweak to a make file and checking on the other platforms would have to happen
anyway.
Many OS projects maintain Makefiles for different platforms such as zlib where they are named like Makefile.win, Makefile.linux etc. You could follow their lead.