I'll start to write a automated build/deploy script for our software team, and I'm not sure which scripting language or tool to use. I'm always eager to learn something new and improve my value on the current software market, so what language do you propose?
In the past, I used windows batch files, which are kind of ugly to read and script.
Here are few words as of my requirements and environment:
developing under Win XP using Visual
Studio 2008 (C++)
subversion
looking for easy to learn & use
scripting language
programming experience in PHP & Ruby & C++
I want the script to
checkout
clean & build many projects
check for errors rinse and repeat
until no errors or timeout
run unittests and evaluate results (text parsing)
zip a bunch of dlls/exe and copy to
server
brew a cup of coffee if possible
Thanks for any suggestions. Could be a community wiki since there's no definite answer, not sure about that...
EDIT:
Found those discussions, but they are a little older and perhaps outdated:
How do you automate a Visual Studio build?
Best .NET build tool
I'd suggest setting up a CI server (Hudson?) and use this to control all deployments to both your QA and production servers. This forces you to automate all aspects of deployment and ensures that the are no ad-hoc restarts of the system by developers.
I'd further suggest that you consider publishing your build output to a repository manager like Nexus , Artifactory or Archiva. In that way deployment scripts could retrieve any version of a previous build. The use of a repository manager would enable your QA team to certify a release prior to it's deployment onto production.
Finally, consider one of the emerging deployment automation tools. Tools like chef, puppet, ControlTier can be used to further version control the configuration of your infrastructure.
It was briefly mentioned, I like PERL for these sorts of things. On an absolutely humongous project I worked on, we had perl scripts practically running the entire build process. It even built classes based upon messaging templates. It was pretty cool, and this was a Windows project to boot.
CPAN provides an enormous resource.
HTH
I would recommend python for this sort of generic scripting task; it's a relatively easy language to learn and very flexible.
In particular, you would use the subprocess module to call all the external programs that you need; it makes capturing their output and controlling them very straightforward.
I'm currently using Apache Ant combined with MPC (home brewed cross platform project) after successfully using the combination at one of my previous jobs, though XML does not quite qualify as a scripting language. But except for the cup of coffee it can do everything on your list. MPC is written in Perl and it's pretty flexible e.g. it allows you to specify postbuild and prebuild actions for VS projects.
EDIT: I was just browsing the site and stumbled on this question; see the answer about
FinalBuilder (I'm considering giving it a try; you'd have to pay for it though)
Related
I am working on C++ programming with perforce (a version control tool) on VMS.
I need to handle tens or even hundreds of C++ files (managed by perforce) on VMS.
I am familiar with Linux, python but not DCL (a script language) on VMS.
I need to find a way to make programming/debug/code-review as easy as possible.
I prefer using python and kscope (a kde based file search/code-review GUI tool that can generate call graph) or similar tools on VMS.
I do not have sys-adm authorization, so I prefer some code-review GUI tools that can be installed without the authorization.
Would you please give me some suggestions about how to do code-review/debug/programing/compile/test by python on VMS meanwhile using kscope or similar large-scale files management tools for code-review ?
Any help will really be appreciated.
Thanks
Your question is pretty broad so it's tough to give a specific answer.
It sounds like you have big goals in mind which is good, but since you are on VMS, you won't have a whole lot of tools at your disposal. It's unlikely that kscope works on VMS. Correct me if I'm wrong. I believe a semi-recent version of python is functional there.
I would recommend starting off with the basics. Get a basic build system working that let's you build in release and debug. Consider starting with either MMS (an HP provided make like tool) or GNU make. You should also spend some time making sure that your VMS based Perforce client is working too. There are some quirks that may or may not have been fixed by the nice folks at Perforce.
If you have more specific issues in setting up GNU make (on VMS) or dealing with the Perforce client on VMS, do ask, but I'd recommend creating separate questions for those.
Indeed, it's not clear from your question what sort of programming you want to do on VMS: C++ or python??
Assuming your first goal is to get familiar with the code-base, i.e. you want the ease of cross-ref'ing the sources:
If you have Perforce server running on VMS, then you may try to connect to it directly with Linux Perforce client. And do "review" locally on Linux.
If you've no Linux client, I'd try fetching the latest revisions off and importing raw files it into an external repository (svn, git, fossil etc.). Then again using Linux client and tools.
If your ultimate goal is to do all development off VMS, then it may not really be viable -- the code may use VMS specific includes, system/RMS calls, data structs. And sync'ing the changes back and forth to VMS will get messy.
From my experience, once you're familiar with the code-base, it's a lot more effective to make the code-changes directly on VMS using whatever editor is available (EDIT/TPU, EDT, LSE, emacs or vim ports etc.).
As for debugging - VMS native debugger supports X-GUI as well as command-line. Check your build system for debug build, or use /NOOPT /DEBUG compile and /DEBUG link qualifiers.
BTW, have a look into DECset, if installed on your VMS system.
EDIT 2012.03.02: updated build tool section with some other requirements. Addding premake.
EDIT 2012.03.05: updated build tool section. Focused on CMake. Added Ant + CMakeAnt.
I have a lot of small Visual C++ projects and I want to start using continuous integration for that. A lot of tools are involved in the process. I start reading articles here and there and it is hard for me to see which to tool could integrate nicely with the others. So I am looking for an already tested continuous integration workflow that could respect my needs:
Current Context
C++/Qt applications developed with Visual Studio;
Less than 1'000 source files;
Small dev team;
Small budget;
Preferably open source/free software;
Looking for easy to use and simple tools. No need of something heavy and "powerful".
Finally packages software (installers ready for download) should work for windows and OSX. Linux (most popular distributions) is a plus.
Tools needed
Continuous Integration Server:
Needs to be Jenkins. I know it and I like it. I could change for another one if it is really needed.
Source Code Management:
GIT.
Documentation Generator:
Doxygen.
Build Tool:
We will need to generate solutions and configurations for all the platforms we need to support. I have been strongly advised to use CMake for that. Ant + CMakeAnt (http://code.google.com/p/cmakeant/) seems to be a nice addition too.
Unit Testing Framework:
CPPunit, C++Test, Googletest?
Installers Builder
CPack (part of CMake seems great too).
Hardware:
Preferably, one single machine with VirtualBox or VMWare. is it only possible? Would it be better to use Linux for such things? Or would a MAC mini be sufficient?
So, has anyone here already built something similar? With which tools exactly?
Thanks for any comments and suggestions.
I have successfully used CMake/CTest/CPack with Jenkins, there is a CMake builder plugin available.
I found the ctest --output-on-failure option to be useful as the output from the test is shown in the Jenkins log on failure.
I use the Boost.Test framework for testing C++, it works fine with ctest, I run it with the -l all option to get all the logging output when something goes wrong.
You should be able to bootstrap the entire process using a bunch of scripts.
Write them in Perl or Python. There are a million tutorials on how to bootstrap these sorts of processes.
If you're stuck on implementing a specific party of this build-process, ask a specific question about it.
I'm looking to implement a build server for Qt-based C++ projects. The server needs to check out the necessary code / assets from Subversion, build the executable files, assemble the artifacts for installation projects, and build the installation media files. The target platforms and (rough) toolchains are:
Windows (32- and 64-bit): qmake, nmake, msbuild, wix toolchain. The end result is an installer EXE and DVD image.
Mac OS X: qmake, make, custom bash scripts to assemble package. The end result is an application bundle within a disk image and a DVD image.
Ubuntu (32- and 64-bit): qmake, make, debuild-based scripts. The end result is a collection of DEB files and a DVD image.
Fedora (32- and 64-bit): qmake, make, rpmbuild-based scripts. The end result is a collection of RPM files and a DVD image.
So that's at least 4 build agents (maybe more if 32- and 64-bit can't be done on the same box) and 7 configurations. Open-source projects are preferred, but that is not an absolute requirement.
Most of the tools I'm seeing seem to be catered to Java (Jenkins, CruiseControl, etc.) or .Net (CruiseControl.net, etc.) Can those be used with a C++ toolchain, or will I constantly be fighting the system? Anything that you have used in the past and found works well with Qt / C++?
I use Jenkins for building and packaging many C++ projects, based on qmake, cmake, and makefiles.
There are plugins for cmake, qmake, and msbuild, but any command line scripts can be run as well.
I have done packaging using Jenkins with no problems, as it is just another command line step in a project.
There are good plugins for monitoring the number of warnings/errors produced by the compiler (I normally use GCC).
It also has matrix builds which allow you to build a project several times with different combinations of compiler flags, pre-processor variables, platform, etc. One project I set up is a matrix build with 5 boolean preprocessor flags on two platforms, which then does 2^6=64 builds. These can take a bit of setting up to get correct.
Here you can read a quick example:
Continuous Integration Server - Hudson
I think that Hudson, jenkins and builbot are worth a try. Wasting a day or two evaluating and trying them with a quick example will help you to choose confidently.
Most of the tools I'm seeing seem to be catered to Java (Jenkins, CruiseControl, etc.) or .Net (CruiseControl.net, etc.) Can those be used with a C++ toolchain, or will I constantly be fighting the system? Anything that you have used in the past and found works well with Qt / C++?
Any reasonably capable CI system will have a piece that will allow you to execute any program you want for your build command.
Here's what I would consider:
Does the CI system run on your system(s) of choice
Does it allow you an easy way to view your logs
Does it integrate with your test runner
Does it integrate with your code coverage reports (e.g. BullseyeCoverage w/C++ & Qt)
Will it publish your files in a manner sensible for your needs
Will at provide an archive/store of files, if necessary (e.g. pdbs & lib*.so.debug)
If the CI system doesn't support feature X, will you have to write it for each supported OS/system
Is the CI system / UI easy for you to use.
I did the above using CruiseControl and most things were pretty easy. I wrote everything in make or qmake and simply called out to the command that I needed executed. For unit test and code coverage integration I output stuff to XML and transformed it to something supported by CruiseControl.
My recommendation, take a look at the recommended CI systems and examine them based on the criteria above.
I'm using buildbot for this. I've been using it for 4 years, and I feel very happy with it.
It is an application written in python, that runs on a server and can manage multiple clients on various OSes. I'm currently using Windows XP, Windows 7, Debian, Ubuntu and CentOS build slaves. My projects are C++, and one of them (the end user GUI) is made in Python. But we've also integrated with other frameworks, for other features than GUI.
What is really good about buildbot is that it works by running command lines on slaves. With this, you can do whatever you want. Even on Windows systems to compile using Visual Studio! From these command lines, you get all the output centralized on the server, and accessible.
You may also find alternatives on this site that references many of them.
Disclaimer: I looked at it 3 years ago, I don't know if it is still accurate.
Hudson or Jenkins is pretty good.
Jenkins is indeed pretty popular for developing such a custom service, even after all these years, considering the question is already 7 years old.
Felgo also offers a Continuous Integration and Delivery (CI/CD) service for Qt. It supports desktop platforms as well as iOS, Android and embedded targets. The full feature set is described in the blog post.
Disclaimer: I am a software developer at Felgo
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
There's a lot of questions on here regarding various niche build needs (.NET, continuous integration, etc) but, of course, my niche need is different.
Rather than asking a very specific question right now, I'd like a survey of available build tools (such as make, ant, etc) so I can ask a follow up question more intelligently if needed.
In your answer, please include:
ONE build tool
Link to the main page about that tool
If you know some pros and cons (ie, runs on windows with cygwin, or .NET specific, etc) then list some (not required - the name and link are required)
If your build tool is already in an answer, comment on it adding pros/cons/limitations/experiences. Feel free to add as many answers as you like as long as they don't already exist - but don't put several build tools into one answer.
SCons
Build scripts are Python scripts. Supposed to work on Linux, Windows, Mac OS X.
NAnt
http://nant.sourceforge.net/
Similar to Ant, a build tool for .Net
Rake
Ruby version of make
Pros:
Clean idiomatic ruby syntax
Rake files are written in ruby, so leverages ruby itself rather than proprietary file format or XML
Ruby on Rails tightly integrated with it
Cons:
Requires ruby, which is not always available
No built-in packaging support/versioning/dependency management (see rubygems)
premake
Build script generator. Uses Lua to describe the build. It can generate Makefile or project files for Visual Studio, Code::Blocks, CodeLite, SharpDevelop, MonoDevelop, etc. Built-in support for C, C++, C#, including things like resources and ASP.NET. Mono support.
Jam family.
Jam
FTJam
BJam
BJam is based on FTJam, which in turn is based on Jam. Small, fast, portable. Automatic dependency analysis for C, C++. It is not a generator -- it does not generate any Makefile or other build files for secondary build systems.
Apache Maven
Pros:
Automatic dependency management
Utilizes convention over configuration (enforces consistent project directory structures)
Projects that use maven can be built in minutes
Excellent support for standard java/Java EE project builds
Works well in a team environment (supports "team" configurations)
Built in release/versioning concepts
Integrates with popular Java IDEs
Growing support in java OSS community
Cons:
Steep learning curve (esp. compared to Ant)
Poor online documentation (the new book is excellent though)
Sometimes surprising behavior
Very java-centric
XML configuration driven
GNU Make
http://www.gnu.org/software/make/
CMake
Cross-platform build system generator. It can generate Makefile or project files for Visual Studio, Eclipse CDT, KDevelop, Code::Blocks, etc. Automatic dependency analysis for C, C++, Fortran, Java.
Apache Ant
Pro:
many task/plugins
runs on many platforms
very mature
is supported by many IDEs, Continuous-Integration-Tools etc.
Con:
requires Java
FinalBuilder
Pros
Visual and GUI-oriented unlike ant or make
Lots of built-in actions
Script builder for your own actions
Integrates easily with cc.net
Cons
Only runs on Windows
Not free, but you get good value for your money
They also have a tool called Automise that does more system-oriented tasks.
Team City
Pros
All available here
Cons
Can be memory intensive
MSBuild
http://msdn.microsoft.com/en-us/library/wea2sca5(VS.80).aspx
CruiseControl/CruiseControl .Net
http://cruisecontrol.sourceforge.net/
Pros
Can use various build and TDD tools depending on version chosen
Automate when builds happen and what kind - full release build or changed code build
Can automatically run tests (nUnit, jUnit, Fitnesse, etc.) on a build to ensure that all tests still pass when new code is checked into the project.
Integrates with source control
Error reporting and notification when builds fail
Cons
Error reporting isn't always in a nice pretty to read format
Set up of projects should be well thought out and all projects monitored should use certain patterns to make integration setup more efficient.
PSake (pronounced "Sake")
Pros -
Powershell
Cons -
Powershell :)
This project is fairly new, looks interesting, and would be very powerful, since it's ".NET at the command line". Unfortunately, I don't know very many people who take the time to learn Powershell.
cons
Pros:
Uses Perl, if you like that sort of thing
Haven't used it otherwise :-)
Cons:
Hard to Google for!
Not actively developed?
UppercuT -
Some good explanations here: UppercuT
Pros -
Super easy to get started - Automated Builds as easy as (1) solution name, (2) source control path, (3) company name for most projects!!!
Limited knowledge of NAnt necessary.
Cons -
Only available for .NET
NUBuild (pronounced "New-Build")
This is the latest and pretty advanced .Net build tool (a very intelligent one) called NUBuild that allows you to build VB.Net and C# projects. Its extremely lightweight, open source and at the same time easy to setup and provides almost no-touch maintenance.
Easy to use command line interface
Ability to target all .Net framework version i.e. 1.1, 2.0, 3.0 and 3.5
Supports XML based configuration
Supports both project and file references
Automatically generates the “complete ordered build list” for a given project
Ability to detect and display circular dependencies
Perform parallel build
Ability to handle proxy assemblies
Easily integrated with Cruise-Control.Net continuous integration system
Version management capability
Notification feature
http://nubuild.codeplex.com/
It's best the build tool we have seen so far!
Bakefile
Bakefile is cross-platform, cross-compiler native makefiles generator. It takes compiler-independent description (XML) of build tasks as input and generates native makefile (autoconf's Makefile.in, Visual C++ project, bcc makefile etc.).
Bakefile's task is to generate native makefiles, so that people can keep using their favorite tools. There are other cross-platform make solutions, but they either aren't native and require the user to use unfamiliar tools (Boost.Build) or they are too limited (qmake).
Continuous Integration toolchains for .NET, Java, and other languages are relatively well defined, but the C++ market seems to have a lot of diversity.
By CI "toolchain" I specifically mean tools for the build scripts, automated testing, coding standards checking, etc.
What are C++ teams using for CI toolchains?
Another option might be buildbot.
It's written in python, but is not just for python apps. It can execute any script for doing your build. If you look at their success stories, there appear to be a wide variety of languages.
We implemented our C++ cross platform continous integration infrastructure using Parabuild
http://www.viewtier.com/products/parabuild/screenshots.htm
We were able to integrate every sort of Win/Mac/Linux QA tool with it and it's really easy to install and maintain: it's one click installation on every platform and the web interface is very handy.
While evaluating several continous integration servers the main problem was that they were Java-biased: Parabuild, on the other hand, fits well in the C++ cross platform development and QA workflow
Visual Build Professional is my favorite tool for pulling together all the other tools. Windows only, of course, but it integrates with all flavors of Visual Studio and a host of test tools, source controls tools, issue trackers, etc. It is windows only, though. I know that's not the entire stack, but it's a start.
G'day,
We actually faced this problem at a site where I was contracting previously.
One bloke sat down and wrote tools, mainly shell scripts, to
check out the current code base every hour or so and do a build to check if it was broken, and
check out the latest good build and do a complete build and run about 8,000 regression tests.
We just couldn't find anything commercially available to do this and so Charlie sat down and wrote this in bash shell scripts and it was running on HP-UX.
cheers,
Rob
As with seemingly every other task in C++, I'm just barely limping along with continuous integration. My setup starts with Eclipse. I set it to generate make files for my projects. I have ant scripts that do the overall build tasks by running 'make all' or 'make clean' on the appropriate makefiles. These ant scripts are part of my project, and I have to update them when I add a new build configuration or a new piece to the system. It's not that bad though.
I use CruiseControl to actually run the builds. Each project (all one of them) has an ant script of its own that performs build specific tasks (copying artifacts, processing results), calling into the project ant script to do the building.
I had to use cppunit for my testing and process the results with an xslt file I found somewhere. I also have the wrong svn revision label on each build because I can't find a suitable svn labeler. All I can find is half-completed years-old code and people arguing that other people are doing it wrong.
It looks to me like CC is a dying system, but I haven't found anything better for C++. Then again, I also feel like C++ is a dying language, so maybe it's bigger than just this.
We used scons for continuous integration run by a central build server. Some projects migrated to buildbot.
I'm now getting into rake and considering solutions as surveyed in this blog. Fowler mentions that ThoughtWorks occasionally use rake for their build scripting in his Continuous Integration article.