Related
I am working on C++ programming with perforce (a version control tool) on VMS.
I need to handle tens or even hundreds of C++ files (managed by perforce) on VMS.
I am familiar with Linux, python but not DCL (a script language) on VMS.
I need to find a way to make programming/debug/code-review as easy as possible.
I prefer using python and kscope (a kde based file search/code-review GUI tool that can generate call graph) or similar tools on VMS.
I do not have sys-adm authorization, so I prefer some code-review GUI tools that can be installed without the authorization.
Would you please give me some suggestions about how to do code-review/debug/programing/compile/test by python on VMS meanwhile using kscope or similar large-scale files management tools for code-review ?
Any help will really be appreciated.
Thanks
Your question is pretty broad so it's tough to give a specific answer.
It sounds like you have big goals in mind which is good, but since you are on VMS, you won't have a whole lot of tools at your disposal. It's unlikely that kscope works on VMS. Correct me if I'm wrong. I believe a semi-recent version of python is functional there.
I would recommend starting off with the basics. Get a basic build system working that let's you build in release and debug. Consider starting with either MMS (an HP provided make like tool) or GNU make. You should also spend some time making sure that your VMS based Perforce client is working too. There are some quirks that may or may not have been fixed by the nice folks at Perforce.
If you have more specific issues in setting up GNU make (on VMS) or dealing with the Perforce client on VMS, do ask, but I'd recommend creating separate questions for those.
Indeed, it's not clear from your question what sort of programming you want to do on VMS: C++ or python??
Assuming your first goal is to get familiar with the code-base, i.e. you want the ease of cross-ref'ing the sources:
If you have Perforce server running on VMS, then you may try to connect to it directly with Linux Perforce client. And do "review" locally on Linux.
If you've no Linux client, I'd try fetching the latest revisions off and importing raw files it into an external repository (svn, git, fossil etc.). Then again using Linux client and tools.
If your ultimate goal is to do all development off VMS, then it may not really be viable -- the code may use VMS specific includes, system/RMS calls, data structs. And sync'ing the changes back and forth to VMS will get messy.
From my experience, once you're familiar with the code-base, it's a lot more effective to make the code-changes directly on VMS using whatever editor is available (EDIT/TPU, EDT, LSE, emacs or vim ports etc.).
As for debugging - VMS native debugger supports X-GUI as well as command-line. Check your build system for debug build, or use /NOOPT /DEBUG compile and /DEBUG link qualifiers.
BTW, have a look into DECset, if installed on your VMS system.
I'm looking to implement a build server for Qt-based C++ projects. The server needs to check out the necessary code / assets from Subversion, build the executable files, assemble the artifacts for installation projects, and build the installation media files. The target platforms and (rough) toolchains are:
Windows (32- and 64-bit): qmake, nmake, msbuild, wix toolchain. The end result is an installer EXE and DVD image.
Mac OS X: qmake, make, custom bash scripts to assemble package. The end result is an application bundle within a disk image and a DVD image.
Ubuntu (32- and 64-bit): qmake, make, debuild-based scripts. The end result is a collection of DEB files and a DVD image.
Fedora (32- and 64-bit): qmake, make, rpmbuild-based scripts. The end result is a collection of RPM files and a DVD image.
So that's at least 4 build agents (maybe more if 32- and 64-bit can't be done on the same box) and 7 configurations. Open-source projects are preferred, but that is not an absolute requirement.
Most of the tools I'm seeing seem to be catered to Java (Jenkins, CruiseControl, etc.) or .Net (CruiseControl.net, etc.) Can those be used with a C++ toolchain, or will I constantly be fighting the system? Anything that you have used in the past and found works well with Qt / C++?
I use Jenkins for building and packaging many C++ projects, based on qmake, cmake, and makefiles.
There are plugins for cmake, qmake, and msbuild, but any command line scripts can be run as well.
I have done packaging using Jenkins with no problems, as it is just another command line step in a project.
There are good plugins for monitoring the number of warnings/errors produced by the compiler (I normally use GCC).
It also has matrix builds which allow you to build a project several times with different combinations of compiler flags, pre-processor variables, platform, etc. One project I set up is a matrix build with 5 boolean preprocessor flags on two platforms, which then does 2^6=64 builds. These can take a bit of setting up to get correct.
Here you can read a quick example:
Continuous Integration Server - Hudson
I think that Hudson, jenkins and builbot are worth a try. Wasting a day or two evaluating and trying them with a quick example will help you to choose confidently.
Most of the tools I'm seeing seem to be catered to Java (Jenkins, CruiseControl, etc.) or .Net (CruiseControl.net, etc.) Can those be used with a C++ toolchain, or will I constantly be fighting the system? Anything that you have used in the past and found works well with Qt / C++?
Any reasonably capable CI system will have a piece that will allow you to execute any program you want for your build command.
Here's what I would consider:
Does the CI system run on your system(s) of choice
Does it allow you an easy way to view your logs
Does it integrate with your test runner
Does it integrate with your code coverage reports (e.g. BullseyeCoverage w/C++ & Qt)
Will it publish your files in a manner sensible for your needs
Will at provide an archive/store of files, if necessary (e.g. pdbs & lib*.so.debug)
If the CI system doesn't support feature X, will you have to write it for each supported OS/system
Is the CI system / UI easy for you to use.
I did the above using CruiseControl and most things were pretty easy. I wrote everything in make or qmake and simply called out to the command that I needed executed. For unit test and code coverage integration I output stuff to XML and transformed it to something supported by CruiseControl.
My recommendation, take a look at the recommended CI systems and examine them based on the criteria above.
I'm using buildbot for this. I've been using it for 4 years, and I feel very happy with it.
It is an application written in python, that runs on a server and can manage multiple clients on various OSes. I'm currently using Windows XP, Windows 7, Debian, Ubuntu and CentOS build slaves. My projects are C++, and one of them (the end user GUI) is made in Python. But we've also integrated with other frameworks, for other features than GUI.
What is really good about buildbot is that it works by running command lines on slaves. With this, you can do whatever you want. Even on Windows systems to compile using Visual Studio! From these command lines, you get all the output centralized on the server, and accessible.
You may also find alternatives on this site that references many of them.
Disclaimer: I looked at it 3 years ago, I don't know if it is still accurate.
Hudson or Jenkins is pretty good.
Jenkins is indeed pretty popular for developing such a custom service, even after all these years, considering the question is already 7 years old.
Felgo also offers a Continuous Integration and Delivery (CI/CD) service for Qt. It supports desktop platforms as well as iOS, Android and embedded targets. The full feature set is described in the blog post.
Disclaimer: I am a software developer at Felgo
I am currently researching a good setup for a continuous integration server which would build various C++ applications for several Linux distributions.
My primary question is how other users here have handled the differences in system libraries between Linux distributions?
While it might be relatively easy to build direct dependencies such as UI libraries along with an application, "indirect" dependencies such as glibc look like a big pain if they had to be built alongside the application every time. I am therefore thinking of moving the actual build execution into a separate virtual machine for each distribution, e.g. using rlogin to run the commands. My goal is to prevent binary incompatibilities between build-machine library versions and those deployed in the target distributions.
Does anyone here have any experience with such a process and could tell if the above sounds like a feasible approach?
We use Jenkins (Contiguous Integration) and CMake (build system) for this purpose. Jenkins is similar to Buildbot, i.e. it also has buildmaster and buildslaves. Currently I have setup 8 slaves to build for 4 different platforms (FC8, FC10, FC12 and Windows 7). We build both debug and release binaries, so I dedicated one slave for each platform and build type.
As for the third party libraries like Qt & Boost, I compiled them on each platform and checked them into a separate repository.
#esavard: We use CMake 2.8 to do cross compilation, I have not used minigw but a quick google search indicates that it is possible. Here is a link to a tutorial to cross compile for Windows on Linux using CMake and miniGW.
I have not used Buildbot and cannot comment on its features but thought I should mention an alternative that we are currently using.
Hope this helps.
Buildbot has the notion of buildmasters and buildslaves.
A buildmaster takes care of displaying the web GUI, sending email, triggering builds, and other housekeeping. The buildslaves wait on the buildmaster and when commanded perform builds.
We have buildbot set up to build on a number of different platforms, some of them VMs, and it's working well for us.
Certainly buildbot and many virtual machines is the way to go with this. We have VMWare ESX server hosting many build slaves which overnight compile our application. The application is then tested on another virtual machine (not the build slave and just having a default OS install) to verify that it works and all dependency's are packaged.
The nice thing I would like to do is make the testing run time phase an automated step but I haven't been given the time to do that yet.
I have an Open Source app and I have it working on Windows, Linux and Macintosh ( it's in C++ and built with gcc ). I've only tested it on a few different flavors of Linux so I don't know if it compiles and runs on all different Linux versions. Is there a place where I can upload my code and have it tested across a bunch of different systems like other Linux flavors and things like, Solaris, FreeBSD and other operating systems?
What would be great is if I can have it directly connect to my svn repository and grab the latest code and then email me back any compile errors generated and what the OS was that it had a problem with.
I would be happy to just know it compiles as it is a GUI based app so I wouldn't expect it to actually be ran and tested.
There are a few options but there don't appear to be many (any?) free services like this, which isn't surprising considering the amount of effort and resources it requires. Sourceforge used to operate a compile farm like what you describe but it shut down a year or so ago. You might look into some of the following. If you're inclined to pay for a service or roll your own, then some of these links may be useful. If you're just looking for a free open source compile/build farm that covers multiple platforms it looks like you're pretty much out of luck.
OpenSuse Build Service
Mentioned by Ted first, worth repeating - only for Linux currently but does support a number of distros.
GCC Compile Farm
Mainly focused on testing builds for GCC but does also host a few other projects such as coLinux, BTG BitTorrent client, ClamAV, and others. May be something you can take advantage of, though I don't see what OSes are in the compile farm (contains at least Linux and Solaris based on the page notes).
BuildLocker
BuildLocker is a Web-based continuous integration solution for Java and .NET projects. BuildLocker is a virtual dedicated build machine that helps teams find bugs earlier in the development cycle, saving time and money. BuildLocker manages scheduled automated builds of source code in your ProjectLocker Source Control repository. Just check in the source code, and scheduled builds validate the integrity of the code. BuildLocker can even run automated tests, and can alert you anytime a test fails.
CruiseControl
CruiseControl is a framework for a continuous build process. It includes, but is not limited to, plugins for email notification, Ant, and various source control tools. A web interface is provided to view the details of the current and previous builds.
Interesting side note, CruiseControl is actually used by StackOverflow's dev team for automated build testing as well, according to the podcast.
Hudson
Hudson monitors executions of repeated jobs, such as building a software project or jobs run by cron.
RunCodeRun
Mentioned in the other linked question, only supports Ruby projects and is in private beta currently. However, if your project is in Ruby, it might be worth keeping an eye on RunCodeRun.
CI Feature Matrix
There are many Continuous Integration systems available. This page is an attempt to keep an unbiased comparison of as many as possible of them.
Take a look at the OpenSuSE build service, it includes a fairly wide variety of Linux distros (not just SuSE/OpenSuSE).
From a software point of view, there's also buildbot (sourceforge project site), which can be used to set up your own build/continous integration server.
This was suggested and considered to be used for gcc development (as mentioned on the gcc compile farm wiki page posted above).
If you are planning to go commercial with your open source product you might consider our Parabuild. It allows you to run a set of builds on multiple platforms and machines in parallel. The build will success only if all platform-specific builds success.
Does anyone have battle stories to share trying to use Visual Studio to develop applications for Unix? And I'm not talking using .NET with a Mono or Wine virtual platform running beneath.
Our company has about 20 developers all running Windows XP/Vista and developing primarily for Linux & Solaris. Until recently we all logged into a main Linux server and modified/built code the good old fashioned way: Emacs, Vi, dtpad - take your pick. Then someone said, "hey - we're living in the Dark Ages, we should be using an IDE".
So we tried out a few and decided that Visual Studio was the only one that would meet our performance needs (yes, I'm sure that IDE X is a very nice IDE, but we chose VS).
The problem is, how do you setup your environment to have the files available locally to VS, but also available to a build server? We settled with writing a Visual Studio plugin - it writes our files locally and to the build server whenever we hit "Save" and we have a bit fat "sync" button that we can push when our files change on the server side (for when we update to the latest files from our source control server).
The plugin also uses Visual Studio's external build system feature that ultimately just ssh's into the build server and calls our local "make" utility (which is Boost Build v2 - has great dependency checking, but is really slow to start as a result i.e. 30-60 seconds to begin). The results are piped back into Visual Studio so the developer can click on the error and be taken to the appropriate line of code (quite slick actually). The build server uses GCC and cross-compiles all of our Solaris builds.
But even after we've done all this, I can't help but sigh whenever I start to write code in Visual Studio. I click a file, start typing, and VS chugs to catch up with me.
Is there anything more annoying than having to stop and wait for your tools? Are the benefits worth the frustration?
Thoughts, stories, help?
VS chugs to catch up with me.
Hmmm ... you machine needs more memory & grunt. Never had performance problems with mine.
I've about a decade's experience doing exactly what you're proposing, most of it in the finance industry, developing real-time systems for customers in the banking, stock exchanges, stock brokerage niches.
Before proceeding further, I need to confess that all this was done in VS6 + CVS, and of late, SVN.
Source Code Versioning
Developers have separate sourcesafe repositories so that they can store their work and check it packages of work at logical milestones. When they feel they want to do an integration test, we run a script that checks it into SVN.
Once checked into SVN, we've a process that kicks off that will automatically generate relevant makefiles to compile them on the target machines for continuous integration.
We've another set of scripts that synchs new stuff from SVN to the folders that VS looks after. There's a bit of gap because VS can't automatically pick up new files; we usually handle that manually. This only happens regularly the first few days of the project.
That's an overview of how we maintain codes. I have to say, I've probably glossed over some details (let me know if you're interested).
Coding
From the coding aspect, we rely heavily on the pre-processors (i.e. #define, etc) and flags in the makefile to shape compilation process. For cross platform portability, we use GCC. A few times, we were force to use aCC on HP-UX and some other compilers, but we did not have much grief. The only thing that is a constant pain, is that we had to watch out for thread heap spaces across platforms. The compiler does not spare us from that.
Why?
The question is usually, "Why the h*ll would you even what to have such a complicated way of development?". Our answer is usually another question that goes, "Have you any clue how insane it is to debug a multi-threaded application by examining the core dump or using gdb?". Basically, the fact that we can trace/step through each line of code when you're debugging an obscure bug, makes it all worth the effort!
Plus!... VS's intellisense feature makes it so easy to find the method/attribute belonging to classes. I also heard the VS2008 has refactoring capabilities. I've shifted my focus to Java on Eclipse that has both features. You'd be more productive focusing coding business logic rather than devote energy making your mind do stuff like remember!
Also! ... We'd end up with a product that can run on both Windows and Linux!
Good luck!
I feel your pain. We have an application which is 'cross-platform'. A typical client/server application where the client needs to be able to run on windows and linux. Since our client base mostly uses windows we work using VS2008 (the debugger makes life a lot easier) - however we still need to perform linux builds.
The major problem with this was we were checking in code that we didn't know would build under gcc, which would more than likely break the CI stuff we had setup. So we installed MingGW on all our developer's machines which allows us to test that working copy will build under gcc before we commit it back to the repository.
We develop for Mac and PC. We just work locally in whatever ide we prefer, mostly VS but also xcode. When we feel our changes are stable enough for the build servers we check them in. The two build servers (Mac and PC) look for source control checkins, and each does a build. Build errors are emailed back to the team.
Editing files live on the build server sounds precarious to me. What happens if you request a build while another developer has edits that won't build?
I know this doesn't really answer your question, but you might want to consider setting up remote X sessions, and just run something like KDevelop, which, by the way, is a very nice IDE--or even Eclipse, which is more mainstream, and has a broader developer base. You could probably just use something like Xming as the X server on your Windows machines.
Wow, that sounds like a really strange use for Visual Studio. I'm very happy chugging away in vim. However, the one thing I love about Visual Studio is the debugger. It sounds like you are not even using it.
When I opened the question I thought you must be referring to developing portable applications in Visual Studio and then migrating them to Solaris. I have done this and had pleasant experiences with it.
Network shares.
Of course, then you have killer lag on the network, but at least there's only one copy of your files.
You don't want to hear what I did when I was developing on both platforms. But you're going to: drag-n-drop copy a few times a day. Local build and run, and periodically checking it out on Unix to make sure gcc was happy and that the unit tests were happy on that platform too. Not really a rapid turnaround cycle there.
#monjardin
The main reason we use it is because of the re-factoring/search tools provided through Visual Assist X (by Whole Tomato). Although there are a number of other nice to haves like Intelli-sense. We are also investigating integrations with our other tools AccuRev, Bugzilla and Totalview to complete the environment.
#roo
Using multiple compilers sounds like a pain. We have the luxury of just sticking with gcc for all our platforms.
#josh
Yikes! That sounds like a great way to introduce errors! :-)
I've had good experience developing Playstation2 code in Visual Studio
using gcc in cygwin. If you've got cygwin with gcc and glibc, it
should be nearly identical to your target environments. The fact that you
have to be portable across Solaris and Linux hints that cygwin should
work just fine.
Most of my programming experience is in Windows and I'm a big fan of visual studio (especially with Resharper, if you happen to be doing C# coding). These days I've been writing an application for linux in C++. After trying all the IDEs (Netbeans, KDevelop, Eclipse CDT, etc), I found Netbeans to be the least crappy. For me, absolute minimum requirements are that I be able to single-step through code and that I have intellisense, with ideally some refactoring functions as well. It's amazing to me how today's linux IDE's are not even close to what Visual Studio 6 was over ten years ago. The biggest pain point right now is how slow and poorly implemented the intellisense in Netbeans is. It takes 2-3 seconds to populate on a fast machine with 8GB of RAM. Eclipse CDT's intellisense was even more laggy. I'm sorry, but a 2 second wait for intellisense doesn't cut it.
So now I'm looking into using VS from Windows, even though my only build target is linux...
Chris, you might want to look at the free automation build server 'CruiseControl', which integrates with all main source control systems (svn, tfs, sourcesafe, etc.). It's whole purpose is to react to check-ins in a source control system. In general, you configure it so that anytime anyone checks code in, a build is initiated and (ideally) unit tests are run. For some languages there are some great plugins that do code analysis, measure unit test code coverage, etc. Notifications are sent back to the team about successful / broken builds.
Here's a post describing how it can be set up for C++: link (thoughtworks.org).
I'm just getting started with converting from a linux-only simple config (Netbeans + SVN, with no build automation) to using Windows VS 2008 with build automation back-end that runs unit tests in addition to doing builds in linux. I shudder at the amount of time it's going to take me to get that all configured, but the sooner the better, I guess.
In my ideal end-state I'll be able to auto-generate the Netbeans project file from the VS project, so that when I need to debug something in linux I can do so from that IDE. VS project files are XML-based, so that shouldn't be too hard.
If anyone has any pointers for any of this, I'd really appreciate it.
Thanks,
Christophe
You could have developers work in private branches (easier if you're using a DVCS). Then, when you want to checkin some code, you check it into your private branch on [windows|unix], update your sandbox on [unix|windows] and build/test before committing back to the main branch.
We are using a similar solution to what you described.
We have our code stored on the Windows side of the world and UNIX (QNX 4.25 to be exact) has access though an NFS mount (thanks to UNIX services for Windows). We have an ssh into UNIX to run make and the pipe to output into VS. Accessing the code is fast, builds are a little slower than before, but our longest compile is currently less than two minutes, not a big deal.
Using VS for UNIX development has been worth the effort to set it up, because we now have IntelliSense. Less typing = happy developer.
Check out "Final Builder" (http://www.finalbuilder.com/). Select a version control system (e.g. cvs or svn, to be honest, cvs would suit this particular use case better by the sounds of it) and then set up build triggers on FinalBuilder so that checkins cause a compile and send the results back to you.
You can set up rulesets in FinalBuilder that prevent you checking in / merging broken code into the baseline or certain branch folders but allow it to others (we don't allow broken commits to /baseline or /branches/*, but we have a /wip/ branching folder for devs who need to share potentially broken code or just want to be able to commit at the end of the day).
You can distribuite FB over multiple "build servers" so that you don't wind up with 20 people trying to build on one box, or waiting for one box to process all the little bitty commits.
Our project has a Linux-based server with Mac and Win clients, all sharing a common codebase. This set up works ridiculously well for us.
I'm doing the exact same thing at work. The setup I use is VS for Windows development, with a Linux VM running under VirtualBox for local build / execute verification. VirtualBox has a facility where you can make a folder on the host OS (Windows 7 in my case) available as a mountable filesystem in the guest (Ubuntu LTS 12.04 in my case). That way, after I start a VS build, and it's saved the files, I can immediately start a make under Linux to verify it builds and runs OK there.
We use SVN for source control, the final target is a Linux machine (it's a game server), so that uses the same makefile as my local Linux build. That way, if I add a file to the project / change a compiler option, usuall adding / changing a -D, I do the modifications initially under VS, and then immediately change the Linus makefile to reflect the same changes. Then when I commit, the build system (Bamboo) picks up the change, and does its own verification build.
Hard earned experience says this is an order of magnitude easier if you build like this from day one.
The first project I worked on started as Windows only, I was hired to port it to Linux, since they wanted to switch to a homogenous server environment, and everything else was Linux. Retrofitting a Windows project into this sort of a setup was a fairly major expenditure of effort.
Project number two was done "two system build" right from day one. We wanted to maintain the ability to use VS for development / debug since it is a very polished IDE, but we also had the requirement for final deploy to Linux servers. As I alluded to above, when the project was build with this in mind right from the start, it was quite painless. The worst part was a single file: system_os.cpp that contained OS specific routines, things like "get current time since linux epoch start in milliseconds", etc. etc. etc.
At the risk of getting a little off topic, we also created unit tests for this, and having the unit tests for the OS specific pieces provided a great deal of confidence that they worked the same on both platforms.