Assume you have five products, and all of them use one or more of the company's internal libraries, written by individual developers.
It sounds simple but in practice, I found it to be very difficult to maintain.
How do you deal with the following scenarios:
A developer unintentionally introduces a bug and breaks everything in production.
Every library has to mature, That means the API needs to evolve, so how do you deploy the updated version to production if every developer needs to update/test their code while they are extremely busy on other projects? Is this a resource and time issue?
Version control, deployment,and usage. Would you store this in one global location or force each project to use, say, svn:externals to "tie" a library?
I've found that it is extremely hard to come up with a good strategy. My own pet theory is this:
Each common library has to have a super-thorough set of tests or else it should never be common, even if it means someone else duplicates the effort. Duplicate untested code is better than common untested code (you break only one project).
Each common library has to have a dedicated maintainer (can be offset by a really good test suite in a smaller team).
Each project should check out the version of the library that is known to work with it. This means a developer does not have to get pulled away to update API usage, as the common code gets updated. Which it will be. Every non-trivial piece of code evolves over months and years.
Thank you for your thoughts on this!
You have a competing set of goals here. First, a library of reusable components must be open enough that people from the other projects can easily add to it (or submit components to it). If it's too difficult for them to do that, they'll build their own libraries, and ignore the common one, leading to a lot of duplicate code and wasted effort. On the other hand, you want to control the development of the library enough that you can ensure its quality.
I've been in this position. There's no easy answer. However, there are some heuristics that can help.
Treat the library as an internal project. Release it on regular intervals. Ensure that it has a well-defined release procedure, complete with unit tests and quality assurance. And, most important, release often, so that new submissions to the library show up in the product frequently.
Provide incentives for people to contribute to the library, rather than just making their own internal libraries.
Make it easy for people to contribute to the library, and make the criteria clear-cut and well-defined (e.g., new classes must come with unit tests and documentation).
Put one or two developers in charge of the library, and (IMPORTANT!) allocate time for them to work on it. A library that is treated as an afterthought will quickly become an afterthought.
In short, model the development and maintenance of your internal library after a successful open source library project.
I don't agree with this:
Duplicate untested code is better than
common untested code (you break only
one project).
If you are all equally likely to create bugs by implementing the same thing, then you'll all have to fix potentially different bugs in each instance of the "duplicate" library.
It also seems that it'd be much faster/cheaper to write the library once and, instead of having multiple other teams write the same thing, have some resources allocated to testing.
Now to solve your actual problem: I'd mimic what we do with real third-party libraries. We use a particular version until we're ready, or compelled to upgrade. I don't upgrade everything just because I can--there has to be a reason.
Once I see that reason (bug fix, new feature, etc.), then I upgrade with the risk that the new library may have new bugs or breaking changes.
So, you're library project would continue development as necessary, without impacting individual teams until they were ready to "upgrade".
You could publish releases or peg/branches/tag svn to help with all this.
If all teams have access to the bug tracker, they could easily see what known issues exist in the upgrade-candidate before they upgrade, too. Or, you could maintain that list yourself.
#Brian Clapper provides some excellent guidelines for how to run your library as a project in his answer.
I used to work in a similar situation to what you're describing, only my company had dozens of software products. I worked on the team that was responsible for maintaining and upgrading the core set of libraries that everyone else used.
We dealt with those scenarios as follows:
Test the heck out of the core libraries. Maintaining duplicate code is a nightmare. You're not just maintaining the core and one copy. Somewhere in your company's source control there are several copies of the same code. We had dozens of products, so that would have meant dozens of copies. Hunt them down and kill them.
We had a small team of 10-12 developers dedicated to maintaining the core library and its test suites. We were also responsible for fielding calls from the other 1100 developers in the company about how to use the core library, so as you can imagine, we were very busy.
Each other project needs to work with the version of the core library that it is known to work with. You can use version control branches to test new releases of the core library with old products to make sure you don't break code that works. If the core team does a thorough job of testing, this should go very smoothly. The only time this ever got really complicated for us was when the core API changed, or when we flat out screwed something up. Even if you're very confident in your core testing, use branches to test individual products.
I agree - this is difficult. In our small team (consulting .. not a product company - which made it harder), we had one common component that stood out from the others. In this case the recipe for success was:
Make a good developer responsible for developing the component
Make a good developer the gatekeeper for maintaining the component
Make sure all upgrades (there were several) are backward compatible
Make sure there is some basic documentation (or a simple reference application) explaining how the component is to be used
Make sure all developers know that the component exists (!) and where they can find it (along with the code, if they wish to review it)
Give developers the ability to review the code and suggest better implementations or refectoring, but have the final mods go through an experienced gatekeeper. When the component were upgraded, older apps did not have to upgrade. If we did a new release, we evaluated if we wanted to upgrade, and if we did, all we needed to do was swap the libraries - no code needed to change, unless we wanted to use some new features available through the upgrade. Resistance is inevitable, but sometimes it is a good sort of resistance when it comes from good developers who have better ideas for a new generation or refactored component.
Treat the development of the libraries like any other product. Each library has its own repository, its own releases and version numbers. The compiled and officially tested versions of the library are also kept in the repository. Document features and changes from version to version.
Then use the libraries like you would using third party libraries. Your product uses only fixed versions of the compiled libraries. Switch to a new version when you really need to and be aware that this involves more testing. Add the versions you use to your version control.
When you find a bug or require a new feature in a library, a new version or sub-version is created. Using a version control system like svn makes this easy. When you need the source code for debugging purposes, export it and include it in your projects, but do not change it there, but fix problems in the libraries' repositories.
This way, every team can contribute to the libraries without endangering the work of the other teams. Switching versions is done deliberately and not by accident.
Create an Anti-corruption (DDD) layer for the existing library... this is nothing but a facade.. and then write unit-test for this anti-corruption layer... Now even if someone upgrade/update the library you would know if something is broken by running the unit tests...
These tests could also serve as documentation of contract... and not every project that need to use the library has to write this anti -corruption layer, if they are using the same exact functionality..
"Duplication is the root of all evil"
Sounds to me like you need:
An artifact repository like Ivy so you can have the libraries shared and versioned with a distinction between versions that are API stable and ones that are "maturing"
Tests for the libraries
Tests for the projects using them
A continuous integration system so that when an incompatibility or bug is introduced both the project and the original library developer are notified
I think that one shared library is better than 3 duplicate ones (and 1 tested is definitely better than 3 untested). That's because when you find and fix problem, this makes the whole application area more solid (and development and maintenance are more efficient).
BTW, that't one of the reasons (apart from contributing back to the community) why our company exposes our .NET shared libraries to the public as open-source.
Plus, there's less code to write. And you can designate one dev to enforce good development practices on the library and its usages (i.e. through code contracts enforced on the unit tests within library consumers). This improves quality and and reduces maintenance costs.
We store shared libraries as binaries in the solution. That comes from the logical requirement that any solution has to be atomic and independent (this rules out svn:externals links).
API compatibility is not an issue at all. Just let your integration server rebuild and retest the whole product stack (while updating all the inner references and propagating changes) and you'll always be sure that all internal API's are solid. And whomever breaks the API has to either fix it or update the usages.
Duplication is the root of all evil
I would argue that unchecked government is the root of all evil :)
I do get a lot of flack for even suggesting that duplication should be an option. I understand why, but let me complicate this a bit.
Say you have a fairly large library that doesn't actually do anything in particular - it's just a collection of utilities. There are NO tests for this library - at all. You need only one function from it. Say, something that parses out a file's extension.
Pop quiz: do you just write something as small as this in your own project, or you bite the bullet and use the free-for-all untested set of utilities, which WILL break your application if someone breaks the function?
Also, imagine you are in environment where writing tests is not part of the culture, since most projects are very intense and have a very short development span.
Duplicating large systems - such as client registration - would be dumb beyond belief, of course. However, aren't there any cases where it is safer to duplicate something fairly small in your project if the alternative is not safe enough (no system for maintaining common code).
Think of it this way - and this happens all the time - multiple contractors working on different projects, for the same company. They don't even know about each other.
My argument is this:
If a team cannot dedicate to maintaining a solid common codebase, or if the environment does not give them enough time to, it's best to let them work as separate "contractors".
You will STILL need to use large existing systems that simply cannot be duplicated.
Duplicating large systems - such as
client registration - would be dumb
beyond belief,
That's why those systems publish external interfaces.
If you define a library as shared code between projects: in my experience that's almost always bad. A project should be stand alone, and updates for one project should not affect other projects.
Even if you start with libraries, you'll end up duplicating code anyway. Want to hotfix project 1? It was released with library 1.34, so to keep the hotfix as small as possible, you'll go back to library 1.34 and fix that. But hey-- now you did exactly waht the library was supposed to avoid-- you've duplicated the code.
Every developer uses Google to find code and copy it into his application. That's probably how they found Stack Overflow in the first place. Imagine what would happen if Stackoverflow published libraries instead of code snippets, and you'll get an idea of the problems that afflicts many well meaning library creators.
Libraries tend to be generic solutions to specific problems. Typically, the generic solution is more complex than the sum of the two specific solutions. This means you need one good programmer to solve a problem that could have been solved by two morons. Sounds like a bad tradeoff to me :D
I would like to point a problem in the solutions suggested above: treating the library as an internal project with its own versioning scheme.
The problem
If your company has more than one product (lets say two teams - two product: A, B), than each product has its own release schedule. Let's give an example: Team A is working on product A v1.6. Their release schedule is two weeks from now (suppose Oct 30th). Team B is working on product B v2.4. Their release schedule is 1.5 months from now - Nov 30th. Lets assume both are working on acme-commons-1.2-SNAPSHOT. Both are adding changes to acme-commons, as they need it. Couple of days before Oct 30th, team B introduce a change which is buggy, to acme-commons-1.2-SNAPSHOT. Team A is getting into stress mode since they discover the bug 1 day prior to code freeze.
This scenario shows that treating a common library as a third party library is almost impossible. The trivial, but problematic, solution is for each team to have their own copy of the version they are about to change. For example, product A v1.2 will create a branch (and version) for acme-commons named "1.2-A-1.6". Team B will also create a branch in acme-commons called "1.2-B-2.4". Their development will never collide and they will be stress free once they tested their product.
Of course, someone will have to merge their changes back to the original branch (lets say master or 1.2).
The problems I found with this solution is:
Branch inflation - the tree structure will be very puffy and it will be harder to understand the flow of changes/merges.
Merges back to 1.2 will probably never happen - Unless a team/developer is dedicated to this library, the chances that Team A or Team B merges their code back to 1.2 branch is slim. They will always stay focused on their tasks, thus creating and using their own branch space. Allocation of a developer/team is expensive, thus not always a viable solution.
I'm still trying to figure this one out, so any thoughts of this matter are welcome
Related
While posting a comment recently, I found myself remarking that, in my experience, Boost is not widely used in regulated industries (FDA, FAA). In fact, I don't know of any project that uses it or has used it. I realize, though, that my experience may be lacking here, so I wanted to know if anybody had knowledge of a project using boost in a medical device or in an aviation flight system (lighting, cabin controls, cockpit equipment, etc.).
I am not sure this is the right place to ask it (maybe some other SO site), but I thought this would be a good place to start.
This is not a question about whether or not boost should be used in these areas, it is a question about anybody knowing if it has been used.
EDIT Some examples projects that might help clarify this: Aircraft cabin lighting systems, cabin management systems, cockpit instrumentation, infusion/food/insulin pumps, dialysis machines, laboratory diagnostic devices, blood center data collection systems, etc. Some are life sustaining or potentially flight critical, some collect data, some collect data used to make medical decisions, etc., but I believe all are covered as regulated devices by the FAA/FDA.
EDIT Outside (did not come with the development chain) libraries are often brought into these types of projects for other purposes (graphics libraries, drivers, USB stacks, etc.) These are treated as SOUP. The use of boost would fall under this approach. Does anybody know of a project where boost was used this way?
EDIT Boost is a very large framework, with multiple components. I'm looking for any part of it that has been used in a project. For example "Boost smart pointers" or "Boost Enable" or "Boost Array" or "Boost Optional", etc. But used in "whole", not part. Not used by looking at the Boost code and re-using the idea; used as a whole component of the system (i.e. the legal sense).
This is central to the question, because used in this way means that tradeoffs of handling the SOUP must be dealt with. This may place this question outside the scope of this SO site...not sure.
I would say that a lot if it comes down to how the system designer(s) are handling system safety at an architectural level.
Triplication
For instance if the approach taken is triplicate redundancy with a trusted voter then system trials/testing is going to be the major step in approving the implementation. Suppose that one of the triplicates' development team chose to use Boost. If the system as a whole passed all its test vectors then one could argue that one didn't need to trawl through Boost itself looking for implementation errors. Obviously if all three triplicates had chosen to use Boost then that would be cause for concern, because then the scope of the test vectors becomes unmanageable.
Triplication is a standard approach to handling the problem of using software resources like compilers, libraries and programmers all of which are at risk of error. Boost is just another one of these. One could argue that Boost shared pointers are clearly a way of reducing the risk of programmer error. That in the round is most likely to be beneficial to the system as a whole.
Important, but Not Safety Critical
Where it gets interesting is when triplication is not being used, and one is now in the realms of really having to trust stuff. Interestingly the way a lot of systems seem to get round the problem is to say that ultimately there is a human in control who is supervising and able to intervene in the event of system error.
For example the car industry has a set of programming rules called MISRA. The software for an ABS system is supposed to be written to this rule set, and the development tools are supposed to be set to enforce those rules on the source code. The idea is that this will reduce the risk of undetected bugs to an acceptable level. And because ultimately there is a driver driving the car they can always do their own cadence braking. And thus the car industry has avoided having to have a triplicate implementation of ABS.
They are extending the same philosophy to more complex car systems like adaptive cruise control and self driving cars. Personally I think that such an extension is unreasonable for self driving cars. The relevant legislation makes it clear that it is the 'drivers' fault if such a vehicle has a crash (ie you are still 'driving' it), but the glossy advertising won't dwell on that important aspect.
It's the same in the world of medical devices; there's supposed to be a nurse or someone monitoring the patient anyway, so the occasional blip is covered by that supervision. The whole thing is very poor anyway; whilst the software for a medical device may have been written tested and approved, quite often these things run on embedded Windows XP. They all get networked up and end up being infested with computer viruses, etc. The FDA won't let you have an auto update system inplace, only the device vendor can update it, and of course they can't ever hope to keep up. So you end up with a nicely written well tested and good piece of medical software running on top of an OS installation which has had all the world's hackers running around inside it doing god knows what. I think that the use of Boost in these circumstances is not going to add much to the overall system risk.
So if a MISRA compliant toolchain offered Boost as part of that toolchain then I don't see why that would be any different to a toolchain offering a standard C library. If the toolchain vendor is certifying it then it's no different to the situation with anything else.
There are weaknesses with that approach. In my experience I have come across a MISRA compliant tool chain in widespread use whose compiler turned out junk object code when all optimisations were turned on. I was actually able to verify this in the disassembly. I then took a look at the source code for toolchains's standard C library, and it clearly wasn't in itself written to the MISRA rule set, and furthermore it contained glaring, horrible bugs.
And yet there is no regulatory block to building, testing and selling a car ABS system using this tool chain so long as you tick the MISRA checkbox in the project settings. Adding Boost to that toolchain would hardly make matters worse.
Safety Critical Without Triplication
The final approach is no triplication and no human supervision. This is really hard because you then need formal proof of the correctness of every component of the tool chain, OS, drivers, chips, etc. AFAIK it's never been done for a truly safety critical system like nuclear reactors, flight control avionics, or other systems that really will definitely kill people if they go wrong.
The only thing that comes close so far as I can tell is Greenhill's compiler suite and their INTEGRITY operating system. They can give you (for a large fee) formal testing and verification evidence for every single line of the OS, all their libraries and their compiler, everything. If one were ever to attempt a truly safety critical system without triplication that would be a starting point.
I don't think they've done a C++11 yet, though I have added Boost to their toolchain and it worked just fine (it wasn't in a safety critical system I hasten to add).
Conclusion
Certainly if outfits like Greenhills with a well deserved and good reputation for reliable and thoroughly tested toolchains offer Boost then I think one would be in good position to use it in an regulated system. However I doubt that the whole of Boost would be offered that way; they are more likely to follow the compiler standards.
I also know that GCC has in the past been put through formal compiler validation testing so that it could be used in Stuff That Matters. I expect that that will get repeated sooner of later for the more recent incarnations that have taken on aspects of Boost.
I think the best answer we can have here is "yes and no." I will try to explain why.
Boost is a huge umbrella for many constituent libraries. Some of them depend on each other in various ways (e.g. when a higher-level library needs features provided by a lower-level part of Boost like Type Traits). This raises questions about the usefulness of a simple answer to the question, because if three parts of Boost have been used in a regulated project, but they are different parts than you want to use, it is of no little value to know this. And we will never know the full answer regarding all parts, because you cannot prove a negative (and there are too many parts to ever expect a "100% yes" answer).
Boost is (and always has been) rapidly evolving. Entirely new libraries are added all the time. ASIO is a big one that didn't exist at all until somewhat recently. This makes it even more difficult to answer the question, because over time there are parts of Boost which are young and not as well tested as others. Additionally, existing libraries sometimes go through backward-incompatible revisions (e.g. "Boost Filesystem 3" not too long ago).
Many parts of Boost end up in projects not by a traditional dependency but rather by copy-pasting code from Boost, and perhaps modifying it to taste (e.g. adding or removing support for specific compilers). Likewise, many parts of Boost end up in projects via the fact that Boost is sort of a proving ground for many new C++ standard library features, such as shared_ptr (C++11) and unordered_map (TR1). Some features which are part of the language today were originally part of Boost, so many people have used "Boost code" without even knowing it.
Note that code does not somehow become safer when it transitions to official status within the language--GCC has had bugs which did not exist in the Boost equivalent implementations of the same concepts. This matters when considering practical questions like "Should we allow the use of Boost in our project or should we restrict ourselves to what the compiler vendor gives us?" If you are thinking of using a feature which has been implemented very recently by your compiler vendor (say, within the past year), you may well be better off using a third-party (e.g. Boost) implementation which is more mature.
Finally, since it seems that the impetus for this question is to gain some reassurance that using Boost is not a bad idea for a production project: I would certainly say that in general using Boost is fine and good, with the huge caveat that you need a local expert in Boost who knows which parts of Boost should not be used in your domain. For example, Boost Spirit, Phoenix, and Wave are examples of libraries which have been in Boost for a while but which very few people truly, deeply understand. It's one thing to use library code you don't fully understand (we all do), but quite another to use code which almost no person on earth understands.
In summary, I don't think anyone will be able to give you the reassurance that you seek that Boost is OK for safety-critical systems. You need to evaluate it on your own, the same as you need to evaluate your own compiler vendor's software, your other third-party dependencies, and the code you write yourself. I have used all four categories of software quite a lot, and in my experience Boost had fewer critical bugs than any of the others, and fewer regressions than either GCC or my own code.
I have been trying to get our team to migrate a large C++ project to VS2012 from VS2008. I want to do it mostly because I want to start using C++11 and the IDE is much nicer. So my reasons are somewhat selfish.
My team lead is pushing back because he doesn't see the business case for the migration, citing that most performance improvement features we'll get with C++11 we already have with BOOST and other libraries. He also says that this will require changing runtimes on all of our platforms which may change certain behavior. Which would mean we would need to retest on all of the servers that we have deployed to.
The first argument I somewhat understand, although I believe C++11 code will be much cleaner than using BOOST (again not a great business case).
The argument about using different runtimes I don't understand. What runtimes do a native C++ application use? This is not VC++. Would his concern be just that the STL won't be exactly the same implementation?
I don't see what the issue there would be. Is there something I am missing? Are there any other good arguments for migrating that I should cite to help my case?
All 3rd party libraries need to be built with the new compiler
Code might currently be unknowingly reliant on undefined behavior, a new compiler might do something totally different for UB than the current one (and cause problems)
Performance won't change much because you wouldn't have been coding in C++11 style (basically, lots of stuff is passed by value where before it would not have been). If your code base has a lot of...
std::vector<Blah> func(std::vector<Asdf> v); // notice all the pass by value
... C++11 could be a great performance improvement. But in C++98/03 you just wouldn't do that.
You need to lower the barrier to entrance for your team lead. Do the migration yourself and smoke test your product. Then show it to him. After that, here are abstract reasons to upgrade:
C++11 style is less code and simpler
VS2012 has the C++11 standard library additions - you can stop hand rolling 50 bug ridden replacements
Programmers want to work using a modern language and modern tools. This will spark a company wide resurgence of learning and best practices which will improve code quality, employee retention, employee continuing education, etc
It's a delicate balance when to do this sort of upgrade. If you do it too often you're spending money without gaining any business advantage. If you do it too infrequently you're dealing with so much legacy tech and legacy code that maintenance can become a nightmare. When a significant language change comes, that you will ultimately move to, it's best to do it sooner than later (and btw, this isn't particularly soon) - otherwise you just keep accumulating what will later be considered legacy code. Moving to a new compiler for new tools often isn't worth it. Moving for an important language upgrade usually is worth it.
Whether any of that is compelling to your team lead is beyond me. Good luck though
VS2012 is so last year, obsolete already, so good Microsoft replaced it after only 1 year of use, criticisms, eye-blinding whiteness, and ALL CAPS!
But considering that you can build a VS2008 project in the newer IDEs means you can upgrade today to VS2013 and work on upgrading the project to the VS2013 tooling over time.
Your TL is correct though, an upgrade requires a complete re-test, but if you have time between adding features then it is possible to fit such a large test in.
I'd say the main aspect of upgrading is just to keep up to date, its not such a big deal today but in another 5 years time your old VS2008 builds might start to hold you back (as I know having recently upgraded a VS2002 project to 2010), its never a good idea to get so far behind as the longer you leave it, the greater the effort in the upgrade that you will eventually have to do. That's the real reason to do it - chances that Microsoft will not support 2008 builds in the next version, and older IDEs won't run on Windows 9, get greater every year. Best to fix this problem while you have time.
I guess I wouldn't expect much in the way of performance improvements, but regarding testing, I'd recommend full retest on any kind of compiler or tool-chain change regardless of if you change your code or not.
EDIT: I'll add that I'd push to move to the more recent compiler on the next release cycle... that removes the testing argument (since you'll need to test anyway).
Well I'm working since some months ago with Alfresco 4.2b, mainly with Share. I'm preforming some customization works and my colleagues and myself are always wondering about the subject of this post.
We know extension modules is a relatively new feature which is available since the first release of Alfresco 4 but we don't know which way should the developments performed take in terms of the extension mechanism used. We are putting all our efforts on using the extension module mechanism rather than using the web-extension directory. We are doing that because we understand that probably the it is more versatile, maintainable and scalable and who knows, perhaps the use of the web-extension directory could be deprecated in further releases. We are also aware that at the moment you cannot do "everything" with extension modules.
So according to that, I would like to know about pros and cons of both mechanisms based on other developers experience, as well as whether we are doing right with our extension philosophy. At the same time, I would like to find a reasonable answer to the next questions:
Are we doing right prioritizing and "sacrifice" our time trying to perform customization with extension modules? Is it worth?
Could be deprecated the web-extension directory as a customization mechanism in further releases?
Is the extension modules feature going to be improved?
(Roughly) known limitations of extension modules? Mainly the question could be, when should be used one extension mechanism or the other?
I'm looking forward to hear your opinions. It would be strongly appreciated opinions based on your real experience and as honest as possible.
Thanks very much in advance.
Actually both are not exclusive.
I'm using myself for instance the extension mechanism to define new components which webscripts are defined in the web-extension folder.
In fact the extension module itself can be located in the web-extension folder under alfresco/web-extension/site-data/extensions/...
For me using the extension modules is a way to logically package (because of the deployment process of modules) your customization. That's the main advantage because it can be configured through evaluators if you want the customization to be available only in particular conditions.
The two main problems with the web-extension route is that you will end up duplicating core code in your customisation, and that if multiple extensions try to extend the same component, one must win.
Extensibility modules may be a little bit more work but you will more than get that back in terms of a much reduced maintenance burden, and in troubleshooting problems when multiple add-ons clash with each other.
From 4.2 onwards you should not need to override web script components - in Share at least. You should always use an extensibility module.
what would be (rough estimation, average, of course) the initial learning and setup curve and subsequent overhead for using Maven for C++/Eclipse/Linux project of small to medium size?
We are 4 developers at the beginning of the way. We currently have ~20 native eclipse C++ (CDT) "projects", which we compile interactively. We would like to have an automated checkout & build script.
It seems a bit overkill at this stage, but perhaps we should adopt it sooner then later, provided that it does not incur an overhead. We don't have bandwidth for extensive configuration management right now. Thanks a lot!
EDITED / DETAILED:
I realize I haven't described my needs well enough. Having read the references provided below, I see that CI tool seems an overkill for us at the moment. What I'd like to have is a build tool that is well integrated with eclipse on one hand, and allows offline, non-interactive builds on the other. I enjoy the simplicity of working with eclipse projects: you just add files, add references to internal components and 3rd part libs as they add up, and that's it. You don't need to manually maintain makefiles or the like. The trouble with it, as with MSVS a few years ago when I worked with it, is that it does not give you an option of non-interactive builds. So, does such tool exist?
First, while Maven has some support to build C++ projects with the maven-native-plugin or, if you already are using Make, with the maven-make-plugin from the c-builds suite, this is not a common use case and there aren't widely used. So while it should be possible, you won't get support and find resources easily (just Google a bit or browse the maven users list to get an idea).
Second, if you add to this that you'll have to learn Maven in the same time, then it seems reasonable to say that you are not taking the easiest path.
So, instead, I'd stick with more traditional tools and/or Ant. For the continuous integration itself, I've seen several references mentioning the use of CruseControl to build a C++ project. Refer to What continuous integration tool is best for a C++ project? or UsingCruiseControlWithCplusPlus for example. But I guess the principles are transposable to another CI engine (like Hudson that I find much more easy to use than CruiseControl).
I am about to attempt reorganizing the way my group builds a set of large applications that share about 90% of their source files. Right now, these applications are built without any libraries whatsoever involved except for externally linked ones that are not under our control. The applications use the same common source files (we are not maintaining 5 versions of the same .h/.cpp files), but these are not built into any common library. So, at the moment, we are paying the price of building the same code over-and-over per application, each time we intend to release a version. To me, this sounds like a prime candidate for using libraries to capture the shared code and reduce build times. I do not have the option of using DLL's, so the approach is to use static libraries.
I would like to know what tips you would have for how to approach this task. I have limited experience with creating/organizing static libraries, so even the basic suggestions towards organization/gotchas are welcome. Maybe even a good book recommendation?
I have done a brief exercise by finding the entire subset of files that each application share in common. As a proof of concept, I took these files and placed them in a single "Common Monster" static library. Building the full application using this single static library certainly improves the build time for all of the applications, but should I leave it at this? The purpose of the library in this form is not very focused and seems like a lazy attempt at modularity. There is ongoing development with these applications, and I'm afraid this setup will cause problems further down the line.
It's very hard to give general guidelines in this area - how you structure libraries depends very much on how you use them. Perhaps if I describe my own code libraries this may help:
One general purpose library containing code that I expect all applications will have at least a 50/50 chance of needing to use. This includes string utilities, regexes, expression evaluation, XML parsing and ODBC support. Conceivably this should be split up a bit, but it makes distributing my code in FOSS projects easier to keep it monolithic.
A library supporting multi-threading, providing wrappers around threads, mutexes, semaphores etc.
One supporting SQLite via its native interface, rather than via ODBC.
A C++ web server wrapper round the Mongoose C web server.
The general purpose library is used in all the stuff I write, the others in more specialised circumstances. Headers for each library are held in separate directories, as are the library binaries themselves (though they should probably be in a single lib directory).
Make sure that the dependencies of your libraries form acyclic directed graph (a tree). While this is not necessarily a problem for static libs (I'm not sure in fact), it will be a problem if you ever decide to switch to dlls. Depending on your situation, this may require some redesign of interfaces.
Another thing I noticed (for sure on MSVC), which you may consider if build speed is an important concern: DLLs link much faster than static libraries. I assume this is because they don't have to be copied into the new executable and there's no need to search an eliminate unused code. Even if it's no option for production, you may use this trick while developing.
I also have the habit to create my solution files with CMake, because it is easier to overview the entire build process than clicking through an endless list of options in a GUI. It's up to you to decide if you want to walk that path.