I am about to attempt reorganizing the way my group builds a set of large applications that share about 90% of their source files. Right now, these applications are built without any libraries whatsoever involved except for externally linked ones that are not under our control. The applications use the same common source files (we are not maintaining 5 versions of the same .h/.cpp files), but these are not built into any common library. So, at the moment, we are paying the price of building the same code over-and-over per application, each time we intend to release a version. To me, this sounds like a prime candidate for using libraries to capture the shared code and reduce build times. I do not have the option of using DLL's, so the approach is to use static libraries.
I would like to know what tips you would have for how to approach this task. I have limited experience with creating/organizing static libraries, so even the basic suggestions towards organization/gotchas are welcome. Maybe even a good book recommendation?
I have done a brief exercise by finding the entire subset of files that each application share in common. As a proof of concept, I took these files and placed them in a single "Common Monster" static library. Building the full application using this single static library certainly improves the build time for all of the applications, but should I leave it at this? The purpose of the library in this form is not very focused and seems like a lazy attempt at modularity. There is ongoing development with these applications, and I'm afraid this setup will cause problems further down the line.
It's very hard to give general guidelines in this area - how you structure libraries depends very much on how you use them. Perhaps if I describe my own code libraries this may help:
One general purpose library containing code that I expect all applications will have at least a 50/50 chance of needing to use. This includes string utilities, regexes, expression evaluation, XML parsing and ODBC support. Conceivably this should be split up a bit, but it makes distributing my code in FOSS projects easier to keep it monolithic.
A library supporting multi-threading, providing wrappers around threads, mutexes, semaphores etc.
One supporting SQLite via its native interface, rather than via ODBC.
A C++ web server wrapper round the Mongoose C web server.
The general purpose library is used in all the stuff I write, the others in more specialised circumstances. Headers for each library are held in separate directories, as are the library binaries themselves (though they should probably be in a single lib directory).
Make sure that the dependencies of your libraries form acyclic directed graph (a tree). While this is not necessarily a problem for static libs (I'm not sure in fact), it will be a problem if you ever decide to switch to dlls. Depending on your situation, this may require some redesign of interfaces.
Another thing I noticed (for sure on MSVC), which you may consider if build speed is an important concern: DLLs link much faster than static libraries. I assume this is because they don't have to be copied into the new executable and there's no need to search an eliminate unused code. Even if it's no option for production, you may use this trick while developing.
I also have the habit to create my solution files with CMake, because it is easier to overview the entire build process than clicking through an endless list of options in a GUI. It's up to you to decide if you want to walk that path.
Related
I searched for this particular question but could not come up with any results, neither here nor on-line in general (maybe because it is a little harder to phrase for me). If it has already been asked, please point me in the right direction.
I am at a point where I would like my libraries/software to be pluggable. I see all these various libraries and systems where plugins are used extensively and the authors boastfully point out (in a good way!) that their software has plugin support.
So my question is, where do I start? Are there any books/on-line-resources that break the ice and may guide one on the do's and dont's of making your library pluggable, define best practices etc.?
You have to understand some things before starting :
There is no support for modules (static or dynamic) in standard C++. Nope. Not yet. Maybe in 2015.
Dlls (or .so on unix-like systems) are dynamically loaded libraries that are compiler/os dependant. So it's a pragmatic solution that fill the need.
So, you'll have to use shared libraries (whatever the file extension, it's the keyword for searches about this subject) as plugin binaries. If your plugin should contain more than runtime code, like graphic resources, you can include your graphic resources in the binary, or have a file format or compressed archibe that contain the binary file.
Whatever the way you setup your plugin files, in C++ the problem is about the interface.
Depending on wich compiler you use, you'll have different ways to "tag" functions or classes as exported/imported (meaning your plugin source code export the code and the user of the plugin should import the code).
Setup clean and clear interface in C++ for the modules, with no templates (because they are compiler and compiler configuration dependant). Those interfaces should be function declarations and class declarations with no inline code and marked exported/imported.
Now, once you've got this, you can use OS-specific API to load/unload dynamic library binaries while the application is running. Once it's done, you can get pointers to functions, again using the OS-specific API. I let you search for it.
Now, there are libraries that provide ways to abstract this in a cross-platform way. I didn't use them yet and they are known to be unperfect because of lack of definitions in the C++ standard, but they could be useful if you're planning to have your application cross-platform:
boost::extension : it's not yet a boost library, nor even proposed yet, and it's developpement is in pause (until some new standard C++ implementations are done) so it might be a bad idea but a lot of people say they use it with success.
POCO libraries have a library for shared libraries that would be the equivalent of boost::extension. Again lot of people say it's useful so I guess it's good enough to be used.
The other alternative, that is easy to setup if you don't need to support tons of target platforms, is to just write some wrapper code around OS-Specific APIs. That's what I did before knowing about boost::extension for example.
I'm not sure if I am going about this the right way. I am making some c++ classes to use in 2 apps. I am going to have to compile them to be used in a Cocoa app and later be compiled for use with fastcgi.
Should I create a dynamic library?
If the source files have to be compiled with different conditional compilation settings to work in the two applications, you'd better ship the source and compile it along with the apps themselves.
Otherwise, you can create a library and ship the compiled versions along with the headers for compilation of the apps. Whether the library should be dynamic or not depends on your situation. If you don't need to update the library separately without recompiling the executable, a simple static library is probably a better choice.
Don't forget that you have static library as an option too. On some platforms dynamic libs come with a bunch of annoying baggage. They're also slightly slower (though this is usually beside the point). They also can't be replaced without recompiling the program (you might be more concerned about this in a web environment). Finally, when you're only running one of the programs on a computer you gain nothing by making the lib dynamic...and little when there's only two.
If you want to share multiple C++ class among projects you should typically place them in a class library. Not familiar with Cocoa though.
If you have many classes, then shared library. Make sure to use only abstract classes and no code in public headers (templates are OK). Provide factories (or plain functions) to instantiate the objects.
If that sounds like too much coding for you then, well, modern version control makes it rather painless to simply re-use the files in several projects, living in the same repository.
P.S. Another factor to consider is how many people work on the project. Shared library is an extra responsibility. If you have a person to take care of it, then it might be worth doing simply from organizational point of view.
My C++ project is growing larger. We are also moving to using cmake for building now. I want to divide the application into libraries so that they can be linked for testing, preparing the application package, etc. Right now I would divide my code into libraries as follows:
core
GUI
utilities (these are used by core and other components)
io (xml parsing/outputing using print functions of classes in core)
tests (unit tests)
simulator (tests the core)
An alternative would be to divide based on the directory structure - one library for each directory. But from my past experience it leads to too many libraries and then library dependencies become tough to handle during linking.
Are there any best practices in this regard?
Sit down with a piece of paper and decide your library architecture.
The library should be designed as a set of levels.
A libary on level A (the base) should have dependencioes only on system libraries and only if it must on libraries on level A.
A library on level B can have dependencies on libraries at level A and system libararies and only if it must on libraries on level B.
etc
Each library should represent a complete job at its particular level. Things at lower level generally have smaller jobs but lots of them. A library at a higher level should reresent a complete task. ie don't have a lib for windows objects and a lib for events. At this level the job here is handline all interaction with a window (this includes how it interacts with events).
You seem to have identified some resonable functional groups. The only one that see a bit suspicious is io. If you truly have some generic IO routines that provide real functionality fine. But if it is just grouping the IO for a bunch of different objects then I would scrap that (it all depends on the usage).
So the next step is to identify the relationship between them.
As for using directory structures. Usually everything in one directory will be present within the same library, but that does not exclude the posability of other directories also being present. I would avoid putting half the classes in directory in libA and the other half of the classes in libB etc.
You should have a read of Large-Scale C++ Software Design by John Lakos.
You may not be able to read it before you start your work, but you should put this book on your list.
Otherwise Martin York's advise is sound.
One more thing though, I would recommend picking up a tool like doxygen that can give you dependency diagrams of your code base. If your bothering to do this type of restructuring you should rid yourself of circular dependencies between your libraries. Lakos describes a lot of ways to cut dependencies - some obvious, some less so.
I like starting from a package diagram that only has one way arrows.
http://www.agilemodeling.com/style/packageDiagram.htm
Your list looks like a good start.
Seems reasonable.
I'd query what your unit_tests library is supposed to do.
Given a collection of "projects" building libA, libB, libC... I'd expect to see some matching projects testA testB testC (whether they build libraries or executables depends whether the built tests run standalone or are loaded into some test runner).
I'm also slightly wary of "utilities" libraries. These seem to have a surprising ability to cause pain and suffering in the long run. For example, maybe your IO library has no other dependency than the utilities library. One day you want to reuse the IO library in another project on another platform. Only problem is, you now also have to port all of the utilities library (90% of which IO doesn't use), or disentangle the 10% of it which IO actually depends on. Sometimes it's better to have libraries be a bit more dependency free, at the cost of some code duplication.
I am developing a portable C++ application and looking for some best practices in doing that. This application will have frequent updates and I need to build it in such a way that parts of program can be updated easily.
For a frequently updating program, creating the program parts into libraries is the best practice? If program parts are in separate libraries, users can just replace the library when something changes.
If answer for point 1 is "yes", what type of library I have to use? In LINUX, I know I can create a "shared library", but I am not sure how portable is that to windows. What type of library I have to use? I am aware about the DLL hell issues in windows as well.
Any help would be great!
Yes, using libraries is good, but the idea of "simply" replacing a library with a new one may be unrealistic, as library APIs tend to change and apps often need to be updated to take advantage of, or even be compatible with, different versions of a library. With a good amount of integration testing though, you'll be able to 'support' a range of different versions of the library. Or, if you control the library code yourself, you can make sure that changes to the library code never breaks the application.
In Windows DLLs are the direct equivalent to shared libraries (so) in Linux, and if you compile both in a common environment (either cross-compiling or using MingW in Windows) then the linker will just do it the same way. Presuming, of course, that all the rest of your code is cross-platform and configures itself correctly for the target platform.
IMO, DLL hell was really more of a problem in the old days when applications all installed their DLLs into a common directory like C:\WINDOWS\SYSTEM, which people don't really do anymore simply because it creates DLL hell. You can place your shared libraries in a more appropriate place where it won't interfere with other non-aware apps, or - the simplest possible - just have them in the same directory as the executable that needs them.
I'm not entirely convinced that separating out the executable portions of your program in any way simplifies upgrades. It might, maybe, in some rare cases, make the update installer smaller, but the effort will be substantial, and certainly not worth it the one time you get it wrong. Replace all executable code as one in most cases.
On the other hand, you want to be very careful about messing with anything your users might have changed. Draw a bright line between the part of the application that is just code and the part that is user data. Handle the user data with care.
If it is an application my first choice would be to ship a statically-linked single executable. I had the opportunity to work on a product that was shipped to 5 platforms (Win2K,WinXp, Linux, Solaris, Tru64-Unix), and believe me maintaining shared libraries or DLLs with large codebase is a hell of a task.
Suppose this is a non-trivial application which involves use of 3rd Party GUI, Threads etc. Using C++, there is no real one way of doing it on all platforms. This means you will have to maintain different codebases for different platforms anyway. Then there are some wierd behaviours (bugs) of 3rd Party libraries on different platforms. All this will create a burden if application is shipped using different library versions i.e. different versions are to be attached to different platforms. I have seen people shipping libraries to all platforms when the fix is only for a particular platform just to avoid the versioning confusion. But it is not that simple, customer often has a different angle to how he/she wants to upgrade/patch which is also to be considered.
Ofcourse if the binary you are building is huge, then one can consider DLLs/shared-libraries. Even if that is the case, what i would suggest is to build your application in the form of layers like:-
Application-->GUI-->Platform-->Base-->Fundamental
So here some libraries can have common-code for all platforms. Only specific libraries like 'Platform' can be updated for specific behaviours. This will make you life a lot easier.
IMHO a DLL/shared-library option is viable when you are building a product that acts as a complete solution rather than just an application. In such a case different subsystems use common logic simultaneously within your product framework whose logic can then be shared in memory using DLLs/shared-libraries.
HTH,
As soon as you're trying to deal with both Windows and a UNIX system like Linux, life gets more complicated.
What are the service requirements you have to satisfy? Can you control when client systems get upgraded? How many systems will you need to support? How much of a backward-compatibility requirement do you have.
To answer your question with a question, why are you making the application native if being portable is one of the key goals?
You could consider moving to a a virtual platform like Java or .Net/Mono. You can still write C++ libraries (shared libraries on linux, DLL's on windows) for anything that would be better as native code, but the bulk of your application will be genuinely portable.
We have a core library in the form of a DLL that is used by more than one client application. It has gotten somewhat bloated and some applications need only a tiny fraction of the functionality that this DLL provides. So now we're looking at dividing this behemoth into smaller components.
My question is this: Can anyone recommend a path to take to divide this bloated DLL into a set of modules that have some interdepencies but do not necessarily require all other modules?
Here are the options as I see them but I'm hoping someone can offer other possibilities:
Create a "core" dll and several "satellite" dlls which use the core and possibly other satellite DLLs.
Subdivide the contents of the bloated DLL into static libraries that the main DLL uses (to maintain the same functionality) but apps that don't want to use the bloated version can assemble the static libraries they need into their own dll or into the app itself.
I was hesitant to mention this but I think it may be important to note that the app uses MFC.
Thanks for your thoughts.
Somewhat related to your question is this question, about splitting up a very large C module into smaller ones.
How do you introduce unit testing into a large, legacy (C/C++) codebase?
It seems your question has to do with the larger question of breaking some large blob of code into a more modular system. The link above is definitely recommended reading.
Without having all the details it is a little hard to help but here is what I would do in your situation
provide both static and dll versions of whate3ver you release - for MT and single threaded.
try to glean from the disparate clients which items should be grouped together to provide reasonable segmentation - without having layers of dependencies.
having a "core" module sounds like a good idea - and make sure you don't have too many levels of dependencies - you might want to keep it simple.
You may find after the exercise that one big dll is actually reasonable.
Another consideration is that maintaining multiple DLLs and both static libs and DLLs will hugely increase the complexity of maintenance.
Are you going to be releasing them all at once every time, or are they going to be mix and match? Be careful here - and know that you could create testing issues
If no one is complaining about the size of the DLL then you might want to consider leaving it as is.