Related
I'm trying to understand the correct way, or right approach, to provide a reasonably large C++ API for a non-open source project. I do not want to provide a "header only" library as the code base is fairly large and meant to be closed source. The goals are as follows:
Provide a native C++ API that users may instantiate C++ classes, pass data around, all in C++, without a C-only wraper
Allow methods to take as parameters and return C++ objects, especially STL types (std::string, std::vector, etc)
No custom allocators
Most industry standard / canonical method of doing this possible, if such a standard exists
No re-creating COM or using MS COM
Assume all C++ compilers are at least C++11 "compliant"
I am targeting Windows as well as other platforms (Linux).
My understanding is creating a DLL or shared library is out of the question because of the DLL boundary issue. To deal with DLL boundary issue, the DLL and the calling code must be compiled with dynamically linked runtime (and the right version, multithreaded/debug/etc on Windows), and then all compiler settings would need to match with respect to debug symbols (iterator debugging settings, etc). One question I have this is whether, if, say on Windows, we ensure that the compiler settings match in terms of /MD using either default "Debug" and "Releas" settings in Visual Studio, can we really be "safe" in using the DLL in this way (that is passing STL objects back and forth and various things that would certainly be dangerous/fail if there was a mismatch)? Do shared object, *.so in Linux under gcc have the same problem?
Does using static libraries solve this problem? How much do compiler settings need to match between a static library and calling code for which it is linked? Is it nearly the same problem as the DLL (on Windows)?
I have tried to find examples of libraries online but cannot find much guidance on this. Many resources discuss Open Source solution, which seems to be copying header and implementation files into the code base (for non-header-only), which does not work for closed source.
What's the right way to do this? It seems like it should be a common issue; although I wonder if most commercial vendors just use C interfaces.
I am ok with static libraries if that solves the problem. I could also buy into the idea of having a set of X compilers with Y variations of settings (where X and Y are pre-determined list of options to support) and having a build system that generated X * Y shared binary libraries, if that was "safe".
Is the answer is really only to do either C interfaces or create Pure Abstract interfaces with factories? (if so, is there a canonical book or guide for doing this write, that is not implementing Microsoft COM?).
I am aware of Stefanus DuToit's Hourglass Pattern:
https://www.youtube.com/watch?v=PVYdHDm0q6Y
I worry that it is a lot of code duplication.
I'm not complaining about the state of things, I just want to understand the "right" way and hopefully this will serve as a good question for others in similar position.
I have reviewed these Stackoverflow references:
When to use dynamic vs. static libraries
How do I safely pass objects, especially STL objects, to and from a DLL?
Distributing Windows C++ library: how to decide whether to create static or dynamic library?
Static library API question (std::string vs. char*)
Easy way to guarantee binary compatibility for C++ library, C linkage?
Also have reviewed:
https://www.acodersjourney.com/cplusplus-static-vs-dynamic-libraries/
https://blogs.msmvps.com/gdicanio/2016/07/11/the-perils-of-c-interface-dlls/
Given your requirements, you'll need a static library (e.g. .lib under windows) that is linked into your program during the build phase.
The interface to that library will need to be in a header file that declares types and functions.
You might choose to distribute as a set of libraries and header files, if they can be cleanly broken into separate pieces - if you manage the dependencies between the pieces. That's optional.
You won't be able to define your own templated functions or classes, since (with most compilers) that requires distributing source for those functions or classes. Or, if you do, you won't be able to use them as part of the interface to the library (e.g. you might use templated functions/classes internally within the library, but not expose them in the library header file to users of the library).
The main down side is that you will need to build and distribute a version of the library for each compiler and host system (in combination that you support). The C++ standard specifically encourages various types of incompatibilities (e.g. different name mangling) between compilers so, generally speaking, code built with one compiler will not interoperate with code built with another C++ compiler. Practically, your library will not work with a compiler other than the one it is built with, unless those compilers are specifically implemented (e.g. by agreement by both vendors) to be compatible. If you support a compiler that supports a different ABI between versions (e.g. g++) then you'll need to distribute a version of your library built with each ABI version (e.g. the most recent version of the compiler that supports each ABI)
An upside - which follows from having a build of your library for every compiler and host you support - is that there will be no problem using the standard library, including templated types or functions in the standard library. Passing arguments will work. Standard library types can be member of your classes. This is simply because the library and the program using it will be built with the same compiler.
You will need to be rigorous in including standard headers correctly (e.g. not relying on one standard header including another, unless the standard says it actually does - some library vendors are less than rigorous about this, which can cause your code to break when built with different compilers and their standard libraries).
There will mostly be no need for "Debug" and "Release" versions (or versions with other optimisation settings) of your library. Generally speaking, there is no problem with having parts of a program being linked that are compiled with different optimisation settings. (It is possible to cause such things to break,if you - or a programmer using your library in their program - use exotic combinations of build options, so keep those to a minimum). Distributing a "Debug" version of your library will permit stepping through your library with a debugger, which seems counter to your wishes.
None of the above prevents you using custom allocators, but doesn't require it either.
You will not need to recreate COM unless you really want to. In fact, you should aim to ensure your code is as standard as possible - minimise use of compiler-specific features, don't make particular assumptions about sizes of types or layout of types, etc. Using vendor specific features like COM is a no-no, unless those features are supported uniformly by all target compilers and systems you support. Which COM is not.
I don't know if there is a "standard/canonical" way of doing this. Writing code that works for multiple compilers and host systems is a non-trivial task, because there are variations between how different compiler vendors interpret or support the standard. Keeping your code simple is best - the more exotic or recent the language or standard library feature you use, the more likely you are to encounter bugs in some compilers.
Also, take the time to set up a test suite for your library, and maintain it regularly, to be as complete as possible. Test your library with that suite on every combination of compiler/system you support.
Provide a native C++ API that users may instantiate C++ classes, pass data around, all in C++, without a C-only wraper
This excludes COM.
Allow methods to take as parameters and return C++ objects, especially STL types (std::string, std::vector, etc)
This excludes DLLs
Most industry standard / canonical method of doing this possible, if such a standard exists
Not something "standard", but common practises are there. For example, in DLL, pass only raw C stuff.
No re-creating COM or using MS COM
This requires DLL/COM servers
Assume all C++ compilers are at least C++11 "compliant"
Perhaps. Normally yes.
Generally: If source is to be available, use header only (if templates) or h +cpp.
If no source, the best is DLL. Static libraries - you have to build for many compilers and one has to carry on your lib everywhere and link to it.
On linux, gcc uses libstdc++ for the c++ standard library while clang can use libc++ or libstdc++. If your users are building with clang and libc++, they won't be able to link if you only build with gcc and libstdc++ (libc++ and libstdc++ are not binary compatible). So if you wan't to get target both you need two version of you library, one for libstdc++, and another for libc++.
Also, binaries on linux (executable, static library, dynamic library) are not binary compatible between distros. It might work on your distro and not work on someone else distro or even a different version of your distro. Be super careful to test that it work on whichever distro you want to target. Holy Build Box could help you to produce cross-distribution binaries. I heard good thing about it, just never tried it. Worst case you might need to build on all the linux distro you want to support.
https://phusion.github.io/holy-build-box/
For our current project, we are thinking to use Boost framework.
However, the project should be truly cross-platform and might be shipped to some exotic platforms. Therefore, we would like to use only Boost packages (libraries) that does not contain any platform specific code: pure C++ and that's all.
Boost has the idea of header-only packages (libraries).
Can one assume that these packages (libraries) are free from platform specific code?
In case if not, is there a way to identify these kind of packages of Boost?
All C++ code is platform-specific to some extent. On the one side, there is this ideal concept of "pure standard C++ code", and on the other side, there is reality. Most of the Boost libraries are designed to maintain the ideal situation on the user-side, meaning that you, as the user of Boost, can write platform-agnostic standard C++ code, while all the underlying platform-specific code is hidden away in the guts of those Boost libraries (for those that need them).
But at the core of this issue is the problem of how to define platform-specific code versus standard C++ code in the real world. You can, of course, look at the standard document and say that anything outside of it is platform-specific, but that's nothing more than an academic discussion.
If we start from this scenario: assume we have a platform that only has a C++ compiler and a C++ standard library implementation, and no other OS or OS-specific API to rely on for other things that aren't covered by the standard library. Well, at that point, you still have to ask yourself:
What compiler is this? What version?
Is the standard library implementation correct? Bug-free?
Are those two entirely standard-compliant?
As far as I know, there is essentially no universal answer to this and there are no realistic guarantees. Most exotic platforms rely on exotic (or old) compilers with partial or non-compliant standard library implementations, and sometimes have self-imposed restrictions (e.g., no exceptions, no RTTI, etc.). An enormous amount of "pure standard C++ code" would never compile on these platforms.
Then, there is also the reality that most platforms today, even really small embedded systems have an operating system. The vast majority of them are POSIX compliant to some level (except for Windows, but Windows doesn't support any exotic platform anyways). So, in effect, platform-specific code that relies on POSIX functions is not really that bad since it is likely that most exotic platforms have them, for the most part.
I guess what I'm really getting at here is that this pure dividing line that you have in your mind about "pure C++" versus platform-specific code is really just an imaginary one. Every platform (compiler + std-lib + OS + ext-libs) lies somewhere along a continuum of level of support for standard language features, standard library features, OS API functions, and so on. And by that measure, all C++ code is platform-specific.
The only real question is how wide of a net it casts. For example, most Boost libraries (except for recent "flimsy" ones) generally support compilers down to a reasonable level of C++98 support, and many even try to support as far back as early 90s compilers and std-libs.
To know if a library, part of Boost or not, has wide enough support for your intended applications or platforms, you have the define the boundaries of that support. Just saying "pure C++" is not enough, it means nothing in the real world. You cannot say that you will be using C++11 compilers just after you've taken Boost.Thread as an example of a library with platform-specific code. Many C++11 implementations have very flimsy support for std::thread, but others do better, and that issue is as much of a "platform-specific" issue as using Boost.Thread will ever be.
The only real way to ever be sure about your platform support envelope is to actual set up machines (e.g., virtual machines, emulators, or real hardware) that will provide representative worst-cases. You have to select those worst-case machines based on a realistic assessment of what your clients may be using, and you have to keep that assessment up to date. You can create a regression test suite for your particular project, that uses the particular (Boost) libraries, and test that suite on all your worst-case test environments. Whatever doesn't pass the test, doesn't pass the test, it's that simple. And yes, you might find out in the future that some Boost library won't work under some new exotic platform, and if that happens you need to either get the Boost dev-team to add code to support it, or you have to re-write your code to get around it, but that's what software maintenance is all about, and it's a cost you have to anticipate, and such problems will come not only from Boost, but from the OS and from the compiler vendors too! At least, with Boost, you can fix the code yourself and contribute it to Boost, which you can't always do with OS or compiler vendors.
We had "Boost or not" discussion too. We decided not to use it.
We had some untypical hardware platforms to serve with one source code. Especially running boost on AVR was simply impossible because RTTI and exceptions, which Boost requires for a lot of things, aren't available.
There are parts of boost which use compiler specific "hacks" to e.g. get information about class structure.
We tried splitting the packages, but the inter dependency is quite high (at least 3 or 4 years ago).
In the meantime, C++11 was underway and GCC started supporting more and more. With that many reasons to use from boost faded (Which Boost features overlap with C++11?). We implemented the rest of our needs from scratch (with relative low effort thanks to variadic templates and other TMP features in C++11).
After a steep learning curve we have all we need without external libraries.
At the same time we have pondered the future of Boost. We expected the newly standardized C++11 features would be removed from boost. I don't know the current roadmap for Boost, but at the time our uncertainty made us vote against Boost.
This is not a real answer to your question, but it may help you decide whether to use Boost. (And sorry, it was to large for a comment)
Are there any explicit compatibility guarantees that boost::interprocess::managed_shared_memory would work across different boost versions? I'm planning to use it to share an integer or ten between multiple processes (which will essentially act as revision numbers for a piece of data they all read and some write). The processes are released separately and end-of-lifed once in a while.
The question is: am I locking myself down to a given version of boost for eternity because managed_shared_memory from 1.51 won't be able to talk with managed_shared_memory from 1.44 and so on?
According to the Boost FAQ :
How can the Boost libraries be used successfully for important projects?
Many of the Boost libraries are actively maintained and improved, so backward compatibility with prior version isn't always possible. Deal with this by freezing the version of the Boost libraries used by your project. Only upgrade at points in your project's life cycle where a bit of change will not cause problems. Individual bug fixes can always be obtained from the boost repository.
So it seems you are locking yourself to a set of future versions of Boost Interprocess that will be compatible with the version you will use, which is an unpredictable number of versions (compatibility could be broken tomorrow as well as it could be never broken).
If you can afford to invest some time to improve your code when a new version of boost comes and breaks compatibility, you are all fine. In practice i think that is more likely to happen once every few years than once a month, libraries makers tend to take backward compatibility into consideration before publishing updates.
I know that this question has been asked previously, but before you give me a minus and report repeated question, ponder a while on this:
In all previous answers everybody says that object memory layout is compiler dependent. How is it then, that shared libraries (*.dll, *.so) can export and import c++ classes, and they can definitely be combined even if coming from different compilers? Consider a DirectX application written under mingw. DirectX was compiled using MSVC++, so how do those environments agree on memory layout? I know that DirectX relies heavily on C++ classes and polymorphism.
Asking differently: let's say that I have a chosen architecture (eg. Windows, intel x86) and I am trying to write a new compiler. How do I know how to access class instance (vtable, member fields) provided by .dll lib compiled by another compiler? Or is it simply like that: M$ has written VC++, and since then it is unwritten standard, and every other compiler does it the same "for compatibility reasons"? And what about linux or other OS-es?
EDIT:
OK I admit, the example with DirectX was bad because of COM specification...
Another example: QT. I am using QT with mingw, but I know there are also available MSVC versions. I don't know if the difference is only in headers, or if shared libs (dll-s) are also different. If they are, does it mean that I have to distribute my app with qt libs included, so if anybody happens to have ones for a different compiler it will not get mixed-up? (Nice memory and code sharing then, right?). Or are they the same and there is some unwritten law about it anyway?
EDIT2:
I have installed a different qt version (msvc 2010) just to see what is and isn't shared. Seems that shared (are they really shared then) libraries are different. Seems that I really have to provide qt-libs with my app then... And this is no small thing (eg. QtGui 8-9MB). What about other, smaller libs, whose authors weren't so kind to provide versions for other compilers? Does it mean that I am stuck with their original compiler? What if I want to use two different libs that were compiled by different compilers?
Basically you're asking about an ABI.
In some cases (e.g., on the Itanium) there's a document that specifies the ABI, and essentially everybody follows the document.
In the case of DirectX, it's a bit the same: Microsoft has published specs for COM, so anybody who follows those specs can get interop with essentially any COM object (within reason -- a 64-bit compiler probably isn't going to work with a 16-bit COM object written for Windows 3.1).
For most other things, you're more or less on your own to figure things out. There's often at least a little published in the way of documentation, but at least in my experience it often skims over some details that end up important, isn't updated entirely dependably, and in some cases is just plain wrong. In most cases, it's also poorly organized so it's pretty much up to you to glean what you can from wherever you can find it, and when you run out of information (or patience) do some reverse engineering to fill in the missing parts.
Edit: For non-COM things like Qt libraries, you have a couple of choices. One is to link to the Qt libraries statically, so the Qt code you need gets linked directly into your executable. If you want to use a DLL, then yes, you're pretty much stuck with distributing one with your app that's going to be specific to the compiler you use to generate the application itself. A different compiler (or even a different version or set of compilation flags with the same compiler) will generally require a different DLL (possibly the same source code, but built to accommodate the compiler change).
There are exceptions to this, but they're pretty much like I've outlined above. For example, the Intel compiler for Windows normally uses Microsoft's standard library, and can use most (if not all) other libraries built for Microsoft's compiler as well. This is pretty much because Intel has bent over backward to assure that their compiler uses the same calling convention, name mangling scheme, etc., as Microsoft's though. It works because they put in a lot of effort to make it work, not because of any unwritten law, or anything like that.
DirectX is a COM-based technology and that is why it can be used in C with various compilers. Under the hood COM interface is a C-like structure which emulates the VMT.
Technically, DirectX has a midl-autogenerated C++ interface, but the general assertion that one can use classes exported in .dlls across different compilers is wrong.
Edit1:
QT dlls built using MSVC won't be compatible with the ones built by gcc, unfortunately, because of the natural reasons: different C++ compiler - different ABI and different run-time library (mingw uses older MSVCRT and that is why pure-C .dlls can be consumed in MSVC and vice versa). Some compilers incidentally or partially intentionally match their ABIs, but this is definitely not the case with MSVC/gcc. QT can also be built as a static library, so to redistribute things one might just link statically.
The name mangling for C++ classes in DLLs depends largely on the compiler front-end used. Many commercial compilers from well-known companies use EDG's C++ parser and that is why the class names an overloaded functions have similar or matching signatures.
Edit2:
"What if I want to use two different libs that were compiled by different compilers?"
If you desperately need some specific functionality from both libraries (I mean some concrete operation, not the framework overall), then the way to go without having the library's source code is to write some wrapper and compile this wrapper to the C-style .dll.
Think of this like having "two different C++-es, C++-1 and C++-2". The problem with ABI/Runtime is no different from using Pascal code from C or linking with some older Fortran libs.
I am developing a portable C++ application and looking for some best practices in doing that. This application will have frequent updates and I need to build it in such a way that parts of program can be updated easily.
For a frequently updating program, creating the program parts into libraries is the best practice? If program parts are in separate libraries, users can just replace the library when something changes.
If answer for point 1 is "yes", what type of library I have to use? In LINUX, I know I can create a "shared library", but I am not sure how portable is that to windows. What type of library I have to use? I am aware about the DLL hell issues in windows as well.
Any help would be great!
Yes, using libraries is good, but the idea of "simply" replacing a library with a new one may be unrealistic, as library APIs tend to change and apps often need to be updated to take advantage of, or even be compatible with, different versions of a library. With a good amount of integration testing though, you'll be able to 'support' a range of different versions of the library. Or, if you control the library code yourself, you can make sure that changes to the library code never breaks the application.
In Windows DLLs are the direct equivalent to shared libraries (so) in Linux, and if you compile both in a common environment (either cross-compiling or using MingW in Windows) then the linker will just do it the same way. Presuming, of course, that all the rest of your code is cross-platform and configures itself correctly for the target platform.
IMO, DLL hell was really more of a problem in the old days when applications all installed their DLLs into a common directory like C:\WINDOWS\SYSTEM, which people don't really do anymore simply because it creates DLL hell. You can place your shared libraries in a more appropriate place where it won't interfere with other non-aware apps, or - the simplest possible - just have them in the same directory as the executable that needs them.
I'm not entirely convinced that separating out the executable portions of your program in any way simplifies upgrades. It might, maybe, in some rare cases, make the update installer smaller, but the effort will be substantial, and certainly not worth it the one time you get it wrong. Replace all executable code as one in most cases.
On the other hand, you want to be very careful about messing with anything your users might have changed. Draw a bright line between the part of the application that is just code and the part that is user data. Handle the user data with care.
If it is an application my first choice would be to ship a statically-linked single executable. I had the opportunity to work on a product that was shipped to 5 platforms (Win2K,WinXp, Linux, Solaris, Tru64-Unix), and believe me maintaining shared libraries or DLLs with large codebase is a hell of a task.
Suppose this is a non-trivial application which involves use of 3rd Party GUI, Threads etc. Using C++, there is no real one way of doing it on all platforms. This means you will have to maintain different codebases for different platforms anyway. Then there are some wierd behaviours (bugs) of 3rd Party libraries on different platforms. All this will create a burden if application is shipped using different library versions i.e. different versions are to be attached to different platforms. I have seen people shipping libraries to all platforms when the fix is only for a particular platform just to avoid the versioning confusion. But it is not that simple, customer often has a different angle to how he/she wants to upgrade/patch which is also to be considered.
Ofcourse if the binary you are building is huge, then one can consider DLLs/shared-libraries. Even if that is the case, what i would suggest is to build your application in the form of layers like:-
Application-->GUI-->Platform-->Base-->Fundamental
So here some libraries can have common-code for all platforms. Only specific libraries like 'Platform' can be updated for specific behaviours. This will make you life a lot easier.
IMHO a DLL/shared-library option is viable when you are building a product that acts as a complete solution rather than just an application. In such a case different subsystems use common logic simultaneously within your product framework whose logic can then be shared in memory using DLLs/shared-libraries.
HTH,
As soon as you're trying to deal with both Windows and a UNIX system like Linux, life gets more complicated.
What are the service requirements you have to satisfy? Can you control when client systems get upgraded? How many systems will you need to support? How much of a backward-compatibility requirement do you have.
To answer your question with a question, why are you making the application native if being portable is one of the key goals?
You could consider moving to a a virtual platform like Java or .Net/Mono. You can still write C++ libraries (shared libraries on linux, DLL's on windows) for anything that would be better as native code, but the bulk of your application will be genuinely portable.