I've been having a lot of conceptual issues with Microsoft's CRT. For any project you have to compile all required libraries to link against the same version of the CRT.
The first problem is when your project statically links against the CRT (/MT). Then all the dependant libraries must also link their own CRT statically. So each library has its own version of - for example - malloc(). If you compiled one of the libraries last year on system A, that CRT version may be different than the one you're currently using on another system B with service pack 3+. So if you're freeing objects allocated by the library you may run into problems.
So it seems dynamically linked CRT is the way to go (/MD). With dlls all the libraries would get the current implementation of the CRT on the system. Except that with Microsoft's Side by Side mechanism that's not what happens. Instead you get the CRT version that's stamped on the library you compiled and that version of the DLL is supplied to that library. So exactly the same problem I described before can occur. You compile a library on system A a year ago against one CRT. A year later there's a new version with upgrade. Your main program gets the DLL with one version of the CRT, the library gets the DLL with another version of CRT, same problem can occur.
So what do you do? I realize cross library memory allocation is frowned upon. But you can ignore the malloc example and come up with another one. Do you have every developer recompile every dependant library on their machine to make sure everything does use the same CRT? Then for the release you recompile every library again?
How does this work on Linux? That's my main interest. Is there a CRT supplied with GCC or the Linux system itself comes with CRT libraries? I've never seen the CRT linked explicitly in Makefils.
On Linux, what CRT do dynamic libraries link against? The most current one on the machine, or is it more "side by side" mechanism.
On the Linux side I think there are two basic parts of the standard library that are at issue: We have the C-runtime part, which should pretty much be ABI compatible forever. Effectively whichever version links at final link time should be fine, and you can redistribute any needed shared library with your binary if it's an older version needed for compatibility. Usually the libraries just sit side-by-side on *NIX systems.
Secondly, you have the C++ libraries. These are pretty much guaranteed to not be ABI compatible in any way, so you must rebuild every single component of a final binary against the same version of the C++ library. There's just no way around it unfortunately because otherwise you could wind up with a variety of mismatches. This is why many open source libraries don't even bother with premade library binaries: Everyone needs to build their own copy to make sure that it will properly link into their final application code.
Related
I'm writing an application that uses both Intel's TBB library, and an API from a company called Maplink, which also uses TBB. The problem is that both my application and the Maplink API want to load TBB.dll from the directory containing my application's binary. The version of TBB.dll that Maplink provided with their API differs from the one my application requires, and they can't both co-exist in the application's executable directory. Do I have any option here other than statically linking TBB into my application so that it doesn't try to load the wrong version of TBB.dll that the Maplink API is using?
In the real world, it is a bad idea to mix different versions of the same DLL. You should really try and get your platform aligned. It is not called package hell for nothing.
That being said, it is very much up to the TBB.dll if it allows for multiple versions at once. You might be able to statically link your code against your version of TBB, but in doing so you will need to make sure the statically linked-in symbols are not dynamically visible (a compiler collection dependant linker option). The code that you have that depends on TBB must probably also be linked in a separate linker step from the one that includes linking to maplink. And the application will need to be linked without relinking against TBB.dll.
At least that is how it could work for so files in Linux.
As mentioned in the comments, you may put the newer version of tbb.dll into your application directory, and it should work properly for both the application and the 3rd party library it uses. For example, the recent version - TBB 4.2 - is binary compatible with old versions back to TBB 2.0.
I know that if I link my c++ program to a dynamic library (DLL) that was built with a different version of Visual Studio, it won't work because of the binary compatibility issue.
(I have experienced this with Boost library and VS 2005 and 2008)
But my question is: is this true for all versions of MSVS? Does this apply to static libraries(LIB) as well? Is this an issue with GCC & Linux as well? and finally how about linking in VS to a DLL built with MinGW?
By the way aside from cross-platform or cross-compiler, why can't two version of the same compiler(VS) be compatibile?
Hi. I know that if I link my c++ program to a dynamic library (DLL) that was built with a different version of Visual Studio, it won't work because of the binary compatibility issue. (I have experienced this with Boost library and VS 2005 and 2008)
I do not remember ever seeing MS changing the ABI, so technically different versions of the compiler will produce the same output (given the same flags (see below)).
Therefore I don't think this is not incompatibilities in Dev Studio but changes in Boost.
Different versions of boost are not backwards compatible (in binary, source they are backward compatible).
But my question is: is this true for all versions of MSVS?
I don't believe there is a problem. Now if you use different flags you can make the object files incompatible. This is why debug/release binaries are built into separate directories and linked against different versions of the standard run-time.
Does this apply to static libraries(LIB) as well?
You must link against the correct static library. But once the static library is in your code it is stuck there all resolved names will not be re-resolved at a later date.
Is this an issue with GCC & Linux as well?
Yes. GCC has broken backwards compatability in the ABI a couple of times (several on purpose (some by mistake)). This is not a real issue as on Linux code is usually distributed as source and you compile it on your platform and it will work.
and finally how about linking in VS to a DLL built with MinGW?
Sorry I don't know.
By the way aside from cross-platform or cross-compiler, why can't two version of the same compiler(VS) be compatibile?
Well fully optimized code objects may be compressed more thus alignment is different. Other compiler flags may affect the way code is generated that is incompatible with other binary objects (changing the way functions are called (all parameters on the stack or some parameters in registers)). Technically only objects compiled with exactly the same flags should be linked together (technically it is a bit looser than that as a lot of flags don't affect the binary compatibility).
Note some libraries are released with multiple versions of the same library that are compiled in different ways. You usually distinguish the library by the extension on the end. At my last job we used the following convention.
libASR.dll // A Sincgle threaded Relase version lib
libASD.dll // A Single threaded Debug version
libAMR.dll // A Multi threaded Release version
libAMD.dll // A Multi threaded Debug version
If properly built, DLLs should be programming-language and version neutral. You can link to DLLs built with VB, C, C++, etc.
You can use dependency walker to examine the exported functions in the dll.
To answer part of your question, GCC/Linux does not have this problem. At least, not as often. libstdc++ and glibc are the standard C++/C libraries on GNU systems, and the authors of those libraries go to efforts to avoid breaking compatibility. glibc is pretty much always backward compatible, but libstdc++ has broken ABI several times in the past and probably will again in the future.
It is very difficult to write stable ABIs in C++ compared to C, because the automatic features in C++ take away some of the control you need to maintain an ABI. Especially once you get into templates and inline functions, where some of the code gets embedded in your application rather than staying contained in the shared library. That means that the object's structure can't ever change without requiring a recompilation of the application.
In practice, it isn't a huge deal on Windows. It would be fantastic if Microsoft just made the MSI installer know how to grab Microsoft-provided DLLs from Windows Update when an app is installed that needs them, but simply adding the redistributable to an InnoSetup-generated installer works well enough.
I'm writing a cross-platform application which is not GNU GPL compatible. The major problem I'm currently facing is that the application is linked dynamically with glibc and libstdc++, and almost every new major update to the libraries are not backwards compatible. Hence, random crashes are seen in my application.
As a workaround, I distribute binaries of my application compiled on several different systems (with different C/C++ runtime versions). But I want to do without this. So my question is, keeping licensing and everything in mind, can I link against glibc and libstdc++ statically? Also, will this cause issues with rtld?
You don't need to.
Copy the original libraries you linked against to a directory (../lib in this example) in your application folder.
Like:
my_app_install_path
.bin
lib
documentation
Rename you app for something like app.bin. Substitute your app for a little shell script that sets the enviroment variable LD_LIBRARY_PATH to the library path (and concatenate the previous LD_LIBRARY_PATH contents, if any). Now ld should be able to find the dynamic libraries you linked against and you don't need to compile them statically to your executable.
Remember to comply with the LGPL adding the given attribution to the libraries and pointing in the documentation where the source can be downloaded.
glibc is under the LGPL. Under section 6. of LGPL 2.1, you can distribute your program linked to the library provided you comply with one of five options. The first is to provide the source code of the library, along with the object code (source is optional, not required) of your own program, so it can be relinked with the library. You can alternatively provide a written offer of the same. Your own code does not have to be under the LGPL, and you don't have to release source.
libstdc++ is under the GPL, but with a major exception. You can basically just distribute under the license of your choice without providing source for either your own code or libstdc++. The only condition is that you compile normally, without e.g. proprietary modifications or plugins to GCC.
IANAL, and you should consider consulting one if you need real legal advice.
Specifying the option -static-libgcc to the linker would cause it to link against a static version of the C library, if available on the system. Otherwise it is ignored.
I must question what the heck you are doing with the poor library functions?
I have some cross platform software as well. It runs fine on Linux systems of all sorts. Build with the oldest version of software that you want to support. The glibc and libstdc++ libraries are really very backward compatible.
I have built on CentOS 4 and run it on RHEL 6 beta. No problems.
I can build on stable Debian and run it on testing.
Now, I do sometimes have trouble with some libraries if I try to build on, say old Debian and try to run it on CentOS 5.4. That is usually due to distribution configuration choices that are different, like choosing threading or non-threading.
How do you construct a library (static lib or a dll/so) so that it isn't sensitive to future updates to the system's C runtime librarires?
At the end of July, Microsoft updated a bunch of libraries, including the C runtime libraries. Our app is written with a mix of MFC/C++/VB and some third party libraries, including some that are closed source.
I've been busy recompiling all of the libraries we have source to, but I am wondering if it is really necessary? What will happen if we link in or load a library built against an earlier version of the C runtime?
When recompiling this stuff, what compiler and linker settings must be the same between the main application and the supporting libraries? I've discovered that the runtime library setting needs to be the same (we use the multi-threaded version /MD and /MDd) but I'm worried about other settings. I've actually pulled all the settings out into Visual Studio property sheets and I'm using the same sheets for all our different projects, but this doesn't work for 3rd party libraries and I'm thinking it is overkill.
I have noticed that the linker will spit out a warning about conflicting libraries, but it suggests to just ignore the default libraries. Is it safe to do so? It seems like a very ugly solution to the problem.
If you are loading the 3rd party libraries as DLLs, they may depend on different runtime versions than your executable as long as
you are not handing over parameters of types, that depend on the runtime libs (like STL types)
the 3rd party lib is able to load the version of the runtime, that it has been built with or is statically linked to the runtime
So, you don't have to recompile the DLLs.
If you are statically
linking to the libs or if you are handing over types defined in the runtime DLLs, you may get some problems with symbols, that are already imported in your lib, so most likely you will have to recompile it.
If you or your third-party components are statically linking against out-of-date C libraries, you are fine; upgrades to those libraries will not affect your program. Of course, you won't benefit from any bug fixes or performance upgrades or what-have-you either. If you do recompile your code to take advantage of your new settings, all runtime switches must be the same. This is always the case, regardless of when or why your libraries are compiled. It is probably safe to ignore the default libraries (I've been doing so for years without difficulty).
Dynamically-linked libraries are another story. If you rely on the target system having a particular version of a given dll, and it has some other incompatible version instead, then you are screwed. The traditional solution to this problem is to bundle all the dlls you need with your executable. Microsoft's new side-by-side assembly thing might also be able to help, but it's always been a little too hard to set up for me to bother with it. You might have better luck.
I have written some code that makes use of an open source library to do some of the heavy lifting. This work was done in linux, with unit tests and cmake to help with porting it to windows. There is a requirement to have it run on both platforms.
I like Linux and I like cmake and I like that I can get visual studios files automatically generated. As it is now, on windows everything will compile and it will link and it will generate the test executables.
However, to get to this point I had to fight with windows for several days, learning all about manifest files and redistributable packages.
As far as my understanding goes:
With VS 2005, Microsoft created Side By Side dlls. The motivation for this is that before, multiple applications would install different versions of the same dll, causing previously installed and working applications to crash (ie "Dll Hell"). Side by Side dlls fix this, as there is now a "manifest file" appended to each executable/dll that specifies which version should be executed.
This is all well and good. Applications should no longer crash mysteriously. However...
Microsoft seems to release a new set of system dlls with every release of Visual Studios. Also, as I mentioned earlier, I am a developer trying to link to a third party library. Often, these things come distributed as a "precompiled dll". Now, what happens when a precompiled dll compiled with one version of visual studios is linked to an application using another version of visual studios?
From what I have read on the internet, bad stuff happens. Luckily, I never got that far - I kept running into the "MSVCR80.dll not found" problem when running the executable and thus began my foray into this whole manifest issue.
I finally came to the conclusion that the only way to get this to work (besides statically linking everything) is that all third party libraries must be compiled using the same version of Visual Studios - ie don't use precompiled dlls - download the source, build a new dll and use that instead.
Is this in fact true? Did I miss something?
Furthermore, if this seems to be the case, then I can't help but think that Microsoft did this on purpose for nefarious reasons.
Not only does it break all precompiled binaries making it unnecessarily difficult to use precompiled binaries, if you happen to work for a software company that makes use of third party proprietary libraries, then whenever they upgrade to the latest version of visual studios - your company must now do the same thing or the code will no longer run.
As an aside, how does linux avoid this? Although I said I preferred developing on it and I understand the mechanics of linking, I haven't maintained any application long enough to run into this sort of low level shared libraries versioning problem.
Finally, to sum up: Is it possible to use precompiled binaries with this new manifest scheme? If it is, what was my mistake? If it isn't, does Microsoft honestly think this makes application development easier?
Update - A more concise question: How does Linux avoid the use of Manifest files?
All components in your application must share the same runtime. When this is not the case, you run into strange problems like asserting on delete statements.
This is the same on all platforms. It is not something Microsoft invented.
You may get around this 'only one runtime' problem by being aware where the runtimes may bite back.
This is mostly in cases where you allocate memory in one module, and free it in another.
a.dll
dllexport void* createBla() { return malloc( 100 ); }
b.dll
void consumeBla() { void* p = createBla(); free( p ); }
When a.dll and b.dll are linked to different rumtimes, this crashes, because the runtime functions implement their own heap.
You can easily avoid this problem by providing a destroyBla function which must be called to free the memory.
There are several points where you may run into problems with the runtime, but most can be avoided by wrapping these constructs.
For reference :
don't allocate/free memory/objects across module boundaries
don't use complex objects in your dll interface. (e.g. std::string, ...)
don't use elaborate C++ mechanisms across dll boundaries. (typeinfo, C++ exceptions, ...)
...
But this is not a problem with manifests.
A manifest contains the version info of the runtime used by the module and gets embedded into the binary (exe/dll) by the linker. When an application is loaded and its dependencies are to be resolved, the loader looks at the manifest information embedded in the exe file and uses the according version of the runtime dlls from the WinSxS folder. You cannot just copy the runtime or other modules to the WinSxS folder. You have to install the runtime offered by Microsoft. There are MSI packages supplied by Microsoft which can be executed when you install your software on a test/end-user machine.
So install your runtime before using your application, and you won't get a 'missing dependency' error.
(Updated to the "How does Linux avoid the use of Manifest files" question)
What is a manifest file?
Manifest files were introduced to place disambiguation information next to an existing executable/dynamic link library or directly embedded into this file.
This is done by specifying the specific version of dlls which are to be loaded when starting the app/loading dependencies.
(There are several other things you can do with manifest files, e.g. some meta-data may be put here)
Why is this done?
The version is not part of the dll name due to historic reasons. So "comctl32.dll" is named this way in all versions of it. (So the comctl32 under Win2k is different from the one in XP or Vista). To specify which version you really want (and have tested against), you place the version information in the "appname.exe.manifest" file (or embed this file/information).
Why was it done this way?
Many programs installed their dlls into the system32 directory on the systemrootdir. This was done to allow bugfixes to shared libraries to be deployed easily for all dependent applications. And in the days of limited memory, shared libraries reduced the memory footprint when several applications used the same libraries.
This concept was abused by many programmers, when they installed all their dlls into this directory; sometimes overwriting newer versions of shared libraries with older ones. Sometimes libraries changed silently in their behaviour, so that dependent applications crashed.
This lead to the approach of "Distribute all dlls in the application directory".
Why was this bad?
When bugs appeared, all dlls scattered in several directories had to be updated. (gdiplus.dll) In other cases this was not even possible (windows components)
The manifest approach
This approach solves all problems above. You can install the dlls in a central place, where the programmer may not interfere. Here the dlls can be updated (by updating the dll in the WinSxS folder) and the loader loads the 'right' dll. (version matching is done by the dll-loader).
Why doesn't Linux have this mechanic?
I have several guesses. (This is really just guessing ...)
Most things are open-source, so recompiling for a bugfix is a non-issue for the target audience
Because there is only one 'runtime' (the gcc runtime), the problem with runtime sharing/library boundaries does not occur so often
Many components use C at the interface level, where these problems just don't occur if done right
The version of libraries are in most cases embedded in the name of its file.
Most applications are statically bound to their libraries, so no dll-hell may occur.
The GCC runtime was kept very ABI stable so that these problems could not occur.
If a third party DLL will allocate memory and you need to free it, you need the same run-time libraries. If the DLL has allocate and deallocate functions, it can be ok.
It the third party DLL uses std containers, such as vector, etc. you could have issues as the layout of the objects may be completely different.
It is possible to get things to work, but there are some limitations. I've run into both of the problems I've listed above.
If a third party DLL allocates memory that you need to free, then the DLL has broken one of the major rules of shipping precompiled DLL's. Exactly for this reason.
If a DLL ships in binary form only, then it should also ship all of the redistributable components that it is linked against and its entry points should isolate the caller from any potential runtime library version issues, such as different allocators. If they follow those rules then you shouldn't suffer. If they don't then you are either going to have pain and suffering or you need to complain to the third party authors.
I finally came to the conclusion that the only way to get this to work (besides statically linking everything) is that all third party libraries must be compiled using the same version of Visual Studios - ie don't use precompiled dlls - download the source, build a new dll and use that instead.
Alternatively (and the solution we have to use where I work) is that if the third-party libraries that you need to use all are built (or available as built) with the same compiler version, you can "just" use that version. It can be a drag to "have to" use VC6, for example, but if there's a library you must use and its source is not available and that's how it comes, your options are sadly limited otherwise.
...as I understand it. :)
(My line of work is not in Windows although we do battle with DLLs on Windows from a user perspective from time to time, however we do have to use specific versions of compilers and get versions of 3rd-party software that are all built with the same compiler. Thankfully all of the vendors tend to stay fairly up-to-date, since they've been doing this sort of support for many years.)