I'm having a strange problem with LoadLibrary on Windows. First some background. This application depends on Qt, and Qt is split into several libraries. I am trying to upgrade versions of Qt, but without breaking anyone. The newer Qt library is backwards compatible with the old one. That means that an application built with the older version can run if it loads the newer version. The opposite is not true -- an application built with the newer version will have missing symbols if the older one is loaded.
The Qt DLLs are in version specific directories (say c:\qt\qt-4.5.2\lib and c:\qt\qt-4.8.1\lib as examples). There is also a common directory that most developers have in their PATH that contains the "current" version of all the third party libraries we use (call it c:\common\lib). This is where the Qt libraries would normally be found when running an application.
I put the new Qt version libraries in the common location, and everything seemed to be working OK, except for one case. The application in question is split into multiple libraries, some of which are loaded by calling LoadLibrary(). Some of these runtime loaded DLLs depend on Qt libraries. In one case, the loaded DLL depends on QtXml, which itself depends on QtCore.
Here's where it gets weird. An application depends on QtCore and also loads a library that depends on QtXml. The application and library were built linking with the old version of Qt. If this application is run with just the common directory in the PATH, everything works because the new Qt version DLLs are loaded from the common directory. However, if the PATH contains the directory where the old Qt version DLLs are stored before the common directory, then loading the runtime DLL fails with missing symbols.
(This situation arises when doing automated unit testing, with scripts explicitly setting the PATH to use the specific library version.)
As near as I can figure, the application is loading the old version of QtCore.dll and the runtime loaded DLL is (somehow) loading the new version of QtXml.dll, which fails because the already loaded QtCore doesn't have the symbols it needs.
But this seems impossible, since the PATH is something like c:\qt\qt-4.5.2\lib;c:\common\lib (plus other unrelated paths). If I remove the newer QtXml from the common lib directory (not replace it with the old version, just remove it), then LoadLibrary() succeeds, because it loads the 4.5.2 version of all the Qt libraries. But that's not a good long term solution, since running without the Qt specific version directory in the PATH (common) will fail to find QtXml.
How could this be? How could LoadLibrary() (or whatever it calls recursively to resolve the library's dependencies) load a library from later in the PATH? I cannot find anything that would indicate that the common library directory is given special consideration (it's not a set DLL directory). It's not mentioned in the build process, it's just something developers have in their PATH for convenience.
btw, a similar situation exists on Linux with LD_LIBRARY_PATH and dlopen(), and it works just fine there. This is something Windows is doing differently that I don't understand. Does anyone have any insight into what could be going wrong?
LoadLibrary has lots of surprising behaviors. Make sure you fully grok all of the Remarks for it in MSDN.
If there's already a library loaded (any version) with the same name, LoadLibrary just returns a handle to the already-loaded DLL. That may be coming into play in your scenario.
Next, if you've specified a relative path or just a file name, LoadLibrary applies arcane search rules. Your PATH variable is typically the last place to search. It's likely that some other rule is finding the "wrong" DLL before it even gets to checking the PATH. A good guideline is to always use an absolute path the file you want to load to be sure that its search rules don't grab the wrong file. A common security flaw is to not control where LoadLibrary searches, and an attacker convinces your application to load a doctored DLL.
And, finally, it's possible for an installer to apply DLL redirection that can override what you ask for and where it might be found. I'm not sure if this is common for Qt DLLs or not, but you might want to check your registry.
I've occasionally used ProcMon from SysInternals to observe a program while it's loading DLLs. You can see each place it checks, which may give you a clue as to why it's finding the wrong version.
Related
I have an application that uses the windows dbghelp.dll(mainly to read the PE file header of dlls). To work with the dll, I had to include dbghelp.lib (import library) in my linker options.
I do understand the functionality of import libraries. My question is , when I distribute my application to users, is the .lib file also packaged with it? As the .lib file is part of sdk from VS, I dont expect my users to have this file.
since dbghelp.dll is provided by default in windows, I expect my application to work correctly on any windows machine. Am I correct in assuming this?
Note: I do know about the various versions of dbghelp.dll being on different versions and how that can sometimes cause issues during runtime on different machines.
My question is , when I distribute my application to users, is the
.lib file also packaged with it?
No, you don't need to distribute the dbghelp.lib file to end users as it is only required to build your application (during the link phase).
Note that the same also applies for developers if your application is open-source. They are expected to have the SDK to build your application, thus having already the dbghelp.lib file at disposition.
OTOH, if your end product is a DLL, you might provide the *.lib file of your product so developers can link against your DLL.
since dbghelp.dll is provided by default in windows, I expect my
application to work correctly on any windows machine. Am I correct in
assuming this?
That's a good question and the answer is yes and no. dbghelp.dll is a little bit special as it is often upgraded but never as a part of system updates (except for bugs). So your application will work fine in most of the cases but if you use the latest version you cannot expect the end user to have it on its system. Thus your application might not work correctly.
Quoting the MSDN:
Although this DLL is included in all supported versions of Windows, it
is rarely the most current version of DbgHelp available
Thus you are free (unlike most system DLLs) to redistribute the dbghelp.dll file (along symbol server files, namely SymSrv.dll, and SrcSrv.dll which are not included on systems.):
The redistribution policies for these included DLLs were specifically
designed to make it as easy as possible for people to include these
files in their own packages and releases.
Note that you must distribute the files from the Debugging Tools For Windows (not the one from your system).
I'm writing an application that uses both Intel's TBB library, and an API from a company called Maplink, which also uses TBB. The problem is that both my application and the Maplink API want to load TBB.dll from the directory containing my application's binary. The version of TBB.dll that Maplink provided with their API differs from the one my application requires, and they can't both co-exist in the application's executable directory. Do I have any option here other than statically linking TBB into my application so that it doesn't try to load the wrong version of TBB.dll that the Maplink API is using?
In the real world, it is a bad idea to mix different versions of the same DLL. You should really try and get your platform aligned. It is not called package hell for nothing.
That being said, it is very much up to the TBB.dll if it allows for multiple versions at once. You might be able to statically link your code against your version of TBB, but in doing so you will need to make sure the statically linked-in symbols are not dynamically visible (a compiler collection dependant linker option). The code that you have that depends on TBB must probably also be linked in a separate linker step from the one that includes linking to maplink. And the application will need to be linked without relinking against TBB.dll.
At least that is how it could work for so files in Linux.
As mentioned in the comments, you may put the newer version of tbb.dll into your application directory, and it should work properly for both the application and the 3rd party library it uses. For example, the recent version - TBB 4.2 - is binary compatible with old versions back to TBB 2.0.
I am using C++ with Visual Studio 2008 Express.
We are supplying a binary-only Windows library to a client, which itself uses a number of other libraries. Therefore we shipped both our dll file, along with the dll files that we use. Now the issue is that our client uses some of the libraries that we also use, but in another version. Therefore he can not use our library, since the libraries we both depend on are incompatible.
Technically I think it should be possible that both dependency versions are loaded into the process space. However, I am unsure how to do this, as both their application, as well as our dll look for the same dependency dll file name. Can anyone tell me what the best/cleanest way to deal with this problem is?
Thanks!
Generally speaking, it won't work. This is due to the fact that the third party DLL versions might interfere with each other when loaded into memory. One example could be if there is an exclusive resource like e.g. a file in a specific directory. Or a specific device. The problem is, nobody knows probably not even the manufacturer of the 3rd party DLLs - so extensive testing is necessary.
But maybe you're lucky and it works anyway. My recipe:
Put your DLL "DTAG.DLL" and all needed DLLs in a subdirectory of the applications directory with a fixed name e.g. "DTAG_LIB".
Write a import library by hand (there are other possibilities using DELAYLOAD). In that library load your DLL with LoadLibraryEx. Provide an absolute path ending with "DTAG_LIB\DTAG.DLL" and the flag LOAD_WITH_ALTERED_SEARCH_PATH. Windows will then load your DTAG.DLL from this directory and all needed DLLs from that directory also. Don't set the PATH to "DTAG_LIB"!
Your customer has to link against your manual import lib.
You could solve this kind of problem using a (new) additional DLL you would deliver and that would take care of handling the versions conflict (at runtime) - being a kind of proxy between your app and its dependencies.
An alternative would be to use the Windows Forwarded Libraries mechanism.
Forwarders are a handy way to accommodate functionality moving from one DLL to another
You can use several ways to declare forwarders, such as a module definition (.def) file and a #pragma:
#pragma comment(linker, "/export:function=otherdll.function")
My (C++, cross-platform) app is heavily using Boost libraries (say version 1.x), and I want to also link against a 3rd-party (vendor)'s SDK (no source), itself using Boost (but version 1.y).
So, we both link dynamically against our own version of Boost DLLs, CRT being identical. Consequently, at run-time my app would have to load both DLL of Boost 1.x & 1.y.
What are the potential issues & gotchas associated?
I can't change vendor's SDK, but I can change my app. Maybe I should try to link statically against my Boost 1.x?
PS: Name of Boost's DLL include their version, so no name collision, both are identifiable. Not the usual DLL-hell.
As far as using the DLLs for different versions there should be no problem. At least not on Windows.
This is true if the SDK is using boost internally. If the SDK uses boost constructs in its interface, for example: it has a function that returns a boost::optional, then having multiple versions can cause problems. It could still work fine, dependent on the changes between the versions, but that will definitely be a risk. I don't know of any good solution in that case. This is also true if you include a SDK header file that includes a boost header file.
This is a big problem.
Do a search on DLL hell.
Basically the DLL (or shared libs in Linux) are loaded but not all the names are resolved at load time. What happens is a lazy evaluation, so the names are evaluated on first use. The problem is that if 2 dll have the same name then the location where the name is resolved to depends on the what order the DLL are searched in (which depends on load order).
If you statically link then you will not have problems with method calls as yours will all be resolved at compile time and the third party will be resolved at runtime from the DLL. But what about structures that are created by version-1 boost. If you then pass these to the third party library that then passes it to the version-x boost. Are the structures layed out in the same way?
This is a very tricky area and when problems occur very hard to de-bug.
So try and use the same version.
If you write a function foo, and export it from F.dll, and another function foo exported from G.dll, would you expect problems?
When AF.exe is linked, the linker is told: put some code in there that loads the address of function foo from F.dll. Now BG.dll is linked to retrieve the foo address from G.dll. I still see no problem.
Now replace AF.exe with your app, BG.dll with your vendor's app, F.dll with your boost version, G.dll with the vendor's boost version.
Concluding: I see no problems if the dll names are different.
I have written some code that makes use of an open source library to do some of the heavy lifting. This work was done in linux, with unit tests and cmake to help with porting it to windows. There is a requirement to have it run on both platforms.
I like Linux and I like cmake and I like that I can get visual studios files automatically generated. As it is now, on windows everything will compile and it will link and it will generate the test executables.
However, to get to this point I had to fight with windows for several days, learning all about manifest files and redistributable packages.
As far as my understanding goes:
With VS 2005, Microsoft created Side By Side dlls. The motivation for this is that before, multiple applications would install different versions of the same dll, causing previously installed and working applications to crash (ie "Dll Hell"). Side by Side dlls fix this, as there is now a "manifest file" appended to each executable/dll that specifies which version should be executed.
This is all well and good. Applications should no longer crash mysteriously. However...
Microsoft seems to release a new set of system dlls with every release of Visual Studios. Also, as I mentioned earlier, I am a developer trying to link to a third party library. Often, these things come distributed as a "precompiled dll". Now, what happens when a precompiled dll compiled with one version of visual studios is linked to an application using another version of visual studios?
From what I have read on the internet, bad stuff happens. Luckily, I never got that far - I kept running into the "MSVCR80.dll not found" problem when running the executable and thus began my foray into this whole manifest issue.
I finally came to the conclusion that the only way to get this to work (besides statically linking everything) is that all third party libraries must be compiled using the same version of Visual Studios - ie don't use precompiled dlls - download the source, build a new dll and use that instead.
Is this in fact true? Did I miss something?
Furthermore, if this seems to be the case, then I can't help but think that Microsoft did this on purpose for nefarious reasons.
Not only does it break all precompiled binaries making it unnecessarily difficult to use precompiled binaries, if you happen to work for a software company that makes use of third party proprietary libraries, then whenever they upgrade to the latest version of visual studios - your company must now do the same thing or the code will no longer run.
As an aside, how does linux avoid this? Although I said I preferred developing on it and I understand the mechanics of linking, I haven't maintained any application long enough to run into this sort of low level shared libraries versioning problem.
Finally, to sum up: Is it possible to use precompiled binaries with this new manifest scheme? If it is, what was my mistake? If it isn't, does Microsoft honestly think this makes application development easier?
Update - A more concise question: How does Linux avoid the use of Manifest files?
All components in your application must share the same runtime. When this is not the case, you run into strange problems like asserting on delete statements.
This is the same on all platforms. It is not something Microsoft invented.
You may get around this 'only one runtime' problem by being aware where the runtimes may bite back.
This is mostly in cases where you allocate memory in one module, and free it in another.
a.dll
dllexport void* createBla() { return malloc( 100 ); }
b.dll
void consumeBla() { void* p = createBla(); free( p ); }
When a.dll and b.dll are linked to different rumtimes, this crashes, because the runtime functions implement their own heap.
You can easily avoid this problem by providing a destroyBla function which must be called to free the memory.
There are several points where you may run into problems with the runtime, but most can be avoided by wrapping these constructs.
For reference :
don't allocate/free memory/objects across module boundaries
don't use complex objects in your dll interface. (e.g. std::string, ...)
don't use elaborate C++ mechanisms across dll boundaries. (typeinfo, C++ exceptions, ...)
...
But this is not a problem with manifests.
A manifest contains the version info of the runtime used by the module and gets embedded into the binary (exe/dll) by the linker. When an application is loaded and its dependencies are to be resolved, the loader looks at the manifest information embedded in the exe file and uses the according version of the runtime dlls from the WinSxS folder. You cannot just copy the runtime or other modules to the WinSxS folder. You have to install the runtime offered by Microsoft. There are MSI packages supplied by Microsoft which can be executed when you install your software on a test/end-user machine.
So install your runtime before using your application, and you won't get a 'missing dependency' error.
(Updated to the "How does Linux avoid the use of Manifest files" question)
What is a manifest file?
Manifest files were introduced to place disambiguation information next to an existing executable/dynamic link library or directly embedded into this file.
This is done by specifying the specific version of dlls which are to be loaded when starting the app/loading dependencies.
(There are several other things you can do with manifest files, e.g. some meta-data may be put here)
Why is this done?
The version is not part of the dll name due to historic reasons. So "comctl32.dll" is named this way in all versions of it. (So the comctl32 under Win2k is different from the one in XP or Vista). To specify which version you really want (and have tested against), you place the version information in the "appname.exe.manifest" file (or embed this file/information).
Why was it done this way?
Many programs installed their dlls into the system32 directory on the systemrootdir. This was done to allow bugfixes to shared libraries to be deployed easily for all dependent applications. And in the days of limited memory, shared libraries reduced the memory footprint when several applications used the same libraries.
This concept was abused by many programmers, when they installed all their dlls into this directory; sometimes overwriting newer versions of shared libraries with older ones. Sometimes libraries changed silently in their behaviour, so that dependent applications crashed.
This lead to the approach of "Distribute all dlls in the application directory".
Why was this bad?
When bugs appeared, all dlls scattered in several directories had to be updated. (gdiplus.dll) In other cases this was not even possible (windows components)
The manifest approach
This approach solves all problems above. You can install the dlls in a central place, where the programmer may not interfere. Here the dlls can be updated (by updating the dll in the WinSxS folder) and the loader loads the 'right' dll. (version matching is done by the dll-loader).
Why doesn't Linux have this mechanic?
I have several guesses. (This is really just guessing ...)
Most things are open-source, so recompiling for a bugfix is a non-issue for the target audience
Because there is only one 'runtime' (the gcc runtime), the problem with runtime sharing/library boundaries does not occur so often
Many components use C at the interface level, where these problems just don't occur if done right
The version of libraries are in most cases embedded in the name of its file.
Most applications are statically bound to their libraries, so no dll-hell may occur.
The GCC runtime was kept very ABI stable so that these problems could not occur.
If a third party DLL will allocate memory and you need to free it, you need the same run-time libraries. If the DLL has allocate and deallocate functions, it can be ok.
It the third party DLL uses std containers, such as vector, etc. you could have issues as the layout of the objects may be completely different.
It is possible to get things to work, but there are some limitations. I've run into both of the problems I've listed above.
If a third party DLL allocates memory that you need to free, then the DLL has broken one of the major rules of shipping precompiled DLL's. Exactly for this reason.
If a DLL ships in binary form only, then it should also ship all of the redistributable components that it is linked against and its entry points should isolate the caller from any potential runtime library version issues, such as different allocators. If they follow those rules then you shouldn't suffer. If they don't then you are either going to have pain and suffering or you need to complain to the third party authors.
I finally came to the conclusion that the only way to get this to work (besides statically linking everything) is that all third party libraries must be compiled using the same version of Visual Studios - ie don't use precompiled dlls - download the source, build a new dll and use that instead.
Alternatively (and the solution we have to use where I work) is that if the third-party libraries that you need to use all are built (or available as built) with the same compiler version, you can "just" use that version. It can be a drag to "have to" use VC6, for example, but if there's a library you must use and its source is not available and that's how it comes, your options are sadly limited otherwise.
...as I understand it. :)
(My line of work is not in Windows although we do battle with DLLs on Windows from a user perspective from time to time, however we do have to use specific versions of compilers and get versions of 3rd-party software that are all built with the same compiler. Thankfully all of the vendors tend to stay fairly up-to-date, since they've been doing this sort of support for many years.)