I have a lot of basic questions about DLLs and how to bundle third-party libraries with your software, but most everything I find when searching is specific to C# (such as this question) and I'm looking for more general information about how DLLs work and what their purpose is.
I have, for example, a CMake-managed C++ application that links to one or more third-party DLLs.
Their DLLs are installed to C:/Program Files/Intel/SomeLibrary/bin/somelibrary.dll. I can build my app fine by having an environment variable take care of finding their headers.
Obviously, I can't actually run my executables out of the box, because somelibrary.dll can't be found. I have to either copy it into the same folder as the executable, or add Intel/SomeLibrary/bin to my Path.
Questions
How do other programs link to their required DLLs? Do they just package copies of all the third-party DLLs they use in with their own software?
Is it a different process for some third-party libraries -- like maybe Intel libraries you'd bundle with your software, but for Microsoft's VS Redistributables you'd require the user to install them? It looks like your installer (WiX example) handles installing them if they don't exist (and then does it install a new copy for every piece of software that needs them?).
Could you potentially have many copies on your computer of the same DLL because a bunch of separate programs needed it but had no way of knowing if you had it somewhere already?
Put another way, could two programs, written by different people, ever link to the same DLL file? How do they find that DLL, or install it if it doesn't exist?
I apologize in advance for whatever fundamental misunderstandings I've made in asking these -- I'm pretty new to the idea of packaging already-built software.
The short answer is that the dll loader will look for dlls in the local application folder and then searching a bunch of system locations. There are complexities with manifests, the SafeDllSearchMode feature (affects directory search order) and developers loading dlls from custom locations by override (LoadLibraryEx) and several others (see below).
Dependencty Types: There are many types of dll dependencies: Module / dll dependencies mind map.
Dynamic-Link Library Search Order: Here is an extract from Microsoft Docs on the topic: "A system can contain multiple versions of the same dynamic-link library (DLL). Applications can control the location from which a DLL is loaded by specifying a full path or using another mechanism such as a manifest."
In other words you can 1) load a dll from a specific location or 2) you can let the OS find it for you via standard mechanisms. The "SafeDllSearchMode" registry setting affects when the users current directory is searched. It now defaults to on, but can be disabled. See the link above for the authoritative details. Beware of OS changes and security fixes - all can affect this order in the future.
Q1: Always use the runtime provided for your dependent dll if available. Don't just include a few dlls if you are not supposed to.
Q2: It is correct that runtimes are often very different. Sometimes they are just a single or a few dlls that you can just install in your local application folder. Other times you install a runtime setup which installs files to appropriate folders for shared use. Some runtimes come as "merge modules" that can merge files into your own MSI installer, and some come as separate MSI installers or setup.exe installers.
Q3: The whole idea of using dlls is essentially to be able to load the same binary component into many processes in order to be able to update a single dll to fix problems in all "consumers" of the dll and also to share them in memory (important back in the day with little memory). Dlls can also be "delay loaded" - which means they won't be loaded until they are necessary - avoiding memory bloat. That is the idea anyway.
Windows OS Version: The search order (where to search on disk) for the right dlls to load is a pretty complicated topic. How the OS loader mechanism locates DLLs can be affected by a number of things, for example 1) the OS version, 2) manifests, 3) hard coded altered load paths (LoadLibraryEx), or 4) dll redirects, and 5) previously loaded dlls in memory (here be dragons), etc...
UWP & .NET: There are also new development with regards to UWP applications. For .NET assemblies the resolution is different again (which you state), please see this article: How the Runtime Locates Assemblies.
WinSxS: The manifest is a newer concept in Windows - in the olden days there would be no such thing and the search order would be simpler, but less flexible. With the advent of manifests and side-by-side Win32 assemblies (several versions of the same dll installed in different versions) the manifest is embedded in the executable (or it is added externally) and it tells the binary where to look for the runtime dlls - in the WinSxS folder hierarchy of Windows (Normally: C:\Windows\WinSxS) where multiple versions of the same dll can reside (in different sub folders).
Most Microsoft runtimes - basically all modern ones - reside here and hence require a manifest to instruct the binary what version to look for and load. There are ways to redirect such manifests too... "Publisher Configuration Files" and 2. Sigh... This can be used to enforce a certain new runtime version when the old one is undesired for use. Unused runtimes can also be "garbage collected" via ways to shrink the WinSxS folders (See this answer - section "Scavenging / Putting WinSxS on a Diet")... Hmmm... Never hard code paths to WinSxS.
Dll Search Order: The old and outdated tool Dependency Walker has an outdated, but very good description of the DLL Search Order in its help file.
Tip: The updated documentation for this is: Microsoft Docs: Module Search Order for Desktop Applications. However, to get a rough idea of search order, you can just launch Dependency Walker and select: Options => Configure Module Search Order.... Then make sure the Expand button is deselected. What you see - outdated as it might be - can help to recollect how module search order works:
Links:
DLL search on windows (old Stackoverflow answer on the same topic)
The path-searching algorithm is not a backtracking algorithm
The case of the DLL that refuses to load
Related
As a background: my application requires:
admin privileges
access to WinAPI DLLs
be able to run on all OSs: Win7-Win10
Normally, to use API, I can just link required *.lib files. However it uses default search order, that means (according to https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic-link-library-search-order) it firstly loads DLLs from "The directory from which the application loaded."
As a result, if the DLL exists in the same directory, running my app by double-click loads also that DLL.
I want to look for DLLs only in system directories (similarly to https://stackoverflow.com/a/46182665/9015013 ).
I know I can try to create some kind of proxy, like
BOOL WinAPIFunction(WinAPIType param) {
return reinterpret_cast<decltype(&WinAPIFunction)>(
reinterpret_cast<void*>(GetProcAddress(manually_loaded_module, "WinAPIFunction")))(param);
}
But it is hard to maintain all these functions. Is there any better method to force windows to look only in system32? I thought about manifest file but it requires version for each DLL that can break "capability" requirement (DLLs have different version for Win7 and Win10)
The solution is posted by #Eryk Sun in comment above.
It is sufficient to add all DLLs not listed in known dlls to delayed loaded libraries and call SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32); at the beginning of the WinMain.
Thanks
You can try to use "Known DLLs" feature: If Windows "knows" that DLL, Windows doesn't search dll file. Known feature is described into link, you written in question.
Some of our users have been complaining about a libmmd.dll not found error on our plugins loading.
Some background:
I'm talking about an issue occurring on Windows (8, for that matter).
We develop plugins for Digital Audio Workstations (for digital sound
processing purposes).
Our plugin is a DLL coded in VS2012 and compiled with Intel Compiler 2015 via Perl scripts calling icl.exe
from the cmd.
One of its dependencies is an Intel-supplied math
library provided in the 2015 flavor of its redistributables (which we
install together with our plugins) called libmmd.dll.
For a matter of convenience, let's call the plugin DLL plugin.dll (!!!).
I didn't have much time on one of the affected users' machine but from what I saw, reinstalling Intel's redistributables package didn't help, only moving libmmd.dll to the same folder as plugin.dll.
Whatever the reason for this (sudden and unexpected) behavior, we want to be able to deliver software protected against this kind of issues (DLL Hell).
What we want is for plugin.dll to look for libraries first in a specific directory (configurable would be even better), then in its usual search order.
I tried playing with manifests, registry, library names, linker options...
The only things that actually worked were switching the /MD option with /MT (but recompiling with static libs really adds to the size of plugin.dll) and copying the library to the folder plugin.dll is in.
Any ideas ?
Thanks !
The SetDllDirectory function lets you add a folder path to the DLL search order. It will be added in second place, immediately behind the program folder itself.
This only affects DLLs loaded via LoadLibrary however; if your DLLs are loaded statically it won't make any difference.
I have read these two SO questions: Which runtime libraries to ship? and License of runtime libraries included in GCC? - both were very helpful but not quite what I was looking for.
I have always just written programs for use on my own machine, which has never caused me any problems, but now I want to start running software on other machines and I'm wary of the runtime requirements.
EDIT: See below example instead, this was misleading.
Specifically, if I write a C++ program on a Windows machine, compiled with gcc through MinGW, and want to run it on another machine:
Do I have to send the libstdc++.dll with my program?
Is this single file (I assume placed in the executable's directory) sufficient to allow the program to run?
Also, an identical example, except this time it is an Objective-C program. Is sending the libobjc.dll file to the other machine sufficient to allow the program to execute properly?
I am used to running programs on machines which have developer tools, etc, installed, but now I'm looking to run them on general purpose machines (friends', colleagues' etc), and I'm not quite sure what to do!
EDIT: In response to edifice's answer, I feel I should clarify what it is I'm looking for. I know how to identify the necessary DLL(s) (/dylibs, etc) that my programs use, (although I am accustomed to doing that work manually; I had not heard of any of the tools). My question was more "What do I do now?"
A more general example is probably needed:
Let's say I have written a program which has object files derived from C++, C and/or Objective-C(2) code. I have used some Windows API code which compiled successfully using MinGW's gcc. I also have a custom DLL I wrote in Visual Studio (C++).
I have identified which DLL's my program will use at runtime (one of which may be GCC's libobjc.dll, I'm not sure if this would/should make a difference on a Windows machine, but I want to make this as general as possible) - The "prerequisite DLLs".
I would like to run it on my colleagues' computers, most of which run Windows 7, but some now run Windows 8. Starting at the very start for the sake of completeness:
Do I need to transfer the prerequisite DLLs to my colleagues' computers?
What directory should I place them in? (exe directory / a system directory?)
Once in place, will the presence of these DLLs allow the program to execute correctly? (Assuming it knows where to find them)
Are there any other files that should be transferred with the DLLs?
Basically I'm trying to determine the entire thought-process for developing and running an application on another machine in terms of system runtime requirements.
When loading DLLs, the first place Windows looks is the directory that the exe is in. So it will probably work just fine to put the DLLs there.
For the Microsoft DLLs though, I think it makes more sense to ask your colleague to install the Visual C++ runtime, which is a redistributable package from Microsoft. Ideally you would make an installer using something like WiX and it would install that prerequisite for you, but it is OK to just tell your colleague to do it.
Be sure to include a license file with your software if you include DLLs from gcc, because the GPL requires it.
libstdc++ isn't necessarily sufficient. You almost certainly need libgcc too, but actual dependencies are liable to vary with your particular application.
The best way to determine what you need to ship with your application is to load your EXE into a program like Dependency Walker.
Just as an example, I've compiled a test C++ program which simply prints a std::string. As you can see, it depends directly on two modules other than those that come with Windows; libgcc_s_dw2-1.dll in addition to libstdc++-6.dll.
You should remember to expand the tree under each DLL to make sure that it itself doesn't have any other dependencies (if A depends on B, B might depend on C even if A doesn't directly depend on C).
If you're worried and want the strongest assurances, you could install Windows into a virtual machine (VirtualBox is free) and test your application inside it. If you use Microsoft APIs, you may wish to check the MSDN documentation to see with what version of Windows they were introduced and ensure that it aligns with your target minimum Windows version.
Update: As xtofl points out this won't cover libraries loaded dynamically using LoadLibrary. If you want to cover this base, use Process Monitor to examine what DLL files are touched when you run the application. (Add an 'Image Path' criterion with the path to your EXE in order not to get flooded.) This has the added advantage that it covers all files, registry entries, etc. that your application depends on, not just DLLs.
My basic issue is this: my program (MyProgram.exe) has a dependency on a DLL from another program (OtherProgram), and I'm trying to avoid repackaging a new DLL every time OtherProgram updates. I'd like to have MyProgram.exe link in OtherProgram's DLL when it launches, but I'm not completely sure that Windows allows for this. So if there is some kind of workaround that would also be acceptable.
And just for some background, the platform is Windows 7 x64, and MyProgram.exe runs fine when I create a symlink in the MyProgram.exe project directory to the DLL in OtherProgram's install directory. When I try to run it without the symlink, I get the "program can't start because OtherProgramDLL.dll is missing from your computer" error.
Any advice or links to relevant info is greatly appreciated!
EDIT: Clarification: the DLL is not linked at compile-time, this issue crops up at runtime
There are two types of dynamic linking in the Windows world:
Load-Time linking is when a DLL is loaded automatically when your program starts up. Windows finds this DLL using a specific algorithm I'll discuss below.
Run-Time linking is when you specifically load a DLL by calling LoadLibrary in your code. Similar rules apply as to how the library is found, but you can specify a fully-qualified or relatively-qualified path to control the search.
In the case of Load-Time linking, MS recommends that your program's DLLs are stored in and loaded from the same directory where your application is loaded from. If this is at all workable, this is probably your best option.
If that doesn't work, there are several other options, outlined here. One is to leverage the search order by putting the DLL in either the working directory or the directory where the application was loaded from.
You can change the working directory of an application by:
Create a shortcut to your application.
Bring up the shortcut's properties
Edit the "Start in" property with the directory where the DLL is located.
When you launch your application using the shortcut, it will load the right DLL.
Other options for load-time linking include:
Adding a manifest to your application which specifies where your dependent assemblies are, or,
Setting the PATH.
You could use LoadLibrary, but you would need a way to guarantee the DLL's location. This Wikipedia article provides good example on how to use the DLL after it has been loaded.
You can add the directory where the dll is located to the PATH environment variable.
I have struggled with the same problem and also found a dead end with the suggested methods like LoadLibrary, SetDllDirectory, Qt's addLibraryPath and others. Regardless of what I tried, the problem still remained that the application checked the libraries (and didn't find them) before actually running the code, so any code solution was bound to fail.
I almost got desperate, but then discovered an extremely easy approach which might also be helpful in cases like yours: Use a batch file! (or a similar loader before the actual application)
A Windows batch file for such a purpose could look like this:
#echo off
PATH=%PATH%;<PATH_TO_YOUR_LIB>
<PATH_TO_YOUR_APP_EXE>
/edit: Just saw #SirDarius comment in Luchian's answer which describes that way, so just take my batch code bit as a reference and all credits go to him.
I have the same problem with one application I am working on.
I do not want to use runtime loading because there are tens of functions I would need to manually create function pointer for.
Mr Dibling's mention of manifest file opened a new door for me but I sadly found out that the oldest version of windows that supports the feature is Windows 7. It won't even work on Vista.
Long story short, a friend familiar with Windows Application development told me to look up Delay-Loaded DLL, which turns out to solve the problem perfectly with minimal effort. It delays the loading of DLL library to either the point you manually do, or the first time its function is called. So you just need to add your DLL path to the search path before that happens, where SetDllDirectory helps.
Here is the steps to make it work:
1) Specify the DLL to be delay-loaded to linker, either through your makefile, cmake or VS property page (Linker->Input of VS2015)
2) Call SetDllDirectory at the beginning of your program, before any call to the DLL is made.
Delay-loaded DLL is supported all the way back to VC6.
SetDllDirectory is supported after XP SP1.
Use Symbolic Links to the 3rd Party Executables
I found the approach advocated by Aaron Margosis useful. See:
Using NTFS Junctions to Fix Application Compatibility Issues on 64-bit Editions of Windows
Essentially, create symbolic links to each of the dependent 3rd Party executables. Place these symbolic link files in and amongst your own dependent executable files. Except for filename changes to the targets, the 'soft' symbolic links will resolve the load-time dependencies even as the target of the links are changed by future updates.
I've got an SDK I'm working on and the previous developer just dropped the DLLs in System32 (Apparently a serious offense: see here)
So assuming I move them out into \Program Files\\SDK (or whatever), how do I make sure that all the apps that needs those DLLs can access them? And to clarify, all apps that access these are doing early (static) binding to the DLLs at compile time so I can't pass the full path to them or anything. They need to be able to find it just given the DLL filename only.
Along the same lines, what about including a particular version of MSVCR80.dll? They all depend on this but I need to make sure they get a specific version (the one I include).
Any ideas?
An SDK is by definition a development kit. It's not a deployment patch...
What this means is that the applications that depend on those assemblies should ship with them and install them into their local \program files.. directories.
The reason for this is let's say you decide to do a breaking change by eliminating an entry point for example. By installing your "SDK", it has the potential to stop older programs from functioning.
You could take a play from the Java handbook and update the PATH environment variable. Whenever a program makes a call to an external assembly it searches along that environment variable until it finds it.
Of course, this could still result in the problem showing up. So your best bet is to just install the SDK into Program Files and let the developers of the products that depend on your toolkit decide whether they want to update their versions or not.
UPDATE
As I'm thinking about this, one last possibility is to GAC your assemblies. In the event you do so, bear in mind that they should be strongly named and properly versioned so as not to step on each other. I don't recommend this route because it hides the actual locations of the assemblies and makes uninstalling a little more difficult then simply hitting delete on your directory.
I can't tell you about your own DLLs, but you should never redistribute Microsoft DLLs alone.
You always have to use Microsoft Redistributable Package.
For example, if your application depends on dll from Dev Studio 2005 SP1, you should redistribute your application with Microsoft Visual Studio 2005 SP1 redistributable. The same applies to 2008. MS provide MSI based installer and Merge Module to include in your own product installer.
You are asking about "DLL Hell", something I had thought every Windows developer was familiar with. The order of search for DLLs is:
the directory the exex that calls them was loaded from
the current directory
various Windows directories (as discussed in your previous question)
directories in the PATH variable
As the Windows directories should be ruled out, that leaves you with three options.
You can put your install path in the search path, which will allow the applications to find them.
Alternatively, you can deploy the DLL's into the same directory as the application that depends on them.
I believe the first is better from an SDK perspective - it'll make development easier. But I think the second is better for when the application gets deployed to end-users, unless you expect there may be many consumers on a single system so the disk and memory footprint of having copies of the DLL are prohibitive.
If you can't install the dlls into the same directory as the exe using them you could append your directory to the PATH environment variable.
You don't say which version of Windows you're using, as the details are slightly different from what I remember.
You could also put your version of MSVCR80.dll in the same folder. However, you'd have to ensure that your folder was before the system one on the path otherwise the linker would pick up the "standard" one first. However, if you adopted the "local" dlls approach then you wouldn't have this problem as Windows searches the local directory first and so will pick up your version of MSVCR80.dll.
Is your version the latest or a previous version? You might be better off getting your app to work with that version or later and then allow the users to update their machines as required. This also illustrates why you should never mess with the dlls in \Windows or \Windows\system32 as, as others have pointed out, you could break other applications by changing the version of this dll.