Some of our users have been complaining about a libmmd.dll not found error on our plugins loading.
Some background:
I'm talking about an issue occurring on Windows (8, for that matter).
We develop plugins for Digital Audio Workstations (for digital sound
processing purposes).
Our plugin is a DLL coded in VS2012 and compiled with Intel Compiler 2015 via Perl scripts calling icl.exe
from the cmd.
One of its dependencies is an Intel-supplied math
library provided in the 2015 flavor of its redistributables (which we
install together with our plugins) called libmmd.dll.
For a matter of convenience, let's call the plugin DLL plugin.dll (!!!).
I didn't have much time on one of the affected users' machine but from what I saw, reinstalling Intel's redistributables package didn't help, only moving libmmd.dll to the same folder as plugin.dll.
Whatever the reason for this (sudden and unexpected) behavior, we want to be able to deliver software protected against this kind of issues (DLL Hell).
What we want is for plugin.dll to look for libraries first in a specific directory (configurable would be even better), then in its usual search order.
I tried playing with manifests, registry, library names, linker options...
The only things that actually worked were switching the /MD option with /MT (but recompiling with static libs really adds to the size of plugin.dll) and copying the library to the folder plugin.dll is in.
Any ideas ?
Thanks !
The SetDllDirectory function lets you add a folder path to the DLL search order. It will be added in second place, immediately behind the program folder itself.
This only affects DLLs loaded via LoadLibrary however; if your DLLs are loaded statically it won't make any difference.
Related
I have a lot of basic questions about DLLs and how to bundle third-party libraries with your software, but most everything I find when searching is specific to C# (such as this question) and I'm looking for more general information about how DLLs work and what their purpose is.
I have, for example, a CMake-managed C++ application that links to one or more third-party DLLs.
Their DLLs are installed to C:/Program Files/Intel/SomeLibrary/bin/somelibrary.dll. I can build my app fine by having an environment variable take care of finding their headers.
Obviously, I can't actually run my executables out of the box, because somelibrary.dll can't be found. I have to either copy it into the same folder as the executable, or add Intel/SomeLibrary/bin to my Path.
Questions
How do other programs link to their required DLLs? Do they just package copies of all the third-party DLLs they use in with their own software?
Is it a different process for some third-party libraries -- like maybe Intel libraries you'd bundle with your software, but for Microsoft's VS Redistributables you'd require the user to install them? It looks like your installer (WiX example) handles installing them if they don't exist (and then does it install a new copy for every piece of software that needs them?).
Could you potentially have many copies on your computer of the same DLL because a bunch of separate programs needed it but had no way of knowing if you had it somewhere already?
Put another way, could two programs, written by different people, ever link to the same DLL file? How do they find that DLL, or install it if it doesn't exist?
I apologize in advance for whatever fundamental misunderstandings I've made in asking these -- I'm pretty new to the idea of packaging already-built software.
The short answer is that the dll loader will look for dlls in the local application folder and then searching a bunch of system locations. There are complexities with manifests, the SafeDllSearchMode feature (affects directory search order) and developers loading dlls from custom locations by override (LoadLibraryEx) and several others (see below).
Dependencty Types: There are many types of dll dependencies: Module / dll dependencies mind map.
Dynamic-Link Library Search Order: Here is an extract from Microsoft Docs on the topic: "A system can contain multiple versions of the same dynamic-link library (DLL). Applications can control the location from which a DLL is loaded by specifying a full path or using another mechanism such as a manifest."
In other words you can 1) load a dll from a specific location or 2) you can let the OS find it for you via standard mechanisms. The "SafeDllSearchMode" registry setting affects when the users current directory is searched. It now defaults to on, but can be disabled. See the link above for the authoritative details. Beware of OS changes and security fixes - all can affect this order in the future.
Q1: Always use the runtime provided for your dependent dll if available. Don't just include a few dlls if you are not supposed to.
Q2: It is correct that runtimes are often very different. Sometimes they are just a single or a few dlls that you can just install in your local application folder. Other times you install a runtime setup which installs files to appropriate folders for shared use. Some runtimes come as "merge modules" that can merge files into your own MSI installer, and some come as separate MSI installers or setup.exe installers.
Q3: The whole idea of using dlls is essentially to be able to load the same binary component into many processes in order to be able to update a single dll to fix problems in all "consumers" of the dll and also to share them in memory (important back in the day with little memory). Dlls can also be "delay loaded" - which means they won't be loaded until they are necessary - avoiding memory bloat. That is the idea anyway.
Windows OS Version: The search order (where to search on disk) for the right dlls to load is a pretty complicated topic. How the OS loader mechanism locates DLLs can be affected by a number of things, for example 1) the OS version, 2) manifests, 3) hard coded altered load paths (LoadLibraryEx), or 4) dll redirects, and 5) previously loaded dlls in memory (here be dragons), etc...
UWP & .NET: There are also new development with regards to UWP applications. For .NET assemblies the resolution is different again (which you state), please see this article: How the Runtime Locates Assemblies.
WinSxS: The manifest is a newer concept in Windows - in the olden days there would be no such thing and the search order would be simpler, but less flexible. With the advent of manifests and side-by-side Win32 assemblies (several versions of the same dll installed in different versions) the manifest is embedded in the executable (or it is added externally) and it tells the binary where to look for the runtime dlls - in the WinSxS folder hierarchy of Windows (Normally: C:\Windows\WinSxS) where multiple versions of the same dll can reside (in different sub folders).
Most Microsoft runtimes - basically all modern ones - reside here and hence require a manifest to instruct the binary what version to look for and load. There are ways to redirect such manifests too... "Publisher Configuration Files" and 2. Sigh... This can be used to enforce a certain new runtime version when the old one is undesired for use. Unused runtimes can also be "garbage collected" via ways to shrink the WinSxS folders (See this answer - section "Scavenging / Putting WinSxS on a Diet")... Hmmm... Never hard code paths to WinSxS.
Dll Search Order: The old and outdated tool Dependency Walker has an outdated, but very good description of the DLL Search Order in its help file.
Tip: The updated documentation for this is: Microsoft Docs: Module Search Order for Desktop Applications. However, to get a rough idea of search order, you can just launch Dependency Walker and select: Options => Configure Module Search Order.... Then make sure the Expand button is deselected. What you see - outdated as it might be - can help to recollect how module search order works:
Links:
DLL search on windows (old Stackoverflow answer on the same topic)
The path-searching algorithm is not a backtracking algorithm
The case of the DLL that refuses to load
I'm totally new in using Qt and I don't know a lot of stuff.
As a test I created a simple application using Visual Studio 2012 and Qt-VS-Add-in based on the newest Qt5.1
After I compiled the application it didn't work for me (gave errors), I searched all over the internet and found a lot of people saying that I have to copy those dlls mentioned below from the directory:
C:\Qt\Qt5.1.0\5.1.0\msvc2012\bin\
DLL's I had to copy to make my application work:
icudt51.dll
icuin51.dll
icuuc51.dll
libEGL.dll
libGLESv2.dll
Qt5Core.dll
Qt5Gui.dll
Qt5Widgets.dll
My problem is the size of these dlls, they're about "37 MB" and my application itself is only "30 KB"! So, those Qt libraries will add at least 37 MB to my application [ Which I don't see it happens with other Qt-based applications I download ]. Is there any solution can make me end up with a single small .exe file?!
And I heard some people saying that I have to also include a dll for Microsoft C++ Compiler, can you explain this for me?
Note: I've come across a lot of questions here on StackOverFlow but I couldn't find anything can help me, so please do not flag this as a duplication because if I found a clear answer I wouldn't post this question!
Any help would be appreciated.
UPDATE: Use windeployqt.exe! It works really well.
http://doc.qt.io/qt-5/windows-deployment.html#the-windows-deployment-tool
The simplest way to use windeployqt is to add the bin directory of
your Qt installation (e.g. ) to the PATH variable and then
run:
windeployqt <path-to-app-binary>
UPDATE: Upon Further testing, windeployqt did not copy over all the MingW dlls for me. (Tested with Qt 5.4 on Windows 10 with MingW 4.9.1). So you need to manually get the last 3 dlls before deploying:
libgcc_s_dw2-1.dll
libstdc++-6.dll
libwinpthread-1.dll
From
C:\Qt\5.4\mingw491_32\bin
I think you may have a few extras in your list... I would double check the docs in the links below...
Here is the definitive documentation on it:
http://doc.qt.io/qt-5/windows-deployment.html
http://doc.qt.io/qt-5/windows-deployment.html#application-dependencies
Size of Qt DLLs
The amazing Qt Libraries can do a lot, but they are kind of big. Some of the older versions of Qt might be a little smaller.
For Qt 4.8 msvc QtCore4.dll is 2.5 MB, and QtGui4.dll is 8.4 MB.
How Windows Resolves Shared Libraries/Dynamic Link Libraries (DLL)
Here is how Windows tracks down a library at runtime:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586(v=vs.85).aspx
Single Small EXE
If you statically link, then your EXE should grab the libraries it needs and gets built into a stand alone exe. It still may be dependent on msvc redistributables. See the next section for more info on it. But it now compiles down the .libs that you reference into your EXE and your exe no longer is pointing at other dynamically linked libraries. It does take more time to get your statically linked exe environment setup.
Your exe will certainly get bigger as it now includes the binary information for the libraries that you referenced before.
https://www.google.com/search?q=qt+static+linking
EDIT:
Statically building the exe, means that you aren't using the LGPL version.
means that you have to have your object files easy to access to end users if you are using LGPL.
I think #peppe described it well (see comment below):
Technically, you are allowed to statically link when using Qt under LGPL, even if your application is not using LGPL. The only tricky requirement is keeping the ability for a third party to relink your application against a different Qt version. But you can comply with that easily, f.i. by providing a huge object file (.o) of your application, that only needs to be linked against any Qt version.
http://blog.qt.io/blog/2009/11/30/qt-making-the-right-licensing-decision/
Look at the chart near the bottom. If you are doing the commercial version, then you can statically link, without worrying about the object files.
MSVC Redistributables
Redistributable dependencies have to do with the run-time library linker options.
http://msdn.microsoft.com/en-us/library/aa278396(v=vs.60).aspx
/MD, /ML, /MT, /LD (Use Run-Time Library)
To find these options in the development environment, click Settings on the Project menu. Then click the C/C++ tab, and click Code Generation in the Category box. See the Use Run-Time Library drop-down box.
These two links below talk about some older versions of visual studio, but the reasoning should still stand.
http://www.davidlenihan.com/2008/01/choosing_the_correct_cc_runtim.html
How do I make a fully statically linked .exe with Visual Studio Express 2005?
Hope that helps.
Just open your terminal execute your_qt_installpath/version/compiler/bin/windeployqt.exe YourApplication.exe. It will automatically copy all the required libs and stuff into the folder, where your exe is located and you can just distribute it.
For Windows you need to include qminimal.dll and qwindows.dll, you will have to put them in folder called platforms.
Even if you program is small you still call huge libraries to do the graphical interface. If the size is really important you should do a console project.
PS : You can check all the libraries you really need by opening your exe with the dependency walker.
I found another workaround without recompiling Qt again!
[ This solution may affect application execution time ]
First we need to use UPX to compress each one of Qt Libraries required by our application, they're often the dll's mentioned in the question. However, avoid compressing them too much because you'll notice that your application takes longer time to run.
[ Optional ]: If your application binary is large, you may find it useful to compress it using UPX.
After compressing all binaries, we want to get a single .exe file, so we can use
Enigma Virtual Box [ http://enigmaprotector.com/en/downloads.html ] to merge all .dll files with the main executable and we'll end up with a single tiny .exe file!
I'll just do it like this for now since I'm not able to recompile Qt with my own configurations on my current machine.
it looks to me that Qt5.2 requires fewer dll.
Qt5Core.dll
Qt5Gui.dll
Qt5Widgets.dll
in windows you also need "qwindows.dll" in folder "platforms".
give it a try.
A possibility for reducing the size of the DLLs is by compressing them with UPX as mentioned by Alaa Alrufaie. Another method is to wrap it into an installer (e.g. Inno Setup). The latter one is particularly useful if you want to distribute it to end users). I had a simple application requiring Qt5Core.dll, Qt5Gui.dll, Qt5Widgets.dll and qwindows.dll (in the folder "platforms") taking about 17 MB. After creating a setup file, it shrank to 5 MB.
My basic issue is this: my program (MyProgram.exe) has a dependency on a DLL from another program (OtherProgram), and I'm trying to avoid repackaging a new DLL every time OtherProgram updates. I'd like to have MyProgram.exe link in OtherProgram's DLL when it launches, but I'm not completely sure that Windows allows for this. So if there is some kind of workaround that would also be acceptable.
And just for some background, the platform is Windows 7 x64, and MyProgram.exe runs fine when I create a symlink in the MyProgram.exe project directory to the DLL in OtherProgram's install directory. When I try to run it without the symlink, I get the "program can't start because OtherProgramDLL.dll is missing from your computer" error.
Any advice or links to relevant info is greatly appreciated!
EDIT: Clarification: the DLL is not linked at compile-time, this issue crops up at runtime
There are two types of dynamic linking in the Windows world:
Load-Time linking is when a DLL is loaded automatically when your program starts up. Windows finds this DLL using a specific algorithm I'll discuss below.
Run-Time linking is when you specifically load a DLL by calling LoadLibrary in your code. Similar rules apply as to how the library is found, but you can specify a fully-qualified or relatively-qualified path to control the search.
In the case of Load-Time linking, MS recommends that your program's DLLs are stored in and loaded from the same directory where your application is loaded from. If this is at all workable, this is probably your best option.
If that doesn't work, there are several other options, outlined here. One is to leverage the search order by putting the DLL in either the working directory or the directory where the application was loaded from.
You can change the working directory of an application by:
Create a shortcut to your application.
Bring up the shortcut's properties
Edit the "Start in" property with the directory where the DLL is located.
When you launch your application using the shortcut, it will load the right DLL.
Other options for load-time linking include:
Adding a manifest to your application which specifies where your dependent assemblies are, or,
Setting the PATH.
You could use LoadLibrary, but you would need a way to guarantee the DLL's location. This Wikipedia article provides good example on how to use the DLL after it has been loaded.
You can add the directory where the dll is located to the PATH environment variable.
I have struggled with the same problem and also found a dead end with the suggested methods like LoadLibrary, SetDllDirectory, Qt's addLibraryPath and others. Regardless of what I tried, the problem still remained that the application checked the libraries (and didn't find them) before actually running the code, so any code solution was bound to fail.
I almost got desperate, but then discovered an extremely easy approach which might also be helpful in cases like yours: Use a batch file! (or a similar loader before the actual application)
A Windows batch file for such a purpose could look like this:
#echo off
PATH=%PATH%;<PATH_TO_YOUR_LIB>
<PATH_TO_YOUR_APP_EXE>
/edit: Just saw #SirDarius comment in Luchian's answer which describes that way, so just take my batch code bit as a reference and all credits go to him.
I have the same problem with one application I am working on.
I do not want to use runtime loading because there are tens of functions I would need to manually create function pointer for.
Mr Dibling's mention of manifest file opened a new door for me but I sadly found out that the oldest version of windows that supports the feature is Windows 7. It won't even work on Vista.
Long story short, a friend familiar with Windows Application development told me to look up Delay-Loaded DLL, which turns out to solve the problem perfectly with minimal effort. It delays the loading of DLL library to either the point you manually do, or the first time its function is called. So you just need to add your DLL path to the search path before that happens, where SetDllDirectory helps.
Here is the steps to make it work:
1) Specify the DLL to be delay-loaded to linker, either through your makefile, cmake or VS property page (Linker->Input of VS2015)
2) Call SetDllDirectory at the beginning of your program, before any call to the DLL is made.
Delay-loaded DLL is supported all the way back to VC6.
SetDllDirectory is supported after XP SP1.
Use Symbolic Links to the 3rd Party Executables
I found the approach advocated by Aaron Margosis useful. See:
Using NTFS Junctions to Fix Application Compatibility Issues on 64-bit Editions of Windows
Essentially, create symbolic links to each of the dependent 3rd Party executables. Place these symbolic link files in and amongst your own dependent executable files. Except for filename changes to the targets, the 'soft' symbolic links will resolve the load-time dependencies even as the target of the links are changed by future updates.
I've got an SDK I'm working on and the previous developer just dropped the DLLs in System32 (Apparently a serious offense: see here)
So assuming I move them out into \Program Files\\SDK (or whatever), how do I make sure that all the apps that needs those DLLs can access them? And to clarify, all apps that access these are doing early (static) binding to the DLLs at compile time so I can't pass the full path to them or anything. They need to be able to find it just given the DLL filename only.
Along the same lines, what about including a particular version of MSVCR80.dll? They all depend on this but I need to make sure they get a specific version (the one I include).
Any ideas?
An SDK is by definition a development kit. It's not a deployment patch...
What this means is that the applications that depend on those assemblies should ship with them and install them into their local \program files.. directories.
The reason for this is let's say you decide to do a breaking change by eliminating an entry point for example. By installing your "SDK", it has the potential to stop older programs from functioning.
You could take a play from the Java handbook and update the PATH environment variable. Whenever a program makes a call to an external assembly it searches along that environment variable until it finds it.
Of course, this could still result in the problem showing up. So your best bet is to just install the SDK into Program Files and let the developers of the products that depend on your toolkit decide whether they want to update their versions or not.
UPDATE
As I'm thinking about this, one last possibility is to GAC your assemblies. In the event you do so, bear in mind that they should be strongly named and properly versioned so as not to step on each other. I don't recommend this route because it hides the actual locations of the assemblies and makes uninstalling a little more difficult then simply hitting delete on your directory.
I can't tell you about your own DLLs, but you should never redistribute Microsoft DLLs alone.
You always have to use Microsoft Redistributable Package.
For example, if your application depends on dll from Dev Studio 2005 SP1, you should redistribute your application with Microsoft Visual Studio 2005 SP1 redistributable. The same applies to 2008. MS provide MSI based installer and Merge Module to include in your own product installer.
You are asking about "DLL Hell", something I had thought every Windows developer was familiar with. The order of search for DLLs is:
the directory the exex that calls them was loaded from
the current directory
various Windows directories (as discussed in your previous question)
directories in the PATH variable
As the Windows directories should be ruled out, that leaves you with three options.
You can put your install path in the search path, which will allow the applications to find them.
Alternatively, you can deploy the DLL's into the same directory as the application that depends on them.
I believe the first is better from an SDK perspective - it'll make development easier. But I think the second is better for when the application gets deployed to end-users, unless you expect there may be many consumers on a single system so the disk and memory footprint of having copies of the DLL are prohibitive.
If you can't install the dlls into the same directory as the exe using them you could append your directory to the PATH environment variable.
You don't say which version of Windows you're using, as the details are slightly different from what I remember.
You could also put your version of MSVCR80.dll in the same folder. However, you'd have to ensure that your folder was before the system one on the path otherwise the linker would pick up the "standard" one first. However, if you adopted the "local" dlls approach then you wouldn't have this problem as Windows searches the local directory first and so will pick up your version of MSVCR80.dll.
Is your version the latest or a previous version? You might be better off getting your app to work with that version or later and then allow the users to update their machines as required. This also illustrates why you should never mess with the dlls in \Windows or \Windows\system32 as, as others have pointed out, you could break other applications by changing the version of this dll.
When I run the program from IDE (VS2008) it works perfectly. When I want to launch it from Windows Commander it doesn't work because it needs DLL that wasn't found.
Used both debug and release builds.
Make sure the Qt DLL's are in your PATH. Two simple solutions are:
Copy Qt's DLL's to your EXE's directory.
Copy Qt's DLL's to %WINDOWS%\System32 (e.g. C:\WINDOWS\System32) (possibly unsafe, especially if you install another versions of Qt system-wide. Also, requires administrative privilages).
You should make sure that the DLLs needed by the executable (probably QT Dlls) are in the PATH or in the same directory as the executable. You can find which dlls are needed by using depends.exe
another solution would be to place path to qt bin subdir in VS tools->options->projects and solutions->VC++ directories.
You could link it statically with all the libraries it needs. Saves messing around with DLLs and stuff. On the down side, you can't update DLLs on your installed PCs to get updated/improved/fixed functionality, and will need to rebuild and redeploy.
So whether or not this is viable depends on what your installation targets are - a few PCs controlled by you, or every man+dog who decides to download your program?
Statically compiling in a library bug (security hole for example) and shipping that to your clients would be very poor form. Doing the same on a secure corporate intranet may make it worth doing just so that you know that each install is running exactly the same.