Long story of what I'm trying to achieve
I'm working on a program that dynamically loads DLLs as plugins. I'm compiling the program using Microsoft Visual C++ 2008. Still, let's assume that any Visual C++ version with which Qt works should be supported. The program directory layout is following:
| plugins/
| plugin1.dll
| plugin2.dll
| QtCore4.dll
| QtGui4.dll
| program.exe
program.exe discovers all plugins DLL files, performs LoadLibrary() on them and calls a certain signature function to find out if it's actually a plugin or not. This works pretty well on computers which have vcredist for MSVC90 installed. Naturally, to make the program work on all computers I have to redistribute it with the msvc*.dll files and with the appropriate manifest file. Qt DLLs also require the redist to run.
Now, I've set up cmake to automatically copy over appropriate redist DLLs and manifest depending on the selected Visual Studio version. For the sake of simplicity let's keep assuming that I'm working with MSVC90. When the redist gets copied to the program directory the layout looks like this:
| plugins/
| plugin1.dll
| plugin2.dll
| QtCore4.dll
| QtGui4.dll
| msvcm90.dll
| msvcp90.dll
| msvcr90.dll
| Microsoft.VC90.CRT.manifest (I'm also aware that this file is bugged in VS2008)
| program.exe
Regarding the bug in manifest file: http://www.cmake.org/pipermail/cmake/2008-September/023822.html
The problem
The program with this layout now works on computers which do not have the redist installed, but the plugins are not getting loaded. In order to get the plugins to load I have to do one of the following:
Copy over the manifest file to plugins/ directory. Remove all references to msvc*.dll files from the manifest file. This works but it's not nice because I'd have to support different versions of edited manifest files depending on the version of used MSVC. Also, I have no idea if this won't break with Visual Studio other than 2008.
Copy entire redist to plugins/ directory. This doesn't require any modification to the manifest file, but now program.exe stupidly tries to load the msvc*.dll files thinking they are plugins. Naturally, this fails gracefully so no big harm is done. The other drawback is that the size of the program package grows by over 1 MB. Both these issues are something that I can live with, though.
Compile plugins with /MT switch. Brief testing has shown that this actually works but I'm not sure if it won't break anything in the future if both Qt and program.exe are /MD.
The question(s)
What's the best solution? What's the correct solution? If there is more than one correct solution then which is the best practice? Am I the first person to ever try to do this?
Update 1 (18 Nov. 2012)
While the question remains unanswered I decided to go for the route that causes the least headache. Until now I've been using solution number 1 and I decided to stick with it. If CMake detects that user is using a different MSVC version than 2008 it will display a warning message saying that automatic packaging is not fully supported.
If your target OS has _WIN32_WINNT >= 0x0502 than you can use function
SetDllDirectory()
before you do loading the plugins.
Put path to main program folder.
The call overrides system load order:
The directory from which the application loaded.
The directory specified by the path in SetDllDirectory() call.
So, you may call the function after application starts up. It is safe in all cases. Good luck!
You can provide full file paths to "LoadLibrary", so you can load your plugins with their paths. I've used this exact layout to load multiple versions of the same library from subdirectories of the current dll in Visual Studio 2005.
You first need to get the current path of the current dll using:
static LPSTR strDLLPath1 = new TCHAR[_MAX_PATH+1];
::GetModuleFileName((HINSTANCE)&__ImageBase, strDLLPath1, _MAX_PATH);
Although if your program.exe is already discovering these plugin files, I would assume you already have access to their full paths.
You can create hardlinks to VC dlls with CreateHardLink() function at installation process. With method (1) that you have described, there may be some problems with different copies of VCRT dlls. Hardlinks or SetDllDirectory() seems the best solution.
Don't mix in a single process static and dynamic link to MSVCRT - it ALWAYS gives you problems!
Related
Today I build my application and packaged the installer with QtIF. It worked nice on my computer but complained about missing msvcp140_1.dll in another computer.
Then I run find . -iname "msvcp140_1.dll" and found more than five different ones on my computer, I checked the md5sum.
Then I spend the time to try all of them on the other computer and all seem to work fine. No more complains about missing DLL.
How should I choose between the DLLs? Just pick any seems too easy.
Is there someway to inspect the DLLs, to check for a version?
I believe that DLL was a dependency from another binary included in my application, was not that the QtIF failed to include it.
Call the MS Redist Installer from your Installer. This can be done quietly, so that the end user does not notice it.
Find the vcredist_x64.exe file (or vcredist_x32 for 32 Bit applications), add it to your installer,
let it extract to the "TEMP" folder
and then call vcredist_x64.exe /quiet at the end of your install.
This has several advantages:
You will definitely copy all required files to the users computer.
Should new versions of the runtime library be released by Microsoft and should they already be on your users computer, your code will use the newer versions, which may include bugfixes.
Windows Update may also update the libraries.
That said, it is possible to copy the DLLs themselves, but you should make sure
a) you choose the right ones
b) they come from a trustworthy place, i.e. your VS installation folder
c) reside in the same directory as the executable - otherwise you will run into trouble with manifests.
The reason you might want to include the DLLs directly is if you want to reduce the overall size of your installer.
We did this a couple of years with our products, but finally gave in and simply used the vcredist_x64.exe, even if that increased the installer binary another couple of MB in size. But in the long run that's the easiest way.
I think (not sure), msvcp140_1.dll is an additional DLL for the VS 2019 runtime. VS 2017 runtime does not need this, but the new one does.
The non-redistributable method is to ship the DLL's that come with your compiler, i.e. the compiler that built your executable. After all, the DLL will be loaded into the same process as your EXE. It's only logical that these should match.
You'll find those DLL's in \Program Files*\Microsoft Visual Studio *\VC\Redist
Some of our users have been complaining about a libmmd.dll not found error on our plugins loading.
Some background:
I'm talking about an issue occurring on Windows (8, for that matter).
We develop plugins for Digital Audio Workstations (for digital sound
processing purposes).
Our plugin is a DLL coded in VS2012 and compiled with Intel Compiler 2015 via Perl scripts calling icl.exe
from the cmd.
One of its dependencies is an Intel-supplied math
library provided in the 2015 flavor of its redistributables (which we
install together with our plugins) called libmmd.dll.
For a matter of convenience, let's call the plugin DLL plugin.dll (!!!).
I didn't have much time on one of the affected users' machine but from what I saw, reinstalling Intel's redistributables package didn't help, only moving libmmd.dll to the same folder as plugin.dll.
Whatever the reason for this (sudden and unexpected) behavior, we want to be able to deliver software protected against this kind of issues (DLL Hell).
What we want is for plugin.dll to look for libraries first in a specific directory (configurable would be even better), then in its usual search order.
I tried playing with manifests, registry, library names, linker options...
The only things that actually worked were switching the /MD option with /MT (but recompiling with static libs really adds to the size of plugin.dll) and copying the library to the folder plugin.dll is in.
Any ideas ?
Thanks !
The SetDllDirectory function lets you add a folder path to the DLL search order. It will be added in second place, immediately behind the program folder itself.
This only affects DLLs loaded via LoadLibrary however; if your DLLs are loaded statically it won't make any difference.
So I've finished my Qt application, and I need to implement testing using the Squish testing application (first time using). Apparently I require a working exe file, but I can't get the executable to run. I added all the .dll files to the same directory, only to get the error:
Prior to that I was getting errors saying that XXXX.dll is missing, but like I said, I've added them to the directory. I've tried using both debug and release builds of my project with the same results. I've also tried building a stand-alone executable, but that has it's own problems (one thing at a time). The program runs great in Qt Creator and VS2013...just not on its own.
Any solutions to this?
EDIT:
From Dependency Walker...
0x7B is the error code for invalid image format.
You're either trying to run a 64-bit application on a 32-bit system, or linking to a 64-bit library (ie you copied the wrong DLLs).
Or your binaries are just corrupted.
If you run the application standalone (i.e. not from Qt Creator) you also need the Qt library DLLs. which one you need, depends on the components you are using.
Dependency Walker is also a useful tool to find missing DLLs under Windows.
As for me it seems that something is missing. Qt on windows has the script windeployqt, it will provide all needed dependencies. See documentation http://doc.qt.io/qt-5/windows-deployment.html about use of this. On Windows you will be able to run cmd with loaded qt environment variables ( on Windows 7 see under windows applications menu - it will be available if qt is installed ). As Simon stated Dependency Walker is good tool.
I'm totally new in using Qt and I don't know a lot of stuff.
As a test I created a simple application using Visual Studio 2012 and Qt-VS-Add-in based on the newest Qt5.1
After I compiled the application it didn't work for me (gave errors), I searched all over the internet and found a lot of people saying that I have to copy those dlls mentioned below from the directory:
C:\Qt\Qt5.1.0\5.1.0\msvc2012\bin\
DLL's I had to copy to make my application work:
icudt51.dll
icuin51.dll
icuuc51.dll
libEGL.dll
libGLESv2.dll
Qt5Core.dll
Qt5Gui.dll
Qt5Widgets.dll
My problem is the size of these dlls, they're about "37 MB" and my application itself is only "30 KB"! So, those Qt libraries will add at least 37 MB to my application [ Which I don't see it happens with other Qt-based applications I download ]. Is there any solution can make me end up with a single small .exe file?!
And I heard some people saying that I have to also include a dll for Microsoft C++ Compiler, can you explain this for me?
Note: I've come across a lot of questions here on StackOverFlow but I couldn't find anything can help me, so please do not flag this as a duplication because if I found a clear answer I wouldn't post this question!
Any help would be appreciated.
UPDATE: Use windeployqt.exe! It works really well.
http://doc.qt.io/qt-5/windows-deployment.html#the-windows-deployment-tool
The simplest way to use windeployqt is to add the bin directory of
your Qt installation (e.g. ) to the PATH variable and then
run:
windeployqt <path-to-app-binary>
UPDATE: Upon Further testing, windeployqt did not copy over all the MingW dlls for me. (Tested with Qt 5.4 on Windows 10 with MingW 4.9.1). So you need to manually get the last 3 dlls before deploying:
libgcc_s_dw2-1.dll
libstdc++-6.dll
libwinpthread-1.dll
From
C:\Qt\5.4\mingw491_32\bin
I think you may have a few extras in your list... I would double check the docs in the links below...
Here is the definitive documentation on it:
http://doc.qt.io/qt-5/windows-deployment.html
http://doc.qt.io/qt-5/windows-deployment.html#application-dependencies
Size of Qt DLLs
The amazing Qt Libraries can do a lot, but they are kind of big. Some of the older versions of Qt might be a little smaller.
For Qt 4.8 msvc QtCore4.dll is 2.5 MB, and QtGui4.dll is 8.4 MB.
How Windows Resolves Shared Libraries/Dynamic Link Libraries (DLL)
Here is how Windows tracks down a library at runtime:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms682586(v=vs.85).aspx
Single Small EXE
If you statically link, then your EXE should grab the libraries it needs and gets built into a stand alone exe. It still may be dependent on msvc redistributables. See the next section for more info on it. But it now compiles down the .libs that you reference into your EXE and your exe no longer is pointing at other dynamically linked libraries. It does take more time to get your statically linked exe environment setup.
Your exe will certainly get bigger as it now includes the binary information for the libraries that you referenced before.
https://www.google.com/search?q=qt+static+linking
EDIT:
Statically building the exe, means that you aren't using the LGPL version.
means that you have to have your object files easy to access to end users if you are using LGPL.
I think #peppe described it well (see comment below):
Technically, you are allowed to statically link when using Qt under LGPL, even if your application is not using LGPL. The only tricky requirement is keeping the ability for a third party to relink your application against a different Qt version. But you can comply with that easily, f.i. by providing a huge object file (.o) of your application, that only needs to be linked against any Qt version.
http://blog.qt.io/blog/2009/11/30/qt-making-the-right-licensing-decision/
Look at the chart near the bottom. If you are doing the commercial version, then you can statically link, without worrying about the object files.
MSVC Redistributables
Redistributable dependencies have to do with the run-time library linker options.
http://msdn.microsoft.com/en-us/library/aa278396(v=vs.60).aspx
/MD, /ML, /MT, /LD (Use Run-Time Library)
To find these options in the development environment, click Settings on the Project menu. Then click the C/C++ tab, and click Code Generation in the Category box. See the Use Run-Time Library drop-down box.
These two links below talk about some older versions of visual studio, but the reasoning should still stand.
http://www.davidlenihan.com/2008/01/choosing_the_correct_cc_runtim.html
How do I make a fully statically linked .exe with Visual Studio Express 2005?
Hope that helps.
Just open your terminal execute your_qt_installpath/version/compiler/bin/windeployqt.exe YourApplication.exe. It will automatically copy all the required libs and stuff into the folder, where your exe is located and you can just distribute it.
For Windows you need to include qminimal.dll and qwindows.dll, you will have to put them in folder called platforms.
Even if you program is small you still call huge libraries to do the graphical interface. If the size is really important you should do a console project.
PS : You can check all the libraries you really need by opening your exe with the dependency walker.
I found another workaround without recompiling Qt again!
[ This solution may affect application execution time ]
First we need to use UPX to compress each one of Qt Libraries required by our application, they're often the dll's mentioned in the question. However, avoid compressing them too much because you'll notice that your application takes longer time to run.
[ Optional ]: If your application binary is large, you may find it useful to compress it using UPX.
After compressing all binaries, we want to get a single .exe file, so we can use
Enigma Virtual Box [ http://enigmaprotector.com/en/downloads.html ] to merge all .dll files with the main executable and we'll end up with a single tiny .exe file!
I'll just do it like this for now since I'm not able to recompile Qt with my own configurations on my current machine.
it looks to me that Qt5.2 requires fewer dll.
Qt5Core.dll
Qt5Gui.dll
Qt5Widgets.dll
in windows you also need "qwindows.dll" in folder "platforms".
give it a try.
A possibility for reducing the size of the DLLs is by compressing them with UPX as mentioned by Alaa Alrufaie. Another method is to wrap it into an installer (e.g. Inno Setup). The latter one is particularly useful if you want to distribute it to end users). I had a simple application requiring Qt5Core.dll, Qt5Gui.dll, Qt5Widgets.dll and qwindows.dll (in the folder "platforms") taking about 17 MB. After creating a setup file, it shrank to 5 MB.
My basic issue is this: my program (MyProgram.exe) has a dependency on a DLL from another program (OtherProgram), and I'm trying to avoid repackaging a new DLL every time OtherProgram updates. I'd like to have MyProgram.exe link in OtherProgram's DLL when it launches, but I'm not completely sure that Windows allows for this. So if there is some kind of workaround that would also be acceptable.
And just for some background, the platform is Windows 7 x64, and MyProgram.exe runs fine when I create a symlink in the MyProgram.exe project directory to the DLL in OtherProgram's install directory. When I try to run it without the symlink, I get the "program can't start because OtherProgramDLL.dll is missing from your computer" error.
Any advice or links to relevant info is greatly appreciated!
EDIT: Clarification: the DLL is not linked at compile-time, this issue crops up at runtime
There are two types of dynamic linking in the Windows world:
Load-Time linking is when a DLL is loaded automatically when your program starts up. Windows finds this DLL using a specific algorithm I'll discuss below.
Run-Time linking is when you specifically load a DLL by calling LoadLibrary in your code. Similar rules apply as to how the library is found, but you can specify a fully-qualified or relatively-qualified path to control the search.
In the case of Load-Time linking, MS recommends that your program's DLLs are stored in and loaded from the same directory where your application is loaded from. If this is at all workable, this is probably your best option.
If that doesn't work, there are several other options, outlined here. One is to leverage the search order by putting the DLL in either the working directory or the directory where the application was loaded from.
You can change the working directory of an application by:
Create a shortcut to your application.
Bring up the shortcut's properties
Edit the "Start in" property with the directory where the DLL is located.
When you launch your application using the shortcut, it will load the right DLL.
Other options for load-time linking include:
Adding a manifest to your application which specifies where your dependent assemblies are, or,
Setting the PATH.
You could use LoadLibrary, but you would need a way to guarantee the DLL's location. This Wikipedia article provides good example on how to use the DLL after it has been loaded.
You can add the directory where the dll is located to the PATH environment variable.
I have struggled with the same problem and also found a dead end with the suggested methods like LoadLibrary, SetDllDirectory, Qt's addLibraryPath and others. Regardless of what I tried, the problem still remained that the application checked the libraries (and didn't find them) before actually running the code, so any code solution was bound to fail.
I almost got desperate, but then discovered an extremely easy approach which might also be helpful in cases like yours: Use a batch file! (or a similar loader before the actual application)
A Windows batch file for such a purpose could look like this:
#echo off
PATH=%PATH%;<PATH_TO_YOUR_LIB>
<PATH_TO_YOUR_APP_EXE>
/edit: Just saw #SirDarius comment in Luchian's answer which describes that way, so just take my batch code bit as a reference and all credits go to him.
I have the same problem with one application I am working on.
I do not want to use runtime loading because there are tens of functions I would need to manually create function pointer for.
Mr Dibling's mention of manifest file opened a new door for me but I sadly found out that the oldest version of windows that supports the feature is Windows 7. It won't even work on Vista.
Long story short, a friend familiar with Windows Application development told me to look up Delay-Loaded DLL, which turns out to solve the problem perfectly with minimal effort. It delays the loading of DLL library to either the point you manually do, or the first time its function is called. So you just need to add your DLL path to the search path before that happens, where SetDllDirectory helps.
Here is the steps to make it work:
1) Specify the DLL to be delay-loaded to linker, either through your makefile, cmake or VS property page (Linker->Input of VS2015)
2) Call SetDllDirectory at the beginning of your program, before any call to the DLL is made.
Delay-loaded DLL is supported all the way back to VC6.
SetDllDirectory is supported after XP SP1.
Use Symbolic Links to the 3rd Party Executables
I found the approach advocated by Aaron Margosis useful. See:
Using NTFS Junctions to Fix Application Compatibility Issues on 64-bit Editions of Windows
Essentially, create symbolic links to each of the dependent 3rd Party executables. Place these symbolic link files in and amongst your own dependent executable files. Except for filename changes to the targets, the 'soft' symbolic links will resolve the load-time dependencies even as the target of the links are changed by future updates.