Assume, I have a folder with my program and also another folder with external library.
bin
myprog.exe
etc
lib.dll
sublib.dll
In my case I want to load the lib.dll from my main program myprog.exe. The problem is that lib.dll linked with sublib.dll.
So I try to do that in this way:
QCoreApplication a(argc, argv);
QLibrary lib;
QString path = "C:/etc/lib.dll";
a.addLibraryPath(path);
if(QLibrary::isLibrary(path)) {
lib.setFileName(path);
lib.load();
if(lib.isLoaded())
qDebug() << "Ok\n";
else
qDebug() << "Error " << lib.errorString() << "\n";
} else
qDebug() << "Not a library\n";
return a.exec();
After running the app I get the error:
cannot load library lib.dll the specified module could not be found
If I put both lib.dll and sublib.dll inside bin directory it works without error. But that is not I want to do.
I've tried
a.addLibraryPath("C:/etc");
but that doesn't work.
As I understand QCoreApplication::addLibraryPath() sets path for Qt program, not as system wide setting. So, in this case, lib.dll still can't find sublib.dll although it locates in same directory.
So my question - how can I load external shared library inside Qt program in case that this library has its own dependencies?
That is Windows issue. The DLL will be looked at the current process directory and then in the system PATH. The code that is contained in some C:\etc\lib.dll is behaving in its own process and unless very specific logic implemented will behave according to the system rule.
Please refer to MSDN article Dynamic-Link Library Search Order for details. If the source code for that lib.dll is available, it makes sense to examine LoadLibrary call. If there is no specific path provided then:
The first directory searched is the directory containing the image
file used to create the calling process (for more information, see the
CreateProcess function). Doing this allows private dynamic-link
library (DLL) files associated with a process to be found without
adding the process's installed directory to the PATH environment
variable. If a relative path is specified, the entire relative path is
appended to every token in the DLL search path list. To load a module
from a relative path without searching any other path, use
GetFullPathName to get a nonrelative path and call LoadLibrary with
the nonrelative path. For more information on the DLL search order,
see Dynamic-Link Library Search Order.
Nothing prevents you from explicitly preloading the libraries that lib.dll depends on. Once pre-loaded, they're ready for use by any library that you'll subsequently open. After all, you know where they are so it's a simple matter to iterate them and attempt to load them. Due to possible dependencies between these libraries, you have to keep loading them until there's no more progress:
QString path;
QSet<QString> libraries;
QDirIterator it{path, {"*.dll"}};
while (it.hasNext())
libraries << it.next();
bool progress = true;
while (progress) {
progress = false;
for (auto i = libraries.begin(); i != libraries.end();) {
QLibrary lib{*i};
if (lib.load()) {
progress = true;
i = libraries.erase(i);
} else
++i;
}
}
It's either that or to use a PE library of your choice to build the dependency tree yourself and only open the necessary libraries, in dependency order.
Side note: you don't own C:\Windows and you should never write anything there (nor in any subfolder) in any sort of a modern installer.
Related
I have a library (let's call it mydll.so) that is dependent to an external application, i.e. Matlab. To dynamically load mydll.so, I have written a code like this (Ubuntu, g++ 4.8.5, qt 5.12.6):
// update LD_LIBRARY_PATH with ALL required paths (Matlab, boost, etc.)
bool res = qputenv("LD_LIBRARY_PATH", required_path.toStdString().c_str());
assert(res);
// loading the dll
QLibrary my_dll;
my_dll.setFileName(dll_path);
if (!my_dll.load())
{
std::cout << my_dll.errorString().toStdString() << std::endl;
}
The above code fails with this message:
Cannot load library /home/user/code/test/lnx_x64/debug/mydll.so: (libMatlabDataArray.so: cannot open shared object file: No such file or directory)
It is strange because the load() function is complaining about a library from Matlab, i.e. libMatlabDataArray.so that its path is already included in LD_LIBRARY_PATH. But, if I run ldd on the same environment, I have:
user#everest:~/code$ ldd /home/user/code/test/lnx_x64/debug/mydll.so
linux-vdso.so.1 (0x00007ffcb4da2000)
libMatlabDataArray.so => /usr/local/MATLAB/extern/bin/glnxa64/libMatlabDataArray.so (0x00007f6af95f2000)
libMatlabEngine.so => /usr/local/MATLAB/extern/bin/glnxa64/libMatlabEngine.so (0x00007f6af93e7000)
That means the libMatlabDataArray.so can be found by ldd command and the contents of LD_LIBRARY_PATH is correct. Then what can be the reason behind this issue in my case?
Update 1: If I set the LD_LIBRARY_PATH before the application is started, everything works fine. What is the difference between setting the the LD_LIBRAARY_PATH before launching the application and inside it?
The problem is that you change the environment variable LD_LIBRARY_PATH from within the process. However the process still uses the "old" environment variables block with old values. As a result it cannot properly find the dependent libraries and eventually fails.
Therefore your approach will not work. I would propose the following solutions:
Set the LD_LIBRARY_PATH variable before starting the process, so that the process can take the updated block into account,
Try to use QCoreApplication::setLibraryPaths() function to set the library search paths.
I have a C++ CMake project that has multiple sub-projects that I package into shared libraries. Then, the project itself, which is an executable, links with all these shared libraries. This is a project that is being ported from Windows to Ubuntu. What I do is have the exectable, EXE, use a one subproject, Core, to open all other libraries. Problem is that this isn't working on Linux.
This is EXE:
int main(int argc, char *argv[])
{
core::plugin::PluginManager& wPluginManager = core::plugin::PluginManagerSingleton::Instance();
wPluginManager.loadPlugin("libcore.so");
wPluginManager.loadPlugin("libcontroller.so")
wPluginManager.loadPlugin("libos.so")
wPluginManager.loadPlugin("libnetwork.so")
wPluginManager.loadPlugin("liblogger.so")
}
This is core::plugin::PluginManager::loadPlugin():
bool PluginManager::loadPlugin(const boost::filesystem::path &iPlugin) {
void* plugin_file = dlopen(plugin_file_name, RTLD_LAZY);
std::cout << (plugin_file ? " success" : "failed") << std::endl;
return true;
}
What happens is that libcore gets loaded properly, but then all other libraries fail with no no error message. I cannot find out why it's not working. However, when I do the same thing, but instead of having Core load the libraries, I simply do it in main and it works.
Basically, I can load libraries from an exe, but I can't from other shared libraries. What gives and how can I fix this?
The most likely reason for dlopen from the main executable to succeed and for the exact same dlopen from libcore.so to fail is that the main executable has correct RUNPATH to find all the libraries, but libcore.so does not.
You can verify this with:
readelf -d main-exe | grep R.*PATH
readelf -d libcore.so | grep R.PATH
If (as I suspect) main-exe has RUNPATH, and libcore.so doesn't, the right fix is to add -rpath=.... to the link line for libcore.so.
You can also gain a lot of insight into dynamic loader operation by using LD_DEBUG envrironment variable:
LD_DEBUG=libs ./main-exe
will tell you which directories the loader is searching for which libraries, and why.
I cannot find out why it's not working
Yes, you can. You haven't spent nearly enough effort trying.
Your very first step should be to print the value of dlerror() when dlopen fails. The next step is to use LD_DEBUG. And if all that fails, you can actually debug the runtime loader itself -- it's open-source.
I managed to find a fix for this issue. I don't quite understand the inner workings nor the explanation of my solution, but it works. If someone who has a better understanding than my very limited experience with shared libraries could comment on my answer with the real explanation, I'm sure it could help future viewers of this question.
What I was currently doing is dlopen("libcore.so"). I simply changed it to an absolute path dlopen("/home/user/project/libcore.so") and it now works. I have not yet tried with relative paths, but it appears we should always use relative or absolute paths instead of just the filename with dlopen.
If absolute path was help, maybe problem is local dependencies of shared libraries. Another words, maybe libcontroller.so is depend from libos.so or other your library, but cannot find it. Linux loader means that all shared libraries are placed in /lib, /usr/lib, etc. You need to specify path for find dynamic libraries with environment variable LD_LIBRARY_PATH.
Try to run your app this way:
LD_LIBRARY_PATH=/path/to/your/executable/and/modules ./yourapp
bool PluginManager::loadPlugin(const boost::filesystem::path &iPlugin) {
void* plugin_file = dlopen(plugin_file_name, RTLD_LAZY);
std::cout << (plugin_file ? " success" : "failed") << std::endl;
return true;
}
The flags to use with dlopen depend upon the distro. I think Debian and derivatives use RTLD_GLOBAL | RTLD_LAZY, while Red Hat and derivatives use RTLD_GLOBAL. Or maybe it is vice-versa. And I seem to recall Android uses RTLD_LOCAL, too.
You should just try both to simplify loading on different platforms:
bool PluginManager::loadPlugin(const boost::filesystem::path &iPlugin) {
void* plugin_file = dlopen(plugin_file_name, RTLD_GLOBAL);
if (!plugin_file) {
plugin_file = dlopen(plugin_file_name, RTLD_GLOBAL | RTLD_LAZY);
}
const bool success = plugin_file != NULL;
std::cout << (success ? "success" : "failed") << std::endl;
return success ;
}
What happens is that libcore gets loaded properly, but then all other libraries fail with no no error message
This sounds a bit unusual. It sounds like the additional libraries from the sub-projects are not in the linker path.
You should ensure the additional libraries are in the linker path. Put them next to libcore.so in the filesystem since loading libcore.so seems to work as expected.
If they are already next to libcore.so, then you need to provide more information, like the failure from loadPlugin, the RUNPATH used (if present) and the output of ldd.
but then all other libraries fail with no no error message. I cannot find out why it's not working.
As #Paul stated in the comments, the way to check for a dlopen error is with dlerror. It is kind of a crappy way to do it since you can only get a text string and not an error code.
The dlopen man page is at http://man7.org/linux/man-pages/man3/dlopen.3.html, and it says:
RETURN VALUE
On success, dlopen() and dlmopen() return a non-NULL handle for the
loaded library. On error (file could not be found, was not readable,
had the wrong format, or caused errors during loading), these
functions return NULL.
On success, dlclose() returns 0; on error, it returns a nonzero value.
Errors from these functions can be diagnosed using dlerror(3).
In many cases to load some newer API one would use a construct as such:
(FARPROC&)pfnZwQueryVirtualMemory = ::GetProcAddress(::GetModuleHandle(L"ntdll.dll"), "ZwQueryVirtualMemory");
But then, considering a chance of Dll hijacking, is it better to specify a DLL's absolute path, as such. Or is it just an overkill?
WCHAR buff[MAX_PATH];
buff[0] = 0;
if(::GetSystemDirectory(buff, MAX_PATH) &&
::PathAddBackslash(buff) &&
SUCCEEDED(::StringCchCat(buff, MAX_PATH, L"ntdll.dll")))
{
(FARPROC&)pfnZwQueryVirtualMemory = ::GetProcAddress(::GetModuleHandle(buff), "ZwQueryVirtualMemory");
}
else
{
//Something went wrong
pfnZwQueryVirtualMemory = NULL;
}
The problem with the latter method is that it doesn't always work (for instance with Comctl32.dll.)
You don't have to do anything special for ntdll.dll and kernel32.dll because they are going to be loaded before you get the chance to do anything, they are also on the known-dlls list.
The dll hijacking issues often include auxiliary libraries. Take version.dll for example, it is no longer on the known-dlls list so explicitly linking to it is problematic, it needs to be loaded dynamically.
The best solution is a combination of 3 things:
Call SetDefaultDllDirectories(LOAD_LIBRARY_SEARCH_SYSTEM32) if it is available (Win8+ and updated Win7).
Call LoadLibrary with the full path (GetSystemDirectory) before calling GetModuleHandle.
Don't explicitly link to anything other than kernel32, user32, gdi32, shlwapi, shell32, ole32, comdlg32 and comctl32.
If SetDefaultDllDirectories is not available then it is really hard to protect yourself if you don't control the application directory because various Windows functions will delay-load dlls like shcore.dll without full paths (especially the shell APIs). SetDllDirectory("") helps against the current/working directory but there is no good application directory workaround for unpatched pre-Win8 systems, you just have to test with Process Monitor and manually load the problematic libraries early in WinMain.
The application directory is a problem because some users just put everything in the downloads folder and run it from there. This means you might end up with a malicious dll in your application directory.
Update Sorry, guys, I was incorrectly determined the problem. All is working, error was in "other program logic". Please delete or close the question.
Qt5 is designed to load libeay32.dll and ssleay32.dll on program start before any instruction in main() (because it is static).
(Details: it is located in qtbase\src\network\ssl\qsslsocket_openssl_symbols.cpp:
static QPair<QSystemLibrary*, QSystemLibrary*> loadOpenSslWin32()
)
Issue:
my program starts with not its exe dir as working directory
libeay32.dll and ssleay32.dll resides in its exe dir
user cannot install OpenSSL in system dir
user cannot change PATH variable
I cannot recompile Qt, i.e. ship program with static Qt compiled with openssl-linked
Qt loads searches dlls in this order (from qtbase\src\corelib\plugin\qsystemlibrary.cpp):
Application path.
System libraries path.
Trying all paths inside the PATH environment variable.
No. 1 is Application path, but in fact it does not search there.
My program:
int main()
{
// at this point Qt5 already checked and tried to load the DLLs
// so this:
ChangeCurrentWorkingDirectoryToExeDir(); // some function to change cwd to current exe dir
// does not work :-(
// ... other program logic ..
}
How to force Qt5 to reload OpenSSL DLLs after changing working directory?
May be someone already faced this problem...
Update Sorry, guys, I was incorrectly determined the problem. All is working, error was in "other program logic". Please delete or close the question.
QSystemLibrary::load is called with onlySystemDirectory = false for SSL, so QFileInfo(qAppFileName()).path() is the first place where the DLLs are searched. Search order:
application dir
system path (e.g. C:\Windows\System32)
all paths in PATH
I don't find documentation for that, but in our software, Qt finds SSL libeay32.dll and ssleay32.dll when they are in the same directory as the application .exe, given that
the .dll files are not in PATH
the .dll files are not in the working directory
no qt.conf exists
If you have a qt.conf, the default library value might apply, which is .\lib.
Library search orders are discussed on MSDN at Dynamic-Link Library Search Order.
There's a few ways to handle loading libraries, but it sounds like most of them won't apply.
Its not clear to me why you can call ChangeCurrentWorkingDirectoryToExeDir, but you can't call SetDllDirectory. I'm probably missing something obvious.
It seems like your last option is to create an Qt.exe.local file. This is called Dynamic-Link Library Redirection, and will cause the linker to load the DLL specified in the local file.
I'm trying to load a dll (through LoadLibraryA) from another dll.
This is the problem:
c:\**EXE_DIR**\myExe.exe // this exe load the MY_DLL_N1.dll
c:\**DLLS_DIR**\MY_DLL_N1.dll // this dll load the MY_DLL_N2.dll
c:\**DLLS_DIR**\MY_DLL_N2.dll
the exe load the MY_DLL_N1.dll ... fine.
MY_DLL_N1.dll try to load (below the code) the MY_DLL_N2.dll (same dir) ... and here is my problem!
int LoadMyDLL()
{
// ...
// same path of the MY_DLL_N1.dll ... right?
handle = LoadLibraryA ("MY_DLL_N2.dll");
// ...
}
that's all .... any help is welcome!
Everything you need to know is located here: Dynamic-Link Library Search Order.
Consider using SetDllDirectory to add your DLL path to the LoadLibrary search path.
handle = LoadLibraryA ("MY_DLL_N2.dll");
Because you do not supply a path, the DLL search order is used. This will look in the executable's directory, but will not search in the directories of any DLLS that are loaded. Hence the failure to find the DLL.
You have a number of options:
Put all the DLLS in the same directory as the executable.
Pass the full path to the DLL when calling LoadLibrary.
Call SetDllDirectory to add the DLL directory to the search path. Make this call from the executable before loading the first DLL. Once you do this you won't need to use the full path when loading the first DLL.
Unless you have a need to share the DLLS between different applications, option 1 is always preferred. This makes it easy for you to be sure that the DLLs you load are the right ones. That's because the executable directory is always searched first.