Cannot dlopen a shared library from a shared library, only from executables - c++

Question

I have a C++ CMake project that has multiple sub-projects that I package into shared libraries. Then, the project itself, which is an executable, links with all these shared libraries. This is a project that is being ported from Windows to Ubuntu. What I do is have the exectable, EXE, use a one subproject, Core, to open all other libraries. Problem is that this isn't working on Linux.
This is EXE:
int main(int argc, char *argv[])
{
core::plugin::PluginManager& wPluginManager = core::plugin::PluginManagerSingleton::Instance();
wPluginManager.loadPlugin("libcore.so");
wPluginManager.loadPlugin("libcontroller.so")
wPluginManager.loadPlugin("libos.so")
wPluginManager.loadPlugin("libnetwork.so")
wPluginManager.loadPlugin("liblogger.so")
}
This is core::plugin::PluginManager::loadPlugin():
bool PluginManager::loadPlugin(const boost::filesystem::path &iPlugin) {
void* plugin_file = dlopen(plugin_file_name, RTLD_LAZY);
std::cout << (plugin_file ? " success" : "failed") << std::endl;
return true;
}
What happens is that libcore gets loaded properly, but then all other libraries fail with no no error message. I cannot find out why it's not working. However, when I do the same thing, but instead of having Core load the libraries, I simply do it in main and it works.
Basically, I can load libraries from an exe, but I can't from other shared libraries. What gives and how can I fix this?

Answer 1

The most likely reason for dlopen from the main executable to succeed and for the exact same dlopen from libcore.so to fail is that the main executable has correct RUNPATH to find all the libraries, but libcore.so does not.
You can verify this with:
readelf -d main-exe | grep R.*PATH
readelf -d libcore.so | grep R.PATH
If (as I suspect) main-exe has RUNPATH, and libcore.so doesn't, the right fix is to add -rpath=.... to the link line for libcore.so.
You can also gain a lot of insight into dynamic loader operation by using LD_DEBUG envrironment variable:
LD_DEBUG=libs ./main-exe
will tell you which directories the loader is searching for which libraries, and why.
I cannot find out why it's not working
Yes, you can. You haven't spent nearly enough effort trying.
Your very first step should be to print the value of dlerror() when dlopen fails. The next step is to use LD_DEBUG. And if all that fails, you can actually debug the runtime loader itself -- it's open-source.

Answer 2

I managed to find a fix for this issue. I don't quite understand the inner workings nor the explanation of my solution, but it works. If someone who has a better understanding than my very limited experience with shared libraries could comment on my answer with the real explanation, I'm sure it could help future viewers of this question.
What I was currently doing is dlopen("libcore.so"). I simply changed it to an absolute path dlopen("/home/user/project/libcore.so") and it now works. I have not yet tried with relative paths, but it appears we should always use relative or absolute paths instead of just the filename with dlopen.

Answer 3

If absolute path was help, maybe problem is local dependencies of shared libraries. Another words, maybe libcontroller.so is depend from libos.so or other your library, but cannot find it. Linux loader means that all shared libraries are placed in /lib, /usr/lib, etc. You need to specify path for find dynamic libraries with environment variable LD_LIBRARY_PATH.
Try to run your app this way:
LD_LIBRARY_PATH=/path/to/your/executable/and/modules ./yourapp

Answer 4

bool PluginManager::loadPlugin(const boost::filesystem::path &iPlugin) {
void* plugin_file = dlopen(plugin_file_name, RTLD_LAZY);
std::cout << (plugin_file ? " success" : "failed") << std::endl;
return true;
}
The flags to use with dlopen depend upon the distro. I think Debian and derivatives use RTLD_GLOBAL | RTLD_LAZY, while Red Hat and derivatives use RTLD_GLOBAL. Or maybe it is vice-versa. And I seem to recall Android uses RTLD_LOCAL, too.
You should just try both to simplify loading on different platforms:
bool PluginManager::loadPlugin(const boost::filesystem::path &iPlugin) {
void* plugin_file = dlopen(plugin_file_name, RTLD_GLOBAL);
if (!plugin_file) {
plugin_file = dlopen(plugin_file_name, RTLD_GLOBAL | RTLD_LAZY);
}
const bool success = plugin_file != NULL;
std::cout << (success ? "success" : "failed") << std::endl;
return success ;
}
What happens is that libcore gets loaded properly, but then all other libraries fail with no no error message
This sounds a bit unusual. It sounds like the additional libraries from the sub-projects are not in the linker path.
You should ensure the additional libraries are in the linker path. Put them next to libcore.so in the filesystem since loading libcore.so seems to work as expected.
If they are already next to libcore.so, then you need to provide more information, like the failure from loadPlugin, the RUNPATH used (if present) and the output of ldd.
but then all other libraries fail with no no error message. I cannot find out why it's not working.
As #Paul stated in the comments, the way to check for a dlopen error is with dlerror. It is kind of a crappy way to do it since you can only get a text string and not an error code.
The dlopen man page is at http://man7.org/linux/man-pages/man3/dlopen.3.html, and it says:
RETURN VALUE
On success, dlopen() and dlmopen() return a non-NULL handle for the
loaded library. On error (file could not be found, was not readable,
had the wrong format, or caused errors during loading), these
functions return NULL.
On success, dlclose() returns 0; on error, it returns a nonzero value.
Errors from these functions can be diagnosed using dlerror(3).