Why does this dynamic library loading code work with gcc? - c++

Background:
I've found myself with the unenviable task of porting a C++ GNU/Linux application over to Windows. One of the things this application does is search for shared libraries on specific paths and then loads classes out of them dynamically using the posix dlopen() and dlsym() calls. We have a very good reason for doing loading this way that I will not go into here.
The Problem:
To dynamically discover symbols generated by a C++ compiler with dlsym() or GetProcAddress() they must be unmangled by using an extern "C" linkage block. For example:
#include <list>
#include <string>
using std::list;
using std::string;
extern "C" {
list<string> get_list()
{
list<string> myList;
myList.push_back("list object");
return myList;
}
}
This code is perfectly valid C++ and compiles and runs on numerous compilers on both Linux and Windows. It, however, does not compile with MSVC because "the return type is not valid C". The workaround we've come up with is to change the function to return a pointer to the list instead of the list object:
#include <list>
#include <string>
using std::list;
using std::string;
extern "C" {
list<string>* get_list()
{
list<string>* myList = new list<string>();
myList->push_back("ptr to list");
return myList;
}
}
I've been trying to find an optimal solution for the GNU/Linux loader that will either work with both the new functions and the old legacy function prototype or at least detect when the deprecated function is encountered and issue a warning. It would be unseemly for our users if the code just segfaulted when they tried to use an old library. My original idea was to set a SIGSEGV signal handler during the call to get_list (I know this is icky - I'm open to better ideas). So just to confirm that loading an old library would segfault where I thought it would I ran a library using the old function prototype (returning a list object) through the new loading code (that expects a pointer to a list) and to my surprise it just worked. The question I have is why?
The below loading code works with both function prototypes listed above. I've confirmed that it works on Fedora 12, RedHat 5.5, and RedHawk 5.1 using gcc versions 4.1.2 and 4.4.4. Compile the libraries using g++ with -shared and -fPIC and the executable needs to be linked against dl (-ldl).
#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
#include <list>
#include <string>
using std::list;
using std::string;
int main(int argc, char **argv)
{
void *handle;
list<string>* (*getList)(void);
char *error;
handle = dlopen("library path", RTLD_LAZY);
if (!handle)
{
fprintf(stderr, "%s\n", dlerror());
exit(EXIT_FAILURE);
}
dlerror();
*(void **) (&getList) = dlsym(handle, "get_list");
if ((error = dlerror()) != NULL)
{
printf("%s\n", error);
exit(EXIT_FAILURE);
}
list<string>* libList = (*getList)();
for(list<string>::iterator iter = libList->begin();
iter != libList->end(); iter++)
{
printf("\t%s\n", iter->c_str());
}
dlclose(handle);
exit(EXIT_SUCCESS);
}

As aschepler says, its because you got lucky.
As it turns out, the ABI used for gcc (and most other compilers) for both x86 and x64 returns 'large' structs (too big to fit in a register) by passing an extra 'hidden' pointer arg to the function, which uses that pointer as space to store the return value, and then returns the pointer itself. So it turns out that a function of the form
struct foo func(...)
is roughly equivlant to
struct foo *func(..., struct foo *)
where the caller is expected to allocate space for a 'foo' (probably on the stack) and pass in a pointer to it.
So it just happens that if you have a function that is expecting to be called this way (expecting to return a struct) and instead call it via a function pointer that returns a pointer, it MAY appear to work -- if the garbage bits it gets for the extra arg (random register contents left there by the caller) happen to point to somewhere writable, the called function will happily write its return value there and then return that pointer, so the called code will get back something that looks a like a valid pointer to the struct it is expecting. So the code may superficially appear to work, but its actually probably clobbering a random bit of memory that may be important later.

Related

Lua: Cant open shared library when lua is wrapped in C++

EDIT: Nearly got the answer, I just dont completely understand it, see last paragraph.
I try to build a shared lua library and use it within a larger project. When calling the script which loads the shared library from shell everything works. However, when I wrap the script within another shell, I get a runtime error when loading the library. Dependent on the script it is just any call to a lua function from c (i.e. lua_pushnumber). Here is a minimal example.
totestlib.cpp:
extern "C" {
#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"
}
int init(lua_State *L) {
lua_toboolean(L, -1);
return 0;
}
static const struct luaL_Reg testlib[] = {
{"init", init},
{NULL, NULL}
};
extern "C"
int luaopen_libtotestlib(lua_State *L) {
luaL_newlib(L, testlib);
return 1;
}
Compiled with: g++ -shared -fPIC -I./lua-5.4.4/src -L./lua-5.4.4/src totestlib.cpp -o libtotestlib.so
testlib.lua (testing shared library):
testlib.lua
print("start")
testlib = require("libtotestlib")
print("done")
testlib.init(true)
print("called")
Calling the lua script using ./lua-5.4.4/src/lua testlib.lua works. Everything is printed. Wrapping script in the following c++ code does not work:
call_testlib.cpp
extern "C" {
#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>
}
#include <unistd.h>
static lua_State *L;
int main(int argc, char *argv[]) {
L = luaL_newstate();
luaL_openlibs(L);
int tmp = luaL_loadfile(L, "testlib.lua");
if(tmp != 0) {
return 1;
}
tmp = lua_pcall(L, 0, 0, 0);
if(tmp != 0) {
printf("error pcall\n");
return 1;
}
}
Compiled with g++ call_testlib.cpp -o ./call_testlib -I./lua-5.4.4/src -L./lua-5.4.4/src -llua it prints "error pcall". If I print the error message on the lua stack, I get:
string error loading module 'libtotestlib' from file './libtotestlib.so':
./libtotestlib.so: undefined symbol: luaL_checkversion_
In this case the undefined symbol is luaL_checkversion_ (which I dont call myself), but with other scripts it is usually the first lua_... function that I call.
I have tried several things to fix this. For example, linking -llua when compiling the shared library, but this does not work (and should not be the problem as calling the script itself works). I also tried to load preload the library from c++ (as done in this question) instead of from lua, but I guess it does not really make a difference and I am getting the same error. I also uninstalled all lua versions from my path to make sure I always use the same version.
What is the difference between calling the script directly from shell and calling it inside a c function? Am I doing something wrong?
EDIT: Nearly got the answer. When using MYCFLAGS= -fPIC when compiling lua I can link lua to the shared library. At least this one works, but does not seem like a good solution to me and does not really answer my question: Why can lua itself (from shell) somehow add these symbols to the library while the wrapped c version can not? Additionally, my program has lua once linked in the shared library and once in the compiled C++ project (not optimal imo).

Obtaining filename at runtime for a shared library c++

I have some code that is compiled as a shared library and used with a universal driver, which can be used with other shared libraries that are specific to a particular application.
My question pertains to obtaining some sort of indicator of the name of the binary containing a code that lives in that shared library.
For example, let's say I have 3 files, the first is driver.cpp, the universal driver:
#include "interface.h"
#include <stdio.h>
int main(int argc, char *argv[]) {
//perform a function from the shared library
std::cout << foobar() << std::endl;
}
The second is sharedlibrary.cpp, the specific implementation for one case of many:
#include "interface.h"
char* foobar() {
return x;
}
Where x is some indicator that this function is defined in sharedlibrary.cpp, or that this function is linked from sharedlibrary.so, or the current stack frame is using the specific binary rather than just being included in driver.cpp.
The last file is interface.h, which provides the interface to the library via extern "C"
extern "C" {
char foobar();
}
I would like to reiterate, for clarity, that I am looking for some indication that this function is being linked from sharedlibrary.so. Many solutions looking for runtime filenames give the executable name using either argv[0] or readlink(), but I have no control over the actual naming of driver.cpp or its executable name. Rather, I can distribute sharedlibrary.so, and would like to be able to use its name from within itself, if possible.
If it helps, I know that a microsoft-specific solution could be to use AfxGetApp()->m_pszAppName to obtain the DLL name. However, I am looking for a linux solution that does not necessarily need to be portable.
EDIT: I do not know or control the names of driver.cpp, sharedlibrary.cpp, or sharedlibrary.h at compile time. I wish to discover the name of sharedlibrary.cpp at run time.
The updated sharedlibrary.cpp with x replaced with the solution looks like this
#include "interface.h"
#include <dlfcn.h>
void func() {
//Some function that is defined in sharedlibrary.cpp
}
char* foobar() {
Dl_info DlInfo;
if(!dladdr((void*)func, &DlInfo)) {
return "default_name";
}
return DlInfo.dli_fname;
}
Obtaining filename at runtime for a shared library c++
My question pertains to obtaining some sort of indicator of the name of the binary containing a code that lives in that shared library.
You can use int dladdr(void *addr, Dl_info *info. It fills a following structure for you:
typedef struct {
const char *dli_fname; /* Pathname of shared object that contains address */
void *dli_fbase;
const char *dli_sname;
void *dli_saddr;
} Dl_info;
You can pass the address of a function exported by the shared library as the argument addr. Or within such function, you could use the instruction pointer value of the current stack frame - if you know how to obtain it.
I believe you must link with the libdl library.
You can use the buildsystem to generate the dynamic library name for linking and preprocess that inside of a header with a function that return a defined macro, in cmake you can see how to do that here.
Then you use the configured-file to return the defined value in a function that's exported from within the dll.
#include "library_name_macro.h"
auto __dllexport libraryName() -> std::string { return LIBRARY_NAME_MACRO; }
I hope, I have understood your question correctly. I hope my answer helps. You know the shared library name, you link that shared library to your program, Later in run time you want to figure out whether a particular function is present in library or not and this logic should be part of shared library itself.
Let's take an example that you have shared library called librandom.so, You have linked this library to your application. You can implement the following function in a librandom.so library, You can pass function name which you want to check whether it is present or not. I have not tested this code, there may be errors. The idea I am proposing is library loads itself again to check whether the function is present when this function is called. May not be ideal method but should serve your purpose.
int isFuncPresent(char funcName[])
{
int isFuncFound = 1;
void *lib_handle;
int x;
char *error;
lib_handle = dlopen("librandom.so", RTLD_LAZY);
if (!lib_handle)
{
fprintf(stderr, "%s\n", dlerror());
isFuncFound = 0;
}
fn = dlsym(lib_handle, funcName);
if ((error = dlerror()) != NULL)
{
fprintf(stderr, "%s\n", error);
isFuncFound = 0;
}
dlclose(lib_handle);
return isFuncFound;
}

std::any across shared library bounding in mingw

I stumbled about an issue while using libstdc++'s std::any implementation with mingw across a shared library boundary. It produces a std::bad_any_cast where it obviously should not (i believe).
I use mingw-w64, gcc-7 and compile the code with -std=c++1z.
The simplified code:
main.cpp:
#include <any>
#include <string>
// prototype from lib.cpp
void do_stuff_with_any(const std::any& obj);
int main()
{
do_stuff_with_any(std::string{"Hello World"});
}
lib.cpp:
Will be compiled into a shared library and linked with the executable from main.cpp.
#include <any>
#include <iostream>
void do_stuff_with_any(const std::any& obj)
{
std::cout << std::any_cast<const std::string&>(obj) << "\n";
}
This triggers a std::bad_any_cast although the any passed to do_stuff_with_any does contain a string. I digged into gcc's any implementation and it seems to use comparison of the address of a static inline member function (a manager chosen from a template struct depending on the type of the stored object) to check if the any holds an object of the requested type.
And the address of this function seems to change across the shared library boundary.
Isn't std::any guaranteed to work across shared library boundaries? Does this code trigger UB somewhere? Or is this a bug in the gcc implementation? I am pretty sure it works on linux so is this only a bug in mingw? Is it known or should i report it somewhere if so? Any ideas for (temporary) workarounds?
While it is true that this is an issue on how Windows DLLs work, and that as of GCC 8.2.0, the issue still remains, this can be easily worked around by changing the __any_caster function inside the any header to this:
template<typename _Tp>
void* __any_caster(const any* __any)
{
if constexpr (is_copy_constructible_v<decay_t<_Tp>>)
{
#if __cpp_rtti
if (__any->type().hash_code() == typeid(_Tp).hash_code())
#else
if (__any->_M_manager == &any::_Manager<decay_t<_Tp>>::_S_manage)
#endif
{
any::_Arg __arg;
__any->_M_manager(any::_Op_access, __any, &__arg);
return __arg._M_obj;
}
}
return nullptr;
}
Or something similar, the only relevant part is the comparison line wrapped in the #if.
To elaborate, there is 2 copies of the manager function one on the exe and one on the dll, the passed object contains the address of the exe because that's where it was created, but once it reaches the dll side, the pointer gets compared to the one in the dll address space, which will never match, so, instead type info hash_codes should be compared instead.

How to manipulate/return data with dynamically loaded functions (dlopen)?

I've spent days reading and re-reading every tutorials I've found on the subject, and spent hours (and even days) browsing related questions here at SO, but I still can't get the following to work. Accept my apologies if this is a duplicate: chances are that I've seen and re-read many times the duplicate questions but couldn't understand the relevance of the answers to my problem. With that out of the way...
I'm trying to implement a plugin architecture for my Application. Plugins are compiled and installed as libraries. At run time, the Application then uses dlopen() / dlsym() to load and link to the plugin's functions.
The idea is that plugins (libraries) will implement a set of functions to return data to the main Application, or manipulate data passed from the Application.
In order to test this idea, I tried to implement a function (inside the plugin) that would return the (human readable) name of the plugin itself (as a std::string). I thought that would be something simple to start with.... :-/
Here is what I got so far:
// Plugin.cpp
extern "C" void plugin_name(std::string *name) {
name = new std::string("Example plugin name");
}
// Application.cpp
void* handle = dlopen("libplugin.so", RTLD_LAZY);
typedef void (*plugin_t)(std::string*);
dlerror(); // Reset errors.
plugin_t call_plugin_name = (plugin_t) dlsym(handle, "plugin_name");
// ... Some error handling code.
std::string my_plugin_name;
call_plugin_name(&my_plugin_name);
dlclose(handle);
// More code that displays my_plugin_name.
I've tried many different combinations, including one that seemed more straigtforward (but didn't work any better) where the plugin name is returned:
// Plugin.cpp
extern "C" std::string plugin_name(void) {
return std::string("Example plugin name");
}
I know I'm close: the code compiles and the Application stopped crashing ;)
However, I've got an empty space where I'd expect seeing the actual plugin name.
All the tutorials I've read so far go very quickly over the mechanism by which data is passed both ways: plugin <=> Application. What I'm trying to do with a "simple" std::string, I wish to do later with much more complex objects (i.e. a plugin function would take an object by reference and change some of its properties). The tutorials more or less all stop at the point of creating a pointer with dlsym() and do not give much examples on how to use this pointer.
So, how to do all that?
Another pertinent question: do I use a common header that I'd use both with the Application and with the plugin and where I'd define the function calls signature? How would I do this and how would that help?
The signature of a function is generated from its name and argument types (return value type doesn't matter). When you declare function with extern "C", C symbol naming scheme is used which apparently can't handle C++ types like std::string. That's why passing std::string as an arguments doesn't work.
I can't explain why returning std::string doesn't work. Maybe different calling conventions are used.
Anyway the correct way of importing C++ code from a shared library is to return pointers to C++ types from entry points. And this entry points have to have arguments with types available in C. (Entry point is a documented function exported from a shared library)
Here is a good article on basic aspects of loading C++ classes from shared libraries. This article will answer your question throughly.
Please note that there are pitfalls when using exceptions thrown from a shared library to the main applications. And with dynamic_cast of objects created inside a library. I've mentioned this topics so that you could be somewhat prepared when you face this problems.
[edit]
To make my answer more clear I'll add a couple of examples.
To get the plugin name you can use:
extern "C" const char * plugin_name() {
return "Example plugin name";
}
// main.cc:
void* handle = dlopen("libplugin.so", RTLD_LAZY);
// ...
typedef const char * (*plugin_t)();
plugin_t call_plugin_name = (plugin_t) dlsym(handle, "plugin_name");
// ...
std::string my_plugin_name(call_plugin_name());
// use it
To really use the plugin functionality you should declare a base class in a header:
// plugin.h
class Plugin {
public:
virtual void doStuff() = 0;
virtual ~Plugin() = 0;
};
// plugin.cc
Plugin::~Plugin() {
}
// myplugin.cc
class MyPlugin : public Plugin {
virtual void doStuff() {
std::cout << "Hello from plugin" << std::endl;
}
};
extern "C" Plugin *createMyPluginInstance() {
return new MyPlugin;
}
Try:
extern "C" void plugin_name(std::string **name) {
*name = new std::string("Example plugin name");
}
...
std::string *my_plugin_name;
call_plugin_name(&my_plugin_name);
As you are assigning a copy of the pointer you passed as the argument, not the one you intended to assign.
EDIT Here you go:
File main.cpp
#include <iostream>
#include <dlfcn.h>
#include <string>
// Application.cpp
int main() {
void* handle = dlopen("libplugin.so", RTLD_LAZY);
typedef void (*plugin_t)(std::string**);
dlerror(); // Reset errors.
plugin_t call_plugin_name = (plugin_t) dlsym(handle, "plugin_name");
// ... Some error handling code.
std::string *my_plugin_name;
call_plugin_name(&my_plugin_name);
dlclose(handle);
// More code that displays my_plugin_name.
std::cout << "Plugin name is " << *my_plugin_name << std::endl;
delete my_plugin_name;
return 0;
}
File plugin.cpp
#include <string>
extern "C" void plugin_name(std::string **name) {
*name = new std::string("example plugin name");
}
Just a word of warning. Although this compiles and runs, passing C++ types across the dll boundry is risky and the above code is just your code fixed enough to compile and run, it is not safe and has very explicit memory handling. You may want to attack the problem in a different way.
Please have a read of this question and its answers. There are many opportunities for incompatibilities across the shared lib boundaries in C++.

How do I load a shared object in C++?

I have a shared object (a so - the Linux equivalent of a Windows dll) that I'd like to import and use with my test code.
I'm sure it's not this simple ;) but this is the sort of thing I'd like to do..
#include "headerforClassFromBlah.h"
int main()
{
load( "blah.so" );
ClassFromBlah a;
a.DoSomething();
}
I assume that this is a really basic question but I can't find anything that jumps out at me searching the web.
There are two ways of loading shared objects in C++
For either of these methods you would always need the header file for the object you want to use. The header will contain the definitions of the classes or objects you want to use in your code.
Statically:
#include "blah.h"
int main()
{
ClassFromBlah a;
a.DoSomething();
}
gcc yourfile.cpp -lblah
Dynamically (In Linux):
#include <stdio.h>
#include <stdlib.h>
#include <dlfcn.h>
int main(int argc, char **argv) {
void *handle;
double (*cosine)(double);
char *error;
handle = dlopen ("libm.so", RTLD_LAZY);
if (!handle) {
fprintf (stderr, "%s\n", dlerror());
exit(1);
}
dlerror(); /* Clear any existing error */
cosine = dlsym(handle, "cos");
if ((error = dlerror()) != NULL) {
fprintf (stderr, "%s\n", error);
exit(1);
}
printf ("%f\n", (*cosine)(2.0));
dlclose(handle);
return 0;
}
*Stolen from dlopen Linux man page
The process under windows or any other platform is the same, just replace dlopen with the platforms version of dynamic symbol searching.
For the dynamic method to work, all symbols you want to import/export must have extern'd C linkage.
There are some words Here about when to use static and when to use dynamic linking.
It depends on the platform. To do it at runtime, on Linux, you use dlopen, on windows, you use LoadLibrary.
To do it at compile time, on windows you export the function name using dllexport and dllimport. On linux, gcc exports all public symbols so you can just link to it normally and call the function. In both cases, typically this requires you to have the name of the symbol in a header file that you then #include, then you link to the library using the facilities of your compiler.
You need to #include any headers associated with the shared library to get the declrarations of things like ClassFromBlah. You then need to link against the the .so - exactly how you do this depends on your compiler and general instalation, but for g++ something like:
g++ myfile.cpp -lblah
will probably work.
It is -l that link the archive file like libblah.a or if you add -PIC to gcc you will get a 'shared Object' file libblah.so (it is the linker that builds it).
I had a SUN once and have build this types of files.
The files can have a revision number that must be exact or higher (The code can have changed due to a bug). but the call with parameters must be the same like the output.