Is it possible to compile (C++) code for the GPU with nvcc into a shared object (.so file) and load it dynamically from a C++ program (in this case, Cern's ROOT, which is essentially a C++ interpreter ("CINT")).
A simple example that I would like to run is:
extern "C"
void TestCompiled() {
printf("test\n");
exit(0);
}
This code was compiled with nvcc --compiler-options '-fPIC' -o TestCompiled_C.so --shared TestCompiled.cu. Loading the shared object into ROOT with:
{ // Test.C program
int error, check;
check = gROOT->LoadMacro("TestCompiled_C.so", &error);
cout << "check " << check << " " << " error: " << error << endl;
TestCompiled(); // run macro
exit(0);
}
loads the library OK, but does not find TestCompiled():
$ root -b -l Test.C
root [0]
Processing Test.C...
check 0 error: 0
Error: Function Hello() is not defined in current scope Test.C:11:
*** Interpreter error recovered ***
Doing the same by compiling the first test script with ROOT (without the extern line, compiling with root TestCompiled.C++) works… What can I try in order to make the C++ program find the test function when nvcc does the compilation?
I am assuming that the shared object file being output is like any other shared library, such as one created with GCC using the shared option. In this case, to load the object dynamically, you will need to use the dlopen function to get a handle to the shared object. Then, you can use the dlsym function to look for a symbol in the file.
void *object_handle = dlopen("TestCompiled_C.so", RTLD_NOW);
if (object_handle == NULL)
{
printf("%s\n", dlerror());
// Exit or return error code
}
void *test_compiled_ptr = dlsym(object_handle, "TestCompiled");
if (!test_compiled)
{
printf("%s\n", dlerror());
// Exit or return error code
}
void (*test_compiled)() = (void (*)()) test_compiled_ptr;
test_compiled();
You will need to include dlfcn.h and link with -ldl when you compile.
The difference between this and what you are doing now is that you are loading the library statically rather that dynamically. Even though shared objects are "dynamically linked libraries," as they are called in the windows world, doing it the way you are now is loading all of the symbols in the object when the program is launched. To dynamically load certain symbols at runtime, you need to do it this way.
I'm copying, for reference, the salient points of the answer from the RootTalk forum that solved the problem:
A key point is that the C interpreter of ROOT (CINT) requires a "CINT dictionary" for the externally compiled function. (There is no problem when compiling through ROOT, because ACLiC creates this dictionary when it pre-compiles the macro [root TestCompiled.C++]).
So, an interface TestCompiled.h++ must be created:
#ifdef __cplusplus
extern "C" {
#endif
void TestCompiled(void);
#ifdef __cplusplus
} /* end of extern "C" */
#endif
The interface must then be loaded inside ROOT along with the shared object:
{ // Test.C ROOT/CINT unnamed macro (interpreted)
Int_t check, error;
check = gROOT->LoadMacro("TestCompiled_C.so", &error);
std::cout << "_C.so check " << check << " error " << error << std::endl;
check = gROOT->LoadMacro("TestCompiled.h++", &error);
std::cout << "_h.so check " << check << " error " << error << std::endl;
TestCompiled(); // execute the compiled function
}
ROOT can now use the externally compiled program: root -b -l -n -q Test.C works.
This can be tested with, e.g., g++ on the following TestCompiled.C:
#include <cstdio>
extern "C" void TestCompiled(void) { printf("test\n"); }
compiled with
g++ -fPIC -shared -o TestCompiled_C.so TestCompiled.C
Related
In a main program, I dlopen and dlclose (LoadLibrary and FreeLibrary respectively) a shared library. The shared library contains a static variable that is instantiated upon dlopen, and destroyed upon dlclose. This behavior is consistent on MSVC 2008 and 2013, GCC 3.4.6, and Sunstudio 12.1.
With GCC 4.9.1 and GCC 5.2.1 however, the destructor was no longer called on dlclose. Instead, it was called before program exit.
The particularity of the static variable's class, is that in the constructor, there is a call to a templated function get (of global scope) that returns a local static variable.
I was able to reproduce this behavior with the following one cpp file linked into a shared library:
#include <iostream>
template <typename T> // In my actual code, i is of type T, however, this has no effect
int get()
{
static int i = 0;
return i;
}
class Dictionary {
public:
Dictionary()
{
std::cout << "Calling Constructor" << std::endl;
get<int>();
}
~Dictionary(){
std::cout << "Calling Destructor" << std::endl;
}
private:
Dictionary(const Dictionary&);
Dictionary& operator=(const Dictionary&);
};
static Dictionary d;
I investigated the tweaks that can be made in order to have the destructor called on dlclose, and concluded the following:
If the function get was not templated
else if the variable i in the function get was not static
else if the function get is made static
The main program's code is the following:
#include <dlfcn.h>
#include <cassert>
#include <string>
#include <iostream>
void* LoadLib(std::string name)
{
void* libInstance;
name = "lib" + name + ".so";
libInstance = dlopen(name.c_str(), RTLD_NOW);
if ( ! libInstance ) std::cout << "Loading of dictionary library failed. Reason: " << dlerror() << std::endl;
return libInstance;
}
bool UnloadLib(void* libInstance)
{
int ret = dlclose(libInstance);
if (ret == -1)
{
std::cout << "Unloading of dictionary library failed. Reason: " << dlerror() << std::endl;
return false;
}
return true;
}
int main()
{
void* instance = LoadLib("dll");
assert(instance != 0);
assert(UnloadLib(instance));
std::cout << "DLL unloaded" << std::endl;
}
I built the binaries with the following commands:
g++ -m64 -g -std=c++11 -shared -fPIC dll.cpp -o libdll.so
g++ -m64 -g -std=c++11 -ldl main.cpp -o main.out
The output I get when the destructor is called before program exit is the following:
Calling Constructor
DLL unloaded
Calling Destructor
The output I get when the destructor is called on dlclose is the following:
Calling Constructor
Calling Destructor
DLL unloaded
Questions:
If the change of behavior between the versions of GCC is not a bug, can you please explain why is the destructor not called on dlclose?
Can you please explain for each tweak: Why is the destructor called on dlclose in this case?
There is no guarantee that unloading (destructors are invoked) happens on dlclose. On musl (as opposed to glibc), constructors only run the first time a library is run, and destructors only run on exit. For portable code, dlclose cannot be assumed to unload the symbols immediately.
The unload behavior depends on glibc's symbol binding when doing dynamic linking, and is independent of GCC.
The static variable get::i has a STB_GNU_UNIQUE binding. For static variables in inline functions, the uniqueness of the object is assured by the ELF linker. However, for dynamic loading, the dynamic linker assures uniqueness by marking the symbol STB_GNU_UNIQUE. Hence, another attempt to dlopen the same shared library by some other code will lookup the symbol and find that it is unique and return the existent one from the unique symbols table. A symbol with unique binding cannot be unloaded.
Unique binding can be disabled with -fno-gnu-unique if not needed.
References
Bug that I raised to GCC
STB_GNU_UNIQUE
Right off the bat, I want to say that I've never worked with dynamic libraries so It's possible that I don't even understand how they work properly.
I want to have a fully loaded code running and after some trigger (probably user interaction) I want to load a specific library and execute a function inside that library. Preferably close it afterwards. Essentially allowing me to change it and re-load it during run time.
This is the simple dynamic library (called dynlib.so located in the same directory as the main code):
int getInt(int arg_0)
{
return (arg_0 + 7);
}
And this is the main program:
#include <iostream>
#include <dlfcn.h>
int main() {
void *lib_handle = dlopen("./dynlib.so", RTLD_LAZY | RTLD_NOW);
if (!lib_handle) {
fprintf(stderr, "%s\n", dlerror());
exit(EXIT_FAILURE);
}
typedef int (*func_ptr)(int);
func_ptr func = (func_ptr)dlsym(lib_handle, "getInt");
std::cout << func(13);
dlclose(lib_handle);
}
I'm compiling it using: g++ -std=c++11 -ldl loadlibtest.cpp -o main.
The error I'm catching is ./libshared.so: file too short In my if (!lib_handle) {.
It works fine for me. I've compiled dynlib.so with
$ gcc dynlib.c -fPIC -shared -o dynlib.so
(Obviously, you need to either compile it as C or C++ with extern "C" to avoid name mangling).
and I needed to place -ldl after the source file in the g++ invocation.
gcc: 4.8.5; g++: 5.3.0
dlsym may fail too and casting from void* to function pointers is technically UB. You should base it on the usage snippet from the
manpage(modified for your function):
dlerror(); /* Clear any existing error */
/* Writing: func = (int (*)(int)) dlsym(handle, "getInt");
would seem more natural, but the C99 standard leaves
casting from "void *" to a function pointer undefined.
The assignment used below is the POSIX.1-2003 (Technical
Corrigendum 1) workaround; see the Rationale for the
POSIX specification of dlsym(). */
*(void **) (&func) = dlsym(handle, "getInt");
if ((error = dlerror()) != NULL) {
fprintf(stderr, "%s\n", error);
exit(EXIT_FAILURE);
}
After some great replies I discovered what I'm doing wrong.
1) I wasn't using extern "C" for my library functions, so dlsym was unable to find the function.
2) I didn't know that dynamic libraries had to be compiled << pretty stupid of me.
I still want to know if there is a way to use uncompiled code as a library, but my initial problem was solved, thanks to everyone.
I'm trying to call some C++ functions through a function pointer table which is exported as a C symbol from a shared object. The code is actually working but Clang's undefined behavior sanitizer (= UBSan) sees the call I made is illegal as follows:
==11410==WARNING: Trying to symbolize code, but external symbolizer is not initialized!
path/to/HelloWorld.cpp:25:13: runtime error: call to function (unknown) through pointer to incorrect function type 'foo::CBar &(*)()'
(./libFoo.so+0x20af0): note: (unknown) defined here
Due to Clang's undefined behavior sanitizer, it is legal to indirectly call a function which returns a reference of a C++ standard class object through a function pointer but it's illegal for a user-defined class. Somebody could you please tell me what's wrong with it?
I've been trying to build the project on Ubuntu 14.04 with Clang-llvm 3.4-1ubuntu3 and CMake 2.8.12.2. To reproduce the phenomenon, please place the following 5 files in the same directory and invoke build.sh. It will create a makefile and build the project, and run the executable.
Foo.h
#ifndef FOO_H
#define FOO_H
#include <string>
//
#define EXPORT __attribute__ ((visibility ("default")))
namespace foo {
class CBar
{
// empty
};
class CFoo
{
public:
static CBar& GetUdClass();
static std::string& GetStdString();
};
// function pointer table.
typedef struct
{
CBar& (*GetUdClass)();
std::string& (*GetStdString)();
} fptr_t;
//! function pointer table which is exported.
extern "C" EXPORT const fptr_t FptrInFoo;
}
#endif
Foo.cpp
#include "Foo.h"
#include <iostream>
using namespace std;
namespace foo
{
// returns reference of a static user-defined class object.
CBar& CFoo::GetUdClass()
{
cout << "CFoo::GetUdClass" << endl;
return *(new CBar);
}
// returns reference of a static C++ standard class object.
std::string& CFoo::GetStdString()
{
cout << "CFoo::GetStdString" << endl;
return *(new string("Hello"));
}
// function pointer table which is to be dynamically loaded.
const fptr_t FptrInFoo = {
CFoo::GetUdClass,
CFoo::GetStdString,
};
}
HelloWorld.cpp
#include <iostream>
#include <string>
#include <dirent.h>
#include <dlfcn.h>
#include "Foo.h"
using namespace std;
using namespace foo;
int main()
{
// Retrieve a shared object.
const string LibName("./libFoo.so");
void *pLibHandle = dlopen(LibName.c_str(), RTLD_LAZY);
if (pLibHandle != 0) {
cout << endl;
cout << "Info: " << LibName << " found at " << pLibHandle << endl;
// Try to bind a function pointer table:
const string SymName("FptrInFoo");
const fptr_t *DynLoadedFptr = static_cast<const fptr_t *>(dlsym(pLibHandle, SymName.c_str()));
if (DynLoadedFptr != 0) {
cout << "Info: " << SymName << " found at " << DynLoadedFptr << endl;
cout << endl;
// Do something with the functions in the function table pointer.
DynLoadedFptr->GetUdClass(); // Q1. Why Clang UBSan find this is illegal??
DynLoadedFptr->GetStdString(); // Q2. And why is this legal??
} else {
cout << "Warning: Not found symbol" << endl;
cout << dlerror() << endl;
}
} else {
cout << "Warning: Not found library" << endl;
cout << dlerror() << endl;
}
cout << endl;
return 0;
}
CMakeLists.txt
project (test)
if(COMMAND cmake_policy)
cmake_policy(SET CMP0003 NEW)
endif(COMMAND cmake_policy)
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,-rpath,$ORIGIN")
add_library(Foo SHARED Foo.cpp)
add_executable(HelloWorld HelloWorld.cpp)
target_link_libraries (HelloWorld dl)
build.sh
#!/bin/bash
# 1. create a build directory.
if [ -d _build ]; then
rm -rf _build
fi
mkdir _build
cd _build
# 2. generate a makefile.
CC=clang CXX=clang++ CXXFLAGS="-fvisibility=hidden -fsanitize=undefined -O0 -g3" cmake ..
# 3. build.
make
# 4. and run the executable.
./HelloWorld
I've been trying to find a clue to dig into the issue and realized the issue was caught by "function" option of the sanitizer (-fsanitize=function) but it's not so much documented. I'd appreciate if you guys could give me a reasonable explanation for such a runtime error message which looks like coming from another planet. Thanks.
What was Clang pointing out as "unknown" in the output?
Below is the output from addr2line to check what was "unknown" for the sanitizer:
$ addr2line -Cfe _build/libFoo.so 0x20af0
foo::CFoo::GetUdClass()
path/to/Foo.cpp:12
Hmm, it really looks like the function I was expecting to call for me. Can you guess how did it look different for Clang?
CBar's typeinfo needs to have default visibility for the function's type be considered the same by Clang on Linux across the executable and the dynamic library; change Foo.h to:
class EXPORT CBar
{
...
}
I'm experimenting with making a kind of plugin architecture for a program I wrote, and at my first attempt I'm having a problem. Is it possible to access symbols from the main executable from within the shared object? I thought the following would be fine:
testlib.cpp:
void foo();
void bar() __attribute__((constructor));
void bar(){ foo(); }
testexe.cpp:
#include <iostream>
#include <dlfcn.h>
using namespace std;
void foo()
{
cout << "dynamic library loaded" << endl;
}
int main()
{
cout << "attempting to load" << endl;
void* ret = dlopen("./testlib.so", RTLD_LAZY);
if(ret == NULL)
cout << "fail: " << dlerror() << endl;
else
cout << "success" << endl;
return 0;
}
Compiled with:
g++ -fPIC -o testexe testexe.cpp -ldl
g++ --shared -fPIC -o testlib.so testlib.cpp
Output:
attempting to load
fail: ./testlib.so: undefined symbol: _Z3foov
So obviously, it's not fine. So I guess I have two questions:
1) Is there a way to make the shared object find symbols in the executable it's loaded from
2) If not, how do programs that use plugins typically work that they manage to get code in arbitrary shared objects to run inside their programs?
Try:
g++ -fPIC -rdynamic -o testexe testexe.cpp -ldl
Without the -rdynamic (or something equivalent, like -Wl,--export-dynamic), symbols from the application itself will not be available for dynamic linking.
From The Linux Programming Interface:
42.1.6 Accessing Symbols in the Main Program
Suppose that we use dlopen() to dynamically load a shared library, use dlsym() to obtain
the address of a function x() from that library, and then call x(). If
x() in turn calls a function y(), then y() would normally be sought in
one of the shared libraries loaded by the program. Sometimes, it is
desirable instead to have x() invoke an implementation of y() in the
main program. (This is similar to a callback mechanism.) In order to
do this, we must make the (global-scope) symbols in the main program
available to the dynamic linker, by linking the program using the
––export–dynamic linker option:
$ gcc -Wl,--export-dynamic main.c (plus further options and arguments)
Equivalently, we can write the
following:
$ gcc -export-dynamic main.c
Using either of these options
allows a dynamically loaded library to access global symbols in the
main program.
The gcc –rdynamic option and the gcc –Wl,–E option are
further synonyms for –Wl,––export–dynamic.
Since I observed some strange behavior of global variables in my dynamically loaded libraries, I wrote the following test.
At first we need a statically linked library: The header test.hpp
#ifndef __BASE_HPP
#define __BASE_HPP
#include <iostream>
class test {
private:
int value;
public:
test(int value) : value(value) {
std::cout << "test::test(int) : value = " << value << std::endl;
}
~test() {
std::cout << "test::~test() : value = " << value << std::endl;
}
int get_value() const { return value; }
void set_value(int new_value) { value = new_value; }
};
extern test global_test;
#endif // __BASE_HPP
and the source test.cpp
#include "base.hpp"
test global_test = test(1);
Then I wrote a dynamically loaded library: library.cpp
#include "base.hpp"
extern "C" {
test* get_global_test() { return &global_test; }
}
and a client program loading this library: client.cpp
#include <iostream>
#include <dlfcn.h>
#include "base.hpp"
typedef test* get_global_test_t();
int main() {
global_test.set_value(2); // global_test from libbase.a
std::cout << "client: " << global_test.get_value() << std::endl;
void* handle = dlopen("./liblibrary.so", RTLD_LAZY);
if (handle == NULL) {
std::cout << dlerror() << std::endl;
return 1;
}
get_global_test_t* get_global_test = NULL;
void* func = dlsym(handle, "get_global_test");
if (func == NULL) {
std::cout << dlerror() << std::endl;
return 1;
} else get_global_test = reinterpret_cast<get_global_test_t*>(func);
test* t = get_global_test(); // global_test from liblibrary.so
std::cout << "liblibrary.so: " << t->get_value() << std::endl;
std::cout << "client: " << global_test.get_value() << std::endl;
dlclose(handle);
return 0;
}
Now I compile the statically loaded library with
g++ -Wall -g -c base.cpp
ar rcs libbase.a base.o
the dynamically loaded library
g++ -Wall -g -fPIC -shared library.cpp libbase.a -o liblibrary.so
and the client
g++ -Wall -g -ldl client.cpp libbase.a -o client
Now I observe: The client and the dynamically loaded library possess a different version of the variable global_test. But in my project I'm using cmake. The build script looks like this:
CMAKE_MINIMUM_REQUIRED(VERSION 2.6)
PROJECT(globaltest)
ADD_LIBRARY(base STATIC base.cpp)
ADD_LIBRARY(library MODULE library.cpp)
TARGET_LINK_LIBRARIES(library base)
ADD_EXECUTABLE(client client.cpp)
TARGET_LINK_LIBRARIES(client base dl)
analyzing the created makefiles I found that cmake builds the client with
g++ -Wall -g -ldl -rdynamic client.cpp libbase.a -o client
This ends up in a slightly different but fatal behavior: The global_test of the client and the dynamically loaded library are the same but will be destroyed two times at the end of the program.
Am I using cmake in a wrong way? Is it possible that the client and the dynamically loaded library use the same global_test but without this double destruction problem?
g++ -Wall -g -ldl -rdynamic client.cpp libbase.a -o client
CMake adds -rdynamic option allowing loaded library to resolve symbols in the loading executable... So you can see that this is what you don't want. Without this option it just misses this symbol by accident.
But... You should not do any stuff like that there. Your libraries and executable should
not share symbols unless they are really should be shared.
Always think of dynamic linking as static linking.
If using shared libraries you must define the stuff you want to export with macro like here. See DLL_PUBLIC macro definition in there.
By default, the linker won't combine a global variable (a 'D') in the base executable with one in a shared library. The base executable is special. There might be an obscure way to do this with one of those obscure control files that ld reads, but I sort of doubt it.
--export-dynamic will cause a.out 'D' symbols to be available to shared libs.
However, consider the process. Step 1: you create a DSO from a .o with a 'U' and a .a with a 'D'. So, the linker incorporates the symbol in the DSO. Step 2, you create the executable with a 'U' in one of the .o files, and 'D' in both a .a and the DSO. It will try to resolve using the left-to-right rule.
Variables, as opposed to functions, pose certain difficulties for the linker across modules in any case. A better practice is to avoid global var references across module boundaries, and use function calls. However, that would still fail for you if you put the same function in both the base executable and a shared lib.
My first question is if there is any particular reason for which you both statically and dynamically (via dlopen) link the same code?
For your problem: -rdynamic will export the symbols from your program and what probably is happening is that dynamic linker resolves all references to your global variable to the first symbol it encounters in symbol tables. Which one is that I don't know.
EDIT: given your purpose I would link your program that way:
g++ -Wall -g -ldl client.cpp -llibrary -L. -o client
You may need to fix the order.
I would advise to use a dlopen(... RTLD_LAZY|RTLD_GLOBAL); to merge global symbol tables.
I would propose to compile any .a static library which you plan to link to a dinamic library, with -fvisibility=hidden parameter, so:
g++ -Wall -fvisibility=hidden -g -c base.cpp