Just for the purpose of learning, I've made a small example of a main program tentatively loeading a shared library via dlopen (and then a symbol from it via dlsym) and using a default one if the former is not avalable.
On my machine, to make the non-default library available to the main program, I need to compile the former via g++ -fPIC -shared MyLib.cpp -o libMyLib.so, whereas both main.cpp and DefaultLib.cpp are compiled simply by g++ -c main.cpp -o main.o and g++ -c DefaultLib.cpp -o DefaultLib.o. How can I provide the options -fPIC -shared to the compilation of MyLib.cpp in Compiler Explorer?
The current attempt is here, where, I believe, MyLib.cpp is compiled just like the other two cpp files, i.e. without providing options -fPIC and -shared, and maybe most importantly without generating a file with the name libMyLib.so, thus resulting dlopen failing to load; indeed, the foo from the other, default library DefaultLib is called.
Can I compile and dlopen a dynamic library in Compiler Explorer?
Yes, it's certainly possible.
In CMakeLists.txt:
add_library(MyLib SHARED MyLib.cpp)
...and remove MyLib.cpp from add_executable.
Then in main.cpp:
void * lib = dlopen("build/libMyLib.so", RTLD_LAZY);
Because the library is placed in the build subdirectory.
Demo
Related
(Note: This question had been closed, citing that this had an answer. However, my question is not generic, I am asking why this works in ubuntu/redhat, but not in macos/cygwin. So I have edited this question, by modifying the title, mentioning the words macos and ubuntu.)
I have the following c++ code:
// main.cpp
#include<iostream>
#include<cstdio>
#include "defs.h" // has the function headers only
int func0(int a0) {
printf("func0-%d\n", a0);
return a0+1;
}
int func1(int a1) {
int x;
x=func0(a1);
printf("func1-%d\n", x);
return a1+1;
}
int func2(int a2) {
int x;
x=func1(a2);
printf("func2-%d\n", x);
return x+5;
}
int main() {
func1(5);
func2(8);
}
I can compile and run this code as:
g++ main.cpp; ./a.out
Now I would like to move the functions to different files (func1 to f1.cpp,
fun0 and func2 to f2.cpp, and main to main.cpp), and create shared libraries
like this:
g++ -c -pipe -std=c++11 -fPIC main.cpp
g++ -c -pipe -std=c++11 -fPIC f1.cpp
g++ -c -pipe -std=c++11 -fPIC f2.cpp
g++ -shared -o libx1.so f1.o
g++ -shared -o libx2.so f2.o
g++ main.o -L. -lx1 -lx2 -o exe
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
./exe
The above instructions work in redhat linux and ubuntu. But when I run the same commands in other variants of linux (eg macos or cygwin) I get errors during creation of the shared library like this:
g++ -shared -o libx1.so f1.o
undefined reference to func0(int)
g++ -shared -o libx2.so f2.o
undefined reference to func1(int)
Why is this error happening only in some OS versions, and not happening in redhat/ubuntu? Is it due to the gcc versions, or something to do with the OS?
(The above instructions work with g++ in redhat(gcc version 8.3.1) and ubuntu (9.4.0). It does not work with g++ in cygwin(11.3.0) and in macos(11.2.0).)
The problem is caused by cyclic dependencies between the two libraries. Before doing anything else, you should ask yourself whether it is acceptable to have cyclic dependencies for your project. It is never a good idea, but if the alternative involves massive refactoring, it could be the lesser of two evils. Still, refactoring should probably be the default answer in most cases. If you cannot refactor, the rest of this answer is for you.
How are cyclic dependencies handled on different OSes?
On both Linux and Mac OS X (and on FreeBSD and on most commercial Unixes of old), references are resolved at load time. The loader uses the first suitable symbol definition it encounters, be it it in the main executable, in the shared object itself, or in a different shared object. It is not known until load time where that definition will be found.
So when the executable from the question is loaded, the dynamic loader finds the definition of func1 in libx1 and the definitions of func0 and func2 in libx2, and all is well.
The difference between Linux and Mac OS X lies in the linker (ld) behaviour. Both GNU ld and LLVM ld by default allow unresolved references when building a shared library. Mac OS X ld appears to be of a different breed and unresolved references are not allowed by default. One can either list the dependencies on the link line, or explicitly allow unresolved references using the Mac-specific ld option -undefined dynamic_lookup. But of course when the dependencies are cyclic, the first option is problematic. For code in question:
g++ -shared -o libx1.so f1.o -Wl,-undefined,dynamic_lookup
g++ -shared -o libx2.so f2.o -Wl,-undefined,dynamic_lookup
Windows DLLs work very differently. Each symbol must be resolved at link time. Unlike the Unix-y loaders, the loader must know exactly which DLL to search for each imported symbol. There is no option to allow unresolved references in DLLs because there is no mechanism to resolve them at load time from an unknown source.
Windows still allows cyclic dependencies between DLLs, but the mechanism is a bit different. The linker must use separate import libraries in this case (they are usually optional when using GNU or LLVM toolchains). The linking is done in two phases. First, the .lib files are generated for each future .dll, and then .dll themselves are produced using the .lib files from the first stage. For code in question:
# first stage
g++ -shared -Wl,--out-implib=x1.lib -o x1.dll f1.o
g++ -shared -Wl,--out-implib=x2.lib -o x2.dll f2.o
# second stage
g++ -shared -o x1.dll f1.o x2.lib
g++ -shared -o x2.dll f2.o x1.lib
The first stage will report undefined symbols but will still produce the .lib file needed for the second stage.
I was trying to follow along with the C++ headers tutorial here, and as the tutorial says I have the files main.cpp, add.cpp, and add.h. The only is that up until now I haven't been using an IDE and compiling straight from the command line.
But I can't seem to figure out how I would compile add.h and and add.cpp into a library.
As of now, if give the command: g++ -o main main.cpp add.h add.cpp, it compiles just fine and gives me a main.exe. But how would I make it so the library (containing add.h and add.cpp) would be precompiled, and saved as a dll? Is this something that's relatively straight forward with the command line?
Thanks for any help guys, cheers.
There are two types of libraries: static and dynamic libraries. Static libraries are linked together with the resulting program, so each program that uses that library will get its own copy of the library code.
A more memory-efficient way is to use shared libraries (on windows called DLL), which are loaded on demand from a location that is specific for each platform, but the advantage is that only one instance of the library code needs to be loaded to memory when different programs that use the library are running simultaneously, and the resulting binary code of those programs do not contain the library code. it resides in a separate file that needs to be shipped together with the application and installed to a proper location.
If you use unix-like build tools (even on a windows system), this could be a typical sequence of commands you would use to produce a library that contains the code in your add.cpp file:
for a static library:
g++ -c add.cpp
ar crf libadd.a add.o
g++ -o main main.cpp -L. -ladd
the first will compile the add.cpp into add.o, the second will create a static library libadd.a from add.o file. If you want to include more object files into your library, add them to the end of that command line. The last command compiles your main.cpp program while linking it with the static library file libadd.a. The -L. option instructs the linker to search for the library file in the current directory. Alternately, you may want to put the library file in some other directory and use the -Lyour_directory option.
for a shared library (a dll):
g++ -shared -o libadd.so add.cpp
g++ -o main main.cpp -L. -ladd
but to run it, the system must be able to locate the shared library. You can help it by adding the directory where your library is located by adding it to the LD_LIBRARY_PATH environment variable, for instance:
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
for the Windows platform, you may need to use a few more qualifiers, which are nicely explained in the mingw tutorial: http://www.mingw.org/wiki/sampledll
g++ -c main.cpp
g++ -c add.cpp
g++ - o x.dll main.o add.o
I'm trying to compile an executable file which i want also to use as shared library. When i'm clearly compile and linking it as "executable" - everything fine - file could start and work correctly. At this phase i cant correctly linking other libraries with it (tons of redefinitions in log). When i'm trying to add options -Fpic -shared - program copiles successfully, but starting with segmentation fault. How can i make it executable and "sharedlibrary" at the same time?
A single file cannot be a shared library and an executable at the same time. But you can link your object files twice to make both. It'd go something like this:
g++ -c -o module.o module.cpp # create an object that has no main()
g++ -shared -fPIC -o libmodule.so module.o # build shared library
g++ -o program module.o main.cpp # build executable
Or instead, the last line could link the shared library (in which case you'll need the library present when you run the executable):
g++ -o program -l module main.cpp
I have a C shared object I'm loading with dlopen. The C shared object includes another library as a static archive (fully specified path /usr/local/.../libsomelib.a). libsomelib.a is a C++ library and it has global and static locals.
On Ubuntu, the static initializers do not appear to run when opening the shared library with RTLD_GLOBAL and RTLD_GLOBAL | RTLD_LAZY. The symptom I am seeing is a program crash.
The behavior I am seeing seems similar to linking with -nostartfiles or -nostdlib (but I'm not using them). I found a similar thread at C++ Static Constructors and dlopen'd Shared Libraries, but its for a NetBSD system.
If the EXE explicitly includes libsomelib.a and calls a function from it, the C++ library will initialize and the program no longer crashes when calling through the function pointer.
EDIT: here's how the shared object is being built (its the simplest case that I've experienced, without mixing/matching C and C++). cryptopp-so-test.exe calls dlopen:
CXXFLAGS = -g -ggdb -fPIC -DDEBUG -O1 -Wall -Wextra -Wno-unused -DUSE_PRECOMPILED_HEADERS=1 -I. -I/usr/local/include/cryptopp
...
precompile:
$(CXX) $(CXXFLAGS) pch.h -o pch.h.gch
cryptopp-so-test.exe: precompile $(EXEOBJECTS)
$(CXX) $(CXXFLAGS) -o $# $(EXESOURCES) -ldl -lpthread
dsotest: precompile $(DLLOBJECTS)
$(CXX) $(CXXFLAGS) $(DLLSOURCES) -o dsotest-1.so -shared /usr/local/lib/libcryptopp.a
While the code above build one EXE (cryptopp-so-test.exe) and one SO (dsotest-1.so), I actually build and load 4 shared objects (they are built identically).
What flags (or other methods) should I use to ensure the static initializers are run when a C shared object with C++ components is dlopen'd?
On Ubuntu, the static initializers do not appear to run when opening the shared library with RTLD_GLOBAL and RTLD_GLOBAL | RTLD_LAZY.
When you dlopen a shared library, the global constructors are called. You are likely jumping to the wrong conclusions.
The symptom I am seeing is a program crash.
That symptom could be caused by anything. You need to look at the crash in the debugger, and understand what is causing it, rather than blindly guess "static initializers".
One difference between linking libsomelib.a into the main executable and linking it into a shared library is that depending on which code calls which functions, you may end up with wildly different parts of libsomelib.a included into each (the linker will only pull in the parts of libsomelib.a that it can see are necessary).
You could try linking the entire libsomelib.a into the shared library as follows:
g++ $(OBJS) -o dsotest-1.so -shared \
-Wl,--whole-archive -lsomelib -Wl,--no-whole-archive
I have a shared library used by a another application beyond my control which requires *.so objects. My library makes use of sqlite3 which needs to be statically linked with it (I absolutely need a self-contained binary).
When I try to compile and link my library:
-fpic -flto -pthread -m64
-flto -static -shared
I end up with the following error:
/usr/bin/ld: /usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.6.1/crtbeginT.o: relocation R_X86_64_32 against `__DTOR_END__' can not be used when making a shared object; recompile with -fPIC
/usr/local/lib/gcc/x86_64-unknown-linux-gnu/4.6.1/crtbeginT.o: could not read symbols: Bad value
collect2: ld returned 1 exit status
What is recompile with -fPIC related to? My code or CRT?
I have already tried to compile my object with -fPIC with the same result.
Thanks.
EDIT:
The problem does not seem to be related to SQLite3.
I wrote a simple one-line-do-nothing library which compiles and links like this:
g++ -c -fPIC -o bar.o bar.cpp
g++ -shared -o bar.so bar.o
but not like this:
g++ -c -fPIC -o bar.o bar.cpp
g++ -static -shared -o bar.so bar.o
The problem seems to be related to CRT (crtbeginT.o). Am I supposed to recompile GCC --with-pic or anything?
You shouldn't use the -static flag when creating a shared library, it's for creating statically linked executables.
If you only have a static version of the library, you can just link it in using -lsqlite3. But if there's both a dynamic version(.so) and a static version, the linker will prefer the dynamic one.
To instruct the linker to pick the static one, give the linker the -Bstatic flag, and make it switch back to dynamic linking for other stuff (like libc and dynamic runtime support) with -Bdynamic. That is, you use the flags:
-Wl,-Bstatic -lsqlite3 -Wl,-Bdynamic
Alternativly, you can just specify the full path of the .a file, e.g. /usr/lib/libsqlite3.a instead of any compiler/linker flags.
With the GNU ld, you can also use -l:libsqlite3.a instead of -lsqlite3. This will force the use of the library file libsqlite3.a instead of libsqlite3.so, which the linker prefers by default.
Remember to make sure the .a file have been compiled with the -fpic flag, otherwise you normally can't embed it in a shared library.
Any code that will somehow make its way into a dynamic library should be relocatable. It means that everything that is linked with your .so, no matter statically or dynamically, should be compiled with -fPIC. Specifically, static sqlite library should also be compiled with -fPIC.
Details of what PIC means are here: http://en.wikipedia.org/wiki/Position-independent_code
I had the same problem. Apparently -static is not the same as -Bstatic. I switched to -Bstatic and everything worked.