I have a bunch of static libraries (*.a), and I want to build a shared library (*.so) to link against those static libraries (*.a). How can I do so in gcc/g++?
You can (just extract all the .o files and link them with -shared to make a .so), but whether it works, and how well it works, depends on the platform and whether the static library was compiled as position-independent code (PIC). On some platforms (e.g. x86_64), non-PIC code is not valid in shared libraries and will not work (actually I think the linker will refuse to make the .so). On other platforms, non-PIC code will work in shared libraries, but the in-memory copy of the library is not sharable between different programs using it or even different instances of the same program, so it will result in HUGE memory bloat.
I can't see why you couldn't just build the files of your dynamic library to .o files and link with;
gcc -shared *.o -lstaticlib1 -lstaticlib2 -o mylib.so
Related
I have a C++ project and I want to compile it into a single binary file which contains all the shared libraries, third-parties, and so on.
I want to move the compiled file into different servers without installing any dependencies. That's why I need an all-in-one/bundled binary file.
For example, I've tried to compile this:
g++ sample.cpp -o sample -Wl,-Bstatic -l:libyaml-cpp.so
But I got this error, while I really have libyaml-cpp.so file in the /usr/lib directory:
/usr/bin/ld: attempted static link of dynamic object `/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/../../../../lib/libyaml-cpp.so'
collect2: error: ld returned 1 exit status
What's the best way to fix the issue and generally have a compiled file with all dependencies in GNU/Linux platform?
Normally you would use g++ sample.cpp -o sample -static -lyaml-cpp.
But it seems Arch tends to not include static libraries in their packages. Your options are:
Build yaml-cpp yourself to get a static library.
Distribute the dynamic library along with your executable.
To avoid having to install it in system directories, you'll can do one of:
Set LD_LIBRARY_PATH env variable when running your program, to the location of the .so.
Add -Wl,-rpath='$ORIGIN' to the linker flags, then the resulting binary will automatically try to load libraries from its directory, before checking system directories.
You can also modify rpath in existing executables and shared libraries using patchelf. This is useful when you want to include shared libraries that other shared libraries depend on, and the latter weren't compiled with -Wl,-rpath='$ORIGIN'.
You're explicitly demanding linkage against a shared object (.so)!
-l:libyaml-cpp.so
You say your linker can't find -lyaml-cpp, so you seem to be missing the necessary files; that should usually not be the case; the installation of the development files for libyaml-cpp should have included the necessary files.
Generally speaking, there are two routes to go to ensure your program only consists of a single (executable) file:
Static Linking
You can compile a single binary, provided you have access to static libraries for all your dependencies (typically named something like lib*.a). Static libraries behave very similar to ordinary object files when linking the program. Note that your question explicitly refers to a dynamic library (lib*.so), which is the other kind of library that is compiled to be loaded from a file.
Be aware that the glibc does not like to be statically linked, and may not support it at all.
Embed Shared Object as Resource
Alternatively, you can embed shared objects (and other resources) into your executable and unpack it for execution. This can be done in a variety of ways (think self-extracting zip archives, files stored in global variables as byte arrays, this question may get you started).
You will then have to dump the shared libraries into files (e.g., in tempfs) and dlopen them.
I know -fPIC is necessary for shared libraries and know why.
However, I am not clear on the question:
Should -fPIC never be used during building an executable or a static library?
Should -fPIC never be used during building an executable or a static library?
Never is a strong word, and the statement above is false.
Code built with -fPIC is (slightly) less optimal, so why would you want to put it into anything other than a shared library?
Let's start with a static library, which has an easy answer.
Suppose you want to give your users a static library that can be linked into either an executable, or into a shared library of their own?
In that case you must either give them 3 separate archive libraries (one built with -fPIC for linking into shared libraries, one built with -fPIE for linking into PIE executables, and a "regular" one), or you can give them a single archive library (which must have code built with -fPIC).
Now, it could be argued that you should instead give them a shared library, but that forces your end users to distribute 2 binaries, and they may prefer to not do that.
But suppose you want to build a regular (non-PIE) executable. What could be the reason to link in -fPIC code into such executable?
Well, suppose you are in the development stage, and don't care that much about optimizing the code yet. Suppose further that you want to test your code as both a shared library, and as part of PIE and non-PIE executable.
Under above conditions, you could either compile your code 3 times (with and without -fPIC, and with -fPIE), or you could compile it once (with -fPIC) and link it into all 3 of shared library, PIE and non-PIE executable. Doing this saves a lot of compilation time, and some build system complexity.
TL;DR: putting -fPIC objects into executables and static libraries has its place, and you should understand the reason why you are doing it (if you end up doing it).
Update:
Code in an object file is always relocatable
Correct.
is it position-independent code?
No: not all relocatable code is position-independent.
Position-independent code is a subset of relocatable code. Relocatable code can have relocations that apply to any section. Position-independent code must not have any relocations against .text (and .rodata).
I am working on a collection of reusable libraries that need to be made available both as static libraries (.a & .lib) and as dynamic libraries (.so & .dll).
I want dependency management for the dynamic libraries to be as simple as possible (you only take one dynamic library for each bit of functionality that you need), so all of the functional dependencies that each dynamic library has are actually statically linked into it. Thus, the dynamic libraries offer their functionality to downstream clients dynamically, but their upstream dependencies are satisfied statically.
The upshot of all this is that all of my static libraries need to be compiled with -fPIC so that their code is suitable for linkage into a shared library. The same goes for any third-party library that we use. It has to be a static library, compiled with -fPIC.
(I could, I suppose, build both PIC and non-PIC variants of my libraries - but I really do not want to compile the libraries a third time for each target platform -- twice is quite (more than) enough!).
So, here is my problem:
I have been trying to compile boost_system as a static library with -fPIC, but I am not sure if I am succeeding:
/b2 --build-type=complete variant=release link=static threading=multi runtime-link=static --layout=versioned --cxxflags=-fPIC
This build produces .a files as output, as expected. However, when I try to link the boost static library into one of my shared libraries, I start getting an error message that indicates that boost_system is not Position Independent Code:
.../dependencies/external/boost/1_54_0/stage/lib/linux_x86_64/libboost_system-gcc46-s-1_54.a(error_code.o):
relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
However, I have (attempted) to build boost with -fPIC. Is there any test that I can use to determine if libboost_system is actually PIC code? I.e. if the problem is with building boost - or with linking it to my application.
I believe your problem can be solved by removing the command line option "runtime-link=static" which enables static linking of the C++ runtime libraries. Since you are building a dynamic shared library object, you want to avoid this behavior, especially if clients are to link to your library from different Linux OS configurations. However, the option "link=static" is ok and should remain.
I have mostly worked in Windows, and recently I started working in Linux. I have a doubt. I have used Visual Studio as IDE in Windows and used Makefile in Linux.
There are two types of libraries in Windows (VC++), static library (.lib) and DLL. It is quite obvious (isn't it?) if I link with lib file I am using static linking else dynamic linking.
Now when I use g++ compiler, why I need to explicitly mention -Bstatic/-static or Bdynamic/-dynamic flags. Because if file is .a file then I must be using static -linking and if file is .so I am using dynamic linking.
There are times when you want to "force" the compiler to do what it normally wouldn't. In particularly -static is useful when you are building against a library that may not be installed [or not have the same version installed] on another machine where you want your code to run.
the -Bdynamic is useful if you want to have ONE library linked statically, but not EVERY library that your code uses.
e.g. gcc -o myprog myprog.o -Wl,-Bstatic -lspecial -Wl,-Bdynamic
will link myprog using static linking for libspecial (which may be something that isn't widely distributed, for example something you have built yourself)
For general, local development, you don't need either.
In the GNU toolchain, and other Unix compilers, you generally don't specify the full path to a library; you give the linker a flag such as
-lfoo
and let it figure out whether it should link with libfoo.a, libfoo.so, and where these files are located. When only a static library is available, it will link with that.
So, you don't really need to specify -Bstatic except in a few specific cases. Dynamic linking is the default and if you don't want a custom library to be linked dynamically, then just don't build a .so out of it.
Besides, you can explicitly link with a static library by giving the full filename to the linker
gcc -o some_binary main.o libfoo.a
to get a dynamically linked binary with libfoo statically linked in.
I think you've got it backwards. Whether I'm linking with
a dynamic library or with a static one, under Windows, I need
a .lib file; they just aren't generated in the same manner,
and don't contain the same thing. Under Unix, when I link with
a shared object, I link directly against the .so file, and not
against some additional file generated at the same time.
The options -Bstatic and -Bdynamic only come into play when
both a .so and a .a are present, in the same directory; they
tell the linker which one to choose for libraries specified
using the -l option. In Windows, this situation cannot occur;
since you need a .lib for the .dll, you cannot have a .lib
to be statically linked in the same directory.
Because if file is .a file then I must be using static -linking and if file is .so I am using dynamic linking.
These filename "extensions" are only human conventions — there is no reason that they have to be .a and .so respectively. Quite rightly does GCC not force us into that convention.
I have a software library and I used to create .a files, so that people can install them and link against them: g++ foo.o -L/path/to -llibrary
But now I often encounter third-party libraries where only .so files are available (instead of .a), and you just link against them without the -l switch, e.g. g++ foo.o /path/to/liblibrary.so.
What are the differences between these solutions? Should I prefer creating .so files for the users of my library?
Typically, libfoo.a is a static library, and libfoo.so is a shared library. You can use the same -L/-l linker options against either a static or shared. Or you can name the full path to the lib with static or shared. Often libraries are built both static and shared to provide application developers the choice of which they want.
All the code needed from a static lib is part of the final executable. This obviously makes it bigger, but it also means it's self-contained. Once it is compiled, you can run your app without the lib.
Code from a shared lib is not part of the executable. There are just some hooks in place to make the executable aware of the name of the lib it needs. In order to run your app, the shared lib has to be present in the lib search path (e.g. $LD_LIBRARY_PATH).
If you have two apps that share the same code, they can each link against a shared lib to keep the binary size down. If you want to upgrade parts of the app without rebuilding the whole thing, shared libs are good for that too.
Good overview of static, shared dynamic and loadable libraries at
http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html
Some features that aren't really called out from comments I've seen so far.
Static linkage (.a/.lib)
Sharing memory between these compilation units is generally ok because they should(?will) all be using the same runtime.
Static linkage means you avoid 'dll hell' but the cost is recompilation to make use of any change at all. static linkage into Shared libraries (.so) can lead to strange results if you have more than 1 such shared library used by the final executable - global variables may exist multiple times and which one is used and when they are initialised can cause an entirely different hell.
The library will be part of the shipped product but obfuscated and not directly usable.
Shared/Dynamic libraries (.so/.dll)
Sharing memory between these compilation units can be hazardous as they may choose to use different runtime. This can mean you provide different Shared/Dynamic libraries based on the debug/release or single/multi threaded or...
Shared libraries (.so) are less prone to 'dll hell' then Dynamic libraries (.dll) as they include options for quite specific versioning.
Compiling against a .so will capture version information internal to the file (hard to fake) so that you get quite specific .so usage. Compiling against the .lib/.dll only gives a basic file name, any versioning is done managed by the developer (using naming or manually loading the library and checking version details by hand)
The library will have to ship with the final product (somebody else can pick it up and use it)
But now I often encounter third-party libraries where only .so files are available [...] and you just link against them without the -l switch, e.g. g++ foo.o /path/to/liblibrary.so.
JFYI, if you link to a shared library which does not have a SONAME set (compare with readelf -a liblibrary.so), you will end up putting the specified path of liblibrary.so into your target object (executable or another shared library), and which is usually undesired, for users have their own ideas of where to put a program and its associated files. The preferred way is to use -L/path/to -llibrary, perhaps together with -Wl,-rpath,/whatever/path/to if this is the final path (such pathing decisions are made by Linux distributions for example).
Should I prefer creating .so files for the users of my library?
If you distribute source code, the user will make the particular choice.