Static and Dynamic/Shared Linking with MinGW - c++

I want to start with a simple linking usage to explain my problem. Lets assume that there is a library z which could be compiled to shared library libz.dll(D:/libs/z/shared/libz.dll) or to static library libz.a (D:/libs/z/static/libz.a).
Let I want to link against it, then I do this:
gcc -o main.exe main.o -LD:/libs/z/static -lz
According to this documentation, gcc would search for libz.a, which is
archive files whose members are object files
I also can do the following:
gcc -o main.exe main.o -LD:/libs/z/shared -lz
It is not mentioned in the documentation above that -l flag will search for lib<name>.so.
What will happen if I libz.a and libz.dll will be in the same directory? How the library will be linked with a program? Why I need the flags -Wl,-Bstatic and -Wl,-Bdynamic if -l searches both for shared and static libraries?
Why some developers provide .a files with .dll files for the same modules, if I compile a shared library distribution?
For example, Qt provides .dll files in bin directory with .a files in lib directory. Is it the same library, but built like shared and static, respectively? Or .a files are some kind of dummy libraries which provide linking with shared libraries, where there are real library implementations?
Another example is OpenGL library on Windows. Why every compiler must provide the static OpenGL lib like libopengl32.a in MingW?
What are files with .dll.a and .la extensions used for?
P.S. There are a lot of questions here, but I think each one depends on the previous one and there is no need to split them into several questions.

Please, have a look at ld and WIN32 (cygwin/mingw). Especially, the direct linking to a dll section for more information on the behavior of -l flag on Windows ports of LD. Extract:
For instance, when ld is called with the argument -lxxx it will attempt to find, in the first directory of its search path,
libxxx.dll.a
xxx.dll.a
libxxx.a
cygxxx.dll (*)
libxxx.dll
xxx.dll
before moving on to the next directory in the search path.
(*) Actually, this is not cygxxx.dll but in fact is <prefix>xxx.dll, where <prefix> is set by the ld option -dll-search-prefix=<prefix>. In the case of cygwin, the standard gcc spec file includes -dll-search-prefix=cyg, so in effect we actually search for cygxxx.dll.
NOTE: If you have ever built Boost with MinGW, you probably recall that the naming of Boost libraries exactly obeys the pattern described in the link above.
In the past there were issues in MinGW with direct linking to *.dll, so it was advised to create a static library lib*.a with exported symbols from *.dll and link against it instead. The link to this MinGW wiki page is now dead, so I assume that it should be fine to link directly against *.dll now. Furthermore, I did it myself several times with the latest MinGW-w64 distribution, and had no issues, yet.
You need link flags -Wl,-Bstatic and -Wl,-Bdynamic because sometimes you want to force static linking, for example, when the dynamic library with the same name is also present in a search path:
gcc object1.o object2.o -lMyLib2 -Wl,-Bstatic -lMyLib1 -Wl,-Bdynamic -o output
The above snippet guarantees that the default linking priority of -l flag is overridden for MyLib1, i.e. even if MyLib1.dll is present in the search path, LD will choose libMyLib1.a to link against. Notice that for MyLib2 LD will again prefer the dynamic version.
NOTE: If MyLib2 depends on MyLib1, then MyLib1 is dynamically linked too, regardless of -Wl,-Bstatic (i.e. it is ignored in this case). To prevent this you would have to link MyLib2 statically too.

Related

Using --whole-archive linker option with CMake and libraries with other library dependencies

I've got a project that used to be a giant set of source files that all got compiled and then linked as one executable. As a first step in making the project more modular, I am breaking up the build into several smaller chunks and making them static libraries. There's a hierarchy, so Exe1 will link against static libs Lib2A and Lib2B. Lib2A will depend on static Lib3A, lib3B, lib3C, etc. The numbers here show their layer in the hierarchy.
The problem is that I need to use --whole-archive when linking or else some symbols from the underlying libraries are not found.
When I add the below for the linking of Exe1:
target_link_libraries(Exe1 -Wl,--whole-archive Lib2A Lib2B -Wl,--no-whole-archive)
I end up with an actual link stage command like:
g++ -o Exe1 -Wl,--whole-archive libLib2A.a libLib2B.a -Wl,--no-whole-archive libLib3A.a libLib3B.a libLib3C.a
Inevitably, symbols from some of the layer 3 static libraries get lost and I get missing symbol errors.
I expected that because Lib2A has Lib3* libraries as dependencies, that they would also be "inside" the --whole-archive part of the linker command, but they show up outside.
I've tried many different combinations (e.g. putting the --whole-archive stuff at lower layers), but haven't come across an approach that works using CMake. What am I doing wrong?
Thanks
For 3.12 and newer versions of CMake, I would use object libraries.
The workaround I found for versions earlier than that was to create an intermediate static library that used some property magic to place all linkage dependencies inside the --whole-archive section. For me, the top-level static library was called 'source'. It contained actually nothing itself, but had linkage dependencies on a bunch of other static libraries. I created 'source-combined' as follows:
add_library(source-combined STATIC "")
set_target_properties(source-combined PROPERTIES LINKER_LANGUAGE CXX)
target_link_libraries(source-combined PUBLIC
-Wl,--whole-archive
$<TARGET_PROPERTY:source,INTERFACE_LINK_LIBRARIES>
-Wl,--no-whole-archive
)
Now when I create an executable or a shared library by linking against this souce-combined library, I get the --whole-archive and --no-whole-archive as bookends around the entire set of static libraries that were the link dependencies of 'source'. It took forever to stumble across this technique, so I'm sharing it.
The following worked for me. Consider two libraries:
my_platform
my_clib
We want the whole archive of my_clib, and my_platform links to it.
add_library(my_platform INTERFACE) # this could also be a regular library
add_library(my_clib STATIC)
target_sources(my_clib
PRIVATE
gcc_newlib_nano.c
gcc_newlib_nano_cpp.cc
)
# Link my_clib and any other libs
target_link_libraries(my_platform
INTERFACE
my_clib
)
# Ensure the whole archive is linked
target_link_options(my_platform
INTERFACE
-Wl,--whole-archive ${CMAKE_CURRENT_BINARY_DIR}/libmy_clib.a -Wl,--no-whole-archive
)
As an alternative to the above answer, I needed to get something quick and dirty to see if the effort to add whole archive target flags (or convert the code base to object libraries...) was the right solution. By following the CMake Source Code for the default link command, I modified my project's command to:
set(CMAKE_CXX_LINK_EXECUTABLE "<CMAKE_CXX_COMPILER> <FLAGS> <CMAKE_CXX_LINK_FLAGS> <LINK_FLAGS> <OBJECTS> -o <TARGET> -Wl,--start-group -Wl,--whole-archive <LINK_LIBRARIES> -Wl,--no-whole-archive -Wl,--end-group")
It worked a treat! While not the greatest solution, it will at least get some results quickly.
If you need to use the linker option --whole-archive, then you definably should use object libraries:
# Lib2A/CMakeLists.txt
add_library(Lib2A OBJECT ${Lib2A_SRC})
# Lib2B/CMakeLists.txt
add_library(Lib2B OBJECT ${Lib2B_SRC})
It is portable and does not require use the linker option --whole-archive.

C++ using functions from shared library

I have the following problem:
I have two separate c++ projects, and want to use certain functions from one of them in the other. I am compiling using g++, on Linux.
I decided to do this by creating a shared library out of the project from which to use the functions. I added -fPIC to the compiler flags in the Makefile and then created a shared library like this:
g++ -shared -Wl,-soname,libmyproject.so.1 -o libmyproject.so a.o b.o c.o -lc
Then I simply copied the .so file and the header files into the (parent) directory of the new project and added the following to its Makefile:
To LIBS:
-L/../external_proj/libmyproject.so
To CXXFLAGS:
-I/../external_proj
Then I #include the appropriate header file in the destination project code and try to call certain functions from the original project. However, when I compile the destination project I get an error "undefined reference" for the imported function.
My question is then: is there something I'm missing in this setup? Is there perhaps something that needs to be added to the headers in the original project in order to export the functions I want to use?
I should note this is the first time I have attempted to use a shared library in this way. Thanks.
The -L option only specifies the directory where the linker will search for libraries to link with. Then you will need to use the -l option to specify the base name of the shared library (without the "lib" prefix and the ".so" suffix).
But even that will unlikely to be enough. The runtime loader needs to find the shared library, when you attempt to try to execute it. -L and -l will be sufficient to successfully link, but the runtime loader only searches /usr/lib(64)?, and a few other places by default. It does NOT search the current directory, and the ELF binary only records the names of the shared libraries that must be loaded, and not their full pathnames. You have to explicitly record any extra directories to search for any shared libraries, which is the -rpath option.
To finish the job you will also need to pass -rpath to the linker, but g++ does not support this option directory, you will have to use -W to do that.
The full set of options you will likely need are:
-L/../external_proj -lmyproject -Wl,-rpath -Wl,`cd ../external_proj && pwd`
See gcc documentation for more information on the -W option.
Absolute pathnames should be used with -rpath, hence the need to obtain the full pathname to the directory where the shared library is.
The -L flag is to add a path to search libraries in. The -l (lower-case L) is for linking with a library in the search path.
Or you can skip the flags and link with the library directly, almost like you do now (but without the -L option).
If you use the -l option, then remember that for a file libname.so you use only name as the library name. As in -lname. The linker will search for the correct files with the added prefix and suffix.
And lastly an important note about the paths used when linking: If you use -L and -l to link with a shared library, it's only the linker which will find the library. The OS runtime-loader will not be able to see the path used and will not find the library, if it's in a non-standard location. For that you must also set the runtime-path using the special linker option -rpath.
Unfortunately the GCC frontend program g++ doesn't recognize that option, you have to use -Wl to tell g++ to pass on an option to the actual linker. As in -Wl,-rpath,/path/to/libraries.
To summarize, here are the different variants you can use:
Link directly with the library: g++ your_source.cpp ../external_proj/libmyproject.so
Use the -L and -l options: g++ your_source.cpp -L../external_proj -lmyproject
To set the runtime linker path: g++ your_source.cpp -L../external_proj -lmyproject -Wl,-rpath,../external_proj

Error when static linking g++

I have a problem, i want to compile my application with static linking of mysql connector.
My command line:
g++ -o newserver stdafx.cpp ... -lboost_system -lboost_thread
-lpthread -lmysqlcppconn -static /usr/lib/libmysqlcppconn-static.a -std=c++0x
But i have error:
/usr/bin/ld: cannot find -lmysqlcppconn
/tmp/ccxpOfdZ.o: In function `IsEqualsDns(unsigned long, std::basic_string<char, std::char_traits<char>, std::allocator<char> >)':
Server.cpp:(.text+0x356e): warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
collect2: ld returned 1 exit status
How can i fix this?
Thanks!
Where is the library libsqlcppconn.a or libsqucppconn.so
(static or dynamic)? The compiler is looking for it, and
doesn't find it.
Presumably, this is the same library as
/usr/lib/mysqlcppconn-static.a. If so, just drop the
-lmysqlcppconn. Or just use -lmysqlcppconn-static (no
spaces), and forget about the /usr/lib/libmysqlconn-static.a.
With a name like that, there shouldn't be a corresponding .so,
which means that g++ will link it statically, even without the
-static. You only need the -static if there is both
a libmysqlconn-static.so and a libmysqlconn-static.a in the
same directory.
With regards to the second error (which is just a warning, but
will cause problems if you try to run the linked program on
other machines, or even after an upgrade of your machine): if
you use -static anywhere in your command line (as you
currently do), then it applies to all files linked afterwards.
Including the system libraries, which you don't want to link
statically. My guess is that the -static isn't necessary (see
above); if it is, place it immediately before the library you
want to link statically, and place a -dynamic immediately
after (so that any following libraries, including the system
libraries, will be dynamically linked).
You could try g++ -static YOUR ARGUMENTS.
If you are coming from a Windows platform, linking against Boost can give a few surprises. The typicall Boost installation (e.g. after ./b2 install) will make both dynamic and static libraries and put them in the same directory. Typically, the two library forms only differ in their extension (.so or .a).
Windows supports auto-linking, which basically means that library files contain some flags in their first few bytes indicating whether they are for dynamic or for static linking. On Linux platforms, this is not the case and the linker gets confused which file to load (since you don't provide the extension of the library name). Therefore, you need to tell your linker which form of linking you want.

Linux based OS with static libraries: what can I do?

I have a linux based OS with a lot of system libraries compiled as static libraries.
How can I use such libraries in my application, and link them to my final binary?
You use them as you do use shared libraries, except that you link against statically. An introduction to GCC - shared libraries and static libraries article will get you started.
I've trouble to understand. If you are linking with something like
g++ -o myprog myprog.o obj1.o obj2.o -L/path/to/lib -L/path2/to/lib -llib1 -llib2 -lib3
the linker called through the gcc or g++ wrapper will do "the right thing(tm)", if liblib1.so exist in the library path (/path/to/lib, /path2/to/lib plus a set of system specific directories where system libraries probably are), it will be linked dynamically, if not liblib1.a will be linked statically. The only thing to be aware of, is that if there are mutual dependencies in static libaries (lib1 needs lib2 and lib2 needs lib1), you may need to repeat them several times or use the --start-group and --end-group options of ld to mark libraries which needs to be considered together.

Creating dummy shared object (.so) to depend on other shared objects

I'm trying to create a shared object (.so) that will make it so, by including one shared object with -lboost, I implicitly include all the boost libraries. Here's what I tried:
#!/bin/sh
BOOST_LIBS="-lboost_date_time-gcc43-mt -lboost_filesystem-gcc43-mt"
#truncated for brevity
g++ $BOOST_LIBS -shared -Wl,-soname,libboost.so.1 -o libboost.so.1.0
ln -si libboost.so.1.0 libboost.so.1
ln -si libboost.so.1 libboost.so
After placing all 3 created files (libboost.so libboost.so.1 libboost.so.1.0) in the same directory as all the boost libraries, I tried compiling a test program with it (which depends on -lboost_date_time-gcc43-mt):
g++ -lboost test.cpp
Doing this, I got the same undefined reference message as not having -lboost. Having -lboost_date_time-gcc43-mt works, but that's too wordy :) How do I get -lboost to automatically bring in the other shared libraries?
You don't. Not really, anyway.
The linker is stripping out all of the symbol dependencies because the .so doesn't use them.
You can get around this, perhaps, by writing a linker script that declares all of the symbols you need as EXTERN() dependencies. But this implies that you'll need to list all of the mangled names for the symbols you need. Not at all worth the effort, IMO.
I don't have a solution for creating a dummy '.so', but I do have something that will simplify your life... I highly suggest that you try using cross-platform make (CMake). In CMake, linking against those libraries is easy:
FIND_PACKAGE(Boost 1.37 COMPONENTS date_time filesystem REQUIRED)
ADD_EXECUTABLE(myexecutable ${myexecutable_SRCS})
TARGET_LINK_LIBRARIES(myexecutable ${Boost_LIBRARIES})
The commands above, if placed in a "CMakeLists.txt" file, is all you need to:
Verify that Boost 1.37 or later is installed, with the "date_time" and "filesystem" libraries installed.
Create an executable named "myexecutable" from the sources listed in the corresponding variable.
Link the executable "myexecutable" against the boost "date_time" and "filesystem" libraries.
See also: Why the KDE project switched to CMake.
Actually, making one .so depend on all boost .so files is quite possible (but might not actually help you). I've just tried this:
$ export BOOST_ROOT=/home/ghost/Work/Boost/boost-svn
$ g++ -shared -Wl,-soname,libboost.so -o libboost.so $BOOST_ROOT/stage/lib/libboost_program_options.so
$ g++ -L . -I $BOOST_ROOT first.cpp -lboost -Wl,-R$BOOST_ROOT/stage/lib
$ LD_LIBRARY_PATH=.:$BOOST_ROOT/stage/lib ./a.out
And it did work. However, note that dancing with -R and LD_LIBRARY_PATH. I don't know an way how you can include the path to Boost .so inside your libboost.so so that they are used both for linking and actually running the application. I can include rpath inside libboost.so just fine, but it's ignored when resolving symbols for the application.