I have a solaris shared object (common.so file) that runs as part of a third party application(app.exe). I do not have access to the source code of the application. To the so, I need to add a capability to post http requests. My plan is to use libcurl with openssl. The tricky part is that app.exe already has a dependency on an older version of curl (7.14) which does not support ssl with tls v1.2.
I downloaded the source code and built curl (7.55.1) and openssl .a files. I was also able to build common.so with static dependency on these archive files. ldd does not show dependency on curl or ssl .so files and it also does not report any "Symbol not found" errors.
With this result, I was expecting my version of curl to be invoked when the so runs as part of the application but it did not. Instead curl_version() displays older version and I get the error unknown ssl protocol error
I am using solaris studio compiler. The application does not depend on curl libraries directly but depends on a different .so file which exports symbols with the same names as curl. I realized from nm and I am assuming that this .so file also links curl statically.
When app.exe loads the two SO in question, it adds the functions of each to its symbol table. Now one of two possible scenarios must occur (which one actually is OS detail but irrelevant here...):
The newer version is loaded before the older version and the older one overwrites the newer one's entries.
The newer version is loaded after the older one, and as there are already entries in the table, it is not updated any more.
Now the cleanest solution – if applicable, i. e. if you have access to the sources – would be updating the other SO to use the newer version of curl. If doing so, consider creating curl as a new SO instead of linking it statically into both common.so and the other SO.
Otherwise, to solve the issue directly, you would have to switch the order the app loads the SO, which probaly would mean to decompile app.exe and rebuild it with the linkage order of the SO changed. Problem then: there might be two versions of app.exe around then, and you must make sure that only the correct one is distributed with your common.so. Potential source of trouble, too...
Apart from this, best I can think up is a workaround:
You might modify the curl prefix from curl_ to e. g. curl_755_. Do not try this by hand, though, you most likely would go crazy this way... Use a script (e. g. perl or python) for instead. And if you ever update the curl sources, you can just run it again then...
Faster version, but potentially unsafe: just replace any occurences. Safer: Identify the externally visible functions (and possibly global objects) in a first run and keep them in a map, then replace any occurence of a string contained in the map with the corresponding value.
The latter approach (map) would additionally allow to generate macros in the header files of the following form:
#define curl_xyz curl_755_xyz
These macros allow the sources of common.so to just look like as if the original sources of curl were used...
Related
First I'll explain the big picture :
I am creating an application where I separated most of the features in different libraries. One of them contains some classes that use curl. I actually use Ubuntu 64 bit to develop and test it, but my production environment is a NAS with an ARM processor. I intend to, later, also make it for windows.
Where I am now :
My application is running on linux and the ARM-based NAS. However, I don't link with curl, I use curl from command-line internally to do what I need. This has some drawbacks :
As a programmer, I consider it an ugly practice. I should link to libcurl, this is the normal and clean way of using features from other software components.
This implies that curl executable is installed on the target. Not only I don't want to rely on this but also after a system upgrade on the NAS, I found out that I can't rely anymore on this.
What I want
As I intended anyway to use curl as a library, I first tried to do it the "soft way" : dynamic linking. Although it worked on my development environment, it didn't work on the production one because I found out that the curl library installed there doesn't work as expected.
So my next try was using libcurl as a static library. I also considered it to be the best future-proof option for me as it would make sure that, either on the NAS or on any other system, the library I will be using will always be the same.
The problems I've solved so far
Including a static library in another static library
This is already well documented in other answers here in StackOverflow : How to merge two "ar" static libraries into one
I did this to create a combined library of my own one and libcurl and as far as I've checked, it worked.
Building libcurl statically from source
There also are other answers that cover this topic, and I managed to create a libcurl.a that has libcurl features only.
The problems I am still trying to solve
Building libcurl statically with all its dependencies
There are some information regarding this, for example here. I did what was suggested, calling the configure script with --disable-shared and --enable-static. I also did the "rm src/curl" before make and called make with LDFLAGS=-all-static, but the resulting libcurl still missed its dependencies (openssl, pthreads, zlib...).
If I could solve this problem, it would answer my question. But not having succesfully done that, I tried another approach :
Manually merging all libcurl dependencies in a final lib
As I did for merging my library with libcurl into a new library, I also tried do add to it curl dependencies : zlib and openssl. So I compiled both from source to create static libraries and included them in the merge. I was not able to fully check the result as it seems that another one is missing : pthread. And I was not able to find pthread for downloading - compiling - static linking.
Looking at the big picture, my main problem is : how to I include curl in my final application so that there is no external dependency to it?
I think that if either of my two remaining problems would be solved, I would be solving my main problem. But if it is not the case, I also would be glad to hear from someone who knows a better way of solving this, or ideally already solved a similar issue.
I compiled Intel TBB from source using GCC. It generates a libtbb.so and littbb.so.2. Looks like the .so.2 file is the real shared library, libtbb.so just contains a line of text
INPUT (libtbb.so.2)
What is the purpose to generate these two files instead of one? For the INPUT (libtbb.so.2), what is the syntax? I want to know more about it.
Usually when you build shared objects (.so) then you also take care of versions by adding suffixes such as mylib.so.2.3.1. To make sure your programs can load this lib or other later versions you create links with names
mylib.so -> mylib.so.2.3.1
mylib.so.2 -> mylib.so.2.3.1
mylib.so.2.3 -> mylib.so.2.3.1
So, everything after .so represents version.sub-version.build (or similar)
Also, it is possible for more than one version of the same lib to coexist with this scheme, and all that is necessary to switch programs to using a particular version is to have the appropriate links in place.
Dynamically linked ELF binary (whether another library or an executable) uses shared-object name or soname to identify the library that the executable should be linked against upon execution.
When a library created as an ELF shared library, the compile-time link editor inserts a DT_SONAME field in the executable which the library's SONAME into the library itself. The DT_SONAME is defined in the ELF standard as:
This element holds the string table offset of a null-terminated
string, giving the name of the shared object. The offset is an index
into the table recorded in the DT_STRTAB entry. See ‘‘Shared Object
Dependencies’’ below for more information about these names.
So now when an executable is create the SONAME is embedded into it. When when the executable is run is used by the linker to look for the library in the files in the predifined locations for dynamic library. The predefined location in windows would be wherever DLLs reside. In Linux and Mac OS X and other System V compatible systems they would be /lib and /usr/lib and possibly other spots, it depends on the linker used, and can be defined in linkers own configurations.
In all events the linker looks to see if the library named in soname entry is present in any of those locations, if it is it will use it.
Note that the standard says that the soname is a STRING and the versioning conventions became a defacto standard after the fact and goes something like this:
Make the soname to be libmyname.so.A and make the library filename be libmyname.so.A.B or libmyname.so.A.B.C (under MacOSX it's libmyname.A.B.dylib). Create a softlink from libmyname.so.A.B[.C]? to libmyname.so.A.
A is kept the same while the library's ABI stays the same.
B (or B.C) becomes the minor version.
Under Linux it's really common that the library version would be the same as the package version number. This has its pros and cons.
libtool formalization
GNU libtool is used a lot to build dynamic libraries, and has a more formal versioning system and has strong logic for it. The libtool versioning system for sonames works very well and is adopted by complex libraries to keep things straight.
Under libtool, the versioning is as under:
libmylib-current.release.age
Under libtool the idea is that as libraries evolve they will add and remove functionality.
Let's say you are developing a library. Start by using a version as 0.0.0.
Now let's say you fix a few bugs, you would only increase the release number.
So new name would be come libmylib.0.1.0 or libmylib.0.2.0 etc.. for every release that just fixes bugs but doesn't change any of the ABI.
Along the way you say. Ugh! I could've done this subfunctionality better, So you add a new set of functions to do something better, but because others are still using your library so you still leave the old (deprecated) functionality in there.
The rules are as under:
Start with version information of ‘0:0:0’ for each libtool library.
Update the version information only immediately before a public
release of your software. More frequent updates are unnecessary, and
only guarantee that the current interface number gets larger faster.
If the library source code has changed at all since the last update,
then increment revision (‘c:r:a’ becomes ‘c:r+1:a’).
If any interfaces have been added, removed, or changed since the last
update, increment current, and set revision to 0.
If any interfaces have been added since the last public release, then increment age.
If any interfaces have been removed or changed since the last public
release, then set age to 0.
You can read more about it in the libtool documentation
Update ...
The following was a comment that my explanation has an error. It does not, which, requires a bit more detail than can be put into an answer comment, so see below.
Original objection
There is an error here: on linux, the version is of the form
libmylib.(current-age).release.age, where the parentheses indicate an
expression to be evaluated. For example GLPK 4.54 with
current:revision:age = 37:1:1 on linux installs the library file
libglpk.so.36.1.1. For more info, see, e.g.,
<autotools.io/libtool/version.html>.
Rebuttal
TLDR: autotools.io's not authortative source.
Explanation
Whilst the Flameeyes is an amazing developer and he is one of Gentoo maintainers, it was he who made the mistake, and created a "rule of thumb" loose interpretation of the libtool spec. While this is not going to break systems 99% of the time, if we were to follow the ad-hoc way of updating current:
The rules of thumb, when dealing with these values are:
Always increase the revision value.
Increase the current value whenever an interface has been added,
removed or changed.
Increase the age value only if the changes made to the ABI are
backward compatible.
he then goes on to say that maintaining multiple versions of Gtk it would be best to just append the library version into the library NAME and simply dump the version number. (as they do in GTK+):
In this situation, the best option is to append part of the library's
version information to the library's name, which is exemplified by
Glib's libglib-2.0.so.0 soname. To do so, the declaration in the
Makefile.am has to be like this:
lib_LTLIBRARIES = libtest-1.0.la
libtest_1_0_la_LDFLAGS = -version-info 0:0:0
Well that's just crockpot approach to mucking up the power of dynamic linking and symbol resolution versioning completely moot!. He's saying just turn it off. Horse boogers! No wonder even experienced developers have had a hard time building and maintaining open source projects and we are constantly running into binaries dying every time new versions of libraries are installed (because they clobber each other).
The libtool versioning approach is VERY WELL THOUGHT OUT. It is an algorithm and its steps are ordered instructions 1 to 6 are to be followed every time there is an update to the code of a dynamic linked library.
For new and current developers, please read them carefully and visualize what will happen to the library version number throughout the life of your amazing software. If you do you will notice that every piece of previously linked software will always use the most current and accurate version of your amazing library correctly, and none of them will ever clobber or stomp on each other, AND you never have to add a blooming number in the name of your library (unless it's for pleasure or esthetics).
I can't be the only one to run into this.
I have a C++ application that needs to link with one third-party and another static library set in an SDK. The SDK has, for some hideously frustrating reason, recompiled a subset of that same third-party library into their own (renamed) lib, although the symbols themselves are named the same and they are not encapsulated within a namespace. My application itself depends upon the same third-party library.
I've considered a few options, but maybe I'm missing something and hopefully a fresh look will help me out. Perhaps I'm close and someone will know the next step for one of these . I'll enumerate what I've tried and the shortcomings of each solution so far:
Link with both.
I get about 2500 lines of symbol redefinition / size change warnings and errors. This is when I first found that they defined the same symbols. I'm trying to recompile OpenSSL with g++ and drop it into a namespace at the moment...see edit below...
Link with the SDK only.
I get undefined symbols that my own code depends upon - this is when I found that their recompile of the third party lib is a subset, or at least was configured with one module disabled.
Link with the third party lib only.
I have a couple of undefined symbols reported by the SDK - one of them is actually a #define in a header file within the third party lib, so all references in the third party lib resolve to the definition, but references outside there do not. I moved that into the c file, which resolves that, however I still have two unresolved functions I can't find anywhere. This is the closest I've gotten so far.
Strip conflicting symbols from one lib and link in both.
So far this hasn't worked. It could be a version issue between the lib statically linked in the SDK and the versions I've tried using of the third-party lib, but it looks like some functions were moved between symbols, so by removing a symbol, I inadvertently remove a function that I need elsewhere. There doesn't seem to be a perfect mapping between functions in symbols in the SDK vs functions in symbols in the third-party lib. Is it plausible to strip functions without having to manually adjust addresses?
I've been examining symbols in libs with:
nm -C --defined-only lib<name>.a
And extracting entire objects with:
ar -x lib<name>.a <objname>.o
Hopefully this will also help others who have had to link with third-party libs that conflict with one another. For the sake of specifics, the third-party lib is OpenSSL, and the SDK is Opsec - libcpopenssl.a is the offending lib in Opsec.
**EDIT- A late entry possible workaround may be to recompile OpenSSL with g++ and put the whole thing in a namespace, and then link both libs. I'm trying that now...more to come...
A Google search indicate that SSL_get_peer_dh and DH_dup are really additions from libcpopenssl.a, and they don't exist in my copy of OpenSSL either. So you'll really have to link that library in. Mixing both libraries together (Approach 4 above) at binary level is unlikely to work -- OpenSSL is very picky about its ABI (they usually have .so files versioned down to the minor number) so you'd have to be very lucky to have an .so that's ABI-compatible to their .a file.
My suggestion is a variation of Approach 4, but at source level: you'll have link in the Opsec libcpopenssl.a, since it's a modified version of OpenSSL which include extra symbols (and possibly other modifications), and grab the extra functions you need from the OpenSSL sources and recompile those objects with libcpopenssl.a, so they can use the functions from the Opsec version. If you're only using a few OpenSSL functions which are not exported by libcpopenssl.a, this is quite doable.
Granted, that's still a cumbersome approach, but it's a guaranteed way to get symbol compatibility, provided of course that the Opsec SDK didn't make semantic changes to OpenSSL that will break the additional OpenSSL functions that you're pulling into your project.
(I'm new at StackOverflow so I don't know if this suggestion qualifies as a proper answer, but I don't have reputation points to post comments, anyway. I'll remove this if it's inappropriate.)
If you're morbidly curious, 249 files were modified in the most recent version of OpenSSL to get it to compile. The most common issue by far was the abundance of C-style pointer casts, particularly with void*. Now I see "reinterpret_cast" in my dreams.
This didn't solve it alone though - it still needs to be placed into a namespace in its entirety, which means modifying all of the files again as well as my own internal references to it. I think I'm going to pass on this for now.
Thanks for the help everyone.
I have a large source-controlled C++ codebase which compiles and links without error on one Linux server.
I am now trying to set up the same application on a new server, so have checked out the same code on a new box.
However, when I execute an identical make command on identical code on this new box, I get errors. The cause appears to be because on the old box, shared library (.so) files are created. On the new box - which is using identical code and therefore makefiles - makes static libraries (.a).
The compiler being used appears to be the same as well - gcc-3.4.6.
Obviously, I have some config set differently somewhere but can anyone advise or where this config might be? I can't think of any small change which would cause this effect.
Note that the linker ld is part of binutils, which is delivered with the standard binaries as part of the Unix distribution you have, and is not part of the gcc suite.
Therefore, when you get from an old server to a new server, chances are that you pass from an old ld to a new ld.
Since a library is first created by the linker, it would be interested to check it out.
Note that if you suspect the compiler (since it performs the call to ld), you can write a ld executable script that just echoes the arguments it receives and then calls the real ld behind the scenes (meddling with $PATH should get you going).
It sounds natural that it is either a case of different arguments (why ?) or a different binray, figure out which and you'll be one step closer to solving your issue.
configure stuff might have generated slightly different Makefile-s.
And when you link with -lfoo, the linker first try dynamic libfoo.so then static libfoo.a.
GCC is now at version 4.6.2 so your 3.4.6 version is very old. Consider upgrading it, because GCC has made a lot of progress since.
Try using gcc -v (perhaps as make CC='gcc -v') to understand what is going on when building.
And give much more detail if you want real help. What are the actual libraries involved?
I'm building a special-purpose embedded Python interpreter and want to avoid having dependencies on dynamic libraries so I want to compile the interpreter with static libraries instead (e.g. libc.a not libc.so).
I would also like to statically link all dynamic libraries that are part of the Python standard library. I know this can be done using Freeze.py, but is there an alternative so that it can be done in one step?
I found this (mainly concerning static compilation of Python modules):
http://bytes.com/groups/python/23235-build-static-python-executable-linux
Which describes a file used for configuration located here:
<Python_Source>/Modules/Setup
If this file isn't present, it can be created by copying:
<Python_Source>/Modules/Setup.dist
The Setup file has tons of documentation in it and the README included with the source offers lots of good compilation information as well.
I haven't tried compiling yet, but I think with these resources, I should be successful when I try. I will post my results as a comment here.
Update
To get a pure-static python executable, you must also configure as follows:
./configure LDFLAGS="-static -static-libgcc" CPPFLAGS="-static"
Once you build with these flags enabled, you will likely get lots of warnings about "renaming because library isn't present". This means that you have not configured Modules/Setup correctly and need to:
a) add a single line (near the top) like this:
*static*
(that's asterisk/star the word "static" and asterisk with no spaces)
b) uncomment all modules that you want to be available statically (such as math, array, etc...)
You may also need to add specific linker flags (as mentioned in the link I posted above). My experience so far has been that the libraries are working without modification.
It may also be helpful to run make with as follows:
make 2>&1 | grep 'renaming'
This will show all modules that are failing to compile due to being statically linked.
CPython CMake Buildsystem offers an alternative way to build Python, using CMake.
It can build python lib statically, and include in that lib all the modules you want. Just set CMake's options
BUILD_SHARED OFF
BUILD_STATIC ON
and set the BUILTIN_<extension> you want to ON.
Using freeze doesn't prevent doing it all in one run (no matter what approach you use, you will need multiple build steps - e.g. many compiler invocations). First, you edit Modules/Setup to include all extension modules that you want. Next, you build Python, getting libpythonxy.a. Then, you run freeze, getting a number of C files and a config.c. You compile these as well, and integrate them into libpythonxy.a (or create a separate library).
You do all this once, for each architecture and Python version you want to integrate. When building your application, you only link with libpythonxy.a, and the library that freeze has produced.
You can try with ELF STATIFIER. I've been used it before and it works fairly well. I just had problems with it in a couple of cases and then I had to use another similar program called Ermine. Unfortunately this one is a commercial program.