Is it possible to use shared object files in a portable way like DLLs in Windows??
I'm wondering if there is a way I could provide a compiled library, ready to use, for Linux. As the same way you can compile a DLL in Windows and it can be used on any other Windows (ok, not ANY other, but on most of them it can).
Is that possible in Linux?
EDIT:
I've just woke up and read the answers. There are some very good ones.
I'm not trying to hide the source code. I just want to provide an already-compiled-and-ready-to-use library, so users with no experience on compilation dont need to do it themselves.
Hence the idea is to provide a .so file that works on as many different Linuxes as possible.
The library is written in C++, using STL and Boost libraries.
I highly highly recommend using the LSB app / library checker. Its going to tell you quickly if you:
Are using extensions that aren't available on some distros
Introduce bash-isms in your install scripts
Use syscalls that aren't available in all recent kernels
Depend on non-standard libraries (it will tell you what distros lack them)
And lots, upon lots of other very good checks
You can get more information here as well as download the tool. Its easy to run .. just untar it, run a perl script and point your browser at localhost .. the rest is browser driven.
Using the tool, you can easily get your library / app LSB certified (for both versions) and make the distro packager's job much easier.
Beyond that, just use something like libtool (or similar) to make sure your library is installed correctly, provide a static object for people who don't want to link against the DSO (it will take time for your library to appear in most distributions, so writing a portable program, I can't count on it being present) and comment your public interface well.
For libraries, I find that Doxygen works the best. Documentation is very important, it surely influences my choice of library to use for any given task.
Really, again, check out the app checker, its going to give you portability problem reports that would take a year of having the library out in the wild to obtain otherwise.
Finally, try to make your library easy to drop 'in tree', so I don't have to statically link against it. As I said, it could take a couple of years before it becomes common in most distributions. Its much easier for me to just grab your code, drop it in src/lib and use it, until and if your library is common. And please, please .. give me unit tests, TAP (test anything protocol) is a good and portable way to do that. If I hack your library, I need to know (quickly) if I broke it, especially when modifying it in tree or en situ (if the DSO exists).
If you'd like to help your users by giving them compiled code, the best way I know is to give them a statically linked binary + documentation how they can run the binary. (This is possibly in addition to giving the source code to them.) Most statically linked binaries work on most Linux distributions of the same architecture (+ 32-bit (x86) statically linked binaries work on 64-bit (amd64)). It is no wonder Skype provides a statically linked Linux download.
Back to your library question. Even if you are in an expert in writing shared libraries on Linux, and you take your time to minimize the dependencies so your shared library would work on different Linux distributions, including old and new versions, there is no way to ensure that it will work in the future (say, 2 years). You'll most probably end up maintaining the .so file, i.e. making small modifications over and over again so the .so file becomes compatible with newer versions of Linux distributions. This is no fun doing for a long time, and it decreases your productivity substantially: the time you spend on maintaining the library compatibility would have been much better spent on e.g. improving the functionality, efficiency, security etc. of the software.
Please also note that it is very easy to upset your users by providing a library in .so form, which doesn't work on their system. (And you don't have the superpower to make it work on all Linux systems, so this situation is inevitable.) Do you provide 32-bit and 64-bit as well, including x86, PowerPC, ARM etc.? If the .so file works only on Debian, Ubuntu and Red Hat (because you don't have time time to port the file to more distributions), you'll most probably upset your SUSE and Gentoo users (and more).
Ideally, you'll want to use GNU autoconf, automake, and libtool to create configure and make scripts, then distribute the library as source with the generated configure and Makefile.in files.
Here's an online book about them.
./configure; make; make install is fairly standard in Linux.
The root of the problem is that Linux runs on many different processors. You can't just rely on the processor supporting x86 instructions like Windows does (for most versions: Itanium (XP and newer) and Alpha (NT 4.0) are the exceptions).
So, the question is, how to develop shared libraries for Linux? You could take a look at this tutorial or the Pogram Library Howto.
I know what you are asking. For Windows, MSFT has carefully made the DLLs all compatible, so your DLLs are usually compatible for almost every version of Windows, that's why you call it "portable".
Unfortunately, on Linux there are too many variations (and everyone is thinking to be "different" to make money) so that you cannot get same benefits as Windows, and that's why we have lots of same packages compiled for different distributions, distro version, CPU type, ...
Some say the problem is caused by (CPU) architecture, but it is not. Even on same arch, there's still different between distributions. Once you've really tried to release a binary package, you would know how much hard it is - even C runtime library dependency is hard to maintain. Linux OS lacks too much stuff so almost every services involves dependency issue.
Usually you can only build some binary that is compatible to some distribution (or, several distributions if you are lucky). That's why releasing Linux programs in binary always screwed up, unless bound to some distro like Ubuntu, Debian, or RH.
Just putting a .so file into /usr/lib may work, but you are likely to mess up the scheme that your distro has for managing libraries.
Take a look at the linux standard base - That is the closest thing you will find to a common platform amongst linux distros.
http://www.linuxfoundation.org/collaborate/workgroups/lsb
What are you trying to accomplish?
Tinkertim's answer is spot on. I'll add that it's important to understand and plan for changes to gcc's ABI. Things have been fairly stable recently and I guess all the major distros are on gcc 4.3.2 or so. However, every few years some change to the ABI (especially the C++ related bits) seems to cause mayhem, at least for those wanting to release cross-distro binaries and for users who have gotten used to picking up packages from other distros than they actually run and finding they work. While one of these transitions is going on (all the distros upgrade at their own pace) you ideally want to release libs with ABIs supporting the full range of gcc versions in use by your users.
Related
Is static linking in Linux portable? I mean, can I use the -static option in gcc and link with every dependencies statically to have a clean output from ldd, and expect that the resulting executable will run portably in another computer with Linux installed? Of course given that the CPU architecture and the kernel version is compatible.
The short answer: Pretty much.
This will make a binary which will run on a kernel which is the same or compatible with the one for which the software was designed.
It may not take into account directory structure and if the binary expects to be able to load any external dependencies dynamically, that might not work.
Assuming there's nothing too fancy going on though, it will work fine.
This is approximately what Go's compiler does to enable shipment of binaries roughly anywhere. This also is a method for making a build forward compatible if you expect to be making OS upgrades that will be disruptive.
Additionally, these static binaries could be run in a FreeBSD kernel with Linux compatibility. As long as the kernel and user space is compatible, the binary should work.
As always, test.
Yes. The static link means it won't depend on any other library.
Maybe. You won't need to worry about dynamic library dependencies. Your statically linked libraries might use system calls or other kernel interfaces that older kernels don't have, so you'll only be forward compatible (the linux kernel has a pretty strong backwards compat policy). The only thing you might need to worry about are external files that your statically linked libraries might depend on (like localization databases and such).
C++ Buildsystem with ability to compile dependencies beforehand
Java has Maven which is a pleasure to work with, simply specifying dependencies that are already compiled, and deposited to Mavens standard directory, meaning that the location of the dependencies is standardized as opposed to the often used way of having multiple locations (give me a break, like anyone remembers the default installed directories for particular deps) of C/C++ dependencies.
It is massively unproductive for every individual developer having to, more often than not, find, read about, get familiar with the configure options/build, and finally compile for every dependency to simply make a build of a project.
What is the theoretical reason this has not been implemented?
Why would it be difficult to provide packages of the following options with a maven-like declaration format?
version
platform (windows, linux)
src/dev/bin
shared/static
equivalent set of Boost ABI options when applicable
Having to manually go to websites and search out dependencies in the year 2013 for the oldest major programming language is absurd.
There aren't any theoretical reasons. There are a great many practical reasons. There are just too many different ways of handling things in the C++ world to easily standardize on a dependency system:
Implementation differences - C++ is a complicated language, and different implementations have historically varied in how well they support it (how well they can correctly handle various moderate to advanced C++ code). So there's no guarantee that a library could be built in a particular implementation.
Platform differences - Some platforms may not support exceptions. There are different implementations of the standard library, with various pros and cons. Unlike Java's standardized library, Windows and POSIX APIs can be quite different. The filesystem isn't even a part of Standard C++.
Compilation differences - Static or shared? Debug or production build? Enable optional dependencies or not? Unlike Java, which has very stable bytecode, C++'s lack of a standard ABI means that code may not link properly, even if built for the same platform by the same compiler.
Build system differences - Makefiles? (If so, GNU Make, or something else?) Autotools? CMake? Visual Studio project files? Something else?
Historical concerns - Because of C's and C++'s age, popular libraries like zlib predate build systems like Maven by quite a bit. Why should zlib switch to some hypothetical C++ build system when what it's doing works? How can a newer, higher-level library switch to some hypothetical build system if it depends on libraries like zlib?
Two additional factors complicate things:
In Linux, the distro packaging systems do provide standardized repositories of development library headers binaries, with (generally) standardized ABIs and an easy way of specifying a project's build dependencies. The existence of these (platform-specific) solutions reduces the impetus for a cross-platform solution.
With all of these complicating factors and pre-existing approaches, any attempt to establish a standard build system is going to run into the problem described in XKCD's "Standards":
Situation: There are 14 competing standards.
"14? Riculous! We need to develop one universal standard that covers everyone's use cases."
Soon: There are 15 competing standards.
With all of that said:
There is some hope for the future. For example, CMake seems to be gradually replacing other build systems. Some of the Boost developers have started Ryppl, an attempt to do what you're describing.
(also posted in linked question)
Right now I'm working on a tool able to automatically install all dependencies of a C/C++ app with exact version requirement :
compiler
libs
tools (cmake, autotools)
Right now it works, for my app. (Installing UnitTest++, Boost, Wt, sqlite, cmake all in correct order)
The tool, named «C++ Version Manager» (inspired by the excellent ruby version manager) is coded in bash and hosted on github : https://github.com/Offirmo/cvm
Any advices and suggestions are welcomed.
well, first off a system that resolves all the dependencies doesn't makes you productive by default, potentially it can make you even less productive.
Regarding the differences between languages I would say that in Java you have packages, which are handy when you have to organize and give a limited horizon to your code, in C++ you don't have an equivalent concept.
In C++ all the libraries that can solve a symbol are good enough for the compiler, the only real requirement for a library is to have a certain ABI and to solve the required symbols, there are no automated ways that you can work to pick the right library, also solving a symbol it's just a matter of linking your function to the actual implementation, this doesn't even grant you that a correct linking phase will make your app work.
To this you can add important variables such as the library version, different implementations of the same library and different libraries with the same methods name.
An example is the Mesa library VS the opengl lib from the official drivers, or whatever lib you want that offers multiple releases and each one can solve all the symbols but probably there is a release that is more mature than the others and you can ask a compiler to pick the right one because they are all the same for its own purposes .
I have compiled my preload file on Ubuntu server (two files for x32 and x64). Where I can get list, in which I will see with what OS my compiled files are compatible and with what I should recompile for compatibility?
Thanks!
Use Linux App Checker developed by ISPRAS and The Linux Foundation. It's designed to perform cross-distro compatibility checks for Linux applications. See sample reports here.
I would start by attempting to execute the program on various Linux distributions in a virtual machine. Pick the top three most popular Linux distributions or the ones your users are most likely to have.
Also, you may be better off to distribute a statically linked binary and offer the source code to others who wish to build it themselves (if you are allowed to distribute source).
I don't know if I fully understand you, but, if my understanding is not that wrong, I'd start by ldd -v. Any OS that is architecture compatible and has the dependent libraries installed in compatible versions should work.
Next, if you plan to support more architectures, you need explicitly to know it and cross compile for every one of it.
So, you must recompile for:
1. Every different architecture.
2. When library versions are not compatible.
This last one is more tricky, since your code may need specific versions to work, but you must know it anyway from start.
Please tell me if it is not what you wanted.
Is it possible to compile a C/C++ source code that executes in all Linux distributions without recompilation?
If the answer is yes, can I use any external (non-standard C/C++) libraries?
I want distribute my binary application instead of distribute of source code.
No, you can't compile an executable the executes in all Linux distributions. However, you can compile an executable that works on most distributions that people will tend to care about.
Compile 32-bit. Compile for the minimum CPU level you're willing to support.
Build your own version of glibc. Use the --enable-kernel option to set the minimum kernel version you're willing to support.
Compile all other libraries you plan to use yourself. Use the headers from your glibc build and your chosen CPU/compiler flags.
Link statically.
For anything you couldn't link to statically (for example, if you need access to the system's default name resolution or you need PAM), you have to design your own helper process and API. Release the source to the helper process and let them (or your installer) compile it.
Test thoroughly on all the platforms you need to support.
You may need to tweak some libraries if they call functions that cannot work with this mechanism. That includes dlopen, gethostbyname, iconv_open, and so on. (These kinds of functions fundamentally rely on dynamic linking. See step 5 above. You will get a warning when you link for these.)
Also, time zones tend to break if you're not careful because your code may not understand the system's zone format or zone file locations. (You will get no warning for these. It just won't work.)
Most people who do this are building with the minimum supported CPU being a Pentium 4 and the minimum supported kernel version being 2.6.0.
There are two differences which are among installations. Architecture and libraries.
Having one binary for different architectures is not directly possible; there was an attempt to have binary for multiple archs in one file (fatelf), but it is not widely used and unlikely to gain momentum. So at least you have to distribute separate binaries for ia32, amd64, arm, ... (most if not all amd64 distros have kernel compiled with support for running ia32 code, though)
Distributions contain different versions of libraries. You're fine as long as the API does not change, you can link to that library. Some libs ensure inary backwards-compatibility within major number (so GTK2.2 app will run fine with GTK2.30 lib, but not necessarily vice versa). If you want to be sure, you have to link statically with all libs that you use, except the most basic ones (probably only libc6, which is binary-compatible accross distros AFAIK). This can increase size of the binary, and it one of reasons why e.g. Acrobat Reader is relatively big download, although the app itself is not specially rich functionality-wise.
There was a transitional period for c++ ABI, which changed between gcc 2.9 and 3 (IIRC), but the old ABI would be really just on ancient installations. This should no longer be an isse for you, and if you link statically, it is irrelevant anyway.
Generally no.
There are several bariers.
Different architectures
While a 32bit binary will run on a x86_64 system, it won't work vice versa. Plus there is a lot of ARM systems.
Kernel ABI
Kernel ABI changes very slowly, but it does change, therefore you can't really support all possible versions. Note that in some places kernel 2.2 is still in use.
What you can do is to create a statically linked binary. Such binary will include all libraries your app depends on, and it will work on all systems with the same architecture and a reasonably similar kernel version.
I'm writing a cross-platform application which is not GNU GPL compatible. The major problem I'm currently facing is that the application is linked dynamically with glibc and libstdc++, and almost every new major update to the libraries are not backwards compatible. Hence, random crashes are seen in my application.
As a workaround, I distribute binaries of my application compiled on several different systems (with different C/C++ runtime versions). But I want to do without this. So my question is, keeping licensing and everything in mind, can I link against glibc and libstdc++ statically? Also, will this cause issues with rtld?
You don't need to.
Copy the original libraries you linked against to a directory (../lib in this example) in your application folder.
Like:
my_app_install_path
.bin
lib
documentation
Rename you app for something like app.bin. Substitute your app for a little shell script that sets the enviroment variable LD_LIBRARY_PATH to the library path (and concatenate the previous LD_LIBRARY_PATH contents, if any). Now ld should be able to find the dynamic libraries you linked against and you don't need to compile them statically to your executable.
Remember to comply with the LGPL adding the given attribution to the libraries and pointing in the documentation where the source can be downloaded.
glibc is under the LGPL. Under section 6. of LGPL 2.1, you can distribute your program linked to the library provided you comply with one of five options. The first is to provide the source code of the library, along with the object code (source is optional, not required) of your own program, so it can be relinked with the library. You can alternatively provide a written offer of the same. Your own code does not have to be under the LGPL, and you don't have to release source.
libstdc++ is under the GPL, but with a major exception. You can basically just distribute under the license of your choice without providing source for either your own code or libstdc++. The only condition is that you compile normally, without e.g. proprietary modifications or plugins to GCC.
IANAL, and you should consider consulting one if you need real legal advice.
Specifying the option -static-libgcc to the linker would cause it to link against a static version of the C library, if available on the system. Otherwise it is ignored.
I must question what the heck you are doing with the poor library functions?
I have some cross platform software as well. It runs fine on Linux systems of all sorts. Build with the oldest version of software that you want to support. The glibc and libstdc++ libraries are really very backward compatible.
I have built on CentOS 4 and run it on RHEL 6 beta. No problems.
I can build on stable Debian and run it on testing.
Now, I do sometimes have trouble with some libraries if I try to build on, say old Debian and try to run it on CentOS 5.4. That is usually due to distribution configuration choices that are different, like choosing threading or non-threading.