Imagine we have an executable exe that depends on shared library foo, which in turn depends on shared library bar: exe needs libfoo.so and foo needs libbar.so.
Then, bar is changed in a source-compatible manner (i.e. API unchanged), but not in a binary-compatible manner (i.e. ABI changed). We must therefore relink (or face segmentation faults).
The question is: What exactly should be recompiled/relinked?
Let me make that a little more specific. To link exe, we need not add the compiler option "-lbar", whereas we do need that option for linking foo:
gcc -fPIC -shared -I. -Ibar -lbar -o libfoo.so foo.o
gcc -fPIC -I. -Ifoo -Ibar -lfoo -o exe exe.o
Therefore, is it sufficient to relink foo, and leave exe as is?
Do foo and exe also need recompilation (if bar is changed, but its API was unchanged)?
Lastly, I have tried leaving foo unchanged, but relinked exe with the added "-lbar" compiler option:
gcc -fPIC -I. -Ifoo -Ibar -lfoo -lbar -o exe exe.o
This did eliminate the segmentation fault, which could imply that everything is then correctly linked... or it could by shear luck (because segmentation faults do not always occur, even though there is an inherit memory problem).
Is it allowed to do that, i.e. leave foo unchanged, but explicitly link exe against bar?
Somewhat related readings: [1][2][3][4][5]
Until an expert answers, these are my own hypotheses:
Given that the linker does not require "-lbar" for exe, I'm inclined to conclude that exe does not need re-linking as the linker apparently does not need bar. However, I am very unsure about this, as this assumes that the linker does not find bar via foo. (After all, ldd libfoo.so will show me bar, so I'd say that the linker may possess the knowledge that it needs bar.)
No, they do not. The sources are compatible, so the symbols inside the compiled (object) files should still be correct/compatible. The executable does, however, need relinking, because the location of those symbols inside the object files has changed.
It appears that that is allowed... But I'm not sure. Also: any executable/library that depends on foo other than exe will still face segmentation faults, so this would be bad-practice, would it not?
Related
Alternate title: Why does my dylib include extra exported symbols when compiled by Xcode vs Makefile?
My company builds a c++ dynamic library (dylib) on the Mac using clang and we recently ported our hand crafted Makefile to the CMake build system and are now using the generated Xcode projects. After ensuring that all the compiler/linker flags and environment variables matched exactly between the two systems, we noticed that the dylib created by CMake/Xcode was slightly larger. Closer examination showed that it contained some additional exported symbols (from templated functions that were never referenced and therefore should never have been instantiated - the specific templates had their definitions and specializations in the source files as we use explicit instantiation frequently, although in this case they were not explicitly instantiated). Examining the disassembly of some of the object files showed slight instruction differences as well. The only thing that got the libraries to match in size and symbols exactly was to match the order of the compiler flags exactly. This appears to demonstrate some order dependent interaction between compiler flags which seems like a compiler bug or at least poorly documented behavior.
For this specific issue, these were the compiler invocations:
clang++ -fvisibility=hidden -fvisibility-ms-compat -c foo.cpp -o foo.o
clang++ -fvisibility-ms-compat -fvisibility=hidden -c foo.cpp -o foo.o
And this was the linker invocation:
clang++ -dynamiclib -o libfoo.dylib foo.o
Displaying the exported symbols with:
nm -g libfoo.dylib
showed the differences. I submitted this LLVM Bug.
Are there ever any valid situations where compiler flag ordering matters?
Microsoft's compilers and pretty much everyone else's have traditionally had very different models for symbol visibility in the object file. The former has for a long time used C and C++ language extensions to control symbol emission by the compiler, and by default not exporting symbols.
It seems likely that -fvisibility=hidden and -fvisibility-ms-compat are mutually exclusive, and that the compiler honours the last one see on its command-line.
In all fairness, there is little documentation for -fvisibility-ms-compat to be had - other than the commit adding it to clang.
I have a project that I'm working on making run with CUDA. For various reasons, it needs to compile an executable either with or without GTK support, without recompiling all of the associated files. Under C, I accomplished this by compiling a base version of the objects to *.o and a GTK version of the objects to *.gtk.o. Thus, I can link to that library and if it needs to use GTK it will pull in those functions (and their requirements); if it doesn't it won't touch those objects.
Converting to nvcc has caused some issues: it works in either always or never GTK mode; but if I compile the libraries with the additional GTK objects, it refuses to ignore them and link a GTKless executable. (It fails with errors about being unable to find the cairo functions I call.)
I'm guessing that nvcc is linking to (at least one of) its helper functions embedded in the object, which is causing the linker to resolve the entire object.
Running ar d <lib> <objects.gtk.o> to manually strip them from the library will "fix" the problem, so there isn't a real dependency there.
I'm compiling/linking with
/usr/local/cuda/bin/nvcc --compiler-options -Wall --compiler-options -pipe
-rdc=true -O0 -g -G -I inc -I inc/ext -arch compute_20 -o program
program.cu obs/external.o libs/base.a libs/extra.a libs/core.a -lm
How can I get nvcc to ignore the unneeded objects?
How can I get nvcc to ignore the unneeded objects?
Before you can achieve that, you need to understand which symbol is causing the *.gtk.o objects to be pulled in from the library when they shouldn't be.
The way to do that is to run link with -Wl,--print-map, and look for linker messages such as:
Archive member included because of file (symbol)
libfoo.a(foo.o) main.o (foo)
Above, main.o referenced foo, which is defined in libfoo.a(foo.o), which caused foo.o to be pulled in into the main binary.
Once you know which symbols cause xxxx.gtk.o to be pulled into the link, searching the web and/or NVidia documentation may reveal a way to get rid of them.
My requirement is to work on some interface .h files. Right now I have .h and .cpp/.cc files in my project.
I need to compile it into shared 64-bit linux compatible library (*.so), using NetBeans/ Eclipse on Linux Fedora.
Since the GCC C++ ABI conventions did slightly change (in particular because of C++ standard libraries evolution, or name mangling convention) from one GCC version to the next (e.g. from g++-4.4 to g++-4.6) your shared library may be dependent upon the version of g++ used to build it
(In practice, the changes are often small inside g++, so you might be non affected)
If you want a symbol to be publicly accessible with dlsym you should preferably declare it extern "C" in your header files (otherwise you should mangle its name).
Regarding how to make a shared library, read documentation like Program Library Howto.
See also this question
And I suggest building your shared libraries with ordinary command-line tools (eg Makefile-s). Don't depend upon a complex IDE like NetBeans/ Eclipse to build them (they are invoking command-line utilities anyway).
If you are compiling a library from the 3 C++ source files called a.cc, b.cc, and c.cc respectively;
g++ -fpic -Wall -c a.cc
g++ -fpic -Wall -c b.cc
g++ -fpic -Wall -c c.cc
g++ -shared -Wl,-soname,libmylib.so.0 -o libmylib.so.0.0.0 a.o b.o c.o
Then you install the library using ldconfig, see man 8 ldconfig
you can then compile the program that uses the libary as follows (but be sure to prefix extern "C" before the class declarations in the header files included in the source code using the library.)
g++ -o myprog main.cc -lmylib
I have tried these compile options with my own sample code, and have been successful.
Basically What is covered in Shared Libraries applies to C++, just replace gcc with g++.
The theory behind all of this is;
Libraries are loaded dynamically when the program is first loaded, as can be confirmed by doing a system call trace on a running program, e.g. strace -o trace.txt ls which will dump a list of the system calls that the program made during execution into a file called trace.txt. At the top of the file you will see that the program (in this case ls) had indeed mmapped all the library's into memory.
Since libraries are loaded dynamically, it is unknown at link time where the library code will exist in the program's virtual address space during run time. Therefore library code must be compiled using position independent code - Hence the -fpic option which tells the translation stage to generate assembly code that has been coded with position independent code in mind. If you tell gcc/g++ to stop after the translation stage, with the -S (upper case S) option, and then look at resulting '.s' file, once with the -fpic option, and once without, you will see the difference (i.e. the dynamic code has #GOTPCREL and #PLT, at least on x86_64).
The linker, of course must be told to link all the ELF relocatatable object types into executable code suitable for use as a Linux shared library.
Say I had a library called libfoo which contained a class, a few static variables, possibly something with 'C' linkage, and a few other functions.
Now I have a main program which looks like this:
int main() {
return 5+5;
}
When I compile and link this, I link against libfoo.
Will this have any effect? Will my executable increase in size? If so, why? Do the static variables or their addresses get copied into my executable?
Apologies if there is a similar question to this or if I'm being particularly stupid in any way.
It won't do anything in a modern linker, because it knows the executable doesn't actually use libfoo's symbols. With gcc 4.4.1 and ld 2.20 on my system:
g++ linker_test.cpp -static -liberty -lm -lz -lXp -lXpm -o linker_test_unnecessary
g++ linker_test.cpp -static -o linker_test_none
ls -l linker_test_unnecessary linker_test_none
They are both 626094 bytes. Note this also applies to dynamic linking, though the size they both are is much lower.
A library contains previously compiled object code - basically a static library is an archive of .o or .obj files.
The linker looks at your object code and sees if there are any unresolved names and if so looks for these in the library, if it finds them it includes the object file that contains them and repeats this.
Thus only the parts of the static library that are needed are included in your executable.
Thus in your case nothing from libfoo will be added to you executable
EDIT: I suppose I should clarify, in case it matters. I am on a AIX Unix box, so I am using VAC compilers - no gnu compilers.
End edit
I am pretty rusty in C/C++, so forgive me if this is a simple question.
I would like to take common functions out of a few of my C programs and put them in shared libraries or shared objects. If I was doing this in perl I would put my subs in a perl module and use that module when needed.
For the sake of an example, let's say I have this function:
int giveInteger()
{
return 1034;
}
Obviously this is not a real world example, but if I wanted to share that function, how would I proceed?
I'm pretty sure I have 2 options:
Put my shared function in a file, and have it compile with my main program at compile time. If I ever make changes to my shared function, I would have to recompile my main program.
Put my shared function in a file, and compile it as a shared library (if I have my terms correct), and have my main program link to that shared library. Any changes I make to my shared library (after compiling it) would be integrated into my main program at runtime without re-compiling my main program.
Am I correct on that thinking?
If so, how can I complish either/both of those methods? I've searched a lot and I seem to find information how how I could have my own program link to someone else's shared library, but not how to create my own shared functions and compile them in a way I can use them in my own program.
Thanks so much!
Brian
EDIT: Conclusion
Thanks everyone for your help! I thought I would add to this post what is working for me (for dynamic shared libraries on AIX) so that others can benefit:
I compile my shared functions:
xlc -c sharedFunctions.c -o sharedFunctions.o
Then make it a shared object:
xlc -qmkshrobj -qexpfile=exportlist sharedFunctions.o
xlc -G -o libsharedFunctions.so sharedFunctions.o -bE:exportlist
Then link it another program:
xlc -brtl -o mainProgram mainProgram.c -L. -lsharedFunctions
And another comment helped me find this link, which also helped:
http://publib.boulder.ibm.com/infocenter/comphelp/v7v91/topic/com.ibm.vacpp7a.doc/proguide/ref/compile_library.htm
Thanks again to all who helped me out!
Yeah you are correct. The first is called a static library, while the second is called a shared library, because the code is not bound to the executable at compile time, but everytime again when your program is loaded.
Static library
Compile your library's code as follows:
gcc -c *.c
The -c tells the program not to link the object file, but just leaves you with object files for each .c file that was compiled. Now, archive them into one static library:
ar rcs libmystuff.a *.o
man ar will tell you what the rcs options mean. Now, libmystuff.a is a archive file (you can open it with some zip-file viewers) which contain those object files, together with an index of symbols for each object file. You can link it to your program:
gcc *.c libmystuff.a -o myprogram
Now, your program is ready. Note that the order of where the static libraries appear in the command matter. See my Link order answer.
Shared library
For a shared library, you will create your library with
gcc -shared -o libmystuff.so *.c
That's all it takes, libmystuff.so is now a shared object file. If you want to link a program to it, you have to put it into a directory that is listed in the /etc/ld.so.conf file, or that is given by the -L switch to GCC, or listed in the LD_LIBRARY_PATH variable. When linking, you cut the lib prefix and .so suffix from the library name you tell gcc.
gcc -L. -lmystuff *.c -o myprogram
Internally, gcc will just pass your arguments to the GNU linker. You can see what arguments it pass using the -### option: Gcc will print the exact arguments given to each sub process.
For details about the linking process (how some stuff is done internally), view my Linux GCC linker answer.
You've got a third option. In general, your C++ compiler should be able to link C routines. The necessary options may vary from compiler to compiler, so R your fine M, but basically, you should be able to compile with g++ as here:
$ g++ -o myapp myapp.cpp myfunc.c giveint.c
... or compile separately
$ gcc -c myfunc.c
$ gcc -c giveint.c
$ g++ -c myapp.cpp
$ g++ -o myapp myapp.o myfunc.o
You also need to include your declaration of the functions; you do that in C++ as
extern "C" {
int myfunc(int,int);
int giveInterger(void);
}
You need to distinguish between recompiling and relinking.
If you put giveInteger() into a separate (archive) library, and then modify it later, you'll (obviously) need to recompile the source file in which it is defined, and relink all programs that use it; but you will not need to recompile such programs [1].
For a shared library, you'll need to recompile and relink the library; but you will not have to relink or recompile any of the programs which use it.
Building C++ shared libraries on AIX used to be complicated; you needed to use makeC++SharedLib shell script. But with VAC 5.0 and 6.0 it became quite easy. I believe all you need to do is [2]:
xlC -G -o shr.o giveInteger.cc
xlC -o myapp main.cc shr.o
[1] If you write correct Makefile (which is recommended practice), all of this will happen automatically when you type make.
[2] There is a certain feature of AIX which may complicate matters: by default shared libraries are loaded into memory, and "stick" there until subsequent reboot. So you may rebuild the shr.o, rerun the program, and observe "old" version of the library being executed. To prevent this, a common practice is to make shr.o world-unreadable:
chmod 0750 shr.o