I've been programming a while but I still don't fully understand how the linker behaves.
For example, today I downloaded and installed a library that I want to use in my application in Linux. (It was Xerces - for parsing XML files).
I created a makefile and gave it the path to the .so and .a files in my command : -L/usr/local/lib and also told it the name of the library to include : -lxerces-c-3.1.
My application compiled fine but failed during runtime with "cannot open shared object file libxerces-c-3.1.so". Why would this be the case, when I properly give it the path and name in the makefile?
I then added the library path to the LD_LIBRARY_PATH variable in my .bashrc file and then it worked. That's fine , but if I now remove the path to the library in my makefile and don't even include the name of the library , it still works.
I'm confused as to what is going on here. How can it still find the correct library just by assigning the path to the LD_LIBRARY_PATH variable and will only work if I do so? I have read elsewhere to not even use LD_LIBRARY_PATH.
I appreciate any answer for this. The question is a bit long and hopefully not off-topic but I hope others can learn from it too. Thanks
compilation and running are different things. :)
1) A make file contains rules on how to build your application. So when you write a rule like:
-L/usr/local/lib -lxerces-c-3.1
You are passing options to the linker. The -L option tells the linker to add additional libraries (in this case '/usr/local/lib') to the linker's search path. The -l option names the library that should be linked in.
2) When you go to run an executable, the loader needs to find all the required libraries. For example, on a Linux system you can use the ldd command to see what shared libraries are used. For example on my system:
ldd FEParser
linux-vdso.so.1 => (0x00007ffcdc7c9000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f835b143000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f835ae3d000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f835ac27000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f835a862000)
/lib64/ld-linux-x86-64.so.2 (0x00007f835b447000)
From this, you can see the name of the library that is linked to the left of the '=>' token, and the path to the library to the right. If a library is not found it will show up as missing.
Now in your case, you were able to successfully compile and link your program due to giving the path and library name to use. You were not able to run the program due to the loader not being able to find the library at run time.
You have several options here:
1) you can move the library in a directory that is in the loaders search path.
2) you can modify LD_LIBRARY_PATH which adds additional directories to the loaders search path. Additionally, the directories specified in LD_LIBRARY_PATH will be passed to the linker (it will be appended after all -L flags). You do need to be careful with this, as if you set it globally (say in .bashrc), then it will effect all compilations that you do. You may or may not want this behavior.
3) As others have specified you can use -Wl,rpath=...., but it is deprecated, and I rarely use it.
4) If you need the library installed in an unusual location, you can add a create a file under /etc/ld.so.conf.d that contains, for example (file is yaml.conf):
# yaml default configuration
/opt/yaml/lib
At system boot, the loader reads all the files in this directory and appends the paths to the loader path. If you make a modification to this directory you can use the ldconfig command to cause the loader to reprocess /etc/ld.so.con.d. N.B. I know this works on centOS and Ubuntu flavors of GNU/Linux - can't speak authoritatively on other flavors.
Compile your program with -Wl,-rpath=/usr/local/lib. This way you'll add /usr/local/lib to runtime library search patch of your program and you won't need the LD_LIBRARY_PATH
Warning: since modern dynamic linkers consider rpath as deprecated you can set also runpath (which supersedes it) by specifying -Wl,-rpath=/usr/local/lib,--enable-new-dtags
There is no mystery here. Default library paths for linker (one you call to make your executable file, for example, ld) and runtime linker (one which is responsible for loading shared libraries when you execute your program, for example, ld.so) are different. Runtime linker uses LD_LIBRARY_PATH, while linker uses whatever was configured when ld was build.
In your case, looks like /usr/local/lib is part of one, but not another.
If you're using static linking, all you have to do is tell the linker where your library is at compile/link time. The library (or as much of it as is necessary) gets copied into your executable, and your executable is standalone.
But for various reasons, these days we generally use dynamic linking, not static linking. With dynamic linking, you have to tell the linker where to find the library at compile/link time, and the dynamic linker (ld.so) has to be able to find the library at run time.
If the library was in one of the standard places (/lib, /usr/lib, etc.) there's no problem. But if you linked against a library in a nonstandard place, in general, you have to add that nonstandard place to your LD_LIBRARY_PATH, so that the runtime linker will always be able to find it.
Related
I'm using omake to build a native binary executable. After it links and i try to run it, it fails to run giving the following error:
error while loading shared libraries: libprotobuf-c.so.1: cannot open shared object file: No such file or directory
Is there a way, at compile time, to tell the executable to pick up the static version: libprotobuf-c.a and not the shared on?
I'm not familiar with omake but I believe flag to ocamlc you are looking for is dllpath:
-dllpath dir
Adds the directory dir to the run-time search path for shared C libraries. At link-
time, shared libraries are searched in the standard search path (the one corresponding
to the -I option). The -dllpath option simply stores dir in the produced executable
file, where ocamlrun(1) can find it and use it.
If you can configure omake to pass the appropriate -dllpath argument to ocamlc, you should be good to go.
Under the hood I believe this is using the rpath feature (runtime library search path) of ld, the GNU linker. See https://linux.die.net/man/1/ld . There is also a chrpath utility to change the rpath of an already-built executable.
Another option is running your executable with LD_LIBRARY_PATH set so that the shared library is on the load path. You could also install the shared library system wide if appropriate. A final option is to load the library manually when your application boots using dlopen.
The correct choice depends on how you will distribute this and to who you will distribute this, if at all. Keep in mind if you use rpath/dllpath the end user is unlikely to have protobuf installed in the same location you do.
There doesn't seem to be a global flag that can be passed to ld, the linker, that enforces the linker prefer static libraries to dynamic ones when available. In my case, I set the library name explicitly like so:
OCAML_LINK_FLAGS += -cclib -l:libprotobuf-c.a
Is it possible to compile a program with g++ so that shared libraries etc are "included" with the executable?
I have a c++ program that I'd like to compile and run at another location where I'm missing some libraries and don't have install access.
The main reason I couldn't find answers for this is probably that I don't know how to call it..
No, it's not possible.
Either link statically (with -static) so it doesn't use any shared libraries, or copy the shared libraries to the other location along with the executable.
Since the shared libraries will not be in the dynamic loader's usual search paths you'll need to ensure they can be found, either by setting the LD_LIBRARY_PATH environment variable when running the program, or by setting an RPATH in the executable when you build it.
(Assuming you're using the GNU linker ...) To set an RPATH in the executable link with '-Wl,-rpath,$ORIGIN' (the quotes are important, to stop $ORIGIN being expanded by the shell). That means the loader will look for shared libraries in the same directory as the executable itself.
See https://gcc.gnu.org/onlinedocs/libstdc++/faq.html#faq.how_to_set_paths and https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dynamic_or_shared.html#manual.intro.using.linkage.dynamic for more information.
I currently have my Other Linker Options as:
Unfortunately, this means that I can only ever run the executable when libsfml-dev is installed, which is undesirable for game development. Whenever I even change the Other Linker Options or Search Directories>Linker will cause an error along the lines of:
error while loading shared libraries: libsfml-graphics.so.2.2.0: cannot open shared object file: No such file or directory. I assume this error happens because the libraries are relative, so it is completely portable.
EDIT:
I am using Code::Blocks (as you can see from the image), and I would like to know how I can link to the libraries relatively. They link fine when it is run from Code::Blocks, but when I run the executable I get the above error. My current Code::Blocks options are:
Linker Options:
(Other linker options is empty)
Search Directories:
Your first linker options would be the correct ones.
The Library libsfml-dev is only required on the compile machine.
There must also be another package which actually supplies the .so files. That one will also be required on the machine running the executable unless you package those .so files with your executable. In that case you probably need a script that sets LD_LIBRARY_PATH to the local directory containing the packaged .so files and then executes your program.
I'm trying to use boost::filesystem::exists function. When I'm trying to link, I'm getting
/usr/local/include/boost/filesystem/operations.hpp:289: undefined reference to `boost::filesystem::detail::status(boost::filesystem::path const&, boost::system::error_code*)'
error.
I googled for a while and found only "link-with-boost" answers. In my case -lboost_system and -lboost_filesystem specified for linker, but it doesn't seem to work.
Actually I can use boost::filesytem::path (for example), but when I'm trying to use anything, which needs boost/filesystem.hpp header, I'm getting linker errors.
Any ideas?
P.S. I'm using gcc-4.6.4 and boost lib installed from repos, but I assume gcc-4.6.4 is default gcc version for my ubuntu 12.04. I guess I don't need to compile boost from source?
There are many options to the linker (ld) to specify the search path to resolve shared libraries, man ld will give you all the options. Suppose you have boost installed in /usr/local/lib, you could add one of these options to gcc to pass along to the linker:
-L=/usr/local/lib
Directories specified on the command line are searched before the default directories. All -L options apply to all -l options, regardless of the order in which the options appear.
If searchdir begins with "=", then the "=" will be replaced by the sysroot prefix, a path specified when the linker is configured. The -L option only sets a compile-time library search path; if you want a shared library to be found at runtime then its directory must be known at runtime.
-Wl,-rpath,/usr/local/lib
Add a directory to the runtime library search path. This is used when linking an ELF executable with shared objects. All -rpath arguments are concatenated and passed to the runtime linker, which uses them to locate shared objects at runtime. The -rpath option is also used when locating shared objects which are needed by shared objects explicitly included in the link; see the description of the -rpath-link option. If -rpath is not used when linking an ELF executable, the contents of the environment variable "LD_RUN_PATH" will be used if it is defined.
Another alternative is to add to your LD_LIBRARY_PATH the location of your boost libraries.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
The linker (ld) uses LD_LIBRARY_PATH as one of the search paths to locate required shared libraries.
You can read more about the linker and shared libraries here.
To fully understand why your installation isn't finding the boost libraries by default you might find this answer at stackexchange informative.
This SO answer suggests using boost m4.
I am trying to get my head around the way shared libraries work in the c++ unix environment. I understand we only need header files and no shared libraries specification when compiling code. But if I want to create an executable or shared library from my compiled files, do I need to specify shared library dependencies (those are dynamic)? And do the paths of shared libraries need to match the path at runtime loading?
I am using Linux 2.6.18-164.11.1.el5 #1 SMP x86_64 GNU/Linux
I am having a problem where my code is not able to pick up a library at runtime. I have tried setting LD_LIBRARY_PATH and PATH. But at runtime when I run the executable, I get the following error:
Error: librc.so: cannot open shared object file: No such file or directory
Sam
The headers are only for the compile phase. At link time, you usually have to specify which shared libs you are going to link to. You might see -L options to set locations to where shared libraries reside, and/or -l to specify which libraries to link. There is usually also a switch on the command line to alert the linker as to whether you are using thread-safe versions of the libs or the 'regular' ones, and another switch to specify dynamic linking.
At run time, whether you are starting the program that uses the libs, or running ldd to find out what it needs, the OS has a system for locating .so files, and this can vary from one unix version to another. The LD_LIBRARY_PATH variable specifies where to look for .so files, but may not be the full story, depending on the exact unix version in question. Also, you probably don't want to fiddle around with modifying LD_LIBRARY_PATH except from a throw-away shell, since it has system wide effects. A better option is to check it the 'missing' .so files are or are not on the existing path set by LD_LIBRARY_PATH, and if not, try putting copies of them somewhere on that path.
At run time, dynamic libraries are searched:
in a path recorded in the executable (under linux with -rpath at link time, under Solaris with -R, using $ORIGIN in a directory name allows to specify a directory relative to the directory containing the executable)
in the LD_LIBRARY_PATH (or equivalent, there are sometimes 64/32 bits variant). If a path has been recorded in the executable, LD_LIBRARY_PATH may not searched (under Linux it is searched after the recorded path if the executable has been linked with the option --enable-new-dtags; I don't remember Solaris behavior for now)
in a set of system dependant directories (Linux allows to specify them in /etc/ld.so.conf and has a cache, see ldconfig)