I have some C++ code in place, which internally uses multiple third party libraries. The code compiles fine but during execution it is failing to load some shared library(libintbasic.so). Given the condition, I can not install any library into the system, the only way out is to find which function is calling that library. How could I find that out who is calling that library (my code does not call that directly).
I can not install any library into the system,
That appears to be a bogus claim: clearly you can copy your binary onto the system. Installing an additional library into the same directory is not much harder.
How could I find that out who is calling that library
There are two cases to consider:
your binary or one of the libraries it is directly linked with links to libintbasic.so. Your binary will not run at all, or
your binary, or one of the libraries it is directly linked with calls dlopen("libintbasic.so",...) and fails when that dlopen fails.
Debugging the first case is often easiest by setting LD_DEBUG=files,libs. The dynamic loader will then tell you which libraries are being loaded, and why they are required.
Debugging the second case is easy with gdb: set a breakpoint on dlopen, and execute where and info shared commands every time the breakpoint is hit.
Remove the linking option -lintbasic, you will see all the functions that needs this library in the error messages. This is not clean but it should work fine.
You may use ldd utility recursively for find all dependencies between shared libraries.
The following link also may be useful: Does ldd also show dependencies of dependencies?
Related
A shared object library required by my program is present in two paths under LD_LIBRARY_PATH, but only one of the two should be loaded by my program because of version compatibility issue. For example, if LD_LIBRARY_PATH is path1:path2 and the shared library libxxx.so is in both path1 and path2, will path1/libxxx.so be loaded or path2/libxxx.so be loaded when I run the program?
Also I would appreciate it if anyone has a better solution than reordering the paths in LD_LIBRARY_PATH. After searching on the internet I saw some solution using rpath but didn't quite understand how that works.
The directories get searched in the order they appear in LD_LIBRARY_PATH.
Keep in mind that the loader also looks elsewhere, not just LD_LIBRARY_PATH, which may or may not be a factor.
Since you asked for a "better solution", here are two:
Get this broken shared library built correctly
There is a well-established mechanism and convention for versioning shared libraries so that different versions of the same shared library get loaded. A program that needs a particular version of a shared library will load the one that it needs.
In fact, it's a safe bet that you already have a bunch of different versions of multiple shared libraries installed, due to applications in your Linux distribution that need different versions of the same shared library.
This is nothing more complicated than not using the same name for incompatible versions of the same shared library. There is a well-defined convention for naming actual shared library filenames, that work together with the linker in order to make this happen (the -soname link option, see your linker documentation for more information).
Don't use LD_LIBRARY_PATH
It is possible to link an executable and embed in the executable itself a pathname to search, first and foremost, for any shared libraries, either before or after LD_LIBRARY_PATH.
Remove all directories from LD_LIBRARY_PATH.
Use the -rpath, with/without --enable-new-dtags or --disable-new-dtags option when linking your executable. The correct set of options depends on your specific details, and specific versions of your linker. See your linker documentation for more information. You mentioned you tried to find information on this in Google, but all that Google will do is, perhaps, refer you to the same documentation that you already have: the manual pages for the linker. That's the best source for complete information on using this or any other linker option.
The best solution depends on your specific circumstances; whether you're building the executable, and/or shared libraries, how easy/hard it is to change whatever you're building; or whether you're not building anything at all, just black boxes that need to be executed.
I'm building a shared library that is loaded from an application (which I have no control of). My library uses other shared libraries which in turn uses other shared libraries, complex but not unusual.
The problem is that the main application have functions present in one of the libraries further down in the chain, to be more specific it is openLDAP that in turn uses openSSL functions:
Main app->My library->openLDAP libraries->openSSL libraries
My guess is that the main application is implementing openSSL either by a static linkage or a simple copy/paste of source code.
My question is: can I control which functions openLDAP uses from my library or do I have to recompile openLDAP with a static linkage to openSSL?
Since openSSL is updated quite frequently due to security issues I don't want a static copy of it if I don't have to. And why re-distribute a proprietary copy of openLDAP when it's part of most distributions packages...
Right now what you have is the executable overriding what would otherwise be the system's default choice of OpenSSL library. It is within the executable's rights to do that, and you can't really stop it.
Statically linking OpenSSL in your library may not really be a solution either. For one thing, what if the executable really does was to use a different version? For another, what if OpenSSL has some global variables? Now you will have two copies of the library in the same process, which is not a good idea and may cause bugs.
To me, the best answer we have on Linux is to not consider this sort of thing to be a problem. If an executable loads a bad version of OpenSSL, that is not your library's fault. At most you can check which version is loaded and refuse to run if it's known to be incompatible with your library for some reason.
My guess is that the main application is implementing openSSL either
by a static linkage or a simple copy/paste of source code.
This is wrong things. If application developer shoots on his foot then you can not do anythings.
App developer should see that your library is dependent on OpenSSL library (using ldd command) then he should not link OpenSSL again as staticly or copy paste its code.
If some functions from OpenSSL does not creating any messy and if they can be used just like any static method of any java class then only App developer should take risk of implementing that code in app.
The solution was to use RTLD_DEEPBIND in dlopen(3):
RTLD_DEEPBIND (since glibc 2.3.4)
Place the lookup scope of the symbols in this library ahead of the
global scope. This means that a self-contained library will use its
own symbols in preference to global symbols with the same name
contained in libraries that have already been loaded. This flag is not
specified in POSIX.1-2001.
This might not be the the best solution but it works in this case when the process is created by closed source software.
I have shared library code in c++ . I am replacing the existing c++ shared library with newer one with changes in c++ functions. But the application is not executing the latest code path instead it goes through the previous code path i.e I have added a print with additional messages in the newer library but the print that I get is from previous one.
I have used strings command to check the changes are reflected in the newer library and it is giving positive result.
The application is using dynamic load call to open the library. I don't have the application code with me.
Make file has the following flags for creating shared library.
${CXX} -shared -Wl,--exclude-libs,ALL,-soname,${LIB} ${LDFLAGS} -o $# $(OBJECTS)
Is there any shared library flags that I am missing ?
Anyone has idea for this strange behaviour ?
PS:I am running this in a embedded environment.
I am assuming a few things here:
1. You have not linked your shared library with your executable with "-L" option. This would never unload your shared library.
2. The library is opened during run time using dlopen
Run time library swapping can work only if the mentioned 2 points hold true for you. When a shared library is opened (using dlopen) its loaded into program's memory. So, even if you delete the library from disk (after it has been loaded), the program will still continue to run whatsoever.
When you call dlclose it "decrements the reference count on the dynamic
library handle handle. If the reference count drops to zero and
no other loaded libraries use symbols in it, then the dynamic library
is unloaded."(From the man page)
So, what you should do to implement "refresh library mechanism" is to close the existing library with dlclose and reload your new library with dlopen (make sure that the symbol names are same for the new library as that of the old one) and the symbols accessed will be of new library.
So, such things must be supported within the program or end up using dirty hacks like LD_PRELOAD which will load your library upfront before the original library. But thats another thing which I think you are not looking for.
I know this may seem quite basic to geeks. But I want to make it crystal clear.
When I want to use a Win32 DLL, usually I just call the APIs like LoadLibrary() and GetProcAdderss(). But recently, I am developing with DirectX9, and I need to add d3d9.lib, d3dx9.lib, etc files.
I have heard enough that LIB is for static linking and DLL is for dynamic linking.
So my current understanding is that LIB contains the implementation of the methods and is statically linked at link time as part of the final EXE file. While DLL is dynamic loaded at runtime and is not part of the final EXE file.
But sometimes, there're some LIB files coming with the DLL files, so:
What are these LIB files for?
How do they achieve what they are meant for?
Is there any tools that can let me inspect the internals of these LIB files?
Update 1
After checking wikipedia, I remember that these LIB files are called import library.
But I am wondering how it works with my main application and the DLLs to be dynamically loaded.
Update 2
Just as RBerteig said, there're some stub code in the LIB files born with the DLLs. So the calling sequence should be like this:
My main application --> stub in the LIB --> real target DLL
So what information should be contained in these LIBs? I could think of the following:
The LIB file should contain the fullpath of the corresponding DLL; So the DLL could be loaded by the runtime.
The relative address (or file offset?) of each DLL export method's entry point should be encoded in the stub; So correct jumps/method calls could be made.
Am I right on this? Is there something more?
BTW: Is there any tool that can inspect an import library? If I can see it, there'll be no more doubts.
Linking to a DLL file can occur implicitly at compile link time, or explicitly at run time. Either way, the DLL ends up loaded into the processes memory space, and all of its exported entry points are available to the application.
If used explicitly at run time, you use LoadLibrary() and GetProcAddress() to manually load the DLL and get pointers to the functions you need to call.
If linked implicitly when the program is built, then stubs for each DLL export used by the program get linked in to the program from an import library, and those stubs get updated as the EXE and the DLL are loaded when the process launches. (Yes, I've simplified more than a little here...)
Those stubs need to come from somewhere, and in the Microsoft tool chain they come from a special form of .LIB file called an import library. The required .LIB is usually built at the same time as the DLL, and contains a stub for each function exported from the DLL.
Confusingly, a static version of the same library would also be shipped as a .LIB file. There is no trivial way to tell them apart, except that LIBs that are import libraries for DLLs will usually be smaller (often much smaller) than the matching static LIB would be.
If you use the GCC toolchain, incidentally, you don't actually need import libraries to match your DLLs. The version of the Gnu linker ported to Windows understands DLLs directly, and can synthesize most any required stubs on the fly.
Update
If you just can't resist knowing where all the nuts and bolts really are and what is really going on, there is always something at MSDN to help. Matt Pietrek's article An In-Depth Look into the Win32 Portable Executable File Format is a very complete overview of the format of the EXE file and how it gets loaded and run. Its even been updated to cover .NET and more since it originally appeared in MSDN Magazine ca. 2002.
Also, it can be helpful to know how to learn exactly what DLLs are used by a program. The tool for that is Dependency Walker, aka depends.exe. A version of it is included with Visual Studio, but the latest version is available from its author at http://www.dependencywalker.com/. It can identify all of the DLLs that were specified at link time (both early load and delay load) and it can also run the program and watch for any additional DLLs it loads at run time.
Update 2
I've reworded some of the earlier text to clarify it on re-reading, and to use the terms of art implicit and explicit linking for consistency with MSDN.
So, we have three ways that library functions might be made available to be used by a program. The obvious follow up question is then: "How to I choose which way?"
Static linking is how the bulk of the program itself is linked. All of your object files are listed, and get collected together in to the EXE file by the linker. Along the way, the linker takes care of minor chores like fixing up references to global symbols so that your modules can call each other's functions. Libraries can also be statically linked. The object files that make up the library are collected together by a librarian in a .LIB file which the linker searches for modules containing symbols that are needed. One effect of static linking is that only those modules from the library that are used by the program are linked to it; other modules are ignored. For instance, the traditional C math library includes many trigonometry functions. But if you link against it and use cos(), you don't end up with a copy of the code for sin() or tan() unless you also called those functions. For large libraries with a rich set of features, this selective inclusion of modules is important. On many platforms such as embedded systems, the total size of code available for use in the library can be large compared to the space available to store an executable in the device. Without selective inclusion, it would be harder to manage the details of building programs for those platforms.
However, having a copy of the same library in every program running creates a burden on a system that normally runs lots of processes. With the right kind of virtual memory system, pages of memory that have identical content need only exist once in the system, but can be used by many processes. This creates a benefit for increasing the chances that the pages containing code are likely to be identical to some page in as many other running processes as possible. But, if programs statically link to the runtime library, then each has a different mix of functions each laid out in that processes memory map at different locations, and there aren't many sharable code pages unless it is a program that all by itself is run in more than process. So the idea of a DLL gained another, major, advantage.
A DLL for a library contains all of its functions, ready for use by any client program. If many programs load that DLL, they can all share its code pages. Everybody wins. (Well, until you update a DLL with new version, but that isn't part of this story. Google DLL Hell for that side of the tale.)
So the first big choice to make when planning a new project is between dynamic and static linkage. With static linkage, you have fewer files to install, and you are immune from third parties updating a DLL you use. However, your program is larger, and it isn't quite as good citizen of the Windows ecosystem. With dynamic linkage, you have more files to install, you might have issues with a third party updating a DLL you use, but you are generally being friendlier to other processes on the system.
A big advantage of a DLL is that it can be loaded and used without recompiling or even relinking the main program. This can allow a third party library provider (think Microsoft and the C runtime, for example) to fix a bug in their library and distribute it. Once an end user installs the updated DLL, they immediately get the benefit of that bug fix in all programs that use that DLL. (Unless it breaks things. See DLL Hell.)
The other advantage comes from the distinction between implicit and explicit loading. If you go to the extra effort of explicit loading, then the DLL might not even have existed when the program was written and published. This allows for extension mechanisms that can discover and load plugins, for instance.
These .LIB import library files are used in the following project property, Linker->Input->Additional Dependencies, when building a bunch of dll's that need additional information at link time which is supplied by the import library .LIB files. In the example below to not get linker errors I need to reference to dll's A,B,C, and D through their lib files. (note for the linker to find these files you may need to include their deployment path in Linker->General->Additional Library Directories else you will get a build error about being unable to find any of the provided lib files.)
If your solution is building all dynamic libraries you may have been able to avoid this explicit dependency specification by relying instead on the reference flags exposed under the Common Properties->Framework and References dialog. These flags appear to automatically do the linking on your behalf using the *.lib files.
This however is as it says a Common Properties, which is not configuration or platform specific. If you need to support a mixed build scenario as in our application we had a build configuration to render a static build and a special configuration that built a constrained build of a subset of assemblies that were deployed as dynamic libraries. I had used the Use Library Dependency Inputs and Link Library Dependencies flags set to true under various cases to get things to build and later realizing to simplify things but when introducing my code to the static builds I introduced a ton of linker warnings and the build was incredibly slow for the static builds. I wound up introducing a bunch of these sort of warnings...
warning LNK4006: "bool __cdecl XXX::YYY() already defined in CoreLibrary.lib(JSource.obj); second definition ignored D.lib(JSource.obj)
And I wound up using the manual specification of Additional Dependencies to satisfy the linker for the dynamic builds while keeping the static builders happy by not using a common property that slowed them down. When I deploy the dynamic subset build I only deploy the dll files as these lib files are only used at link time, not at runtime.
Here are some related MSDN topics to answer my question:
Linking an Executable to a DLL
Linking Implicitly
Determining Which Linking Method to Use
Building an Import Library and Export File
There are three kinds of libraries: static, shared and dynamically loaded libraries.
The static libraries are linked with the code at the linking phase, so they are actually in the executable, unlike the shared library, which has only stubs (symbols) to look for in the shared library file, which is loaded at run time before the main function gets called.
The dynamically loaded ones are much like the shared libraries, except they are loaded when and if the need arises by the code you've written.
In my mind, there are two method to link dll to exe.
Use dll and the import library (.lib file) implicitly
Use functions like loadlibrary() explicitly
I'm trying to understand how a certain library works. I've compiled it with my added prinfts and everything is great. Now I want to stop the example program during runtime to look at the call stack, but I can't quite figure out how to do it with gdb. The function I want to break on, is inside a shared library. I've reviewed a previous question here on SO, but the approach doesn't work for me. The language in question is C++. I've attempted to provide the filename and line number, but gdb refuses to understand that, it only lists the source files from the demo app.
Any suggestions?
You can do "break main" first. By the time you hit that, the shared library should be loaded, and you can then set a breakpoint in any of its routines.
There are two cases to consider (and your question doesn't make it clear which case you have):
- your executable is linked with the shared library directly:
this means that GDB will "see" the symbols (and sources) from shared library when you stop on main
- your executable dynamically loads the shared library (e.g. via dlopen):
in that case, GDB will not "see" your shared library until after dlopen completes.
Since you can't see the symbols when you stop at main, I am guessing you have the second case.
You can do "set stop-on-solib-events 1" at the (gdb) prompt, and GDB will stop every time a new shared library is loaded (or unloaded).
You can see which libraries GDB "knows" about via info shared command.
Just wait until you see your target library in that list, before attempting to set breakpoints in it.
Check this out:
http://linux.die.net/man/1/ltrace
it will trace your library calls - probably be useful.
And "strace" does the same thing for system calls.
And with that you should be able to find an entry point... You could set a breakpoint in GDB that way (although i can't explain the details myself)