I am building a Java application that uses a shared library written in C++ and compiled for different operating systems. The problem is, that this shared library itself depends on an additional library it normally finds under the appropriate environment variable (PATH, LIBRARY_PATH or LD_LIBRARY_PATH).
I can - but don't want to - set these environment variables. I'd rather load the needed shared libraries from a given path at runtime - just like a plugin. And no - I don't want any starter application that starts a new process with a new environment. Does anybody know how to achieve this?
I know that this must be possible, as one of the libraries I use is capable of loading its plugins from a given path. Of course I'd prefer platform independent code, but if this ain't possible, seperate solutions for Windows, Linux and MacOS would also do it.
EDIT
I should have mentioned that the shared library I'd wish to use is object oriented, which means that a binding of single functions won't do it.
Un UNIX/Linux systems you can use dlopen. The issue then is you have to fetch all symbols you need via dlsym
Simple example:
typedef int (*some_func)(char *param);
void *myso = dlopen("/path/to/my.so", RTLD_NOW);
some_func *func = dlsym(myso, "function_name_to_fetch");
func("foo");
dlclose(myso);
Will load the .so and execute function_name_to_fetch() from in there. See the man page dlopen(1) for more.
On Windows, you can use LoadLibrary, and on Linux, dlopen. The APIs are extremely similar and can load a so/dll directly by providing the full path. That works if it is a run-time dependency (after loading, you "link" by calling GetProcAddress/dlsym.)
I concur with the other posters about the use of dlopen and LoadLibrary. The libltdl gives you a platform-independent interface to these functions.
I do not think you can do it for it.
Most Dlls have some sort of init() function that must be called after it have been loaded, and sometime that init() function needs some parameters and returns some handle to be used to call the dll's functions. Do you know the definition of the additional library?
Then, the first library can not simply look if the DLL X is in RAM only by using its name. The one it needs can be in a different directory or a different build/version. The OS will recognize the library if the full path is the same than another one already loaded and it will share it instead of loading it a second time.
The other library can load its plugins from another path because it written to not depend on PATH and they are his own plugins.
Have you try to update the process's environment variables from the code before loading the Dll? That will not depends on a starter process.
Related
I have a c++ application that is linked against some 3rd party dynamic libraries. Some of my main classes inherit from these libraries, and make function calls to those libraries, etc. My application works without the inclusion of those libraries in theory (ie, if I manually stripped out all code and references pertaining to those libraries, it would still work), it would just be more limited in functionality. If I could craft an analogy, imagine that I've created a Windows Notepad clone, and included a 3rd party library that allows users to embed pictures and video in the document.
When I distribute my application, there is a chance my clients may not have those libraries installed. Is there a way I can have my program detect if the required DLL library exists, and simply ignore all related code if it's not installed?
Currently if I run my application without the 3rd party libraries installed, it displays errors related to the missing DLLs and crashes. One obvious solution is to simply release two versions of my application...one without the external dependencies and one with, but I'd like to avoid managing two products independently like that.
There is such option "delay load dll"
For dll xxx.dll you configure linker to use "delayload"
Till you call any function from dll it will not be loaded and your application will start successfully even if the dll is not present
You use LoadLibrary to check if xxx.dll is available.
If LoadLibrary failed - you disable module using xxx.dll
If LoadLibrary is successful - You unload it (you do not need dynamic loading - it is only use to test the presence of the dll) and use library as if it is linked regularly - no need to modify code using any xxx.dll related functionality
See this answer here:
Dynamically load a function from a DLL
You basically use LoadLibrary to load the DLL, the result will be
null or not if loaded.
Then you use GetProcAddress to get the functions.
If you want to distribute the binary of your application you can use dynamic loading (take a look here)
But if you have the option to build your application on clients system, you can use CMake and set compilation flags if it could not find the library you want. Then your code can behave according to this flag's value (i.e. this way you have the information about weather this library exists or not in your code)
Loading dynamic lib or injecting to the process or ... can be using after this you can :
if (ul_reason_for_call == DLL_PROCESS_ATTACH)
{
inj_hModule = hModule;
DisableThreadLibraryCalls(hModule); // Disable DllMain calls for DLL_THREAD_*
HANDLE hThreadProc = CreateThread(nullptr, NULL, LPTHREAD_START_ROUTINE(ThreadProc), hModule, NULL, nullptr);
CloseHandle(hThreadProc);
Beep(2000, 500);
}
if (ul_reason_for_call == DLL_PROCESS_DETACH)
{
Beep(2000, 500);
FreeLibraryAndExitThread(hModule, 0);
// exit this thread
ExitThread(0);
}
Well, you should revise your software architecture, and go about a Plugin Design Pattern as was introduced by Martin Fowler.
You don't want dynamic linkage for such case, though it's quite similar.
Regarding implementations in C++ that basically means you introduce an
abstract interface first
struct Foo {
virtual void Bar() = 0;
virtual ~Foo() {}
};
and lookup for suitable implementations coming with shared libraries at runtime1.
The exported interface of the shared library implementation needs to match your abstract interface declaration.
You can use a Factory to do that lookup, loading the shared library and create an instance of the implementation.
Here's some more information which OS functions are involved in dynamically loading and binding the Plugin.
1)Note that shared libraries don't need to have the extension .dll or .so if you're loading these actively.
My advice would be to carefully split the code dependent on that DLL. Design a clean interface to that part of your project that can be easily dynamically loaded (LoadLibrary), and have only pure virtual interfaces (no DLL export) on any classes you interact with in the main part of your program. Ideally it has a pure function interface, as those are most stable.
Now, that DLL can depend on the 3rd party DLL like normal. Your main code calls your "wrapping" DLL by loading it manually, and handles load failure gracefully.
This is easier than using a DLL not designed for this, and you can experiment without installing/uninstalling the 3rd party DLL by omitting your wrapping DLL.
I'm using Code::Blocks and hate manually linking DLLs. I found the LoadLibrary() function, and am wondering if it works like a .a or .lib file would. Does this function work like that? If not, what can I do programming-wise (if anything) to link a DLL without having to link a DLL by doing the Project < Build options < Linker settings < add < ... method?
LoadLibrary loads the requested library (and all the libraries it needs) into your process' address space. In order to access any of the code/data in that library, you need to find out the code or data address in the newly loaded region of memory. You need to use GetProcAddress.
The difference between this process and adding a library during build time is that for build-time library the compiler prepares a list of locations that refer to a given function, linker puts that list into .exe, and run-time linker loads the library, does the equivalent of GetProcAddress for the function name and places the address into all locations that the compiler marked.
When you don't have this automated support, you have to declare a pointer to function, call GetProcAddress yourself, and assign the returned value to your pointer to function. You can then call the function like any other C function (note "C" part - the above process is complicated by name mangling when you use C++, so make use of extern "C")
LoadLibrary() loads a DLL at runtime. Normaly you link when you compile the EXE, and link the DLLs like a static library at this time. If you need to load libraries dynamically at runtime, you use LoadLibrary().
For example when you implement a pluginsystem, is this usefull, as you don't know the libraries beforehand.
Not how it works at all. LoadLibrary is used to load a "at compile time unknown" DLL - such as program extensions/plugins or "This DLL for SSE, that DLL for non-SSE" based on "what can the hardware do - one could also consider having a DLL per connection type to email servers or something like that, so that the email program doesn't have to "carry" all the different variants, when only one is used for any particular email address.
Further, to use a DLL that has been loaded this way, you need to use GetProcAddress to get the address of the functions in the DLL. This is very different from linking the DLL into the project at buildtime, where functions simply appear "automagically" by help of the system loader functions that load the DLL's that are added to the project at build-time.
Contrary to static or dynamic library linking, LoadLibrary doesn't make the library's symbols directly available to your program. You need to call GetProcAddress at runtime to get a pointer to the functions you want to call.
As #Devolus mentioned, this is a good way to implement plugin systems and/or to access optional components. However, since the symbols are not available to your program in a transparent way, this is not really practical for ordinary usage.
I'm working on c++ with shared library usage.
Currently I'm using "LD_PRELOAD" and set this environment variable using setenv()
call.
But I want to use dlopen() API to load shared library. That should work same as like setting environment variable (i.e. LD_PRELOAD) using setenv().
can i use dlopen() to get above requirements? or there is difference in the library loading using LD_PRELOAD and dlopen()?
I'm not 100% sure about this, but as I understand it using LD_PRELOAD makes the program loader load all libraries, first, then the library specified by LD_PRELOAD and last your application. This makes it possible to override system libraries with your own.
Using dlopen loads the shared object after your program is loaded, so can not be used to override system objects.
If the environment variable have to be set for the program to work correctly, then it has to be set before the program is loaded, either in the shell or by your LD_PRELOAD file. If the program doesn't need the environment variable immediately then you can either set it in the program or in the "on-load" function in the shared object loaded by dlopen.
I have been involved in some debate with respect to libraries in Linux, and would like to confirm some things.
It is to my understanding (please correct me if I am wrong and I will edit my post later), that there are two ways of using libraries when building an application:
Static libraries (.a files): At link time, a copy of the entire library is put into the final application so that the functions within the library are always available to the calling application
Shared objects (.so files): At link time, the object is just verified against its API via the corresponding header (.h) file. The library isn't actually used until runtime, where it is needed.
The obvious advantage of static libraries is that they allow the entire application to be self-contained, while the benefit of dynamic libraries is that the ".so" file can be replaced (ie: in case it needs to be updated due to a security bug) without requiring the base application to be recompiled.
I have heard some people make a distinction between shared objects and dynamic link libraries (DLL's), even though they are both ".so" files. Is there any distinction between shared objects and DLLs when it comes to C/C++ development on Linux or any other POSIX compliant OS (ie: MINIX, UNIX, QNX, etc)? I am told that one key difference (so far) is that shared objects are just used at runtime, while DLL's must be opened first using the dlopen() call within the application.
Finally, I have also heard some developers mention "shared archives", which, to my understanding, are also static libraries themselves, but are never used by an application directly. Instead, other static libraries will link against the "shared archives" to pull some (but not all) functions/resources from the shared archive into the static library being built.
Thank you all in advance for your assistance.
Update
In the context in which these terms were provided to me, it was effectively erroneous terms used by a team of Windows developers that had to learn Linux. I tried to correct them, but the (incorrect) language norms stuck.
Shared Object: A library that is automatically linked into a program when the program starts, and exists as a standalone file. The library is included in the linking list at compile time (ie: LDOPTS+=-lmylib for a library file named mylib.so). The library must be present at compile time, and when the application starts.
Static Library: A library that is merged into the actual program itself at build time for a single (larger) application containing the application code and the library code that is automatically linked into a program when the program is built, and the final binary containing both the main program and the library itself exists as a single standalone binary file. The library is included in the linking list at compile time (ie: LDOPTS+=-lmylib for a library file named mylib.a). The library must be present at compile time.
DLL: Essentially the same as a shared object, but rather than being included in the linking list at compile time, the library is loaded via dlopen()/dlsym() commands so that the library does not need to be present at build time for the program to compile. Also, the library does not need to be present (necessarily) at application startup or compile time, as it is only needed at the moment the dlopen/dlsym calls are made.
Shared Archive: Essentially the same as a static library, but is compiled with the "export-shared" and "-fPIC" flags. The library is included in the linking list at compile time (ie: LDOPTS+=-lmylibS for a library file named mylibS.a). The distinction between the two is that this additional flag is required if a shared object or DLL wants to statically link the shared archive into its own code AND be able to make the functions in the shared object available to other programs, rather than just using them internal to the DLL. This is useful in the case when someone provides you with a static library, and you wish to repackage it as an SO. The library must be present at compile time.
Additional Update
The distinction between "DLL" and "shared library" was just a (lazy, inaccurate) colloquialism in the company I worked in at the time (Windows developers being forced to shift to Linux development, and the term stuck), adhering to the descriptions noted above.
Additionally, the trailing "S" literal after the library name, in the case of "shared archives" was just a convention used at that company, and not in the industry in general.
A static library(.a) is a library that can be linked directly into the final executable produced by the linker,it is contained in it and there is no need to have the library into the system where the executable will be deployed.
A shared library(.so) is a library that is linked but not embedded in the final executable, so will be loaded when the executable is launched and need to be present in the system where the executable is deployed.
A dynamic link library on windows(.dll) is like a shared library(.so) on linux but there are some differences between the two implementations that are related to the OS (Windows vs Linux) :
A DLL can define two kinds of functions: exported and internal. The exported functions are intended to be called by other modules, as well as from within the DLL where they are defined. Internal functions are typically intended to be called only from within the DLL where they are defined.
An SO library on Linux doesn't need special export statement to indicate exportable symbols, since all symbols are available to an interrogating process.
I've always thought that DLLs and shared objects are just different terms for the same thing - Windows calls them DLLs, while on UNIX systems they're shared objects, with the general term - dynamically linked library - covering both (even the function to open a .so on UNIX is called dlopen() after 'dynamic library').
They are indeed only linked at application startup, however your notion of verification against the header file is incorrect. The header file defines prototypes which are required in order to compile the code which uses the library, but at link time the linker looks inside the library itself to make sure the functions it needs are actually there. The linker has to find the function bodies somewhere at link time or it'll raise an error. It ALSO does that at runtime, because as you rightly point out the library itself might have changed since the program was compiled. This is why ABI stability is so important in platform libraries, as the ABI changing is what breaks existing programs compiled against older versions.
Static libraries are just bundles of object files straight out of the compiler, just like the ones that you are building yourself as part of your project's compilation, so they get pulled in and fed to the linker in exactly the same way, and unused bits are dropped in exactly the same way.
I can elaborate on the details of DLLs in Windows to help clarify those mysteries to my friends here in *NIX-land...
A DLL is like a Shared Object file. Both are images, ready to load into memory by the program loader of the respective OS. The images are accompanied by various bits of metadata to help linkers and loaders make the necessary associations and use the library of code.
Windows DLLs have an export table. The exports can be by name, or by table position (numeric). The latter method is considered "old school" and is much more fragile -- rebuilding the DLL and changing the position of a function in the table will end in disaster, whereas there is no real issue if linking of entry points is by name. So, forget that as an issue, but just be aware it's there if you work with "dinosaur" code such as 3rd-party vendor libs.
Windows DLLs are built by compiling and linking, just as you would for an EXE (executable application), but the DLL is meant to not stand alone, just like an SO is meant to be used by an application, either via dynamic loading, or by link-time binding (the reference to the SO is embedded in the application binary's metadata, and the OS program loader will auto-load the referenced SO's). DLLs can reference other DLLs, just as SOs can reference other SOs.
In Windows, DLLs will make available only specific entry points. These are called "exports". The developer can either use a special compiler keyword to make a symbol an externally-visible (to other linkers and the dynamic loader), or the exports can be listed in a module-definition file which is used at link time when the DLL itself is being created. The modern practice is to decorate the function definition with the keyword to export the symbol name. It is also possible to create header files with keywords which will declare that symbol as one to be imported from a DLL outside the current compilation unit. Look up the keywords __declspec(dllexport) and __declspec(dllimport) for more information.
One of the interesting features of DLLs is that they can declare a standard "upon load/unload" handler function. Whenever the DLL is loaded or unloaded, the DLL can perform some initialization or cleanup, as the case may be. This maps nicely into having a DLL as an object-oriented resource manager, such as a device driver or shared object interface.
When a developer wants to use an already-built DLL, she must either reference an "export library" (*.LIB) created by the DLL developer when she created the DLL, or she must explicitly load the DLL at run time and request the entry point address by name via the LoadLibrary() and GetProcAddress() mechanisms. Most of the time, linking against a LIB file (which simply contains the linker metadata for the DLL's exported entry points) is the way DLLs get used. Dynamic loading is reserved typically for implementing "polymorphism" or "runtime configurability" in program behaviors (accessing add-ons or later-defined functionality, aka "plugins").
The Windows way of doing things can cause some confusion at times; the system uses the .LIB extension to refer to both normal static libraries (archives, like POSIX *.a files) and to the "export stub" libraries needed to bind an application to a DLL at link time. So, one should always look to see if a *.LIB file has a same-named *.DLL file; if not, chances are good that *.LIB file is a static library archive, and not export binding metadata for a DLL.
You are correct in that static files are copied to the application at link-time, and that shared files are just verified at link time and loaded at runtime.
The dlopen call is not only for shared objects, if the application wishes to do so at runtime on its behalf, otherwise the shared objects are loaded automatically when the application starts. DLLS and .so are the same thing. the dlopen exists to add even more fine-grained dynamic loading abilities for processes. You dont have to use dlopen yourself to open/use the DLLs, that happens too at application startup.
I suspect some kind of misunderstanding here, but header files, at least of the .h variety used for compiling source code, are most definitely NOT checked during link time.
.h, and for that matter, .c/.cpp files, are only involved during the compilation phase, which includes preprocessing. Once the object code has been created the header file is long gone well before the linker gets around to dealing with things.
I know this may seem quite basic to geeks. But I want to make it crystal clear.
When I want to use a Win32 DLL, usually I just call the APIs like LoadLibrary() and GetProcAdderss(). But recently, I am developing with DirectX9, and I need to add d3d9.lib, d3dx9.lib, etc files.
I have heard enough that LIB is for static linking and DLL is for dynamic linking.
So my current understanding is that LIB contains the implementation of the methods and is statically linked at link time as part of the final EXE file. While DLL is dynamic loaded at runtime and is not part of the final EXE file.
But sometimes, there're some LIB files coming with the DLL files, so:
What are these LIB files for?
How do they achieve what they are meant for?
Is there any tools that can let me inspect the internals of these LIB files?
Update 1
After checking wikipedia, I remember that these LIB files are called import library.
But I am wondering how it works with my main application and the DLLs to be dynamically loaded.
Update 2
Just as RBerteig said, there're some stub code in the LIB files born with the DLLs. So the calling sequence should be like this:
My main application --> stub in the LIB --> real target DLL
So what information should be contained in these LIBs? I could think of the following:
The LIB file should contain the fullpath of the corresponding DLL; So the DLL could be loaded by the runtime.
The relative address (or file offset?) of each DLL export method's entry point should be encoded in the stub; So correct jumps/method calls could be made.
Am I right on this? Is there something more?
BTW: Is there any tool that can inspect an import library? If I can see it, there'll be no more doubts.
Linking to a DLL file can occur implicitly at compile link time, or explicitly at run time. Either way, the DLL ends up loaded into the processes memory space, and all of its exported entry points are available to the application.
If used explicitly at run time, you use LoadLibrary() and GetProcAddress() to manually load the DLL and get pointers to the functions you need to call.
If linked implicitly when the program is built, then stubs for each DLL export used by the program get linked in to the program from an import library, and those stubs get updated as the EXE and the DLL are loaded when the process launches. (Yes, I've simplified more than a little here...)
Those stubs need to come from somewhere, and in the Microsoft tool chain they come from a special form of .LIB file called an import library. The required .LIB is usually built at the same time as the DLL, and contains a stub for each function exported from the DLL.
Confusingly, a static version of the same library would also be shipped as a .LIB file. There is no trivial way to tell them apart, except that LIBs that are import libraries for DLLs will usually be smaller (often much smaller) than the matching static LIB would be.
If you use the GCC toolchain, incidentally, you don't actually need import libraries to match your DLLs. The version of the Gnu linker ported to Windows understands DLLs directly, and can synthesize most any required stubs on the fly.
Update
If you just can't resist knowing where all the nuts and bolts really are and what is really going on, there is always something at MSDN to help. Matt Pietrek's article An In-Depth Look into the Win32 Portable Executable File Format is a very complete overview of the format of the EXE file and how it gets loaded and run. Its even been updated to cover .NET and more since it originally appeared in MSDN Magazine ca. 2002.
Also, it can be helpful to know how to learn exactly what DLLs are used by a program. The tool for that is Dependency Walker, aka depends.exe. A version of it is included with Visual Studio, but the latest version is available from its author at http://www.dependencywalker.com/. It can identify all of the DLLs that were specified at link time (both early load and delay load) and it can also run the program and watch for any additional DLLs it loads at run time.
Update 2
I've reworded some of the earlier text to clarify it on re-reading, and to use the terms of art implicit and explicit linking for consistency with MSDN.
So, we have three ways that library functions might be made available to be used by a program. The obvious follow up question is then: "How to I choose which way?"
Static linking is how the bulk of the program itself is linked. All of your object files are listed, and get collected together in to the EXE file by the linker. Along the way, the linker takes care of minor chores like fixing up references to global symbols so that your modules can call each other's functions. Libraries can also be statically linked. The object files that make up the library are collected together by a librarian in a .LIB file which the linker searches for modules containing symbols that are needed. One effect of static linking is that only those modules from the library that are used by the program are linked to it; other modules are ignored. For instance, the traditional C math library includes many trigonometry functions. But if you link against it and use cos(), you don't end up with a copy of the code for sin() or tan() unless you also called those functions. For large libraries with a rich set of features, this selective inclusion of modules is important. On many platforms such as embedded systems, the total size of code available for use in the library can be large compared to the space available to store an executable in the device. Without selective inclusion, it would be harder to manage the details of building programs for those platforms.
However, having a copy of the same library in every program running creates a burden on a system that normally runs lots of processes. With the right kind of virtual memory system, pages of memory that have identical content need only exist once in the system, but can be used by many processes. This creates a benefit for increasing the chances that the pages containing code are likely to be identical to some page in as many other running processes as possible. But, if programs statically link to the runtime library, then each has a different mix of functions each laid out in that processes memory map at different locations, and there aren't many sharable code pages unless it is a program that all by itself is run in more than process. So the idea of a DLL gained another, major, advantage.
A DLL for a library contains all of its functions, ready for use by any client program. If many programs load that DLL, they can all share its code pages. Everybody wins. (Well, until you update a DLL with new version, but that isn't part of this story. Google DLL Hell for that side of the tale.)
So the first big choice to make when planning a new project is between dynamic and static linkage. With static linkage, you have fewer files to install, and you are immune from third parties updating a DLL you use. However, your program is larger, and it isn't quite as good citizen of the Windows ecosystem. With dynamic linkage, you have more files to install, you might have issues with a third party updating a DLL you use, but you are generally being friendlier to other processes on the system.
A big advantage of a DLL is that it can be loaded and used without recompiling or even relinking the main program. This can allow a third party library provider (think Microsoft and the C runtime, for example) to fix a bug in their library and distribute it. Once an end user installs the updated DLL, they immediately get the benefit of that bug fix in all programs that use that DLL. (Unless it breaks things. See DLL Hell.)
The other advantage comes from the distinction between implicit and explicit loading. If you go to the extra effort of explicit loading, then the DLL might not even have existed when the program was written and published. This allows for extension mechanisms that can discover and load plugins, for instance.
These .LIB import library files are used in the following project property, Linker->Input->Additional Dependencies, when building a bunch of dll's that need additional information at link time which is supplied by the import library .LIB files. In the example below to not get linker errors I need to reference to dll's A,B,C, and D through their lib files. (note for the linker to find these files you may need to include their deployment path in Linker->General->Additional Library Directories else you will get a build error about being unable to find any of the provided lib files.)
If your solution is building all dynamic libraries you may have been able to avoid this explicit dependency specification by relying instead on the reference flags exposed under the Common Properties->Framework and References dialog. These flags appear to automatically do the linking on your behalf using the *.lib files.
This however is as it says a Common Properties, which is not configuration or platform specific. If you need to support a mixed build scenario as in our application we had a build configuration to render a static build and a special configuration that built a constrained build of a subset of assemblies that were deployed as dynamic libraries. I had used the Use Library Dependency Inputs and Link Library Dependencies flags set to true under various cases to get things to build and later realizing to simplify things but when introducing my code to the static builds I introduced a ton of linker warnings and the build was incredibly slow for the static builds. I wound up introducing a bunch of these sort of warnings...
warning LNK4006: "bool __cdecl XXX::YYY() already defined in CoreLibrary.lib(JSource.obj); second definition ignored D.lib(JSource.obj)
And I wound up using the manual specification of Additional Dependencies to satisfy the linker for the dynamic builds while keeping the static builders happy by not using a common property that slowed them down. When I deploy the dynamic subset build I only deploy the dll files as these lib files are only used at link time, not at runtime.
Here are some related MSDN topics to answer my question:
Linking an Executable to a DLL
Linking Implicitly
Determining Which Linking Method to Use
Building an Import Library and Export File
There are three kinds of libraries: static, shared and dynamically loaded libraries.
The static libraries are linked with the code at the linking phase, so they are actually in the executable, unlike the shared library, which has only stubs (symbols) to look for in the shared library file, which is loaded at run time before the main function gets called.
The dynamically loaded ones are much like the shared libraries, except they are loaded when and if the need arises by the code you've written.
In my mind, there are two method to link dll to exe.
Use dll and the import library (.lib file) implicitly
Use functions like loadlibrary() explicitly