Understanding lua extension dll building/loading in statically linked embedded lua environment - c++

I have a relatively complex lua environment and I'm trying to understand how the following would/could work. The starting setup includes the following two modules:
Main application (no lua environment)
DLL (statically linked to lua lib, including interpreter)
The dll is loaded into the main application and runs a lua console interpreter and a lua API accessible from the console.
Now, let's say I want to expand this setup to include another dll that would extend that lua API, luasql for example. The new dll needs to link against lua in order to build, and my understanding is that I cannot link against lua statically since there would now be two unshared copies of the lua code in process when I load the extension dll. However, even if I built the lua core lib as a dll and linked against it with the extension dll, that lua core dll would not be loaded at runtime by the main application or the primary dll. So my questions are:
What happens if I load that extension dll from the lua intepreter in the primary dll, considering that the lua core dll will not be loaded?
If I loaded the lua core dll at runtime, how would that conflict with the statically linked lua lib?
Would both scenarios (linking statically in extension dll and dynamically linking/loading the lua dll) result in having two copies of the lua core code in process?
In that event, what would happen if I tried to call an API function from the primary dll's lua environment/interpreter that was built/loaded in the extension dll?
OR does lua have some kind of special mechanism for loading native dlls that provide new C API functions that allows it to bypass normal linking rules?
Hopefully I have provided enough details to make the questions specific, if not I will be happy to refine the scenario/questions further.
Edit: I have looked at Bundling additional Lua libraries for embedded and statically linked Lua runtime and I believe it may be helpful in providing a solution ultimately but I'd like to understand it at the linker level.


You can't have the situation when you load one interpreter (let's say it's linked statically) and load a module X that is linked against a dll with a Lua interpreter, which loads another copy of the interpreter. This is likely to cause an application crash. You need to make the loaded dll to use the interpreter that is already loaded, either by linking against that dll with the interpreter or by using a proxy dll (see below).
You have two main options: (1) make dllA that is loaded by the main application that in turn depends on Lua dll; you can then link all other lua modules against Lua dll without any issues; or (2) include Lua dll into dllA, but keep Lua methods exposed so that lua modules can be linked against that dllA.
I think the first option is much simpler and likely not to require any changes to the Lua modules (as long as you can keep the name of the Lua dll the same as the one that the modules are compiled against).
Another option I should mention is that you can still use Lua modules compiled against a Lua DLL even with applications that have the Lua interpreter statically compiled. You need to use a proxy DLL; see this maillist thread for the solution and related discussion.


The answer boils down to this:
Don't try to load any Lua extensions from a dll linked against a different Lua-core. Doing so will cause utter chaos.
As long as any Lua extension loaded resolves all its dependencies to the proper Lua core, it does not matter (aside from bloat) how many Lua cores you use.
Keep in mind that windows always resolves symbols according their name and their providing dll.

Related

How to hide exe dll in C++?

I have my program (.exe) and it depends on 3 DLLs to work, but, I want to hide these DLLs, is there any way for me to "put them in .exe" these dlls? Only have the .exe,is there any method to do this?
Info:
Program's ling: C++
DLLs: {libcrypto-1_1-x64, libcurl-x64, libssl-1_1-x64}

If you use the static versions of the libraries, their code will be compiled directly into your EXE, and then you won't need DLLs at all.
But, if you need to use DLLs, then you can compile them into your EXE's resources, and then extract them at runtime to temp files before using them. You won't be able to statically link to any of the DLL functions at compile-time, or else your app won't run at all. But you can dynamically load them at runtime using LoadLibrary() and GetProcAddress() (or, if you compiler supports it, use delay-loading), which will allow you to extract the DLLs before calling their functions.

Static link an existing windows binary

I was wondering if I can take an existing windows DLL and static link the dynamically-linked files?
I saw a number of projects to do this with Linux/elf
http://magicermine.com/
http://statifier.sourceforge.net/
http://bitwagon.com/jumpstart/jumpstart.html
I imagine this is most likely not possible, but I am running into some issues in WinPE where when I statically linked the DLLs everything started working great.
I don't have the source to the existing DLL.
I guess I could make a pass-through DLL that exposed all of the same functions and static linked?

There is no tool support for linking in the code of a DLL statically.
The problem is that a DLL is a full Windows PE executable, not a C or C++ “library” in any sense. The C++ standard has only one statement that is vaguely in support of DLL-like things (in the para about dynamic initialization after first statement of main). You’re out of luck.
But if you had the source code (as e.g. with MFC), which you say you don’t, then you could just have created static libraries.
Do note that there already is a meaning for “linking statically” a DLL, namely to have it loaded and have its functions resolved automatically.
Which is the usual way of using a DLL.
And which is in contrast to explicitly loading it dynamically and using GetProcAddress to resolve its functions.
Regarding
” when I statically linked the DLLs everything started working great
presumably earlier you have explictly loaded the DLLs dynamically, and used GetProcAddress, and presumably something about that did not work perfectly.
One main problem with GetProcAddress is that it assumes that the provided function name is encoded as Windows ANSI (the machine-dependent encoding reported by GetACP), and then (apparently) translates that to UTF-8 for the function lookup.
One workaround could be to access the function by ordinal rather than name.
One way to find the ordinal with Microsoft's tools, is to use dumpbin /exports.

C++ Winapi Including DLL in the .exe file

I'm using MYSQL library, and libmysql.lib /.dll.
My program cannot be working without the libmysql.dll
When I'm trying to run my project without the dll I'm getting that error message.
What I'm basically want to do is to put that dll in my .exe file.
build the .exe file with that dll and make the program read it from himself.
I mean, give the program to people with that dll inside.
It is possible ?
I tried this section: embed DLL in MFC C++ EXE?
But the program still asking for the dll .. (But I do see that the size of the .exe has been changed) so that dll has been added.
But the program still asking for the libmysql.dll ..
All the point is to use it inside the .exe file..
thanks.

What you are asking for cannot be done if you statically link to the DLL at compile-time. You need to dynamically link to the DLL at run-time instead, either by explicit calls to LoadLibrary() and GetProcAddress() to access the DLL functions directly, or by utilizing your compiler's delay-load functionality (which uses LoadLibrary() and GetProcAddress() internally, but hides that fact from your code). Either way, you can then store the DLL in your EXE's resources at compile-time, then extract the resource to a temporary file at run-time and load/use it as needed (you can't use the DLL from inside the EXE's resources directly. Well, there is a way to do it, but it is a VERY complicated and advanced technique, as it requires implementing your own executable loader that basically mimics what the OS's built-in executable loader already does). When you are done using the DLL, you can unload it from memory and delete the temp file.

C++ - How do i link a C++ resource?

I need to use functions in a compiled C++ resource i have.
How can i use the functions in the compiled resource and have the main program use this library i have?
I have this code:
log("INFO","THIS IS A TEST MESSAGE");
That should pull the function from the file i have. This

Update (old answer removed)
Since you have elaborated on the need to load pre-compiled code in through "plugins". This is typically done through a mechanism known as "dynamic loading". In Windows, this is accomplished through the use of DLL (Dynamic Link Libraries) files. DLL files are libraries of compiled code that a running program "load in" on demand.. The running program can load the DLL and query for addresses to function pointers with known names. For example, a "logging" DLL may export a "Log" function. And if someone else wants to provide an alternate implementation (and can register the path to their DLL with the program in some sort of configuration file), they merely have to supply their own DLL which exports the same set of functions with the exact same signature. And there are useful abstractions known as COM for loading (C++) interfaces, but I digress.
In Unix operating systems, the equivalent of DLLs is known as "shared libraries" or ".so" files.
Here's some relevant links:
http://en.wikipedia.org/wiki/Dynamic-link_library
http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html

How does the Import Library work? Details?

I know this may seem quite basic to geeks. But I want to make it crystal clear.
When I want to use a Win32 DLL, usually I just call the APIs like LoadLibrary() and GetProcAdderss(). But recently, I am developing with DirectX9, and I need to add d3d9.lib, d3dx9.lib, etc files.
I have heard enough that LIB is for static linking and DLL is for dynamic linking.
So my current understanding is that LIB contains the implementation of the methods and is statically linked at link time as part of the final EXE file. While DLL is dynamic loaded at runtime and is not part of the final EXE file.
But sometimes, there're some LIB files coming with the DLL files, so:
What are these LIB files for?
How do they achieve what they are meant for?
Is there any tools that can let me inspect the internals of these LIB files?
Update 1
After checking wikipedia, I remember that these LIB files are called import library.
But I am wondering how it works with my main application and the DLLs to be dynamically loaded.
Update 2
Just as RBerteig said, there're some stub code in the LIB files born with the DLLs. So the calling sequence should be like this:
My main application --> stub in the LIB --> real target DLL
So what information should be contained in these LIBs? I could think of the following:
The LIB file should contain the fullpath of the corresponding DLL; So the DLL could be loaded by the runtime.
The relative address (or file offset?) of each DLL export method's entry point should be encoded in the stub; So correct jumps/method calls could be made.
Am I right on this? Is there something more?
BTW: Is there any tool that can inspect an import library? If I can see it, there'll be no more doubts.

Linking to a DLL file can occur implicitly at compile link time, or explicitly at run time. Either way, the DLL ends up loaded into the processes memory space, and all of its exported entry points are available to the application.
If used explicitly at run time, you use LoadLibrary() and GetProcAddress() to manually load the DLL and get pointers to the functions you need to call.
If linked implicitly when the program is built, then stubs for each DLL export used by the program get linked in to the program from an import library, and those stubs get updated as the EXE and the DLL are loaded when the process launches. (Yes, I've simplified more than a little here...)
Those stubs need to come from somewhere, and in the Microsoft tool chain they come from a special form of .LIB file called an import library. The required .LIB is usually built at the same time as the DLL, and contains a stub for each function exported from the DLL.
Confusingly, a static version of the same library would also be shipped as a .LIB file. There is no trivial way to tell them apart, except that LIBs that are import libraries for DLLs will usually be smaller (often much smaller) than the matching static LIB would be.
If you use the GCC toolchain, incidentally, you don't actually need import libraries to match your DLLs. The version of the Gnu linker ported to Windows understands DLLs directly, and can synthesize most any required stubs on the fly.
Update
If you just can't resist knowing where all the nuts and bolts really are and what is really going on, there is always something at MSDN to help. Matt Pietrek's article An In-Depth Look into the Win32 Portable Executable File Format is a very complete overview of the format of the EXE file and how it gets loaded and run. Its even been updated to cover .NET and more since it originally appeared in MSDN Magazine ca. 2002.
Also, it can be helpful to know how to learn exactly what DLLs are used by a program. The tool for that is Dependency Walker, aka depends.exe. A version of it is included with Visual Studio, but the latest version is available from its author at http://www.dependencywalker.com/. It can identify all of the DLLs that were specified at link time (both early load and delay load) and it can also run the program and watch for any additional DLLs it loads at run time.
Update 2
I've reworded some of the earlier text to clarify it on re-reading, and to use the terms of art implicit and explicit linking for consistency with MSDN.
So, we have three ways that library functions might be made available to be used by a program. The obvious follow up question is then: "How to I choose which way?"
Static linking is how the bulk of the program itself is linked. All of your object files are listed, and get collected together in to the EXE file by the linker. Along the way, the linker takes care of minor chores like fixing up references to global symbols so that your modules can call each other's functions. Libraries can also be statically linked. The object files that make up the library are collected together by a librarian in a .LIB file which the linker searches for modules containing symbols that are needed. One effect of static linking is that only those modules from the library that are used by the program are linked to it; other modules are ignored. For instance, the traditional C math library includes many trigonometry functions. But if you link against it and use cos(), you don't end up with a copy of the code for sin() or tan() unless you also called those functions. For large libraries with a rich set of features, this selective inclusion of modules is important. On many platforms such as embedded systems, the total size of code available for use in the library can be large compared to the space available to store an executable in the device. Without selective inclusion, it would be harder to manage the details of building programs for those platforms.
However, having a copy of the same library in every program running creates a burden on a system that normally runs lots of processes. With the right kind of virtual memory system, pages of memory that have identical content need only exist once in the system, but can be used by many processes. This creates a benefit for increasing the chances that the pages containing code are likely to be identical to some page in as many other running processes as possible. But, if programs statically link to the runtime library, then each has a different mix of functions each laid out in that processes memory map at different locations, and there aren't many sharable code pages unless it is a program that all by itself is run in more than process. So the idea of a DLL gained another, major, advantage.
A DLL for a library contains all of its functions, ready for use by any client program. If many programs load that DLL, they can all share its code pages. Everybody wins. (Well, until you update a DLL with new version, but that isn't part of this story. Google DLL Hell for that side of the tale.)
So the first big choice to make when planning a new project is between dynamic and static linkage. With static linkage, you have fewer files to install, and you are immune from third parties updating a DLL you use. However, your program is larger, and it isn't quite as good citizen of the Windows ecosystem. With dynamic linkage, you have more files to install, you might have issues with a third party updating a DLL you use, but you are generally being friendlier to other processes on the system.
A big advantage of a DLL is that it can be loaded and used without recompiling or even relinking the main program. This can allow a third party library provider (think Microsoft and the C runtime, for example) to fix a bug in their library and distribute it. Once an end user installs the updated DLL, they immediately get the benefit of that bug fix in all programs that use that DLL. (Unless it breaks things. See DLL Hell.)
The other advantage comes from the distinction between implicit and explicit loading. If you go to the extra effort of explicit loading, then the DLL might not even have existed when the program was written and published. This allows for extension mechanisms that can discover and load plugins, for instance.

These .LIB import library files are used in the following project property, Linker->Input->Additional Dependencies, when building a bunch of dll's that need additional information at link time which is supplied by the import library .LIB files. In the example below to not get linker errors I need to reference to dll's A,B,C, and D through their lib files. (note for the linker to find these files you may need to include their deployment path in Linker->General->Additional Library Directories else you will get a build error about being unable to find any of the provided lib files.)
If your solution is building all dynamic libraries you may have been able to avoid this explicit dependency specification by relying instead on the reference flags exposed under the Common Properties->Framework and References dialog. These flags appear to automatically do the linking on your behalf using the *.lib files.
This however is as it says a Common Properties, which is not configuration or platform specific. If you need to support a mixed build scenario as in our application we had a build configuration to render a static build and a special configuration that built a constrained build of a subset of assemblies that were deployed as dynamic libraries. I had used the Use Library Dependency Inputs and Link Library Dependencies flags set to true under various cases to get things to build and later realizing to simplify things but when introducing my code to the static builds I introduced a ton of linker warnings and the build was incredibly slow for the static builds. I wound up introducing a bunch of these sort of warnings...
warning LNK4006: "bool __cdecl XXX::YYY() already defined in CoreLibrary.lib(JSource.obj); second definition ignored D.lib(JSource.obj)
And I wound up using the manual specification of Additional Dependencies to satisfy the linker for the dynamic builds while keeping the static builders happy by not using a common property that slowed them down. When I deploy the dynamic subset build I only deploy the dll files as these lib files are only used at link time, not at runtime.

Here are some related MSDN topics to answer my question:
Linking an Executable to a DLL
Linking Implicitly
Determining Which Linking Method to Use
Building an Import Library and Export File

There are three kinds of libraries: static, shared and dynamically loaded libraries.
The static libraries are linked with the code at the linking phase, so they are actually in the executable, unlike the shared library, which has only stubs (symbols) to look for in the shared library file, which is loaded at run time before the main function gets called.
The dynamically loaded ones are much like the shared libraries, except they are loaded when and if the need arises by the code you've written.

In my mind, there are two method to link dll to exe.
Use dll and the import library (.lib file) implicitly
Use functions like loadlibrary() explicitly