With LD version scripts, it's possible to specify symbols to be exported via wildcard patterns. But I cannot find how to do the same with MSVC module definition (.def) files.
e.g. With the following version script, getFoo along with all symbols inside mynamespace::bar:: will be exported while all other symbols will be hidden.
{
global:
getFoo;
extern "C++" {
mynamespace::bar::*;
};
local: *;
};
With a .def file, I can get getFoo exported with an EXPORTS statement in the .def file; but how can I achieve the same effect of exporting everything inside mynamespace::bar::? Is there a way to get the MSVC linker to recognize wildcard patterns?
Note that I don't necessarily need to make this work with a .def file per se. If I can somehow get the MSVC linker to recognize the version script that is fine too. However, when I try to do this
with a -Wl,--version-script=libfoo.map linker flag, MSVC complains about not recognizing this option.
Related
The C++ modules TS provides an excellent facility for eliminating the preprocessor, improving compile times, and generally supporting much more robust, modular, code development in C++, for non-template code at least.
The underlying machinery provides control over import and export of symbols in ordinary programs.
However, there is a major problem developing libraries for two kinds of dynamic loading: startup time loading, and run time loading. This problem involves the exporting of symbols from the library, which is often discussed in terms of visibility.
Generally, not all the extern symbols of the translation units used to construct a dynamic link library should be made visible to the user. In addition, with run time loading, especially with a plugin concept, the same symbol must be exported from many concurrently loaded libraries.
On Windows the use of language extensions
__declspec(dllexport)
__declspec(dllimport)
attached in source code as attributes of symbols, and more recently on gcc and clang systems on unix platforms, the use of
__attribute__((visibility("default")))
__attribute__((visibility("hidden")))
are intended to support the provision and use of symbols intended to be made public by the library. Using these is complicated and messy: on Windows macros must be used to export the symbols whilst compiling the library, but import them when using it. On the unix platforms, the visibility must be set to default to both export and import the symbols, the compiler deciding itself, based on whether a definition is found or not: the compiler must be invoked with
-fvisibility=hidden
switch. The export/import attributes are not required for static linkage, and probably should be macro'd out to an empty string. Making code and fiddling the build system so that this all works, especially considering that #includes must have the correct symbol visibility set during compilation of library translation units is very hard, the file structure required in repositories is a mess, the source code is littered with macros, and in general .. the whole thing is a disaster. Almost all open source repositories FAIL to correctly export symbols for dynamic linkage, and most programmers have no idea that dynamic library code structure (using two level namespaces) is quite different to static linkage.
An example of how to do it (hopefully correctly) can be seen here:
https://github.com/calccrypto/uint256_t
This repository used to have 2 headers and 2 implementation files, the user of the built library would see 2 headers. There are now 7 headers and 2 implementation files and the user of the built library will see 5 header files (3 with extension include to indicate they're not to be directly included).
So after that long winded explanation, the question is: will the final C++ modules specification help to solve problems with export and import of symbols for dynamic linkage? Can we expect to be able to develop for shared libraries without polluting our code with vendor specific extensions and macros?
Modules don't help you with symbol visibility across DLL boundaries. We can check this with a quick experiment.
// A.ixx
export module A;
export int f() { return 1; }
Here we have a simple module interface file exporting one symbol f in the module interface of module A (happens to share the same name as the file base name, but this isn't necessary). Let's compile this like so:
cl /c /std:c++20 /interface /TP A.ixx
The /c flag avoids invoking the linker (happens automatically by default), c++20 or later is required for module syntax to work, and the /interface flag lets the compiler know we are compiling a module interface unit. The /TP arg says "treat the source input as a C++ input" and is needed when /interface is specified. Finally, we have our input file.
Running the above produces an interface file A.ifc and an object file A.obj. Note that there is no import lib file or exp file you would expect if you were compiling a DLL.
Next, let's write a file that consumes this.
// main.cpp
import A;
int main() { return f(); }
To compile this into an executable, we can use the following command:
cl /std:c++20 main.cpp A.obj
The presence of the A.obj input there is not optional. Without it, we get a classic linker error of f being an unresolved symbol. If we run this, we'll get a main.exe which statically links the code in A.obj.
What happens if we try and compile A.ixx into a DLL? That is, what if we try to produce a DLL with the linker from A.obj? The answer you get a DLL but no import lib or exp. If you try running link /noentry /dll A.obj /out:A.dll you will get an A.dll with the expected /disasm section (visible via dump bin), but no export table.
Dump of file A.dll
File Type: DLL
0000000180001000: B8 01 00 00 00 mov eax,1
0000000180001005: C3 ret
Summary
1000 .rdata
1000 .text
That's the disassembly in A.dll which we expect, but checking the exports section with dumpbin /export A.dll reveals nothing. The reason is of course, we didn't export the symbol!
If we change the source code of A.ixx to the following:
// A.ixx
export module A;
export __declspec(export) int f() { return 1; }
... we can repeat the steps (compile A.obj, link A.dll) to find that this time, the linker produces an import lib and exp file as we'd expect. Invoking dumpbin /exports A.lib on the import lib generated should show the ?f##YAHXZ symbol present.
Now, we can link main.cpp again A.lib (as opposed to A.obj) via cl /std:c++20 main.cpp A.lib to produce a valid executable, this time relying on A.dll for the code instead of having f statically embedded.
We can check that this is in fact happening as expected in WinDbg.
Note on the lower left module pane the presence of A.dll. Note also that in the disassembly view in the center, we are about to call main!f. Uh oh, not good. While this does properly resolve to the !A module, it does so via an extra indirection in the import address table as seen here:
This is the classic problem that happens when you forget to decorate a function or symbol with the __declspec(dllimport) directive. When the compiler encounters the symbol without the dllimport directive that it doesn't recognize, it emits a relocation entry which is expected to be resolved at link time. Along with that entry, it emits a jmp and an unresolved address. This is a classic problem that I won't get into here, but the upshot is that we have an extra unnecessary indirection because the symbol recognized as exported from the module A was expected to be statically linked.
It turns out, we can't fix this easily. If we try to add another declaration of f to main.cpp or some other translation unit, the linker will complain that it sees f with "inconsistent dll linkage." The only way to resolve this is to compile a second version of the A module interface with dllimport decorations (much like how headers typically have macros that expand to dllexport or dllimport depending on the TU using the header).
The moral of the story is that DLL linkage and module linkage, while not completely at odds, aren't particularly compatible either. The module export does not include exported symbols in the export table, needed to resolve symbols across DLL boundaries. Furthermore, putting these symbols in the export table still leaves you the trouble of an extra indirection after the implicit dynamic link is done via the import address table.
I would like to hide exported symbols from a DLL for obfuscation purposes.
That is pretty neatly doable when using a module definition file (.def) looking something like this;
EXPORTS
??0Foo#QAE#XZ #1 NONAME
??1Foo#QAE#XZ #2 NONAME
?Bar#Foo#UAEHXZ #3 NONAME
Trouble is, such solution is highly inflexible and demands manual work. As you can see within my example, I am exporting C++ symbols, hence they are heavily decorated by my compiler.
So my current workflow looks like this;
I have to first create a version of my DLL that exports all symbols in the standard way using __declspec(dllexport), then I need to extract all exported symbol names using dumpbin or alike. After that is done, I need to copy&paste the symbols into my module definition file and add that NONAME directive. Then I have to make sure that my original sources do not use that __declspec(dllexport) anymore. Once all of that is done, I need to activate that .def file within the project settings and then I can finally build the export symbol free version of that DLL. Plenty of work for that rather simple task, I guess.
Before covering all of this using a bunch of scripts and stuff, I thought that maybe, just maybe there is a solution that is much simpler?
Please note that I am using VisualStudio (2012) and hence that nifty GCC pragma hidden wont do, as far as I know.
How do you merge multiple static linked libraries into a single dll given each static lib defines exported functionality (vc++ 2008)?.
In a multi project layout existing out of a single dll project and multiple sub projects that are linked in statically (in the dll project). Despite being marked as __declspec(export) some of the symbols in the sub-projects (.lib) refuse to have their symbols exported in the final dll.
Generating a .def file and marking the symbols explicitly for exportation could solve this problem. However identifying which of the symbols are marked as __declspec( export ) proofs a problem. Due large number of exported classes/function and primarily name mangling maintaining a list by hand is a unfordable process thus generating the list of symbols, that were marked for export, would be the only viable option.
Is there an utility or compiler directive could do this?
Use a DEF file.
Always use a DEF file.
Never fail to use a DEF file.
Just accept that a DEF file is the thing to use.
Stop using __declspec(dllexport), and use a dang-dratted def file already.
Also don't export classes. Export those class members which need to be exported only. And use a DEF file to do it.
Seriously, if you export classes without a DEF file, the function names will be several times longer than the actual program data. You should to use ordinals for exporting C++ member functions.
After bit trial and error I found that using the lib /def command can be utilized to generate an import library and export file. It appears that the export file contains all symbols that are marked with __declspec(dllexport). Subsequently the .exp file can be inspected with dumpbin and used as a reference to generate a module definition file.
Starting with Visual Studio 2015 Update 2 there is a new way of doing this, by using the linker option /WHOLEARCHIVE
It's documented here
The /WHOLEARCHIVE option forces the linker to include every object
file from either a specified static library, or if no library is
specified, from all static libraries specified to the LINK command. To
specify the /WHOLEARCHIVE option for multiple libraries, you can use
more than one /WHOLEARCHIVE switch on the linker command line. By
default, the linker includes object files in the linked output only if
they export symbols referenced by other object files in the
executable. The /WHOLEARCHIVE option makes the linker treat all object
files archived in a static library as if they were specified
individually on the linker command line.
I declared some C++ functions prototyped as follows:
extern "C" void __stdcall function();
I also have some third-party dll with exported function() - no name decorations at all.
I'm unable to build my exe or dll due to undefined reference to function#..., because of MinGW's stdcall #-suffix. How can I obtain object file without #... just plain function names?
It sounds like you're trying to use MinGW to compile a program that uses external C functions from a third-party dll. There's a way to export those external functions into a proper import library that MinGW's gnu ld linker can use but it involves creating a .def definition file. The advantage with this is once you create a proper import library you won't have to fiddle around with switches like --add-stdcall-alias or --kill-at because the import library will contain the symbols expected by the compiler and linker.
Here's a rough outline of the procedure for doing this:
You'll need a tool call dlltool.exe which should be included in the same MinGW/bin directory as the compiler.
You'll need to create a definition file (*.def) listing all the external functions you're interested in importing.
Create the import file stub (*.a) by running dlltool passing in the .def file you created as input.
Pass the newly created import file *.a to the linker when building your project so the symbols can be resolved properly.
Here's how the definition file looks like:
;Run the dlltool like this:
;dlltool -k -d third_party.def -l libthird_party.a
LIBRARY third_party.dll
EXPORTS
dll_function1#0
dll_function2#8
dll_function3#16
; ...
dll_function_n#24
Couple of important things to note. The EXPORTS section has to list the exported symbols/functions in the same name-decorated format as expected by the toolchain. In this case, MinGW compiler and ld linker expects __stdcall C functions to be appended with an '#' follow by the number of bytes in the arguments. The second important thing to note is that dlltool -k will remove the '#', which does the same thing as the --kill-at option you've already seen. The end result of this is you have an import library with the correct internal name-decoration, so things resolve properly, and that internal name will map to the exported visible name found in your 3rd party dll.
One final thing that needs mentioning. Throughout the entire example, we were assuming the undecorated names in the dll used the __stdcall which isn't necessarily true. The following chart(taken from here) shows how different compilers decorate __cdecl vs __stdcall differently:
MSVC DLL
Call Convention | (dllexport) | DMC DLL | MinGW DLL | BCC DLL
----------------------------------------------------------------------------
__stdcall | _Function#n | _Function#n | Function#n | Function
__cdecl | Function | Function | Function | _Function
It's up to you to make sure that the call conventions match-up properly or risk stack corruption and mysterious program crashes.
man ld gives you the --enable-stdcall-fixup option for linking against non-#-decorated libraries. I'm not sure if the leading underscores will be a problem, you'll have to try.
If you want to produce object files or DLLs with mingw and want export names without the "#", --kill-at is your friend.
I am building a shared library on Ubuntu 9.10. I want to export only a subset of my functions from the library. On the Windows platform, this would be done using a module definition (.def) file which would contain a list of the external and internal names of the functions exported from the library.
I have the following questions:
How can I restrict the exported functions of a shared library to those I want (i.e. a .def file equivalent)
Using .def files as an example, you can give a function an external name that is different from its internal name (useful for prevent name collisions and also redecorating mangled names etc)
On windows I can use the EXPORT command (IIRC) to check the list of exported functions and addresses, what is the equivalent way to do this on Linux?
The most common way to only make certain symbols visible in a shared object on linux is to pass the -fvisibility=hidden to gcc and then decorate the symbols that you want to be visible with __attribute__((visibility("default"))).
If your looking for an export file like solution you might want to look at the linker option --retain-symbols-file=FILENAME which may do what you are looking for.
I don't know an easy way of exporting a function with a different name from its function name, but it is probably possible with an elf editor. Edit: I think you can use a linker script (have a look at the man page for ld) to assign values to symbols in the link step, hence giving an alternative name to a given function. Note, I haven't ever actually tried this.
To view the visible symbols in a shared object you can use the readelf command. readelf -Ds if I remember correctly.
How can I restrict the exported functions of a shared library to those I want (i.e. a .def file equivalent)
Perhaps you're looking for GNU Export Maps or Symbol Versioning
g++ -shared spaceship.cpp -o libspaceship.so.1
-Wl,-soname=libspaceship.so.1 -Wl,
--version-script=spaceship.expmap
gcc also supports the VC syntax of __declspec(dllexport). See this.
Another option is to use the strip command with this way:
strip --keep-symbol=symbol_to_export1 --keep-symbol=symbol_to_export2 ... \
libtotrip.so -o libout.so