I have a common scenario:
Some source files for an executable which depend on some standard libraries and some user libraries. All the user libraries are statically linked into the executable whereas the standard libraries are linked dynamically to it.
Problem:
I believe that I have multiply defined symbols in my complete package (executable which already includes the user library code + shared standard libraries). The linker obviously has insight into it, but as I understand the linker won't complain unless it encounters multiple strong named symbols. My fear is that, while I am moving my code from solaris 8/sparc platform to solaris10/sparc platform, some standard unix functions have been implemented in the user libraries which are causing the app to crash at runtime. Note that the app runs fine in solaris 8/sparc platform
I have been facing weird issues which have led me to believe this might be the source
Modifying one variable from one library is changing the value of another variable in another library
Solaris 8-10: host2ip conversion problems
What I need:
Is there a way to easily list all multiply defined symbols?
Is there a way to easily list all multiply defined symbols stemming from user libraries?
Do you guys think the issue #1 might be caused by linking issues, or you feel it might be a sign of some other issue?
Edit1:
Since then I know that on generating map file using ld, it has a section of multiply defined symbol which I am going through to find names that look like standard library call. For people who do not know, the linker will only fail to link if it finds multiple symbols with the same name AND the names are strong names.
You could turn on MAP file generation in the compiler (actually linker) settings and look through the map file for symbols that match the UNIX system functions you are concerned about. You'd probably have to write a script to automate it, but this would be a good starting point. The command line switch is probably -map or something similar, it will depend on which compiler/linker you are using.
The actual problem that was happening is:
The library (let's call it lib1) had an array like below
#define ARRAY_SIZE 1024
SomeStruct* global_array[ARRAY_SIZE];
This array is used by my another library (let's call it lib2) which in turn is used by my application using an extern declaration for it.
While compiling lib2 (or is it the app not sure), we did not define ARRAY_SIZE at all. This somehow caused the compiler of lib2 (or the app) to miscalculate the size of global_array in-turn causing it to allocate the memory for some other variable at a location which was already allocated to the global_array.
By defining ARRAY_SIZE again while compiling my libs and apps, everything starts behaving normal. I do not fully understand what caused the issue and why it gets resolved since extern declaration of arrays do not contain the size. Also, if the library really used the MACRO ARRAY_SIZE, then why wouldn't the compilation fail? Also, there is a possibility, that the name used for the define is a standard name (the actual string was FD_SETSIZE)
My initial gut feeling about the linker was wrong.
Related
I'm struggling with a linking error.
I have 3 modules:
static library A which defines function ole::compound_document::find_storage(const std::string&);
shared library B which is linked to A and uses the function;
executable C which is linked to B and uses functions from B (but does not call directly functions of library A).
During the linking of the executable C, I receive the following error message:
../../bin/B.so: undefined reference to `ole::compound_document::find_storage(std::string const&)'
The function is defined in library A.
If I run utility nm on shared library B, I receive the following output:
0000000001841c70 T ole::compound_document::find_storage(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
U ole::compound_document::find_storage(std::string const&)
It shows two find_storage functions. One of them is defined another is not defined.
I'm trying to understand how can it happens. So far unsuccessful.
The problem appears under Linux (Ubuntu), compiler: clang-9. On Windows, I can build the libraries and the executable without any problem.
I've tried to create a minimal example, just putting 3 simple modules together with only a few functions. Everything works. The compiler uses only the first definition of the function. I don't understand where the second definition comes from. I suspected some mix of c++ standards but cannot find anything.
Any suggestions will be highly appreciated.
You will need to look for problems in library B's code. std::string, verbatim, is not something that's expected to be an actual type referenced from an exported symbol, since std::string is just an alias for a std::basic_string instance. Narrow this down by looking at symbols of all modules that were linked into library B, and once you identified the module that does, you'll need to figure out why.
Your question does not provide sufficient data to decisively identify the linking problem because, of course, that can only be done by inspecting all the object modules involved in the linking, and inspecting all the source code for the likely violations of the One Definition Rule, or ill-formed code that did not produce a diagnostic, but manifested itself as a link failure.
Therefore, the following answer is meant to be as a general guide to isolating these kinds of linking failure. I don't think this question qualifies to be hammered by the canonical answer.
You have a symbol in the linked shared library that's not getting resolved when linked to an executable, specifically:
ole::compound_document::find_storage(std::string const&)
Just like you ran nm on the shared library, you can use it on every module that went into the shared library, individually. This will find which object module the unresolved reference came from. If you don't find it, it must've come from the static library you linked with, so repeat your search there.
That reference came from one of the object modules that you used to build the shared library. You will find it this way, it's unlikely to pop into existence of its own.
Once you find the relevant module, you're then on your own, by looking at the actual code that was compiled, and figure out what's up. If you can't figure it out: divide and conquer. Take the object module, and split the source file into two files, half of the functions in each one, compile them separately, and then look and see where the unresolved reference comes from.
Finally: before doing all that, try the low hanging fruit: make clean, then recompile everything. This has all the hallmarks of a compiler switch, some of the object modules were compiled by a different compiler. If that static library was provided by a third party vendor as a binary blob, it must've been compiled by a different compiler or a different version of your compiler. C++ does not guarantee binary ABI compatibility.
As mentioned by n. 'pronouns' m.The problem was in inconsistent use of -D_GLIBCXX_USE_CXX11_ABI flag. I used c++14 standard, but some projects were compiled with the flag -D_GLIBCXX_USE_CXX11_ABI=1.
It was not easy to find out, because I use a lot of 3rd party libraries with conan package manager.
Situation:
Static library LIB1, compiled from source and linked as lib1.lib (with /MD).
Uses library LIB2 and has inside objects from lib2.lib
Static library LIB2, also compiled with /MD.
EXE that (not directly) depends on both libraries.
Result of linking this EXE on MSVC 15.9.19: a lot of LNK2005 errors like
lib2.lib: error LNK2005: "function <funcsig> already defined in lib1.lib"
Also I get a lot of warnings like
lib1.lib: warning LNK4099: PDB 'lib2.pdb' was not found with 'lib1.lib' or at '<path>'; linking object as if no debug info
The question: why didn't the linker merge duplicate definitions? How do I diagnose the exact reason for this problem?
Thanks!
UPDATE:
The errors are NOT about the standard library. They are about the Google Protobuf functions. LIB2 is Google's libprotobuf.lib. LIB1 is also Google's OR-Tools library that uses Protobuf. But we also use Protobuf, hence the conflict!
This link to MSDN will almost certainly help.
As you say these are not standard C/C++ functions, you can ignore the sections about the mixing Debug and Release code. Unfortunately this still leaves you with plenty of possible causes. This one is a good one to rule out.
This error can occur if you mix use of static and dynamic libraries when you use the /clr option.
If you view the command line options as you compile, either by switching on verbose mode or examining the properties of each file.
Once this is ruled out, all the other causes are that you literally declaring the same thing (function/variable) twice. The most likely way for this to happen is to put something (like a function) in a header file, which is included by more than one module file. Diagnosing this without seeing the code is hard. You must pick one of the errors, and select part of the name that is not mangled, basically the human readable bit. You then need to examine where it is declared, and see if is in a header. If it is not, you must be including a non-header from a source file. A quick (and dirty) way to find where a file is included from is to add a #pragma message to the file, then recompile one file at a time to see when it is printed. To understand why it is included use show includes.
If the symbol is declared from a header file, you must fix it, by making it a forward declare.
If you don't know how to do this, I would suggest starting a new question, along the lines of, "how do I forward declare this?". It should get you answers pretty quickly.
Hope this helps.
Description :
a. Class X contains a static private data member ptr and static public function member getptr()/setptr().
In X.cpp, the ptr is set to NULL.
b. libXYZ.so (shared object) contains the object of class X (i.e libXYZ.so contains X.o).
c. libVWX.so (shared object) contains the object of class X (i.e libVWX.so contains X.o).
d. Executable a.exe contains X.cpp as part of translation units and finally is linked to libXYZ.so, libVWX.so
PS:
1. There are no user namespaces involved in any of the classes.
2. The libraries and executable contain many other classes also.
3. no dlopen() has been done. All libraries are linked during compile time using -L and -l flags.
Problem Statement:
When compiling and linking a.exe with other libraries (i.e libXYZ.so and libVWX.so), I expected a linker error (conflict/occurance of same symbol multiple times) but did not get one.
When the program was executed - the behavior was strange in SUSE 10 Linux and HP-UX 11 IA64.
In Linux, when execution flow was pushed across all the objects in different libraries, the effect was registered in only one copy of X.
In HPUX, when execution flow was pushed across all the objects in different libraries, the effect was registered in 3 differnt copies of X (2 belonging to each libraries and 1 for executable)
PS : I mean during running the program, the flow did passed thourgh multiple objects belonging to a.exe, libXYZ.so and libVWX.so) which interacted with static pointer belonging to X.
Question:
Is Expecting linker error not correct? Since two compilers passed through compilation silently, May be there is a standard rule in case of this type of scenario which I am missing. If so, Please let me know the same.
How does the compiler (gcc in Linux and aCC in HPUX) decide how many copies of X to keep in the final executable and refer them in such scenarios.
Is there any flag supported by gcc and aCC which will warn/stop compilation to the users in these kind of scenarios?
Thanks for your help in advance.
I'm not too sure that I've completely understood the scenario. However,
the default behavior on loading dynamic objects under Linux (and other
Unices) is to make all symbols in the library available, and to only use
the first encountered. Thus, if you both libXYZ.so and libVWX.so
contain a symbol X::ourData, it is not an error; if you load them in
that order, libVWX.so will use the X::ourData from libXYZ.so,
instead of its own. Logically, this is a lot like a template definition
in a header: the compiler chooses one, more or less by chance, and if
any of the definitions is not the same as all of the others, it's
undefined behavior. This behavior can be
overridden by passing the flag RTLD_LOCAL to dlopen.
With regards to your questions:
The linker is simply implementing the default behavior of dlopen (that which you get when the system loads the library implicitely). Thus, no error (but the logical equivalent of undefined behavior if any of the definitions isn't the same).
The compiler doesn't decide. The decision is made when the .so is loaded, depending on whether you specify RTLD_GLOBAL or RTLD_LOCAL when calling dlopen. When the runtime calls dlopen implicitly, to resolve a dependency, it will use RTLD_GLOBAL if this occurs when loading the main executable, and what ever was used to load the library when the dependency comes from a library. (This means, of course, that RTLD_GLOBAL will propagate until you invoke dlopen explicitly.)
The function is "public static", so I assume it's OOP-meaning of "static" (does not need instance), not C meaning of static (file-static; local to compilation unit). Therefore the functions are extern.
Now in Linux you have explicit right to override library symbols, both using another library or in the executable. All extern symbols in libraries are resolved using the global offset table, even the one the library actually defines itself. And while functions defined in the executable are normally not resolved like this, but the linker notices the symbols will get to the symbol table from the libraries and puts the reference to the executable-defined one there. So the libraries will see the symbol defined in the executable, if you generated it.
This is explicit feature, designed so you can do things like replace memory allocation functions or wrap filesystem operations. HP-UX probably does not have the feature, so each library ends up calling it's own implementation, while any other object that would have the symbol undefined will see one of them.
There is a difference beetween "extern" symbols (which is the default in c++) and "shared libary extern". By default symbols are only "extern" which means the scope of one "link unit" e.g. an executable or a library.
So the expected behaviour would be: no compiler error and every module works with its own copy.
That leads to problems of course in case of inline compiling etc...etc...
To declare a symbol "shared library extern" you have to use a ".def" file or an compiler declaration.
e.g. in visual c++ that would be "_declspec(dllexport)" and "_declspec(dllimport)".
I do not know the declarations for gcc at the moment but I am sure someone does :-)
I'll divide this questions in 3 parts:
I would like to produce a static library and strip off its symbols. (Debug info is already not included)
Similar to the strip command in linux. Can it be done?
Is there an equivalent tool in windows env, to the nm tool in linux?
When creating a static library using VS2008. Is it possible to define a script that will exclude some of the produced .obj files out of the build and out of the static lib?
Can it be dynamic? I mean I'd define a compilation mode in the script and this would result in specific object files being excluded from the build
If anything is visible that you feel should not be, try declaring it with the "static" keyword. This tells the compiler that it is accessible only to the current module.
There are cases where it would be convenient to be able to strip out all but a small number of "exported" public symbols, but it's not really feasible.
A static library is little more than a collection of .obj files. The internal dependencies haven't been resolved yet, and they won't be resolved until link time.
For example, if your .lib consists of foo.obj and bar.obj, and there's a call in foo.obj to a function defined in bar.obj, then that symbol must be available at link time, even if nothing outside of the library should be able to see it.
For that reason, you cannot strip the symbols (with the possible exception of file-scope static symbols). Even class methods that are protected or private (in the C++-sense) will exist in the symbol table, since the enforcement of the visibility is a compile-time issue, not a link-time one.
In contrast, a dynamic library is a standalone binary that has already been linked. References from foo.obj to bar.obj have already been resolved. Thus a DLL can be stripped of symbols except for the ones that must be exported (and even those can be renamed or replaced by ordinals).
If your DLL exposes a simple C API, then you're all set. But if you want to expose a C++ class, you're probably going to end up exporting all of its methods, even the protected and private ones (since inlining in the external application might result in direct calls to private methods).
No, how do you think the users of the static library would link to it without knowing where are the symbols they use defined?
Yes, try the DUMPBIN utility.
Well, yes. You can run the LIB utility with /REMOVE:foo.
That said, I think you are doing something that either is not worth doing or could be done a lot simpler than with removing library members.
I kept finding the names of certain (but not all) static functions in .obj files produced by VS2010. Interestingly, they were visible in my Release .obj files but not the Debug .obj files. I just used cygwin strings to perform the search:
$ strings myObjectFile.obj | grep myStaticFunctionName
I tracked it down to the "Whole Program Optimization = Yes" setting ("/GL"). When I switched this to "No" the function names no longer appear.
Update: As a followup test I opened the "cleansed" myObjectFile.obj in vim and I can still find them (with either :set encoding=utf-8 or :set encoding=latin1). I'm not sure why strings was missing the matches. Oh well.
I am writing a hello world c++ application, in the instruction #include help the compiler or linker to import the c++ library. My " cout << "hello world"; " use a cout in the library. The question is after compile and generated exe is about 96k in size, so what instructions are actually contained in this exe file, does this file also contains the iostream library?
Thanks
In the general case, the linker will only bring in what it needs. Once the compiler phase has turned your source code into an object file, it's treated much the same as all other object files. You have:
the C start-up code which prepares the execution environment (sets up argv, argv and so on) then calls your main or equivalent.
your code itself.
whatever object files need to be dragged in from libraries (dynamic linking is a special case of linking that happens at runtime and I won't cover that here since you asked specifically about static linking).
The linker will include all the object files you explicitly specify (unless it's a particularly smart linker and can tell you're not using the object file).
With libraries, it's a little different. Basically, you start with a list of unresolved symbols (like cout). The linker will search all the object files in all the libraries you specify and, when it finds an object file that satisfies that symbol, it will drag it in and fix up the symbol references.
This may, of course, add even more unresolved symbols if, for example, there was something in the object file that relies on the C printf function (unlikely but possible).
The linker continues like this until all symbols are satisfied (when it gives you an executable) or one cannot be satisfied (when it complains to you bitterly about your coding practices).
So as to what is in your executable, it may be the entire iostream library or it may just be the minimum required to do what you asked. It will usually depend on how many object files the iostream library was built into.
I've seen code where an entire subsystem went into one object file so, that if you wanted to just use one tiny bit, you still got the lot. Alternatively, you can put every single function into its own object file and the linker will probably create an executable as small as possible.
There are options to the linker which can produce a link map which will show you how things are organised. You probably won't generally see it if you're using the IDE but it'll be buried deep within the compile-time options dialogs under MSVC.
And, in terms of your added comment, the code:
cout << "hello";
will quite possibly bring in sizeable chunks of both the iostream and string processing code.
Use cl /EHsc hello.cpp -link /MAP. The .map file generated will give you a rough idea which pieces of the static library are present in the .exe.
Some of the space is used by C++ startup code, and the portions of the static library that you use.
In windows, the library or part of the libraries (which are used) are also usually included in the .exe, the case is different in case of Linux. However, there are optimization options.
I guess this Wiki link will be useful : Static Libraries