This question already has answers here:
How does the linker handle identical template instantiations across translation units?
(3 answers)
Closed 4 years ago.
When I include, say <vector> in a C++ file, and create an instance of std::vector<double>, and then call the std::vector<double>::push_back() function, my understanding is that at that point, the compiler creates a definition of the function and generates the necessary code. However, I can call that function from several distinct C++ files which will be linked into the same executable, and presumably each .cpp file will generate a .o file each it's own definition of std::vector<double>::push_back(). Normally, that kind of multiple function definition would generate a linking error, but it doesn't seem to happen for templates- why is that so?
Compilers can flag certain symbols to be autoremoved in case the linker sees redefinition.
The "symbol" in this case will be the unique name (the mangled name) that the compiler assigns to each templated function, or templated member function, or member function of a templated class, after instantiation with a concrete type. In different translation units (different .cpp), the generated unique names will be exactly the same. Consider that, besides exceptions and polimorphic hierarchies, a C++ program is basically translated to a C program before applying the final translation to assembler code, so your program get up being a bunch of functions, some of them being inline functions, others instantiated templated functions, and others usual free functions. All but the thirds are flagged by the compiler to let the linker know redefinitions are possible comming and they must be silently removed.
If a different translation unit uses a same templated method instantiated with a same type, the mangled name will also be the same, and then flagged by the compiler. Finally, in the linking stage, the linker will see two redefinitions of a same function, but flagged; so the linker must remove one of them and only one will survice the purge.
It's the same with inline methods.
This causes no harm since a same method from a same instiantated templated class for a same type, will produce the exact same assembler code, no matter from which translation unit.
Otherwise it would be impossible to have inline or templated methods.
AFAIK, originally linkers didn't allow this kind of flagging; it was explicitely added to make C++ implementable.
Related
Suppose we have this header, with a function:
//template<int does_nothing=0> // <-- !! links correctly if uncommented
void incAndShow() {
static int myStaticVar = 0;
std::cout << ++myStaticVar << " " << std::endl;
}
If this is included in several .cpp files, you'll get a linker error due to the redefinition of incAndShow().
Now, if you uncomment the //template<int does_nothing=0> line above, this code will compile correctly with no linker error. And, the static variable will be shared correctly in all instances of incAndShow<0>().
This is because the C++ standard explicitly states that this should be allowed in its "One Definition Rule" (Page 43). The abbreviated text is:
There can be more than one definition of a... non-static function template (17.5.6)... provided that each definition appears in a different translation unit...
Now, note that the above is only true for templated functions, and not for non-templated functions. This is because, at least in the traditional paradigm, non-templated functions are compiled individually within each translation unit, and then the linker, envisioned as a totally separate process from the compiler, sees multiple definitions of the same symbol in different .obj files and throws the error.
But somehow, this doesn't happen for templated functions.
So my question is: how, in a low-level sense, does this work?
Does the compiler just compile each translation unit independently, meaning it generates multiple versions of the same symbol within each translation unit, but then have some special way to tell the linker to not worry about it?
Does the compiler, on-the-fly, keep a note of the first time some templated function definition is seen in some translation unit, and automatically ignore upon seeing the same definition again when compiling the next translation unit?
Does the compiler do some kind of "pre-compilation process," before compiling anything, where it goes through each file and removes duplicate templated code?
Something else?
This has been brought up on StackOverflow a few times, but this is the first time I've seen anyone ask how the compilation process really happens on a low-level:
Static variable inside template function (where my example came from)
Why C++'s <vector> templated class doesn't break one definition rule?
In C and C++, you can't have a function with two definitions. For example, say we have the following two files:
1.c:
int main(){ return 0}
2.c:
int main(){ return 0}
Issuing the command gcc 1.c 2.c will give you a duplicate symbol linker error.
Why doesn't the same happen with structs and classes? Why are we allowed to have multiple
definitions of the same struct as long as they have the same tokens?
To answer this question, one has to delve into compilation process and what is needed in each part (question why these steps are perfomed is more historical, going back to beginning of C before it's standardization)
C and C++ programs are compiled in multiple steps:
Preprocessing
Compilation
Linkage
Preprocessing is everything that starts with #, it's not really important here.
Compilation is performed on each and every translation unit (typically a single .c or .cpp file plus the headers it includes). Compiler takes one translation unit at a time, reads it and produces an internal list of classes and their members, and then assembly code of each function in given unit (basing on the structures list). If a function call is not inlined (e.g. it is defined in different TU), compiler produces a "link" - "please insert function X here" for the linker to read.
Then linker takes all of the compiled translation units and merges them into one binary, substituting all the links specified by compiler.
Now, what is needed at each phase?
For compilation phase, you need the
definition of every class used in this file - compiler needs to know the size and offset of each class member to produce assembly
declaration of every function used in this file - to produce those "links".
Since function definitions are not needed for producing assembly (as long as they are compiled somewhere), they are not needed in compilation phase, only in linking phase.
To sum up:
One Definition Rule is there to protect programmers from theselves. If they'd accidentally define a function twice, linker will notice that and executable is not produced.
However, class definitions are required in every translation unit, and therefore such a rule cannot be set up for them. Since it cannot be forced by language, programmers have to be responsible beings and not define the same class in different ways.
ODR has also other limitations, e.g. you have to define template functions (or template class methods) in header files. You can also take the responsibility and say to the compiler "Every definition of this function will be the same, trust me dude" and make the function inline.
There is no use case for a function with 2 definitions. Either the two definitions would have to be the same, making it useless, or the compiler wouldn't be able to tell which one you meant.
This is not the case with classes or structures. There is also a large advantage to allowing multiple definitions of them, i.e. if we want to use a class or struct in multiple files. (This leads indirectly to multiple definitions because of includes.)
Structures, classes, unions and enumerations define types that can be used in several compilation units to define objects of these types. So each compilation unit need to know how the types are defined, for example to allocate correctly memory for an object or to be sure that specified member of a class does indeed exist.
For functions (if they are not inline functions) it is enough to have their declaration without their definition to generate for example a function call.
But the function definition shall be single. Otherwise the compiler will not know what function to call or the object code will be too big due to duplication and will be error prone..
It's quite simple: It's a question of scope. Non-static functions are seen (callable) by every compilation unit linked together, while structures are only seen in the compilation unit where they are defined.
For example, it's valid to link the following together because it's clear which definition of struct Foo and which definition of f is being used:
1.c:
struct Foo { int x; };
static void f(void) { struct Foo foo; ... }
2.c:
struct Foo { double d; };
static void f(void) { struct Foo foo; ... }
int main(void) { ... }
But it isn't valid to link the following together because the linker wouldn't know which f to call.
1.c:
void f(void) { ... }
2.c:
void f(void) { ... }
int main(void) { f(); }
Actually every programming element is associated with a scope of its applicability. And within this scope you cannot have the same name associated with multiple definitions of an element. In compiled world:
You cannot have more than one class definition with the same name within a single file. But you can have it in different compilation units.
You cannot have the same function or global variable name within a single link unit (library or executable), but you can potentially have functions named the same within different libraries.
you cannot have shared libraries with the same name situated in the same directory, but you can have them in different directories.
C/C++ compilation is very much after the compilation performance. Checking 2 objects like function or classes for identity is a time-consuming task. So, it is not done. Only names are considered for comparison. It is better to consider that 2 types are different and error out then checking them for identity. The only exception from this rule are text macros.
Macros are a pre-processor concept and historically it is allowed to have multiple identical macro definitions. If a definition changes, a warning gets generated. Comparing macro context is easy, just a simple string comparison, but some macro definitions could be huge.
Types are the compiler concept and they are resolved by the compiler. Types do not exist in object libraries and are represented by the sizes of corresponding variables. So, there is no reason for checking type name collisions at this scope.
Functions and variables on the other hand are named pointers to executable codes or data. They are the building blocks of applications. Applications are assembled from the codes and libraries coming from all around the world in some cases. In order to use someone else's function you'd better now its name and you do not want the same name to be used by some one else. Within a shared library names of functions and variables are usually stored in a hash table. There is no place for duplicates there.
And as I already mention checking functions for identical contents is seldom done, however there are some cases, but not in c or c++.
The reason of impeding two different definitions for the same thing to be used in programming is to avoid the ambiguity of deciding which definition to use at run time.
If you have two different implementations to the same thing to coexist in a program, then there's the possibility of aliasing them (with a different name each) into a common reference to decide at runtime which one of the two to use.
Anyway, in order to distinguish both, you have to be able to indicate the compiler which one you want to use. In C++ you can overload a function, giving it the same name and different lists of parameters, so you can distinguish which one of both you want to use. But in C, the compilers only preserve the name of the function to be able to solve at link time which definition matches the name you use in a different compilation unit. In case the linker ends with two different definitions with the same name, it is uncapable of deciding for you which one to use, so it emits an error and gives up the building process.
What should be the intention of using this ambiguity in a productive way? this is the question you have actually to ask to yourself.
This question already has answers here:
How does C++ link template instances
(4 answers)
Closed 7 years ago.
Templates and inline functions should be provided in the header, so the same code will go inside each translation unit that uses them.
Unless the compiler chooses to actually put the template or inline code in-line, it would be nice for the linker to remove the same code appearing in each translation unit. Does this actully happen? required by the standard? or up to the compiler?
First of all there's no such thing as the linker in the standard, so if you're looking at the standard you have to call it the implementation (which is responsible for turning the source code into an executable in some sense). IIRC there's a requirement that the same template in different translation units should have the same address (but I may be mistaken). Inline function otoh is the same as a static function, just allowed to be declared in every translation unit without complaint (note that the compiler is not required to actually inline the inline function or forbidden to inline other functions).
However normally a linker will do just what you suggest, namely to put only one instance of the same object in the executable (under this circumstance). There are features in the language that relies on the linker being able to do that anyway (IIRC) so there's no reason why the linker shouldn't be that smart anyway.
Then it's good question if you should rely on this fact in your code. Granted it would be nice if the executable were as small as possible, but other than that? How often do you compare if function f1 is the same as function f2 or in other way relies on that (f1 and f2 being pointer to the samely named function retrieved from different translation units)? If you require the executable to be small, you should of course select an implementation (compiler+linker) that produces just that.
inline functions are only suggestion we can ask (follow the defined rules for inline functions), compiler to look at (different compilers have different criteria for handling inline functions).
Now templates! Templates need two phase compilations and they are not treated as inline functions.
vector v1<int>;
vector v2<int>;
both statements require two phase compilations (v1 and v2 have same type, but only we know it, not the compilers). Different optimizations may have been applied but this is general working method for template compilations
This question already has answers here:
Does it make any sense to use inline keyword with templates?
(4 answers)
Closed 8 years ago.
This code will be placed in a header file:
template<typename TTT>
inline Permutation<TTT> operator * (const Cycle<TTT>& cy, const Permutation<TTT>& p)
{
return Permutation<TTT>(cy)*p;
}
Is inline necessary to avoid a linker error?
If this function is not a template and the header file is used in more than one .cpp file, inline is necessary to avoid a liker error complaining about multiple definitions for a function. It seems linker ignores this for templates.
Is inline necessary to avoid a linker error?
On a function template, no. Templates, like inline functions, are subject to a more relaxed One Definition Rule which allows multiple definitions - as long as the definitions are identical and in separate translation units.
As you say, inline would be necessary if you wanted to define a non-template function in a header; non-inline functions are subject to a more strict One Definition Rule, and can only have one definition in a program.
For the gory details, this is specified by C++11 3.2/5:
There can be more than one definition of a class type, inline function with
external linkage, class template, non-static function template, static data member
of a class template, member function of a class template, or template specialization for
which some template parameters are not specified in a program provided that each definition
appears in a different translation unit, and provided the definitions satisfy the following requirements.
(The "following requirements" basically say that the definitions must be identical).
Consider that a template function (or function template if you prefer) is not a function at all. It is rather a recipe to create a function. The actual function is only created when and where the template is instantiated. So you do not need the inline keyword here, because template functions will not result in multiple-definition linker errors because they are not actually defined (from the linker's perspective) until they are used.
Expanding on Mike Seymour's answer -- the paragraph (3.2/5) he cites in in the Standard refers to a concept called "vague linkage". Basically, it's a way of saying "we need this to exist somewhere in the resulting binary, but we don't have a clear-cut home for it in any specific object file we're emitting." On modern platforms (Windows, ELF systems such as Linux, and OS X), this is implemented using a mechanism known as COMDAT support that allows the compiler to simply generate the instantiations and other vague linkage items (vtables, typeinfos, and inline function bodies) as-needed -- the linker then is free to throw out the duplicates:
When used with GNU ld version 2.8 or later on an ELF system such as GNU/Linux or Solaris > 2, or on Microsoft Windows, duplicate copies of these constructs will be discarded at
link time. This is known as COMDAT support.
This is discussed in more detail in the GCC manual (quote snipped as the Cfront model is irrelevant for modern compilers):
C++ templates are the first language feature to require more intelligence from the
environment than one usually finds on a UNIX system. Somehow the compiler and linker
have to make sure that each template instance occurs exactly once in the executable if
it is needed, and not at all otherwise. There are two basic approaches to this problem,
which are referred to as the Borland model and the Cfront model.
Borland model
Borland C++ solved the template instantiation problem by adding the code equivalent
of common blocks to their linker; the compiler emits template instances in each
translation unit that uses them, and the linker collapses them together. The advantage
of this model is that the linker only has to consider the object files themselves; there
is no external complexity to worry about. This disadvantage is that compilation time is
increased because the template code is being compiled repeatedly. Code written for this
model tends to include definitions of all templates in the header file, since they must
be seen to be instantiated.
This question already has answers here:
How does C++ link template instances
(4 answers)
Closed 8 years ago.
In C, partial compilation is possible since the entire *.c file can be compiled into machine code with resolution and relocation left for the linker to handle. This is just an issue of calculating the displacement certain instructions have in the final executable or knowing the absolute address for some global variable.
In C++ it would seem that almost the same can be done - there exists a fairly uncomplicated mapping between C++ code and equivalent C code (as far as mappings between programming languages go). However, templates seem to complicate things.
If I use, for instance, a std::vector<int> in 1.c, then, since the template class was specified by the <vector> header, the compiler can generate machine code for an int specification. Suppose in the same project there is a file 2.c which also relies on a std::vector<int> specialization, and that 1.o and 2.o must be linked. Is partial compilation of 1.c and 2.c to their own *.o files to be linked later possible?
As mentioned in the linked question in the comments below, there are two commonly used methods for this problem: both generate std::vector<int> code, or the linker goes through another round of "dependency compilations" where a single vector<int> is compiled and then linked to both files.
Regarding "greedy compilation" - does this mean that every use of template class methods in every compilation unit must be put in the linker relocation table? Also, certain calls may not use long jumps (i.e., a template class is defined right above the method using it). However, if the linker is going to force a compilation unit to use the specialization it has selected, then a long jump would be necessary - but the instruction size would be too large to patch in.
This is a slightly more complicated question than what most people will realize.
In the general and simplest case, the template definition is present in the header, and it behaves as inline functions. The compiler will generate the code for those functions needed in each translation unit that needs them. Then the linker will resolve the duplicate symbols by removing all but one. Since the standard requires that they are exactly equivalent, the linker can pick any one from the list.
If the template need only work with a couple of types, you can move the definition to a single translation unit and explicitly instantiate the template for those types there. This would behave as a non-inline function in the general case.
Somewhere in between, if the template can be instantiated with any type but it is commonly instantiated with a few of them, the implementor of the template can use a mixed approach, where the template and the members are defined in the header, but explicit instantiations are also declared. Then in a single translation unit, those explicit instantiations can be done.
This approach can be used, for example, to minimize compile and link time when using std::string (which is really std::basic_string<char, std::char_traits<char>, std::allocator<char> >). The compiler can, in a single translation unit provide all of the functions for the common instantiation, but still provide the definition of the template functions in the header so that if you opt to use a different instantiation of the basic_string template it will still work for you. In all translation units that only use std::string, the compiler knows not to generate the code for all members as those will be available to the linker.
The compiler generates code for each instantiation of the template and makes sure that there are no name clashes. It is not like ordinary functions, where you get linker errors if a .cpp file is used in two compilation units.
It is possible to save some compilation time by explicitly instantiating the template in some compilation unit and use that template elsewhere, but this needs quite a bit of manual housekeeping (adding explicit instantiations for each new type that is used in the project). You can also save some compilation time by avoiding some unnecessary conversions by using the keyword explicit on templated constructors.