Suppose we have this header, with a function:
//template<int does_nothing=0> // <-- !! links correctly if uncommented
void incAndShow() {
static int myStaticVar = 0;
std::cout << ++myStaticVar << " " << std::endl;
}
If this is included in several .cpp files, you'll get a linker error due to the redefinition of incAndShow().
Now, if you uncomment the //template<int does_nothing=0> line above, this code will compile correctly with no linker error. And, the static variable will be shared correctly in all instances of incAndShow<0>().
This is because the C++ standard explicitly states that this should be allowed in its "One Definition Rule" (Page 43). The abbreviated text is:
There can be more than one definition of a... non-static function template (17.5.6)... provided that each definition appears in a different translation unit...
Now, note that the above is only true for templated functions, and not for non-templated functions. This is because, at least in the traditional paradigm, non-templated functions are compiled individually within each translation unit, and then the linker, envisioned as a totally separate process from the compiler, sees multiple definitions of the same symbol in different .obj files and throws the error.
But somehow, this doesn't happen for templated functions.
So my question is: how, in a low-level sense, does this work?
Does the compiler just compile each translation unit independently, meaning it generates multiple versions of the same symbol within each translation unit, but then have some special way to tell the linker to not worry about it?
Does the compiler, on-the-fly, keep a note of the first time some templated function definition is seen in some translation unit, and automatically ignore upon seeing the same definition again when compiling the next translation unit?
Does the compiler do some kind of "pre-compilation process," before compiling anything, where it goes through each file and removes duplicate templated code?
Something else?
This has been brought up on StackOverflow a few times, but this is the first time I've seen anyone ask how the compilation process really happens on a low-level:
Static variable inside template function (where my example came from)
Why C++'s <vector> templated class doesn't break one definition rule?
Related
In C and C++, you can't have a function with two definitions. For example, say we have the following two files:
1.c:
int main(){ return 0}
2.c:
int main(){ return 0}
Issuing the command gcc 1.c 2.c will give you a duplicate symbol linker error.
Why doesn't the same happen with structs and classes? Why are we allowed to have multiple
definitions of the same struct as long as they have the same tokens?
To answer this question, one has to delve into compilation process and what is needed in each part (question why these steps are perfomed is more historical, going back to beginning of C before it's standardization)
C and C++ programs are compiled in multiple steps:
Preprocessing
Compilation
Linkage
Preprocessing is everything that starts with #, it's not really important here.
Compilation is performed on each and every translation unit (typically a single .c or .cpp file plus the headers it includes). Compiler takes one translation unit at a time, reads it and produces an internal list of classes and their members, and then assembly code of each function in given unit (basing on the structures list). If a function call is not inlined (e.g. it is defined in different TU), compiler produces a "link" - "please insert function X here" for the linker to read.
Then linker takes all of the compiled translation units and merges them into one binary, substituting all the links specified by compiler.
Now, what is needed at each phase?
For compilation phase, you need the
definition of every class used in this file - compiler needs to know the size and offset of each class member to produce assembly
declaration of every function used in this file - to produce those "links".
Since function definitions are not needed for producing assembly (as long as they are compiled somewhere), they are not needed in compilation phase, only in linking phase.
To sum up:
One Definition Rule is there to protect programmers from theselves. If they'd accidentally define a function twice, linker will notice that and executable is not produced.
However, class definitions are required in every translation unit, and therefore such a rule cannot be set up for them. Since it cannot be forced by language, programmers have to be responsible beings and not define the same class in different ways.
ODR has also other limitations, e.g. you have to define template functions (or template class methods) in header files. You can also take the responsibility and say to the compiler "Every definition of this function will be the same, trust me dude" and make the function inline.
There is no use case for a function with 2 definitions. Either the two definitions would have to be the same, making it useless, or the compiler wouldn't be able to tell which one you meant.
This is not the case with classes or structures. There is also a large advantage to allowing multiple definitions of them, i.e. if we want to use a class or struct in multiple files. (This leads indirectly to multiple definitions because of includes.)
Structures, classes, unions and enumerations define types that can be used in several compilation units to define objects of these types. So each compilation unit need to know how the types are defined, for example to allocate correctly memory for an object or to be sure that specified member of a class does indeed exist.
For functions (if they are not inline functions) it is enough to have their declaration without their definition to generate for example a function call.
But the function definition shall be single. Otherwise the compiler will not know what function to call or the object code will be too big due to duplication and will be error prone..
It's quite simple: It's a question of scope. Non-static functions are seen (callable) by every compilation unit linked together, while structures are only seen in the compilation unit where they are defined.
For example, it's valid to link the following together because it's clear which definition of struct Foo and which definition of f is being used:
1.c:
struct Foo { int x; };
static void f(void) { struct Foo foo; ... }
2.c:
struct Foo { double d; };
static void f(void) { struct Foo foo; ... }
int main(void) { ... }
But it isn't valid to link the following together because the linker wouldn't know which f to call.
1.c:
void f(void) { ... }
2.c:
void f(void) { ... }
int main(void) { f(); }
Actually every programming element is associated with a scope of its applicability. And within this scope you cannot have the same name associated with multiple definitions of an element. In compiled world:
You cannot have more than one class definition with the same name within a single file. But you can have it in different compilation units.
You cannot have the same function or global variable name within a single link unit (library or executable), but you can potentially have functions named the same within different libraries.
you cannot have shared libraries with the same name situated in the same directory, but you can have them in different directories.
C/C++ compilation is very much after the compilation performance. Checking 2 objects like function or classes for identity is a time-consuming task. So, it is not done. Only names are considered for comparison. It is better to consider that 2 types are different and error out then checking them for identity. The only exception from this rule are text macros.
Macros are a pre-processor concept and historically it is allowed to have multiple identical macro definitions. If a definition changes, a warning gets generated. Comparing macro context is easy, just a simple string comparison, but some macro definitions could be huge.
Types are the compiler concept and they are resolved by the compiler. Types do not exist in object libraries and are represented by the sizes of corresponding variables. So, there is no reason for checking type name collisions at this scope.
Functions and variables on the other hand are named pointers to executable codes or data. They are the building blocks of applications. Applications are assembled from the codes and libraries coming from all around the world in some cases. In order to use someone else's function you'd better now its name and you do not want the same name to be used by some one else. Within a shared library names of functions and variables are usually stored in a hash table. There is no place for duplicates there.
And as I already mention checking functions for identical contents is seldom done, however there are some cases, but not in c or c++.
The reason of impeding two different definitions for the same thing to be used in programming is to avoid the ambiguity of deciding which definition to use at run time.
If you have two different implementations to the same thing to coexist in a program, then there's the possibility of aliasing them (with a different name each) into a common reference to decide at runtime which one of the two to use.
Anyway, in order to distinguish both, you have to be able to indicate the compiler which one you want to use. In C++ you can overload a function, giving it the same name and different lists of parameters, so you can distinguish which one of both you want to use. But in C, the compilers only preserve the name of the function to be able to solve at link time which definition matches the name you use in a different compilation unit. In case the linker ends with two different definitions with the same name, it is uncapable of deciding for you which one to use, so it emits an error and gives up the building process.
What should be the intention of using this ambiguity in a productive way? this is the question you have actually to ask to yourself.
This question already has answers here:
How does the linker handle identical template instantiations across translation units?
(3 answers)
Closed 4 years ago.
When I include, say <vector> in a C++ file, and create an instance of std::vector<double>, and then call the std::vector<double>::push_back() function, my understanding is that at that point, the compiler creates a definition of the function and generates the necessary code. However, I can call that function from several distinct C++ files which will be linked into the same executable, and presumably each .cpp file will generate a .o file each it's own definition of std::vector<double>::push_back(). Normally, that kind of multiple function definition would generate a linking error, but it doesn't seem to happen for templates- why is that so?
Compilers can flag certain symbols to be autoremoved in case the linker sees redefinition.
The "symbol" in this case will be the unique name (the mangled name) that the compiler assigns to each templated function, or templated member function, or member function of a templated class, after instantiation with a concrete type. In different translation units (different .cpp), the generated unique names will be exactly the same. Consider that, besides exceptions and polimorphic hierarchies, a C++ program is basically translated to a C program before applying the final translation to assembler code, so your program get up being a bunch of functions, some of them being inline functions, others instantiated templated functions, and others usual free functions. All but the thirds are flagged by the compiler to let the linker know redefinitions are possible comming and they must be silently removed.
If a different translation unit uses a same templated method instantiated with a same type, the mangled name will also be the same, and then flagged by the compiler. Finally, in the linking stage, the linker will see two redefinitions of a same function, but flagged; so the linker must remove one of them and only one will survice the purge.
It's the same with inline methods.
This causes no harm since a same method from a same instiantated templated class for a same type, will produce the exact same assembler code, no matter from which translation unit.
Otherwise it would be impossible to have inline or templated methods.
AFAIK, originally linkers didn't allow this kind of flagging; it was explicitely added to make C++ implementable.
Suppose I have two .cpp files file1.cpp and file2.cpp:
// file1.cpp
#include <iostream>
inline void foo()
{
std::cout << "f1\n";
}
void f1()
{
foo();
}
and
// file2.cpp
#include <iostream>
inline void foo()
{
std::cout << "f2\n";
}
void f2()
{
foo();
}
And in main.cpp I have forward declared the f1() and f2():
void f1();
void f2();
int main()
{
f1();
f2();
}
Result (doesn't depend on build, same result for debug/release builds):
f1
f1
Whoa: Compiler somehow picks only the definition from file1.cpp and uses it also in f2(). What is the exact explanation of this behavior?.
Note, that changing inline to static is a solution for this problem. Putting the inline definition inside an unnamed namespace also solves the problem and the program prints:
f1
f2
This is undefined behavior, because the two definitions of the same inline function with external linkage break C++ requirement for objects that can be defined in several places, known as One Definition Rule:
3.2 One definition rule
...
There can be more than one definition of a class type (Clause 9), enumeration type (7.2), inline function with external linkage (7.1.2), class template (Clause 14),[...] in a program provided that each definition appears in a different translation unit, and provided the definitions satisfy the following requirements. Given such an entity named D defined in more than one translation unit, then
6.1 each definition of D shall consist of the same sequence of tokens; [...]
This is not an issue with static functions, because one definition rule does not apply to them: C++ considers static functions defined in different translation units to be independent of each other.
The compiler may assume that all definitions of the same inline function are identical across all translation units because the standard says so. So it can choose any definition it wants. In your case, that happened to be the one with f1.
Note that you cannot rely on the compiler always picking the same definition, violating the aforementioned rule makes the program ill-formed. The compiler could also diagnose that and error out.
If the function is static or in an anonymous namespace, you have two distinct functions called foo and the compiler must pick the one from the right file.
Relevant standardese for reference:
An inline function shall be defined in every translation unit in which it is odr-used and shall have exactly
the same definition in every case (3.2). [...]
7.1.2/4 in N4141, emphasize mine.
As others have noted, the compilers are in compliance with the C++ standard because the One definition rule states that you shall have only one definition of a function, except if the function is inline then the definitions must be the same.
In practice, what happens is that the function is flagged as inline, and at linking stage if it runs into multiple definitions of an inline flagged token, the linker silently discards all but one. If it runs into multiple definitions of a token not flagged inline, it instead generates an error.
This property is called inline because, prior to LTO (link time optimization), taking the body of a function and "inlining" it at the call site required that the compiler have the body of the function. inline functions could be put in header files, and each cpp file could see the body and "inline" the code into the call site.
It doesn't mean that the code is actually going to be inlined; rather, it makes it easier for compilers to inline it.
However, I am unaware of a compiler that checks that the definitions are identical before discarding duplicates. This includes compilers that otherwise check definitions of function bodies for being identical, such as MSVC's COMDAT folding. This makes me sad, because it is a reall subtle set of bugs.
The proper way around your problem is to place the function in an anonymous namespace. In general, you should consider putting everything in a source file in an anonymous namespace.
Another really nasty example of this:
// A.cpp
struct Helper {
std::vector<int> foo;
Helper() {
foo.reserve(100);
}
};
// B.cpp
struct Helper {
double x, y;
Helper():x(0),y(0) {}
};
methods defined in the body of a class are implicitly inline. The ODR rule applies. Here we have two different Helper::Helper(), both inline, and they differ.
The sizes of the two classes differ. In one case, we initialize two sizeof(double) with 0 (as the zero float is zero bytes in most situations).
In another, we first initialize three sizeof(void*) with zero, then call .reserve(100) on those bytes interpreting them as a vector.
At link time, one of these two implementations is discarded and used by the other. What more, which one is discarded is likely to be pretty determistic in a full build. In a partial build, it could change order.
So now you have code that might build and work "fine" in a full build, but a partial build causes memory corruption. And changing the order of files in makefiles could cause memory corruption, or even changing the order lib files are linked, or upgrading your compiler, etc.
If both cpp files had a namespace {} block containing everything except the stuff you are exporting (which can use fully qualified namespace names), this could not happen.
I've caught exactly this bug in production multiple times. Given how subtle it is, I do not know how many times it slipped through, waiting for its moment to pounce.
POINT OF CLARIFICATION:
Although the answer rooted in C++ inline rule is correct, it only applies if both sources are compiled together. If they are compiled separately, then, as one commentator noted, each resulting object file would contain its own 'foo()'. HOWEVER: If these two object files are then linked together, then because both 'foo()'-s are non-static, the name 'foo()' appears in the exported symbol table of both object files; then the linker has to coalesce the two table entries, hence all internal calls are re-bound to one of the two routines (presumably the one in the first object file processed, since it is already bound [i.e the linker would treat the second record as 'extern' regardless of binding]).
After reading this question I thought I understood everything, but then I saw this file from a popular header-only library.
The library uses the #ifndef line, but the SO question points out that this is NOT adequate protection against multiple definition errors in multiple TUs.
So one of the following must be true:
It is possible to avoid multiple definition linker errors in ways other than described in the SO question. Perhaps the library is using techniques not mentioned in to the other SO question that are worthy of additional explanation.
The library assumes you won't include its header files in more than translation unit -- this seems fragile since a robust library shouldn't make this assumption on its users.
I'd appreciate having some light shed on this seemingly simple curiosity.
A header that causes linking problems when included in multiple translation units is one that will (attempt to) define some object (not just, for an obvious example, a type) in each source file where it's included.
For example, if you had something like: int f = 0; in a header, then each source file into which it was included would attempt to define f, and when you tried to link the object files together you'd get a complaint about multiple definitions of f.
The "technique" used in this header is simple: it doesn't attempt to define any actual objects. Rather, it includes some typedefs, and the definition of one fairly large class--but not any instances of that class or any instance of anything else either. That class includes a number of member functions, but they're all defined inside the function definition, which implicitly defines them as inline functions (so defining separately in each translation unit in which they're used is not only allowed, but required).
In short, the header only defines types, not objects, so there's nothing there to cause linker collisions when it's included in multiple source files that are linked together.
If the header defines items, as opposed to just declaring them, then it's possible to include it in more than one translation unit (i.e. cpp file) and have multiple definitions and hence linker errors.
I've used boost's unit test framework which is header only. I include a specified header in only one of my own cpp files to get my project to compile. But I include other unit test headers in other cpp files which presumably use the items that are defined in the specified header.
Headers only include libraries like Boost C++ Libraries use (mostly) stand-alone templates and as so are compiled at compile-time and don't require any linkage to binary libraries (that would need separate compilation). One designed to never need linkage is the great Catch
Templates are a special case in C++ regarding multiple definitions, as long as they're the same. See the "One Definition Rule" section of the C++ standard:
There can be more than one definition of a class type (Clause 9),
enumeration type (7.2), inline function with external linkage (7.1.2),
class template (Clause 14), non-static function template (14.5.6),
static data member of a class template (14.5.1.3), member function of
a class template (14.5.1.1), or template specialization for which some
template parameters are not specified (14.7, 14.5.5) in a program
provided that each definition appears in a different translation unit,
and provided the definitions satisfy the following requirements. ....
This is then followed by a list of conditions that make sure the template definitions are identical across translation units.
This specific quote is from the 2014 working draft, section 3.2 ("One Definition Rule"), subsection 6.
This header file can indeed be included in difference source files without causing "multiple symbol definition" errors.
This happens because it is fine to have multiple identically named symbols in different object files as long as these symbols are either weak or local.
Let's take a closer look at the header file. It (potentially) defines several objects like this helper:
static int const helper[] = {0,7,8,13};
Each translation unit that includes this header file will have this helper in it. However, there will be no "multiple symbol definition" errors, since helper is static and thus has internal linkage. The symbols created for helpers will be local and linker will just happily put them all in the resulting executable.
The header file also defines a class template connection. But it is also okay. Class templates can be defined multiple times in different translation units.
In fact, even regular class types can be defined multiple times (I've noticed that you've asked about this in the comments). Symbols created for member functions are usually weak symbols. Once again, weak symbols don't cause "multiple symbol definition" errors, because they can be redefined. Linker will just keep redefining weak symbols with names he has already seen until there will be just one symbol per member function left.
There are also other cases, where certain things (like inline functions and enumerations) can be defined several times in different translation units (see ยง3.2). And mechanisms of achieving this can be different (see class templates and inline functions). But the general rule is not to place stuff with global linkage in header files. As long as you follow this rule, you're really unlikely to stumble upon multiple symbol definitions problems.
And yes, include guards have nothing to do with this.
I know that compiler synthesized functions in c++, such as default constructor, copy constructor .etc are implicitly inline, but I am just curious about why?
and, what if the tasks these functions need to do are too complicated to be inappropriate to be inlined?
The first question is what does it really mean to be inline? The answer to that question has little to do with actual inlining, and more to do with the One Definition Rule. When the compiler generates code for an inline function it marks the symbol for the linker to know that having that same symbol in different translation units is fine.
With this in mind, the answer is quite simple. If the functions were not inline and the compiler generated them in two different translation units, linking both translation units together would cause a violation of the One Definition Rule that is outside of the reach of the programmer to fix.
It seems that the answer above is not clear enough regarding what happens when the function is not really inlined. From the first paragraph:
When the compiler generates code for an inline function it marks the symbol for the linker to know that having that same symbol in different translation units is fine.
The compiler does generate a function symbol in each translation unit that uses that function. The linker will then use the mark (weak symbol) to discard all but one of the definitions from the program. In many cases compilers generate an out-of-line definition of inline functions even when the function code is actually inlined.
This is different from marking the function as static (internal linkage) as if it was marked as internal linkage, all of those definitions would remain in the final executable *
* There are some linkers that are able to figure out that different symbols are exactly the same and will remove duplicates, so it might merge void f() { std::cout << "Hi\n"; } with void g() { std::cout << "Hi\n"; }. Linkers that do this can also fold all of thestatic` functions together, but this is an optimization outside of what the standard mandates.