Is it possible for the compiler to duplicate instantiations of a same template across several translation units?
For instance, if you have a.cpp which use a std:vector<int> inside a function, and same thing for b.cpp, is there a risk of having two times std::vector<int> in the final binary?
As sharptooth says, the final binary will only contain one instantiation. But the templates will still be instantiated everytime they are encountered in a compilation unit. If you want some compilation speed, in C++0x we get the extern templates. It works like normal extern variables, in that it has to be specified in at least one compilation unit, but the compiler wont instantiate the template in this unit. See here and this draft (14.7.2 [temp.explicit]) for more infos.
This can happen while the project is being compiled, so different .obj files will have copies of the same instantiation. When the binary is linked the linker will eliminate all redundant copies of an instantiation, so the end binary will have only one copy.
It is possible, but only if you explicitly instantiate them, but then you will get a linker errors :
// header.hpp
template< typename T >
class A
{
};
// source1.cpp
template class A< int >;
// source2.cpp
template class A< int >;
If you are not explicitly instantiating templates, then any decent linker will easily eliminate copies.
I think that the compiler uses the same mechanism as with member functions of ordinary classes. It can make them inline and I presume that it leaves information around that the linker uses to sort it out for the final binary.
The only difference is that the compiler 'writes' the definitions - that is the 'instantiation' of a template - but it manages to make exactly the same instantiation whilst compiling either of a.cpp or b.cpp
Related
I have a template class with a specialization, defined in another file. Therefore it is possible to generate two versions of the same class: one time by replacing the template parameter and one time using the specialization. My current understanding is, that this can lead to two instanciations of the same type to have different sizes in memory, resulting in segmentation faults.
I created a minimal example and the following code is to illustrate the question:
Create a template class:
// - templateexample.h ---------------
#ifndef TEMPLATEEXAMPLE_H
#define TEMPLATEEXAMPLE_H
template<typename T> class Example
{
public:
Example(){}
int doWork() {return 42;}
};
#endif
// -----------------------------------
Template specialization in another file:
// - templatespecialization.h --------
#ifndef TEMPLATESPECIALIZATION_H
#define TEMPLATESPECIALIZATION_H
#include "templateexample.h"
template<> class Example<int>
{
public:
Example() : a(0), b(1), c(2), d(3) {}
int doWork() {return a+b+c+d;}
private:
int a; //<== the specialized object will be larger in memory
int b;
int c;
int d;
};
#endif
// --------------------------------
Have a class which includes only the template class definition, but should include the specialization.
// - a.h --------------------------
#ifndef A_H
#define A_H
#include "templateexample.h"
class A
{
public:
Example<int> returnSmallExample();
};
#endif
// - a.cpp ------------------------
#include "a.h"
Example<int> A::returnSmallExample() {return Example<int>();}
// --------------------------------
The main class now knows two versions of Example<int> the one from A and the one from the templatespecialization.h.
// - main.cpp ---------------------
#include <iostream>
#include "a.h"
#include "templatespecialization.h"
int main()
{
A a;
Example<int> test = a.returnSmallExample();
std::cout<<test.doWork()<<std::endl;
}
// --------------------------------
Please note, this problem will only occur when compiling class A separately, this example from ideone outputs 6, whereas using separate files can result in a segmentation fauls, or output 42 (https://ideone.com/3RTzlC). On my machine the example compiles successfully and outputs 2013265920:
In the production version of the above example everything is build into a shared library which is used by main.
Question 1: Why doesn't the linker detect this problem? This should be easy to spot by comparing the size of objects.
Question 2: is there a way to examine the object files or the shared library to detect multiple implementations of the same type like in the example above?
Edit: please note: the code above is a minimal example to explain the problem. The reason for the situation is that the template class is from one library and I cannot edit the files from this library. Finally the whole thing is used all over the place in the executable and now I need to find out if the problem above occurs.
Edit: code above can be compiled like this:
#!/bin/bash
g++ -g -c a.cpp
g++ -g -c main.cpp
g++ -o test a.o main.o
You have different definition of the same template and its specializations in different translation units. This leads to One Definition Rule violation.
A fix would be to put the specialization in the same header file where the primary class template is defined.
Question 1: Why doesn't the linker detect this problem? This should be easy to spot by comparing the size of objects.
Different types may have the same size (e.g. double and int64_t), so, obviously, just comparing sizes of objects does not work.
Question 2: is there a way to examine the object files or the shared library to detect multiple implementations of the same type like in the example above?
You should use gold linker for linking your C++ applications, if you do not use it already. One nice feature of it is --detect-odr-violations command line switch which does exactly what you ask:
gold uses a heuristic to find potential ODR violations: if the same symbol is seen defined in two different input files, and the two symbols have different sizes, then gold looks at the debugging information in the input objects. If the debugging information suggests that the symbols were defined in different source files, gold reports a potential ODR violation. This approach has both false negatives and false positives. However, it is reasonably reliable at detecting problems when linking unoptimized code. It is much easier to find these problems at link time than to debug cases where the wrong symbol.
See Enforcing One Definition Rule for more details.
Question 1: Why doesn't the linker detect this problem? This should be easy to spot by comparing the size of objects.
Because this is not a linker's problem. In case of templates, the main declaration and all the other specializations (be it class or function) should be visible upfront.
Question 2: is there a way to examine the object files or the shared library to detect multiple implementations of the same type
like in the example above?
At least I am not aware of any.
To further simplify this situation, look at a similar broken code:
// foo.h
inline void foo () {
#ifdef FOO
return;
#else
throw 0;
#endif
}
// foo1.cpp
#define FOO
#include"foo.h"
// ... uses foo() with "return"
// foo2.cpp
#include"foo.h"
// ... uses foo() with "throw"
It's possible that you get different results based on the way of compilation being used.
Update:
Having multiple body definitions for a same function is undefined behavior. That's the reason why you are getting an awkward output like 2013265920 and the same happens in my machine as well. The output should be either 42 or 6. I gave you above example because with inline functions you can create such race conditions.
With my limited knowledge on linking stage, the responsibility undertaken by a typical linker is limited only till rejecting more than 1 non-inline functions definitions with the same signature. e.g.
// Header.h is included in multiple .cpp files
void foo () {} // rejected due to multiple definitions
inline void bar () {} // ok because `inline` keyword is found
Beyond that it doesn't check if the function body is similar or not, because that is already parsed during earlier stage and linker doesn't parse the function body.
Having said above, now pay attention to this statement:
template functions are always inline by nature
Hence linker may not get a chance to reject them.
The safest way is to #include the read-only header into your specialized header and include that specialized header everywhere.
I don't know of a way to do this by analysis of the compiled binary, but you could build a graph of your program's #include relationships — there are tools that can do this, such as Doxygen — and use it to look for files that (directly or indirectly) include the library header but not the specialization header.
You'd need to examine each file to determine whether it actually uses the template in question, but at least you can narrow down the set of files that you have to examine.
I think you managed to deceive the compiler. But I should note that in my humble opinion you understand conception of templates in a wrong way, particularly you are trying to mix up template specialization with inheritance.
I mean, template specialization must not add data members to class, the only aims are to define types for function parameters and class fields. If you want to change algorithms, i.e. rewrite code, or add new date members to class, you should define derived class.
Concerning "several-step" separate compilation and template class in library, C++ reference said (http://www.cplusplus.com/doc/oldtutorial/templates/):
Because templates are compiled when required, this forces a restriction for multi-file projects: the implementation (definition) of a template class or function must be in the same file as its declaration. That means that we cannot separate the interface in a separate header file, and that we must include both interface and implementation in any file that uses the templates.
Since no code is generated until a template is instantiated when required, compilers are prepared to allow the inclusion more than once of the same template file with both declarations and definitions in a project without generating linkage errors.
This question already has answers here:
How does C++ link template instances
(4 answers)
Closed 8 years ago.
In C, partial compilation is possible since the entire *.c file can be compiled into machine code with resolution and relocation left for the linker to handle. This is just an issue of calculating the displacement certain instructions have in the final executable or knowing the absolute address for some global variable.
In C++ it would seem that almost the same can be done - there exists a fairly uncomplicated mapping between C++ code and equivalent C code (as far as mappings between programming languages go). However, templates seem to complicate things.
If I use, for instance, a std::vector<int> in 1.c, then, since the template class was specified by the <vector> header, the compiler can generate machine code for an int specification. Suppose in the same project there is a file 2.c which also relies on a std::vector<int> specialization, and that 1.o and 2.o must be linked. Is partial compilation of 1.c and 2.c to their own *.o files to be linked later possible?
As mentioned in the linked question in the comments below, there are two commonly used methods for this problem: both generate std::vector<int> code, or the linker goes through another round of "dependency compilations" where a single vector<int> is compiled and then linked to both files.
Regarding "greedy compilation" - does this mean that every use of template class methods in every compilation unit must be put in the linker relocation table? Also, certain calls may not use long jumps (i.e., a template class is defined right above the method using it). However, if the linker is going to force a compilation unit to use the specialization it has selected, then a long jump would be necessary - but the instruction size would be too large to patch in.
This is a slightly more complicated question than what most people will realize.
In the general and simplest case, the template definition is present in the header, and it behaves as inline functions. The compiler will generate the code for those functions needed in each translation unit that needs them. Then the linker will resolve the duplicate symbols by removing all but one. Since the standard requires that they are exactly equivalent, the linker can pick any one from the list.
If the template need only work with a couple of types, you can move the definition to a single translation unit and explicitly instantiate the template for those types there. This would behave as a non-inline function in the general case.
Somewhere in between, if the template can be instantiated with any type but it is commonly instantiated with a few of them, the implementor of the template can use a mixed approach, where the template and the members are defined in the header, but explicit instantiations are also declared. Then in a single translation unit, those explicit instantiations can be done.
This approach can be used, for example, to minimize compile and link time when using std::string (which is really std::basic_string<char, std::char_traits<char>, std::allocator<char> >). The compiler can, in a single translation unit provide all of the functions for the common instantiation, but still provide the definition of the template functions in the header so that if you opt to use a different instantiation of the basic_string template it will still work for you. In all translation units that only use std::string, the compiler knows not to generate the code for all members as those will be available to the linker.
The compiler generates code for each instantiation of the template and makes sure that there are no name clashes. It is not like ordinary functions, where you get linker errors if a .cpp file is used in two compilation units.
It is possible to save some compilation time by explicitly instantiating the template in some compilation unit and use that template elsewhere, but this needs quite a bit of manual housekeeping (adding explicit instantiations for each new type that is used in the project). You can also save some compilation time by avoiding some unnecessary conversions by using the keyword explicit on templated constructors.
I'm using Code::Blocks to build my project, which contains three files: main.cpp, TimeSeries.cpp, TimeSeries.h. TimeSeries.h provides declarations for the TimeSeries class as follows:
template<class XType, class YType> class TimeSeries {
public:
TimeSeries(void);
~TimeSeries(void);
};
Then TimeSeries.cpp contains:
#include "TimeSeries.h"
template<class XType, class YType>
TimeSeries<XType, YType>::TimeSeries(void) {
}
template<class XType, class YType>
TimeSeries<XType, YType>::~TimeSeries(void) {
}
And finally, main.cpp contains
#include "TimeSeries.h"
typedef TimeSeries<float, float> FTimeSeries;
int main(int argc, char** argv) {
FTimeSeries input_data;
return 0;
}
When building with C::B, I get the following error:
undefined reference to `TimeSeries<float, float>::TimeSeries()'
What can I do?
Thanks,
CFP.
Basically all templated code should be defined in a header, otherwise it will not be built since nothing uses it in the compiled unit.
Each cpp file is compiled as a separate unit, and thus constructor and destructor are not compiled. The compiler has no way of knowing what type of template argument you will use in main.cpp when it compiles TimeSeries.cpp.
The reason for splitting code into header- and source-files is so that the declaration and the implementation are separated. The compiler can translate the source-file (compilation unit) into an object file, and other compilation-units that want to use the classes and functions just include the header-file, and link the object file. This way, the code has to be compiled only once, and can be reused by linking it.
The problem with templates is that, as long as there are no parameters provided for them, the compiler cannot compile them. The same template instantiated with different parameters results in different types. std::vector<int> and std::vector<float> are, from the compilers perspective, not related in any way. Because of this, template-classes usually have to reside completely in a header-file, because, when the template is used, the compiler needs the complete definition in order to generate the class depending on the parameters.
As #Gabriel Schreiber pointed out in his answer, you can tell the compiler that he should compile the template with a specific set of parameters, making that available to other compilation units just by linking. However, this does not make the template available for other parameter sets.
You need to add this in your .cpp-file (below the definitions):
template class TimeSeries<float, float>;
When the compiler compiles TimeSeries.cpp it doesn't know which for which types the template is need because it is used in another source file. You need to tell the compiler explicitly.
Read about Explicit Template Instantiation in your copy of the Stroustrup or on the internet.
Template code is not compiled until the template function is used. But where does it save the compiled code, is it saved in the object file from which used the template function in the first place?
For example,
main.cpp is calling a template function from the file test.h, the compiler generates an object file main.o,
Is the template function inside the main.o file? because template code is not inlined, is it?
It's totally compiler implementation dependant. Most compilers will generate code around, inline or in cpp-like files and then compile with that. Sometimes, with optimization setup, some compilers will even reuse the same code instead of recreate it for each cpp.
So you have to see your compiler's doc for more details.
Yes, the template function code is emitted in the main.o file. Some of it may be inlined, as any other code may be inlined, but in general, code for templates is emitted in any file in which the template is instantiated. Think of it as first instantiating the template code to produce normal non-template functions, and then compiling those functions with whatever inlining and optimization the compiler applies to any other function.
When the same template instantiation (e.g. std::vector<int>) occurs in multiple compilation units (.cpp files), there is a bit of difficulty, because the code is instantiated in each of them. There are various ways of handling this, sometimes involving a cleanup step in the linking phase where duplicate template instantiations are resolved into a single one; your compiler manual can provide more information on exactly how it handles that situation.
Template code is compiled even if it is never instantiated. Otherwise, compilers couldn't be required to emit a diagnostic for this:
template< typename T >
void f()
{
blah
}
Template compilation happens in two phases. Besides the basic checks, everything that depends on template parameters can only be checked when a template is instantiated and the formal parameters are filled in with actual types. For example, here
template< typename T >
void f()
{
typename T::nested_type x;
}
T::nested_type can only be checked after the template is instantiated and an actual type is given for T. However, a basic check ("given T::nested_type is a type, is this a valid variable definition?") is performed the moment the compiler encounters the template's definition. (That's why that typename is required, BTW. Depending on T, T::nested_type might just as well be the name of member of T or a static data member - which would make T::nested_type x; a syntax error. So we have to tell the compiler to treat T::nested_type as the name of a type.)
You mean instantiated, not compiled. At compile time the compiler finds out each and every version that your code uses and instatiates (in object files) all the required versions.
It is always inlined (meaning, it is always internal linkage, having inline semantics). It may in fact be not inlined after all, just as an inline function, however, template is not code. It is a "template for making code". Therefore, it will normally reside in a header, except special cases, see below.
There was an idea to make something else, codenamed "export keyword". It was removed from standard.
Special cases: you can compile template instantiations into an object file, without having them used. This is the only way to avoid having all template code inlined.
This is how it is done:
template class std::vector<MyClass>;
This will force the compiler to instantiate a template in the current location. C++0x will have a syntax to force compiler not to do it, and have the linker search for template instantiation elsewhere:
extern template class std::vector<MyClass>; // C++0x only
The compiler says it can't find the reference for the function when I do this:
// link.h
template <class T>
T *Link(T *&, T *(*)())
// link.cpp
template <class T>
T c:Link(T *&ChildNodeReference, T *(*ObjectCreator)()){
}
If I implement inside the class on the header it goes smoothly.
Please, I will work on the header until someone enlightens me about this.
There are somethings in C++ that are weirdly annoying. I know, there is a reason for this and etc. Even so, can't the compilers help you out about it -_-"
Templates are essentially semi-type-safe macros, hence the limitation.
Normal (non-template) functions can be compiled to native code residing object/library files, and then referenced with only a prototype available in the header, solely because there's only a single version of such a function.
With templates, C++ compiler has to compile each instantiation of the function separately. Obviously, it cannot do it "in advance", because the set of types for which you can instantiate the function is effectively unbounded (you can always define a new type in your code before calling the function). In fact, two instantiations of the same function template can be completely different. Consider this extreme case:
struct t1 {
template <int>
struct a {};
};
struct t2 {
enum { a = 123 };
};
enum { b = 456, c = 789 };
template <class T>
void foo() {
T::a<b>c;
}
Now if we call foo<t1>(), the statement inside it is a local variable declaration, because t1::a is a class template:
T::a<b> c;
But if we call foo<t2>(), the statement inside is an expression, because t2::a is an integral constant:
(T::a < b) > c;
This is just to show that compiler cannot meaningfully "compile" a template; it really has to mostly preserve the tokens.
Now, all that said, ISO C++ does actually provide the ability to separate declaration and definition of templates, so that they can be treated as normal functions, with declarations in .h files, and definitions in .cpp files. This is called "export templates", because you have to precede both declaration and definiton with keyword export:
// link.h
export template <class T>
T *Link(T *&, T *(*)());
// link.cpp
export template <class T>
T *Link(T *&ChildNodeReference, T *(*ObjectCreator)()) {
}
This is, however, a controversial feature of the Standard because of very high burden on implementation, and most popular implementations refuse to implement it; notably, g++, MSVC, and C++Builder do not implement it. The only compiler I know of that supports it is Comeau C++.
Programming non-template code or non-inlined functions in headers is a Bad Thing™. The reason you have cpp files is to prevent redefinition of the same function code more than once, amongst other things.
The difference with templates is that the compiler practically doesn't touch them until your code instantiates a specialisation of that template which is why they need to have their source inside the header.
When the compiler finds an instantiation of a template specialisation (say List<int>), it goes back to the included template code and compiles it with that specialisation, which is why you don't have issues with redefinition of function code.
What you don't seem to understand is that this doesn't apply to non-templated code. All non-template code is compiled as normal and thus CPP files are needed to only define the code once then link it all together.
If you define functions inside a header, your linker will not link the compiled translation units because they have defined the same function more than once.
Templated implementations (not only definitions) have to be available at compile time.
So, the full template code is normally put in the header file.
Template code must be in the header. Sorry, I completely missed that! (and I thought I'd been coding C++ for years :P)
You forgot the * for the return type. So implementation is not matching definition. Add it and it should work:
T *c:Link(T *&ChildNodeReference, T *(*ObjectCreator)())
{
}
Implementation must be too in the header file under the class definition in order to be available at compile time.