Maybe this has been asked several times but I couldn't find a single question that were focused on static vars storage inside templated functions. I would like to know where statics within templated functions are stored and how what is the compiler doing with them exactly?
I'm going to provide some g++ memory layout just to show why I don't get them.
My first code I cheched was rather simple:
#include <iostream>
using namespace std;
void my_func() {
static int x;
}
int main() {
my_func();
return 0;
}
When I check the memory layout of this program with g++ 4.8.1 I end up with the following sections:
.text: 1845
.data: 620
.bss: 12
Nothing unexpected so far. The uninitialized static variable is stored within the .bss segment. The same goes if I initialize the x variable to 0, while if I initialize it with any non-zero value the; still nothing unexpected:
.text: 1845
.data: 624
.bss: 8
x in this case is stored within the data segment instead of bss. So far so good, so I turned towards my questionable part and changed my_func according to the following:
template <typename T> void my_func() {
static T x;
}
int main() {
my_func<int>();
return 0;
}
Now this was interesting to me but the memory layout became:
.text: 1845
.data: 620
.bss: 4
Where did my static go? Whether I initialize it to any value static declared within templated functions doesn't seem to apprear nor in .DS neither in .BSS... Even if I instantiate another instance of that template function with different type for example my_func<float>() nothing is going to change. How is the compiler doing it? Where will it put those statics and how will these statics behave exactly the same as they weren't in templates - meaning they keep their values for each instantiated template function?
Probably the variable is being optimized away because it is not used. You may try to compile to assembly (gcc -S) and then lookup the variable in the output.
This experiment seems to confirm it:
> cat static.cpp
#include <iostream>
template<class T>
int func()
{
static int x = 0x666;
return 3;
}
int main()
{
std::cout << func<int>();
return 0;
}
> g++ -S static.cpp && c++filt < static.s | grep ::x
> sed -i 's/return 3/return x/' static.cpp
> g++ -S static.cpp && c++filt < static.s | grep ::x
movl int func<int>()::x(%rip), %eax
.weak int func<int>()::x
.section .data._ZZ4funcIiEivE1x,"awG",#progbits,int func<int>()::x,comdat
.type int func<int>()::x, #gnu_unique_object
.size int func<int>()::x, 4
int func<int>()::x:
Related
I am trying to write a simple plugin system for an application and would like to prevent plugins from stomping on each others symbols, however RTLD_DEEPBIND and RTLD_LOCAL don't seem to be enough when it comes to static class members when they happen to have the same name in different plugins.
I wrote a stripped down example to show what I mean.
I compiled and ran it like this:
g++ -c dumb-plugin.cpp -std=c++17 -fPIC
gcc -shared dumb-plugin.o -o dumb1.plugin
cp dumb1.plugin dumb2.plugin
g++ main.cpp -ldl -o main
./main
And the content of the output file for the second plugin showed that it reused the the class from the first plugin.
How can I avoid this?
EDIT: I compiled the plugin with clang(not main just the plugin) and it worked despite all of the RTLD_DEEPBIND stuff being in main.cpp which was still compiled with g++. It didn't work when the plugin was compiled with gcc 10.3 or 11.1 even when I tried -Bsymbolic. Is this a bug?
If I run readelf on the DSO compiled/linked with clang i see these 2 lines:
21: 00000000000040b0 4 OBJECT UNIQUE DEFAULT 26 _ZN9DumbClass7co[...]_ZN9DumbClass7co[...]
25: 00000000000040b0 4 OBJECT UNIQUE DEFAULT 26 _ZN9DumbClass7co[...]
and with gcc i get:
20: 00000000000040a8 4 OBJECT WEAK DEFAULT 24 _ZN9DumbClass7co[...]
27: 00000000000040a8 4 OBJECT WEAK DEFAULT 24 _ZN9DumbClass7co[...]
with WEAK instead of UNIQUE under the BIND column.
dumb-plugin.cpp:
#include <dlfcn.h>
#include <cstdio>
#include <string>
int global_counter = 0;
static int static_global_counter = 0;
std::string replace_slashes(const char * str) {
std::string s;
for (const char* c = str; *c != '\0'; c++)
s += (*c == '/')?
'#' : *c;
return s;
}
void foo() {}
class DumbClass {
public:
static inline int counter = 0;
};
extern "C" void plugin_func() {
static int static_local_counter = 0;
Dl_info info;
dladdr((void*)foo, &info);
std::string path = "plugin_func() from: " + replace_slashes(info.dli_fname);
auto fp = std::fopen(path.c_str(), "w");
fprintf(fp, "static local counter: %d\n", static_local_counter++);
fprintf(fp, "DumbClass::counter: %d\n", DumbClass::counter++);
fprintf(fp, "global counter: %d\n", global_counter++);
fprintf(fp, "static global counter: %d\n", static_global_counter++);
std::fclose(fp);
}
main.cpp:
#include <dlfcn.h>
#include <iostream>
#include <unistd.h>
#include <string.h>
int main () {
char path1[512], path2[512];
getcwd(path1, 512);
strcat(path1, "/dumb1.plugin");
getcwd(path2, 512);
strcat(path2, "/dumb2.plugin");
auto h1 = dlopen(path1, RTLD_NOW | RTLD_LOCAL | RTLD_DEEPBIND);
auto h2 = dlopen(path2, RTLD_NOW | RTLD_LOCAL | RTLD_DEEPBIND);
auto func = (void(*)()) dlsym(h1, "plugin_func");
func();
func = (void(*)()) dlsym(h2, "plugin_func");
func();
}
gcc implements static inline data members (and also static data members of class templates, inline or not, and static variables in inline functions, and perhaps other things as well) as global unique symbols (a GNU extension to the ELF format). There is only one such symbol with a given name per process, by design.
clang implements such things as normal weak symbols. These will not collide when RTLD_LOCAL and RTLD_DEEPBIND are used.
There are several ways to avoid collisions, but all of them require plugin writers to take an action. The best way IMO is to use hidden symbol visibility by default, only opening symbols that are meant to be dlsymd.
How to place a c++ namespace at a particular memory location for an embedded setup? I tried using
attribute((section("sectionName"))) but it is being ignored by the compiler.
You could use linker scripts to map different program sections to different memory regions, etc.. It would give you some flexibility however please note that it is not guaranteed that a namespace would be a single section.
For you to know static variables are placed in different section than global variables and same for automatic variables (on a stack) etc...
Update:
It does not matter if variables are in one file or many because the placement of the symbols happen at the other stage.
Constant variable is a global (I assume you are talking about globals) and it will go to .rodata (read only data) because it is constant.
Consider following example:
➜ ~ cat main.cpp
#include <iostream>
#include <cstring>
const int varGlobal = 0;
static int varStatic;
int varNonconst = 1;
int main(int argc, char **argv) {
And lets peek onto the binary:
➜ ~ objdump -t main | rg var
0000000000002008 l O .rodata 0000000000000004 _ZL9varGlobal
0000000000004138 l O .bss 0000000000000004 _ZL9varStatic
00000000000012bb l F .text 0000000000000019 _GLOBAL__sub_I_varNonconst
0000000000004010 g O .data 0000000000000004 varNonconst
From there you see where each one of them has ended up.
I have a template class, which has two static member variables, one int and another an std::array<volatile uint_fast32_t, 8>. When I instantiate the template with two different classes (which are templates themselves) as template parameters, for one of the instantiations everything works perfectly, i.e. there is exactly one copy of both variables. However, for the other, the array appears in duplicate in the symbol table, and indeed my code has a bug that when I set a value in the array in one compilation unit, the change does not appear in another.
This is for an embedded system, which is the reason for this weird idiom of using static templates for a kind of compile-time polymorphism.
In code: Header declaring the class itself
//dacmux.h
namespace HAL {
template<typename dac_write_sequence_t,
unsigned int chans,
typename sample_t = uint_fast32_t>
struct dacmux {
private:
typedef std::array<volatile sample_t, chans> chans_t;
static chans_t channels;
static unsigned int nextchan;
...
};
//The static variables defined here,
//count on the compiler/linker to make sure
//there is exactly one definition
template<typename dac_write_sequence_t,
unsigned int chans,
typename sample_t>
typename dacmux<dac_write_sequence_t, chans, sample_t>::chans_t dacmux<dac_write_sequence_t, chans, sample_t>::channels{0};
template<typename dac_write_sequence_t, unsigned int chans, typename sample_t>
unsigned int dacmux<dac_write_sequence_t, chans, sample_t>::nextchan = 0;
template<typename dac_t, typename addr_t, typename en_t>
struct muxed_setter {
...
};
template<typename dac_t>
struct dac_setter {
...
};
}//namespace HAL
A header which distributes definitions of the hardware:
//Hardware_types.h
...
//Multiplexer for the internal DAC
typedef HAL::dacmux<HAL::muxed_setter<dac1, mux1_addr, mux1_en>, 8> mux1;
//Sequencer for writing the external DAC values
typedef HAL::dacmux<HAL::dac_setter<extdac1>, 8> extdac_sequencer;
...
The header Hardware_types.h is included in two source files, main.cpp, DebugConsole.cpp, both of which use both mux1 and extdac_sequencer.
As for as I understand, based on answers such as this one and many others, the compiler should take care that each of the static member variables is instantiated exactly once for each instantiation of the template?
However, when I set the values of extdac_sequencer::channels in DebugConsole.cpp, the changes are not reflected in an interrupt handler declared in main.cpp. The same works perfectly for mux1::channels. Indeed, an excerpt from the symbol table, extracted from the .elf by objdump -t:
20000280 l O .bss 00000004 _ZN3HAL6dacmuxINS_10dac_setterINS_6ti_dacINS_3SPIINS_5SPI_2EEEN5GPIOs5pin_tINS6_1BELj12EEENS_12_GLOBAL__N_110xx68_frameENSA_12command_xx68ENSA_12channel_xx68EEEEELj8EjE8nextchanE
...
20000254 l O .bss 00000020 _ZN3HAL6dacmuxINS_10dac_setterINS_6ti_dacINS_3SPIINS_5SPI_2EEEN5GPIOs5pin_tINS6_1BELj12EEENS_12_GLOBAL__N_110xx68_frameENSA_12command_xx68ENSA_12channel_xx68EEEEELj8EjE8channelsE
...
20000288 l O .bss 00000020 _ZN3HAL6dacmuxINS_10dac_setterINS_6ti_dacINS_3SPIINS_5SPI_2EEEN5GPIOs5pin_tINS6_1BELj12EEENS_12_GLOBAL__N_110xx68_frameENSA_12command_xx68ENSA_12channel_xx68EEEEELj8EjE8channelsE
...
20000234 w O .bss 00000020 _ZN3HAL6dacmuxINS_12muxed_setterIN4DACs11DAC_channelILj1EN5GPIOs5pin_tINS4_1AELj4EEEEENS4_12bit_stripe_tINS4_1CELj6ELj3EEENS5_ISA_Lj9EEEEELj8EjE8channelsE
...
2000027c w O .bss 00000004 _ZN3HAL6dacmuxINS_12muxed_setterIN4DACs11DAC_channelILj1EN5GPIOs5pin_tINS4_1AELj4EEEEENS4_12bit_stripe_tINS4_1CELj6ELj3EEENS5_ISA_Lj9EEEEELj8EjE8nextchanE
So the nextchan variable appears once per instantiation, as it should, and for mux1 so does channels. However, for extdac_sequencer, the channels variable is repeated, which I believe explains the bug.
Am I doing something wrong, or is this a compiler or linker bug?
Compiler: GCC arm-none-eabi 5.2.1 20151202
Linker: arm-none-eabi-ld 2.25.90.20151217
Linker options: -T mem.ld -T libs.ld -T sections.ld -nostartfiles -Xlinker --gc-sections -L"../ldscripts" -Wl,-Map,"Synth1Firmware.map" -Xlinker --cref --specs=nano.specs
Update:
I've narrowed down the conditions for this to happen:
If the first template parameter of dacmux is not itself a template, everything works, i.e. no duplicate symbol:
struct extdac1_setter {
template<typename sample_t>
inline static void update(sample_t val, unsigned int addr) {
extdac1::write_and_update(val, addr);
}
};
//Multiplexer for external DAC, works
typedef HAL::dacmux<extdac1_setter, 8> extdac_sequencer;
However, if the template parameter is templated itself, I get the duplicate symbol problem:
template<typename dac_t>
struct dac_setter {
template<typename sample_t>
inline static void update(sample_t val, unsigned int addr) {
dac_t::write_and_update(val, addr);
}
};
//Multiplexer for external DAC, this produces a duplicate symbol
typedef HAL::dacmux<dac_setter<extdac1>, 8> extdac_sequencer;
Here, extdac1 is itself, again, a template:
typedef HAL::DAC8568<dacspi, typename dacspi::nss> extdac1;
...and dacspi is a template, and so on. Also, in the case which does work, with the other instantation, while dac_write_sequence_t is a template, it isn't anymore a template of templates. So I'm starting to think that this is a problem with template recursion depth, i.e. ld isn't looking deep enough.
A further interesting observation: on exactly the same condition as having the duplicate symbol, the Eclipse syntax highlighter says "invalid template parameters" on the line declaring extdac_sequencer, although the actual compilation step goes through.
Turns out this was my sillyness: I was using an unnamed namespace in a header that defines the template HAL::DAC8568, which is defined as
template<typename spi_t, typename nss_t> using DAC8568 = ti_dac<spi_t, nss_t,
xx68_frame,
command_xx68,
channel_xx68>;
Here xx68_frame, command_xx68, and channel_xx68 are all defined in the unnamed namespace (which is of course the wrong thing to do in a header). This of course means that when instantiated from a different compilation unit, I get a different type for each of them and therefore a different type for DAC8568 and so on, so it's perfectly natural to get another instance of the static variable.
Changing the unnamed namespace to namespace detail fixed the problem immediately.
I am still somewhat baffled by the fact that the mangled names in the linker output seem identical. How can that be?
Anyway, we learn from this the following (some of which we knew already):
I'm an instantiation of the simpleton -pattern
unnamed namespaces in headers are truly evil
bugs stemming from the above can be quite subtle
I have such code:
struct Storage
{
static int GetData()
{
static int global_value;
return global_value++;
}
};
int free_func()
{
static int local_value;
return local_value++;
}
int storage_test_func()
{
return Storage::GetData();
}
Compiling it on OSX:
$ clang++ 1.cpp -shared
And running nm:
$ nm | c++filt
I am getting strange results:
0000000000000f50 T storage_test_func()
0000000000000f30 T free_func()
0000000000000f60 unsigned short Storage::GetData()
0000000000001024 bool free_func()::local_value
0000000000001020 D Storage::GetData()::global_value
U dyld_stub_binder
Two symbols (local_value and global_value) have different linkage! One visible difference is that global_value is defined in static member function and local_value is defined in free function.
Can somebody explain why it happens?
UPD:
After comments reading it looks like I should clarify things.
Maybe using of c++filt was a bad idea. Without it it shows:
$ nm
0000000000000f50 T __Z17storage_test_funcv
0000000000000f30 T __Z9free_funcv
0000000000000f60 t __ZN7Storage7GetDataEv
0000000000001024 b __ZZ9free_funcvE11local_value
0000000000001020 D __ZZN7Storage7GetDataEvE5value
U dyld_stub_binder
So yes. __ZZ9free_funcvE11local_value goes to BSS and __ZZN7Storage7GetDataEvE5value goes to data section.
man nm says that:
If the symbol is local (non-external), the symbol's type is instead represented by the corresponding lowercase letter.
And that what I see. __ZZ9free_funcvE11local_value is marked with lowercase b and __ZZN7Storage7GetDataEvE5value is marked with capital D.
And this is the main part of the question. Why it happens like that?
UPD2
One more way:
$ clang++ -c -emit-llvm 1.cpp
$ llvm-dis 1.bc
shows how these variables are represented internally:
#_ZZ9free_funcvE11local_value = internal global i32 0, align 4
#_ZZN7Storage7GetDataEvE5value = global i32 0, align 4
UPD3
Also there were some concerns about different sections symbols belong to. Putting __ZZ9free_funcvE11local_value to the text section does not change its visibility:
struct Storage
{
static int GetData()
{
static int value;
return value++;
}
};
int free_func()
{
static int local_value = 123;
return local_value++;
}
int storage_test_func()
{
return Storage::GetData();
}
Compiling:
$ clang++ 1.cpp -shared
Checking:
$ nm
Gives:
0000000000000f50 T __Z17storage_test_funcv
0000000000000f30 T __Z9free_funcv
0000000000000f60 t __ZN7Storage7GetDataEv
0000000000001020 d __ZZ9free_funcvE11local_value
0000000000001024 D __ZZN7Storage7GetDataEvE5value
Now both symbols are in data section but still one of those is local and another is global.
And the question is why it happens? Can somebody the logic on such compiler's decision?
There is no difference in a local static variable in static member function, and a local static variable in a free function.
Two symbols (local_value and global_value) have different linkage!
In standard nomenclature and from point of view of the standard, both of the variables have no linkage.
The relevant difference between the functions is not that one is a static member function and the other is not. The relevant difference is that the former is an inline function, but the latter is not.
Non-inline functions can only be defined in one translation unit, and therefore their local static variables do not need to be accessible from other translation units.
Inline functions on the other hand must be defined in every translation unit that use them. And, since the local static must refer to the same object globally, that object must be visible to multiple translation units.
The values will be linked in different storage sections.
000000000001020 D Storage::GetData()::global_value
The Dshows that your variable will be linked to a section which will be initialized. From the nm man page:
"D"
"d" The symbol is in the initialized data section.
The local_value will not be initialized via the c-startup code.
After linking I got:
0804a0d0 b free_func()::local_value
0804a0d8 u Storage::GetData()::global_value
Also from nm man page:
"B"
"b" The symbol is in the uninitialized data section (known as BSS).
"U" The symbol is undefined.
"u" The symbol is a unique global symbol. This is a GNU extension to the standard set of ELF symbol bindings. For such a symbol the
dynamic linker will make sure that in the entire process there is just one symbol with this name and type in use.
The reason is simply that the values will initialized via the standard startup initialization ( data section ) or not ( bss section ). The names for the section are not generally specified but used in common implementations.
You can find a lot Q&A to "When and where will which var initialized or not".
e.g.:
When are static and global variables initialized?
Clarification: ( I hope )
That a variable is not initialized via startup code did not mean that is not initialized. A static variable inside a method/function is typically initialized in the first access of the containing code block. If you want to know how your compiler do this job, have a look in the assembly output.
For gcc you will find some __cxa_ labels around the value initialization which protects your vars from multiple init.
I can't understand, why if we define static variable of usual (non-template) class in header, we have linker error, but in case of templates all works fine and moreover we will have single instance of static variable among all translation units:
It's template header (template.h):
// template.h
template<typename T>
class Templ {
public:
static int templStatic;
};
template<typename T> Templ<T>::templStatic = 0;
It's first unit using template (unit1.cpp)
// unit1.cpp
#include "template.h"
int method1() {
return Templ<void>::templStatic++;
}
Second unit here (unit2.cpp):
// unit2.cpp
#include "template.h"
int method2() {
return Templ<void>::templStatic++;
}
And, finally, main.cpp:
// main.cpp
#include <iostream>
int method1();
int method2();
int main(int argc, char** argv) {
std::cout << method1() << std::endl;
std::cout << method2() << std::endl;
}
After compilling, linking and executing this code, we will have following output:
0
1
So, why in case of templates all works fine (and as expected) ? How compiler or linker handle this (we can compile each .cpp file in separated calling of compiler, and then link them with caling to linker, so compiler and linker don't "see" all .cpp files at same time) ?
PS: My compiler: msvcpp 9 (but checked on mingw too)
It's because the definition of the static data member is itself a template. Allowing this is necessary for the same reason you are allowed to have a function template that's not inline multiple times in a program. You need the template to generate the resulting entity (say, a function, or a static data member). If you wouldn't be allowed to put the definition of a static data member, how would you instantiate the following
template<typename T>
struct F {
static int const value;
};
template<typename T>
int const F<T>::value = sizeof(T);
It's not known what T is - the Standard says the definition outside the class template is a template definition, in which the parameters are inherited from its class template owner.
I've made some experiment with GCC. In the following, we have one implicit instantiation of F<float>::value, and one explicit specialization of F<char>::value which has to be defined in a .cpp file to not cause duplicated symbol errors when included multiple times.
// Translation Unit 1
template<typename T>
struct F {
static int value;
};
template<typename T>
int F<T>::value = sizeof(T);
// this would belong into a .cpp file
template<> int F<char>::value = 2;
// this implicitly instantiates F<float>::value
int test = F<float>::value;
int main() { }
The second translation unit contains just another implicit instantiation of the same static data member
template<typename T>
struct F {
static int value;
};
template<typename T>
int F<T>::value = sizeof(T);
int test1 = F<float>::value;
Here is what we get with GCC - it makes each implicit instantiation into a weak symbols and sticks it into its own section here. Weak symbols will not cause errors when there exist multiple of them at link time. Instead, the linker will choose one instance, and discards the other ones assuming all of them are the same
objdump -Ct main1.o # =>
# cut down to the important ones
00000000 l df *ABS* 00000000 main1.cpp
0000000a l F .text 0000001e __static_initialization_and_destruction_0(int, int)
00000000 l d .data._ZN1FIfE5valueE 00000000 .data._ZN1FIfE5valueE
00000028 l F .text 0000001c global constructors keyed to _ZN1FIcE5valueE
00000000 g O .data 00000004 F<char>::value
00000000 g O .bss 00000004 test
00000000 g F .text 0000000a main
00000000 w O .data._ZN1FIfE5valueE 00000004 F<float>::value
So as we can see F<float>::value is a weak symbol which means the linker can see multiple of these at link time. test, main and F<char>::value are global (non-weak) symbols. Linking main1.o and main2.o together, we see in the map output (-Wl,-M) the following
# (mangled name)
.data._ZN1FIfE5valueE
0x080497ac 0x4 main1.o
0x080497ac F<float>::value
This indicates that actually it drops all except one instance.
There is solution, you can create a parent class and put the static variable in it, then make your template class inherit it privately, here's an example:
class Parent
{
protected:
static long count;
};
long Parent::count = 0;
template<typename T>
class TemplateClass: private Parent
{
private:
int mKey;
public:
TemplateClass():mKey(count++){}
long getKey(){return mKey;}
}
int main()
{
TemplateClass<int> obj1;
TemplateClass<double> obj2;
std::cout<<"Object 1 key is: "<<obj1.getKey()<<std::endl;
std::cout<<"Object 2 key is: "<<obj2.getKey()<<std::endl;
return 0;
}
Output will be:
Object 1 key is: 0
Object 2 key is: 1
It's because template code is not source code; it's instructions on how to write source code.
The non-template static variable is actual source code, and the compiler will attempt to do exactly what you say by including something in twice. Hence, you have to initialize the static variable in a .cpp file, and only reference it in the .h file describing the class. It's equivalent to a global variable declared through extern.
When the compiler sees
template<class T> Templ{...};
it does nothing except make a note that the template exists. As far as it is concerned, there is no source code associated with Templ.
The first time you actually refer to
Templ<int> Instance
the compiler looks at all the template<> code associated with Templ and uses it to construct a .h and a .cpp file (which exists only for the duration of compilation). Those files might look like this:
Temple_int.h
class Templ_int{
public:
static int templStatic;
};
Templ_int.cpp
#include "Templ_int.h"
Templ_int::templStatic = 0;
And every
Templ<int>
becomes a Templ_int.
Thus, the source code to initialize the static variable only exists once, in a .cpp file created by the compiler.
(Obviously, the actual compiler-specific implementation of this process would be robust against creating a class with a similar name to the template, etc.)