Initializing std::vector outside of main() causes performance drop (multithreading) - c++

Question

I'm writing a path tracer as a programming exercise. Yesterday I finally decided to implement multithreading - and it worked well. However, once I wrapped the test code I wrote inside main() in a separate renderer class, I noticed a significant and consistent performance drop. In short - it would seem that filling std::vector anywhere outside of main() causes threads using its elements to perform worse. I managed to isolate and reproduce the issue with simplified code, but unfortunately I still don't know why it happens or what to do in order to fix it.
Performance drop is quite visible and consistent:
97 samples - time = 28.154226s, per sample = 0.290250s, per sample/th = 1.741498
99 samples - time = 28.360723s, per sample = 0.286472s, per sample/th = 1.718832
100 samples - time = 29.335468s, per sample = 0.293355s, per sample/th = 1.760128
vs.
98 samples - time = 30.197734s, per sample = 0.308140s, per sample/th = 1.848841
99 samples - time = 30.534240s, per sample = 0.308427s, per sample/th = 1.850560
100 samples - time = 30.786519s, per sample = 0.307865s, per sample/th = 1.847191
The code I originally posted in this question can be found here: https://github.com/Jacajack/rt/tree/mt_debug or in edit history.
I created a struct foo that is supposed to mimic the behavior of my renderer class and is responsible for initialization of path tracing contexts in its constructor.
The interesting thing is, when I remove the body of foo's constructor and instead do this (initialize contexts directly from main()):
std::vector<rt::path_tracer> contexts; // Can be on stack or on heap, doesn't matter
foo F(cam, scene, bvh, width, height, render_threads, contexts); // no longer fills `contexts`
contexts.reserve(render_threads);
for (int i = 0; i < render_threads; i++)
contexts.emplace_back(cam, scene, bvh, width, height, 1000 + i);
F.run(render_threads);
the performance is back to normal. But then, if I wrap these three lines into a separate function and call it from here, it's worse again. The only pattern I can see here is
that filling the contexts vector outside of main() causes the problem.
I initially thought that this was an alignment/caching issue, so I tried aligning path_tracers with Boost's aligned_allocator and TBB's cache_aligned_allocator with no result. It turns out that this problem persists even when there's only one thread running.
I suspect it must be some kind of wild compiler optimization (I'm using -O3), althought that's just a guess. Do you know any possible causes of such behavior and what can be done to avoid it?
This happens on both gcc 10.1.0 and clang 10.0.0. Currently I'm only using -O3.
I managed to reproduce a similar issue in this standalone example:
#include <iostream>
#include <thread>
#include <random>
#include <algorithm>
#include <chrono>
#include <iomanip>
struct foo
{
std::mt19937 rng;
std::uniform_real_distribution<float> dist;
std::vector<float> buf;
int cnt = 0;
foo(int seed, int n) :
rng(seed),
dist(0, 1),
buf(n, 0)
{
}
void do_stuff()
{
// Do whatever
for (auto &f : buf)
f = (f + 1) * dist(rng);
cnt++;
}
};
int main()
{
int N = 50000000;
int thread_count = 6;
struct bar
{
std::vector<std::thread> threads;
std::vector<foo> &foos;
bool active = true;
bar(std::vector<foo> &f, int thread_count, int n) :
foos(f)
{
/*
foos.reserve(thread_count);
for (int i = 0; i < thread_count; i++)
foos.emplace_back(1000 + i, n);
//*/
}
void run(int thread_count)
{
auto task = [this](foo &f)
{
while (this->active)
f.do_stuff();
};
threads.reserve(thread_count);
for (int i = 0; i < thread_count; i++)
threads.emplace_back(task, std::ref(foos[i]));
}
};
std::vector<foo> foos;
bar B(foos, thread_count, N);
///*
foos.reserve(thread_count);
for (int i = 0; i < thread_count; i++)
foos.emplace_back(1000 + i, N);
//*/
B.run(thread_count);
std::vector<float> buffer(N, 0);
int samples = 0, last_samples = 0;
// Start time
auto t_start = std::chrono::high_resolution_clock::now();
while (1)
{
last_samples = samples;
samples = 0;
for (auto &f : foos)
{
std::transform(
f.buf.cbegin(), f.buf.cend(),
buffer.begin(),
buffer.begin(),
std::plus<float>()
);
samples += f.cnt;
}
if (samples != last_samples)
{
auto t_now = std::chrono::high_resolution_clock::now();
std::chrono::duration<double> t_total = t_now - t_start;
std::cerr << std::setw(4) << samples << " samples - time = " << std::setw(8) << std::fixed << t_total.count()
<< "s, per sample = " << std::setw(8) << std::fixed << t_total.count() / samples
<< "s, per sample/th = " << std::setw(8) << std::fixed << t_total.count() / samples * thread_count << std::endl;
}
}
}
and results:
For N = 100000000, thread_count = 6
In main():
196 samples - time = 26.789526s, per sample = 0.136681s, per sample/th = 0.820088
197 samples - time = 27.045646s, per sample = 0.137288s, per sample/th = 0.823725
200 samples - time = 27.312159s, per sample = 0.136561s, per sample/th = 0.819365
vs.
In foo::foo():
193 samples - time = 22.690566s, per sample = 0.117568s, per sample/th = 0.705406
196 samples - time = 22.972403s, per sample = 0.117206s, per sample/th = 0.703237
198 samples - time = 23.257542s, per sample = 0.117462s, per sample/th = 0.704774
200 samples - time = 23.540432s, per sample = 0.117702s, per sample/th = 0.706213
It seems that the results are the opposite of what is happening in my path tracer, but the visible difference is still here.
Thank you

Answer 1

There is a race condition with foo::buf - one thread makes stores into it, anther reads it. This is undefined behaviour, but on x86-64 platform that is harmless in this particular code.
I cannot reproduce your observations on Intel i9-9900KS, both variants print the same per sample stats.
Compiled with gcc-8.4, g++ -o release/gcc/test.o -c -pthread -m{arch,tune}=native -std=gnu++17 -g -O3 -ffast-math -falign-{functions,loops}=64 -DNDEBUG test.cc
With int N = 50000000; each thread operates on its own array of float[N] which occupies 200MB. Such a data set doesn't fit in CPU caches and the program incurs a lot of data cache misses because it needs to fetch the data from memory:
$ perf stat -ddd ./release/gcc/test
[...]
71474.813087 task-clock (msec) # 6.860 CPUs utilized
66 context-switches # 0.001 K/sec
0 cpu-migrations # 0.000 K/sec
341,942 page-faults # 0.005 M/sec
357,027,759,875 cycles # 4.995 GHz (30.76%)
991,950,515,582 instructions # 2.78 insn per cycle (38.43%)
105,609,126,987 branches # 1477.571 M/sec (38.40%)
155,426,137 branch-misses # 0.15% of all branches (38.39%)
150,832,846,580 L1-dcache-loads # 2110.294 M/sec (38.41%)
4,945,287,289 L1-dcache-load-misses # 3.28% of all L1-dcache hits (38.44%)
1,787,635,257 LLC-loads # 25.011 M/sec (30.79%)
1,103,347,596 LLC-load-misses # 61.72% of all LL-cache hits (30.81%)
<not supported> L1-icache-loads
7,457,756 L1-icache-load-misses (30.80%)
150,527,469,899 dTLB-loads # 2106.021 M/sec (30.80%)
54,966,843 dTLB-load-misses # 0.04% of all dTLB cache hits (30.80%)
26,956 iTLB-loads # 0.377 K/sec (30.80%)
415,128 iTLB-load-misses # 1540.02% of all iTLB cache hits (30.79%)
<not supported> L1-dcache-prefetches
<not supported> L1-dcache-prefetch-misses
10.419122076 seconds time elapsed
If you run this application on NUMA CPUs, such as AMD Ryzen and Intel Xeon with multiple sockets, then your observations can probably be explained by adverse placement of threads onto remote NUMA nodes relative to NUMA node where foo::buf is allocated. Those last-level data cache misses have to read memory and if that memory is in a remote NUMA node that takes longer.
To fix that, you may like to allocate memory in the thread that uses it (not in the main thread as the code does) and use a NUMA-aware allocator, such as TCMalloc. See NUMA aware heap memory manager for more details.
When running your benchmark you may like to fix the CPU frequency, so that it doesn't get dynamically adjusted during the run, on Linux you can do that with sudo cpupower frequency-set --related --governor performance.