Setting stack size with setrlimit fails with gcc - c++

Question

I'm using gcc 10.1 on Ubuntu 18.04. I'm getting segfaults when defining a large stack allocated variable, even though my stack seems to be large enough to accommodate it. Here is a code snippet:
#include <iostream>
#include <array>
#include <sys/resource.h>
using namespace std;
int main() {
if (struct rlimit rl{1<<28, 1l<<32}; setrlimit(RLIMIT_STACK, &rl))
cout << "Can not set stack size! errno = " << errno << endl;
else
cout << "Stack size: " << rl.rlim_cur/(1<<20) << "MiB to " << rl.rlim_max/(1<<20) << "MiB\n";
array<int8_t, 100'000'000> a;
cout << (int)a[42] << endl;
}
which segfaults when compiled with gcc, but runs fine when compiled with clang 11.0.1 and outputs:
Stack size: 256MiB to 4096MiB
0
EDIT
Clang was eliding allocation of a. Here is a better example:
#include <iostream>
#include <array>
#include <sys/resource.h>
using namespace std;
void f() {
array<int8_t, 100'000'000> a;
cout << (long)&a[0] << endl;
}
int main()
{
if (struct rlimit rl{1<<28, 1l<<32}; setrlimit(RLIMIT_STACK, &rl))
cout << "Can not set stack size! errno = " << errno << endl;
else
cout << "Stack size: " << rl.rlim_cur/(1<<20) << "MiB to " << rl.rlim_max/(1<<20) << "MiB" << endl;
array<int8_t, 100'000'000> a; // line 21
cout << (long)&a[0] << endl; // line 23
f();
}
which you can find at: https://wandbox.org/permlink/XMaGFMa7heWfI9G8. It runs fine when lines 21 and 23 are commented out, but segfaults otherwise.

Answer 1

Use proc(5) and pmap(1) and strace(1) to understand the limitations of your computer.
array<int8_t, 100'000'000> a;
requires about 100Mbytes of space on call stack, which is generally limited to a few megabytes (perhaps even by your Linux kernel, but I am not sure)
Try also cat /proc/$$/limits in your terminal. On mine I am getting
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
The difference of behavior between compilers might be attributed to various optimizations (e.g. permitted by some C++ standard like n4849). A clever enough compiler is allowed to use just a few words for a inside your function f (e.g. because it would figure out, maybe with some abstract interpretation techniques, that locations a[1024] ... a[99999999] are useless).
If you compile with a recent GCC (e.g. GCC 10), you could invoke it as g++ -O -Wall -Wextra -fanalyzer -Wstack-usage=2048 to get useful warnings. See also this draft report funded by CHARIOT & DECODER projects.
In practice, use dynamic allocation for huge data (e.g. placement new with mmap(2) and smart pointers)
For a real application, consider writing your GCC plugin to get ad-hoc warnings.
Or at least compile your source code foo.cc with g++ -O2 -fverbose-asm -S foo.cc and look inside the generated foo.s and repeat with clang++ : the generated assembler files are different.