Stack Smashing in GCC vs Clang (Possibly due to canaries) - c++

Question

I am trying to understand possible sources for "stack smashing" errors in GCC, but not Clang.
Specifically, when I compile a piece of code with just debug symbols
set(CMAKE_CXX_FLAGS_DEBUG "-g")
and use the GCC C++ compiler (GNU 5.4.0), the application crashes with
*** stack smashing detected ***: ./testprogram terminated
Aborted (core dumped)
However, when I use Clang 3.8.0, the program completes without error.
My first thought was that perhaps the canaries of GCC are catching a buffer overrun that Clang isn't. So I added the additional debug flag
set(CMAKE_CXX_FLAGS_DEBUG "-g -fstack-protector-all")
But Clang still compiles a program that runs without errors. To me this suggests that the issue likely is not a buffer overrun (as you commonly see with stack smashing errors), but an allocation issue.
In any case, when I add in the ASAN flags:
set(CMAKE_CXX_FLAGS_DEBUG "-g -fsanitize=address")
Both compilers yield a program that crashes with an identical error. Specifically,
GCC 5.4.0:
==1143==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
==1143==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v
Aborted (core dumped)
Clang 3.8.0:
==1387==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
==1387==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v
Aborted (core dumped)
Can somebody give me some hints on the likely source of this error? I am having an awefully hard time tracing down the line where this is occurring, as it is in a very large code base.
EDIT
The issue is unresolved, but is isolated to the following function:
void get_sparsity(Data & data) {
T x[n_vars] = {};
T g[n_constraints] = {};
for (Index j = 0; j < n_vars; j++) {
const T x_j = x[j];
x[j] = NAN;
eval_g(n_vars, x, TRUE, n_constraints, g, &data);
x[j] = x_j;
std::vector<Index> nonzero_entries;
for (Index i = 0; i < n_constraints; i++) {
if (isnan(g[i])) {
data.flattened_nonzero_rows.push_back(i);
data.flattened_nonzero_cols.push_back(j);
nonzero_entries.push_back(i);
}
}
data.nonzeros.push_back(nonzero_entries);
}
int internal_debug_point = 5;
}
which is called like this:
get_sparsity(data);
int external_debug_point= 6;
However, when I put a debug point on the last line of the get_sparsity function, internal_debug_point = 5, it reaches that line without issue. However, when exiting the function, and before it hits the external debug point external_debug_point = 6, it crashes with the error
received signal SIGABRT, Aborted.
0x00007ffffe315428 in __GI_raise (sig=sig#entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
My guess is that GCC is only checking the canaries when exiting that function, and hence the error is actually occurring inside the function. Does that sound reasonable? If so, then is there a way to get GCC or clang to do more frequent canary checks?

Answer 1

I suspect ASan is running out of memory.
I don't think the ASan errors mean your program is trying to allocate that memory, it means ASan is trying to allocate it for itself (it says "shadow memory" which is what ASan uses to keep track of the memory your program allocates).
If the number of iterations (and size of array) n_vars is large, then the function will use extra memory for a new std::vector in every loop, forcing ASan to track more and more memory.
You could try moving the local vector out of the loop (which will likely increase the performance of the function anyway):
std::vector<Index> nonzero_entries;
for (Index j = 0; j < n_vars; j++) {
// ...
for (Index i = 0; i < n_constraints; i++) {
if (isnan(g[i])) {
data.flattened_nonzero_rows.push_back(i);
data.flattened_nonzero_cols.push_back(j);
nonzero_entries.push_back(i);
}
}
data.nonzeros.push_back(nonzero_entries);
nonzero_entries.clear();
}
This will reuse the same memory for nonzero_entries instead of allocating and deallcoating memory for a new vector every iteration.

Answer 2

Trying to figure out the source of the stack problems was getting nowhere. So I tried a different approach. Through debugging, I narrowed down the above function get_sparsity as the culprit. The debugger wasn't giving me any hints exactly WHERE the problem was occurring, but it was somewhere inside that function. With that information, I switched the only two stack variables in that function x and g to heap variables so that valgrind could help me find the error (sgcheck was coming up empty). Specifically, I modified the above code to
void get_sparsity(Data & data) {
std::vector<T> x(n_vars, 0);
std::vector<T> g(n_constraints, 0);
/* However, for our purposes, it is easier to make an std::vector of Eigen
* vectors, where the ith entry of "nonzero_entries" contains a vector of
* indices in g for which g(indices) are nonzero when perturbing x(i).
* If that sounds complicated, just look at the code and compare to
* the code where we use the sparsity structure.
*/
for (Index j = 0; j < n_vars; j++) {
const T x_j = x[j];
x[j] = NAN;
Bool val = eval_g(n_vars, x.data(), TRUE, n_constraints, g.data(), &data);
x[j] = x_j;
std::vector<Index> nonzero_entries;
for (Index i = 0; i < n_constraints; i++) {
if (isnan(g[i])) {
data.flattened_nonzero_rows.push_back(i);
data.flattened_nonzero_cols.push_back(j);
nonzero_entries.push_back(i);
}
}
data.nonzeros.push_back(nonzero_entries);
}
int bob = 5;
return;
}
and then valgrinded it to find the offending line. Now that I know where the problem is occurring, I can fix the problem.