Making sense of stacktrace - c++

Question

I have been trying to debug an issue with my service which results in a segmentation fault. I do not have access to the production server, so I have handled the SIGSEGV signal in my service and printed the stacktrace in the Log files. Following is the stacktrace when the service crashes
0# 0x00000000005054DA in ./afiniti_lookup
1# 0x00007F2BBB74A400 in /usr/lib64/libc.so.6
2# 0x00007F2BBB86F9BD in /usr/lib64/libc.so.6
3# 0x000000000041BB52 in ./afiniti_lookup
4# std::string::_M_move(char*, char const*, unsigned long) in ./afiniti_lookup
5# std::string::_M_mutate(unsigned long, unsigned long, unsigned long) in ./afiniti_lookup
6# std::string::_M_replace_safe(unsigned long, unsigned long, char const*, unsigned long) in ./afiniti_lookup
7# std::string::assign(char const*, unsigned long) in ./afiniti_lookup
8# std::string::assign(char const*) in ./afiniti_lookup
9# std::string::operator=(char const*) in ./afiniti_lookup
10# 0x000000000061E8E9 in ./afiniti_lookup
11# 0x0000000000620200 in ./afiniti_lookup
12# 0x000000000055B586 in ./afiniti_lookup
13# 0x00000000004F2BAC in ./afiniti_lookup
14# 0x00000000004F0715 in ./afiniti_lookup
15# 0x000000000051CDBF in ./afiniti_lookup
16# 0x0000000000529869 in ./afiniti_lookup
17# 0x0000000000464968 in ./afiniti_lookup
18# 0x0000000000461369 in ./afiniti_lookup
19# 0x0000000000460D6E in ./afiniti_lookup
20# 0x0000000000460086 in ./afiniti_lookup
21# 0x000000000045FD36 in ./afiniti_lookup
22# 0x000000000046CAB4 in ./afiniti_lookup
23# 0x000000000046B4F6 in ./afiniti_lookup
24# 0x000000000046FF13 in ./afiniti_lookup
25# 0x000000000046FE65 in ./afiniti_lookup
26# 0x000000000046FCDA in ./afiniti_lookup
27# 0x00007F2BBCE5038F in /opt/lib64/libcpprest.so.2.10
28# 0x00007F2BBEDCAEA5 in /usr/lib64/libpthread.so.0\n29# clone in /usr/lib64/libc.so.6
However, this trace is not of much use as I cannot pinpoint the location in my code where the issue is occurring. Can somebody help me better understand and inspect this stacktrace?

Answer 1

Can somebody help me better understand and inspect this stacktrace?
It looks like you have a partially-stripped executable in production.
You should have an unstripped copy (which was produced by linking your executable). If you don't, you'll need to change your ways, and save a copy before you strip.
With an unstripped copy, you can make sense of your stack trace like so:
addr2line -fe afiniti_lookup.unstripped 0x61E8E9 0x620200 0x55B586 ...
Here is example output:
cat foo.c
int foo() { int *ip = 0; return *ip; }
int bar() { return foo(); }
int zoo() { return bar(); }
int main() { return zoo(); }
Compile this with debug info: gcc -g foo.c (produces a.out).
Strip the binary for "production": strip --strip=all a.out -o b.out.
Run the b.out under GDB to simulate production stack trace:
(gdb) run
Starting program: /tmp/b.out
Program received signal SIGSEGV, Segmentation fault.
0x0000000000401112 in ?? ()
(gdb) bt
#0 0x0000000000401112 in ?? ()
#1 0x0000000000401124 in ?? ()
#2 0x0000000000401134 in ?? ()
#3 0x0000000000401144 in ?? ()
#4 0x00007ffff7dfbcca in __libc_start_main (main=0x401136, argc=1, argv=0x7fffffffdc98, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdc88) at ../csu/libc-start.c:308
#5 0x000000000040104a in ?? ()
Now use addr2line on the unstripped binary to make sense of the stack trace above:
addr2line -fe a.out 0x0000000000401112 0x0000000000401124 0x0000000000401134 0x0000000000401144
foo
/tmp/foo.c:1
bar
/tmp/foo.c:2
zoo
/tmp/foo.c:3
main
/tmp/foo.c:4
P.S. For real production use, ideally you would compile your binary with gcc -O2 -g ..., so you have full debug info, and then strip the binary (but keep a full-debug copy). That way you can fairly easily debug core dumps from production with access to functions, files, lines and variables.