Linux - segmentation fault only sometimes - how to debug - c++

Question

I have a Linux program, that from time to time ends with a segmentation fault. The program is running periodically every hour, but the segmentation fault occurs only sometimes.
I have a problem to debug this, because if I run the program again with the same input, no error is reported and all is OK.
Is there a way, how to "report" in which part of the code error occured or what caused the problem?

Answer 1

The usual way is to have the crashing program generate a corefile and analyze this after the crash. Make sure, that:
the maximum corefile-size is big enough (i.e. unlimited) by calling ulimit -c unlimited in the shell, which starts the process.
The cwd is writable by the segfaulting process.
Then you can analyze the file with
gdb <exe> <corefile>

Answer 2

Since your code not crashing every time, you can use backtrace as well. Using this you can see the function call stack at the time of crash. There are many examples available. In my projects I normally use the following code for backtracing.
/*
* call reg_usr2 function from main
* gcc -rdynamic myfile.c -o output
*/
#include <stdio.h>
#include <stdarg.h>
#include <signal.h>
#include <unistd.h>
#include <stdlib.h>
#include <execinfo.h>
#define FILE_NAME "/tmp/debug"
#define MODE 0xFFFF
void dbgprint(int flag, char* fmt, ...)
{
if(flag & MODE) {
char buf[100];
va_list vlist;
FILE *fp = fopen(FILE_NAME,"a");
va_start(vlist, fmt);
vsnprintf( buf, sizeof( buf), fmt, vlist);
va_end( vlist);
fprintf(fp,"[%x]->%s\n", flag, buf);
fclose(fp);
}
}
/** Here is the code to print backtrace **/
void print_stack_trace ()
{
void *array[20];
size_t size;
char **strings;
size_t i;
size = backtrace (array, 20);
strings = backtrace_symbols (array, size);
dbgprint(0xFFFF, "Obtained %zd stack frames.", size);
dbgprint(0xFFFF, "-------------------------");
dbgprint(0xFFFF, "---------Backtrace-------");
for (i = 0; i < size; i++)
dbgprint (0xFFFF, "%s", strings[i]);
dbgprint(0xFFFF, "-------------------------");
free (strings);
}
void sig_handler(int signo)
{
FILE *fp = fopen(FILE_NAME,"a");
if (signo == SIGUSR2){
dbgprint(0xFFFF, "received SIGUSR2");
dbgprint(0xFFFF, "----------------");
}
print_stack_trace();
exit(0);
}
void reg_usr2()
{
if (signal(SIGUSR2, sig_handler) == SIG_ERR)
printf("\ncan't catch SIGUSR2\n");
}
int main()
{
reg_usr2(); //should be first line of main after variables
//Code.....
return 0;
}

Answer 3

You can generate backtrace by catching SIGSEGV signal, and see where your application throw an invalid access.
see https://stackoverflow.com/a/77336/4490542
But there is more easier solution, try running your application with catchsegv
catchsegv './program args'
and better alternative, valgrind
valgrind --tool=none ./program args