Getting a stack trace of a hung process running in GDB - gdb

I am trying to figure out why a program I am using is hanging.
I initially thought the problem was "solved" by running it through GDB but I am getting the exact same symptoms as before - is there any way I can send a signal or something to GDB to print off the stack trace so I can try to see what is happening?
I am using FreeBSD-9.0.

You can interrupt a running process within GDB by simply pressing Control-C.
You can then get a backtrace with the backtrace command.

I ended up getting a backtrace by creating a coredump using gcore(1) and then starting myprogram with the coredump created in GDB.

Related

How do I debug an executable with gdb when it crashes during startup?

I have a C-and-C++-based project I just got to build and link for the first time, and it segfaults on execution. I tried running it in gdb to get a backtrace, and saw this:
gdb) run
Starting program: /home/jon/controlix-code/bin/controlix
During startup program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
No stack.
(gdb)
I assume it is crashing before main() is called, but beyond that I don't have a clue. I haven't been able to find much about this type of situation on Google, so I thought I'd ask here.
One approach is to catch all exceptions before running:
catch throw
run
And if that does not help, you may have to single-step through the assembly from the very beginning. But before you do that,
break main
run
and single-step through the code using step and next should lead you to the culprit.

gcc segmentation fault - how can I find a line where it happened?

I'm using Ubuntu and gcc. My application crashes I only have Segmentation fault message in console. (previously Segmentation fault (core dumped) was reported but now it changed to just Segmentation fault).
There are no hints where the problem is so I do not understand how should I fix the problem. I need some hints to find what caused this - ideally complete stack trace or at least object type/method or something like this.
What would be the correct way of troubleshooting such type of problem? (may be compile with some extra flags, run some tools, collect core dump and analyze it somehow?)
You might well need to enable core dumps with
ulimit -c unlimited
Once you have a core dump, you can look at the program state with GDB:
gdb my_prog core
You should then have the same view that you would have had if you'd run the program under GDB until it crashed - you could just do that, rather than collecting the core dump. In particular, it will show you which line caused the segfault, and the state of the call stack at that point.
To get the best debugging view, you should tell the compiler to include debugging symbols (-g) and disable optimisation (-O0).
you can use the gdb tools to help debuging.
run gdb ./your_app_name on terminal if you have the gdb installed and you will see some infomation as follow:
.....
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./gsvod_client...done.
(gdb)
then input "r" to start your app, if it crashed again, you can type 'bt' to see the line where the problem occured.

GDB - how to find out from where program exited

While debugging a program in GDB, I get an unexpected "program exited normally". So I'm wondering if is there a way to find out from where (which line) the program exited.
Program is multi-threaded, if that matters.
You could try the GDB command break exit to set a breakpoint on the exit(2) library call. If that doesn't get you what you need, maybe break _exit. You might need to start your program with 'sta' before getting the latter breakpoint to take. In either case, you should then be able to use the where command to get a stack trace showing where you were when the program decided to exit.
Usually with the command below when the application has finished executing:
(gdb) thread apply all bt
Of course, if you want to know the exact line you must compile your application with debugging symbols, i.e. -g
Set a breakpoint on _exit and then examine the stack.

Aborted core dumped C++

I have a large C++ function which uses OpenCV library and running on Windows with cygwin g++ compiler. At the end it gives Aborted(core dumped) but the function runs completely before that. I have also tried to put the print statement in the end of the function. That also gets printed. So I think there is no logical bug in code which will generate the fault.
Please explain.
I am also using assert statements.But the aborted error is not due to assert statement. It does not say that assertion failed. It comes at end only without any message.
Also the file is a part of a large project so I cannot post the code also.
gdb results:
Program received signal SIGABRT, Aborted.
0x7c90e514 in ntdll!LdrAccessResource () from /c/WINDOWS/system32/ntdll.dll
It looks like a memory fault (write to freed memory, double-free, stack overflow,...). When the code can be compiled and run under Linux you can use valgrind to see if there are memory issues. Also you can try to disable parts of the application until the problem disappears, to get a clue where the error happens. But this method can also give false positives, since memory related bugs can cause modules to fail which are not the cause of the error. Also you can run the program in gdb. But also here the position the debugger points to may not be the position where the error happened.
You don't give us much to go on. However, this looks like you are running into some problem when freeing resources. Maybe a heap corruption. Have you tried running it under gdb and then looking where it crashes? Also, check if all your new/delete calls match.
Load the core dump together with the binary into gdb to get an idea at which location the problem list. Command line is:
gdb <path to the binary> <path to the core file>
For more details on gdb see GDB: The GNU Project Debugger.
Run it through AppVerifier and cdb.
E.g.
cdb -xd sov -xd av -xd ch <program> <args>

How to run gdb against a daemon in the background?

I'm trying to debug a server I wrote with gdb as it segfaults under very specific and rare conditions.
Is there any way I can make gdb run in the background (via quiet or batch mode?), follow children (as my server is a daemon and detaches from the main PID) and automatically dump the core and the backtrace (to a designated file) once the program crashes?
Assuming you have appropriate permissions, you can have gdb attach to any process. You can do it on the command line with:
gdb /path/to/binary _pid_
or from within gdb with the attach command:
attach _pid_
So, once your daemon has started, you can use either of these techniques to attach to the final PID your daemon is running as. Attaching gdb stops the process which you are tracing so you will need to issue a "continue" to restart it.
I don't know a direct way to get gdb to run arbitrary commands when the program crashes. Here is one workaround I can think of:
Create and register a signal handlers for SIGSEGV.
Tell gdb not to stop on that signal (handle SIGSEGV nostop)
Set a breakpoint at the first line of your signal handler.
Assign commands to the breakpoint from step 3
Why not just run the process interactively in a persistent screen session? Why must it be a daemon when debugging? Or just run gdb in the screen session and attach it to the running process (e.g. gdb /path/to/binary -p PID_of_binary) after it forks.
First, I'd setup your shell / environment to give you a core dump. In bash:
ulimit -c unlimited
Once you have the core dump, you can use gdb to examine the stack trace:
gdb /path/to/app /path/to/core/file
I'm not really a gdb expert but two things come to mind
Tracepoints which might give you the necessary information as your program runs or
Use gdb's remote debugging facility to debug your program while it's running as a daemon.
How to generate a stacktrace when my gcc C++ app crashes answer for this question should do what you want. (assuming you can make changes in your code)
You might want to take a look at how Samba facilitates debugging; it has a configurable "panic action" that can suspend the application, notify the developer, spawn gdb, etc., and is run as part of its signal handler. See lib/util/fault.c in the Samba source tree.
My practice: comment out daemon function call, rebuild binary, then use gdb to run.