My understanding is that gdb can monitor the complete state of a running program. Can I save a gdb session that is paused at a breakpoint and resume the session later?
My first attempt was simply generating a core dump in a first gdb session that was paused at a breakpoint and then using the core dump to start a second gdb session.
Saving core file in gdb
This resulted in the following error.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
So breakpoint information is inserted into the program state, interesting. On my second attempt I did the same but this time I added the same breakpoint to the second session as were in the first session.
Getting gdb to save a list of breakpoints?
Still, I get the same error.
Can I save and restart a gdb session? If so, how?
I don't think this is directly relevant but I'm also getting this warning.
warning: core file may not match specified executable file.
Is gdb simply stating that such a thing is possible in general or does gdb believe this may have happened in the running session? I'm confident that the same executable that produced the core dump is being run under gdb.
Edit: For anyone else who comes along, this question: Save a process' memory for later use? adds to Mats Petersson's answer and links to this article: http://blogs.msdn.com/b/oldnewthing/archive/2004/04/20/116749.aspx which is an interesting read. The linked question also has the suggestion of wrapping the process up in a VM.
I doubt that will ever work. Handles for files and any other resources (semaphores, shared memory, serial ports, network connections and lots of other things) that the program has opened/created will be lost when you save the core-file. You can inspect it, but you can't "continue". A core-file is simply a copy of all the memory that original program was using. Anything else is "lost" when the program terminates. In other words, a core-file will only be useful to inspect things later on, but you can't run, step or continue in a core-file debug session. Only "look at things". And if you can't execute, breakpoints won't really work either... ;)
Related
My project is quite large and multithreaded. There should be a bug which crashes the whole program.
For release version, it stuck sometimes, but does not appear very often.
For debug code, it is more likely to appear. And the stack trace of gdb is the following.
0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:81
1 0x00007dff8270c700 in ?? ()
2 0x00007ffff6dde38d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
This information is not enough for me to locate the buggy code.
So my question is: how to get more information from the crash? any advanced use of gdb or other advanced tools?
============= Update ==============
One more information to add, after printing out all the thread ids, I figure out the thread that crashed. The only difference of the thread is that it is detached from the std thread object. If anyone has any experience with this, please tell me.
============= Update2 ================
This problem is not solved yet, and turn out to be a sever one.
If I run in the terminal, it'll crash the whole terminal and all other programs currently running under my username.
The system is then down and not accessible by ssh for a while. There are some other users getting broken pipe and it seems my program has made sshd not responsive.
After a while I'm able to login again, and find that the binary file of the program is broken (truncated) and need to recompile.
For me it looks like a memory or stack overwrite or access of dead pointers or objects.
To catch these kind of errors I like to use tools like efence or valgrind. With actual compilers you also can use thread sanitizer or the memory sanitizer. Both works with clang and g++.
If you can not catch the problem with that, you also should install the debug library version of the standard libs. Sometimes a wrong value crashes inside the g++lib or some other libs which results in hard to debug situations. With the debug infos installed you can catch this much easier.
I am using GDB in remote configuration. So I have gdbserver running on a ARM HW running linux, I connect to remote gdbserver from local gdb, I am able to put breakpoints in initial part of code and ensure that basically GDB works. However when I let my SW run for a while and break to see threads, I don't get useful stack traces for threads. All stack frames are hex addressed none resolving to symbols present in my binaries. Most of them also look the same with gdb also hinting with "same stack frames, corrupted stack ??"
Since SW runs fine with or without debugger I don't doubt my SW. Anybody seen this issue or any idea what might be going on here.
Thanks
I am trying to extract the execution sequence of my program (something like a program counter) with gdb on my local computer (windows x86) and gdbserver on a remote target (arm-linux). The idea I had was to insert breakpoints at "important" lines of my source files (i.e.: at the beginning of a specific function, and more in general before and after a conditional statement) with a high ignore count for each breakpoint, and then check if a breakpoint was hit or not. I was actually able to receive the informations with this method, but there is a problem: the application behavior I am debugging depends on real-time, and this specific method slows down the program execution too much. Do you think I could use some other method with gdb? I stumbled upon tracepoints, wich seems the exact thing I am looking for, but I was not able to find some property like a "hit counter" for them. The gdb version I am currently using is 7.5.
Thanks a lot in advance.
If your program execution must not be slowed down, you will probably need some HW tool. See these:
Keil real time trace
Lauterbach PowerDebug
(probably other similar solutions)
I just fired up totalview on my "hello world" application (c++) and i only get to view the assembly code.
Is there any settings/flags i need to set to view the source code? Menubar->View->Source As->Source does not work for me.
the application im trying to debug is just a cout << "Hello World" application, just to get the debugger up and running.
Lets start with the simple stuff.
Did you compile your application with the '-g' debugging flag? The debugger relies on the compiler to provide it with a symbol table and line number table to map what happens in the executable back to your source code. Without that -g flag (or if you subsequently strip your application) that info won't be present and assembly debugging is the best you can hope for.
If you did compile with -g are the source and the executable all together in the same directory, or if not have they been moved since you compiled them? The compiler only knows the locations of the source and executable at the time they are created, if you move them around then sometimes the debugger won't be able to locate the source code file. In that case you might need to give it some help by defining a source code search path.
Write back here and let me know if -g fixed your problem. If not we can look into the search path and such.
Cheers,
Chris
I realize that Jason94 has almost certainly solved his problem some other way, but I figured I could chime in here to answer this since it is a good question.
For this particular case it would be interesting to know if the program is multi-threaded. TotalView is designed to let you work with multi-threaded programs and it has a characteristic that may be surprising to users. By default it won't always focus you on the thread that hits the breakpoint. So your program might actually have stopped at your second breakpoint in another thread.
Imagine you have 6 threads (we'll number them 0 - 5) and you set a breakpoint in a routine. Thread 0 is the one you are focused on and you hit "go". The program runs and thread 4 hits the breakpoint first. By default the breakpoint will stop the whole process when the breakpoint is hit. In the debugger you might see assembly representing where thread 0 was when thread 4 hit the breakpoint.
You can check the root window or the thread pane to see what the status of the other threads are and you might see that one of them says "B2" (for breakpoint 2). Then you can click on that thread and TotalView will refocus you to that thread and you'll see it sitting at the breakpoint.
Why do we do that? Well, because we think it is confusing/disconcerting to have your focus "ripped away from you" just because another thread hit a breakpoint. So by default we leave the user in control of their thread focus.
There is a preference that you can change which will tell totalview to refocus the process window to the "site of the event". You can set that if you would prefer to have TotalView refocus your attention to the breakpoint, but be aware that as you do that you may be bouncing from one thread to the next.
The other possibility is that TotalView stopped the process for some reason other than a breakpoint being hit. Did the program segfault? Check the status bar at the top of the process window to see what the status of the thread and process are.
Anyway -- just wanted to post this for the record.
To learn a bit more about FreeBSD and *nix systems in general, I'm starting to look at the binaries from the DEFCON 17 Capture The Flag game. Right now, I'm reversing the tucod binary. Here's some possibly useful information on tucod:
tucod: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 7.2, dynamically linked (uses shared libs), FreeBSD-style, stripped
Some other possibly useful information gained from some brief static analysis is that tucod binds on port 0xDEAD (cute, eh?) and if you give it a specific password ("HANGEMHIGH!") it will play a game of hang-man with you.
The problem that I'm encountering is that I'm not hitting my breakpoints in gdb. Specifically, the breakpoint that I'm trying to reach is in the code that handles the client connection. Without breakpoints, the code executes as expected. When I set a breakpoint on that code, the child exits (instead of breaking into gdb, as expected). If I set breakpoints before the server forks off the child, I can hit those fine but after hitting "continue" the child does not continue to process my connection (that is, it won't ask me for a password or play hang-man).
Since the daemon forks when it receives a new connection, I try to tell gdb to follow the child with this command:
(gdb) set follow-fork-mode child
But after single-stepping the instructions after the fork, it appears that this isn't working.
I've tried looking for calls to signal, thinking they implemented a custom SIGINT handler (or similar), but the only call to signal that I can see handles SIGCHLD.
My breakpoint in gdb currently looks like this:
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y 0x080497d0
And 0x080497d0 is the address I want to break on in the client processing code.
I'm sort of new to analyzing software on *nix systems and could use some pointers. How else should I go about troubleshooting why GDB will not hit my breakpoints? Or is there something major I'm just overlooking?
There's a torrent available with all of the game binaries for those interested in seeing the binary first-hand.
Look here for the answer. In short, it looks like GDB supports child debug mode only on HP-UX and Linux.