I am using GDB in remote configuration. So I have gdbserver running on a ARM HW running linux, I connect to remote gdbserver from local gdb, I am able to put breakpoints in initial part of code and ensure that basically GDB works. However when I let my SW run for a while and break to see threads, I don't get useful stack traces for threads. All stack frames are hex addressed none resolving to symbols present in my binaries. Most of them also look the same with gdb also hinting with "same stack frames, corrupted stack ??"
Since SW runs fine with or without debugger I don't doubt my SW. Anybody seen this issue or any idea what might be going on here.
Thanks
Related
My project is quite large and multithreaded. There should be a bug which crashes the whole program.
For release version, it stuck sometimes, but does not appear very often.
For debug code, it is more likely to appear. And the stack trace of gdb is the following.
0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:81
1 0x00007dff8270c700 in ?? ()
2 0x00007ffff6dde38d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
This information is not enough for me to locate the buggy code.
So my question is: how to get more information from the crash? any advanced use of gdb or other advanced tools?
============= Update ==============
One more information to add, after printing out all the thread ids, I figure out the thread that crashed. The only difference of the thread is that it is detached from the std thread object. If anyone has any experience with this, please tell me.
============= Update2 ================
This problem is not solved yet, and turn out to be a sever one.
If I run in the terminal, it'll crash the whole terminal and all other programs currently running under my username.
The system is then down and not accessible by ssh for a while. There are some other users getting broken pipe and it seems my program has made sshd not responsive.
After a while I'm able to login again, and find that the binary file of the program is broken (truncated) and need to recompile.
For me it looks like a memory or stack overwrite or access of dead pointers or objects.
To catch these kind of errors I like to use tools like efence or valgrind. With actual compilers you also can use thread sanitizer or the memory sanitizer. Both works with clang and g++.
If you can not catch the problem with that, you also should install the debug library version of the standard libs. Sometimes a wrong value crashes inside the g++lib or some other libs which results in hard to debug situations. With the debug infos installed you can catch this much easier.
My understanding is that gdb can monitor the complete state of a running program. Can I save a gdb session that is paused at a breakpoint and resume the session later?
My first attempt was simply generating a core dump in a first gdb session that was paused at a breakpoint and then using the core dump to start a second gdb session.
Saving core file in gdb
This resulted in the following error.
Program terminated with signal SIGTRAP, Trace/breakpoint trap.
So breakpoint information is inserted into the program state, interesting. On my second attempt I did the same but this time I added the same breakpoint to the second session as were in the first session.
Getting gdb to save a list of breakpoints?
Still, I get the same error.
Can I save and restart a gdb session? If so, how?
I don't think this is directly relevant but I'm also getting this warning.
warning: core file may not match specified executable file.
Is gdb simply stating that such a thing is possible in general or does gdb believe this may have happened in the running session? I'm confident that the same executable that produced the core dump is being run under gdb.
Edit: For anyone else who comes along, this question: Save a process' memory for later use? adds to Mats Petersson's answer and links to this article: http://blogs.msdn.com/b/oldnewthing/archive/2004/04/20/116749.aspx which is an interesting read. The linked question also has the suggestion of wrapping the process up in a VM.
I doubt that will ever work. Handles for files and any other resources (semaphores, shared memory, serial ports, network connections and lots of other things) that the program has opened/created will be lost when you save the core-file. You can inspect it, but you can't "continue". A core-file is simply a copy of all the memory that original program was using. Anything else is "lost" when the program terminates. In other words, a core-file will only be useful to inspect things later on, but you can't run, step or continue in a core-file debug session. Only "look at things". And if you can't execute, breakpoints won't really work either... ;)
I am using GDB along with a JTAG device, an Abatron BDI2000, to debug a programs running on a Motorola M68332.
The 68332 does not have any hardware breakpoint registers. It has very primitive debugging features.
The build tools do not generate 'elf' files, so no symbols for GDB to use.
Also the program I'm debugging is running in Flash.
In fact the 68332 has only one debug instruction, ti. ti by itself steps to the next assembly instruction. ti xxx steps until the address xxx is reached. [Yes, this is caveman days, cold hammer and chisel :)]
I am able to use GDB with target remote to connect to the BDI2000 and issue the GDB commands 'nexti'. Due to the limitations of the 68332, 'stepi' is equivalent to 'nexti'.
Single stepping is only command available.
The monitor command 'monitor ti ' states change the program counter to and step.
If one uses a 'monitor' command that changes the registers, then GDB does not know about the command and its register cache become out of sync. I have created GDB functions which have the GDB command 'flushregs' at the end of each of them. This marks the register cache dirty. The GDB command will fetch a new set of registers.
I would like to create a symbol table file for debugging, but have not found any documentation on the GDB symbol file format.
Are there alternatives to what I have setup?
I do have a RAM overlay for the Flash area. Would this allow software breakpoints?
Thanks in advance for any advice.
I found I can use 'convenience' variables as a substitute for symbols, since I'm not using ever symbol in the program all at once.
set $Symbol=(unsigned int*)<address>
Each 'Symbol' is declared a pointer to an unsigned int at an address. One can put these statements in .gdbinit, and add to them over time.
One can then state
break $Symbol
I show a GDB command function that can be passed one of these 'convenience' variables in the question linked below.
How do I write a GDB function to make a comparison to the program counter
I am trying to extract the execution sequence of my program (something like a program counter) with gdb on my local computer (windows x86) and gdbserver on a remote target (arm-linux). The idea I had was to insert breakpoints at "important" lines of my source files (i.e.: at the beginning of a specific function, and more in general before and after a conditional statement) with a high ignore count for each breakpoint, and then check if a breakpoint was hit or not. I was actually able to receive the informations with this method, but there is a problem: the application behavior I am debugging depends on real-time, and this specific method slows down the program execution too much. Do you think I could use some other method with gdb? I stumbled upon tracepoints, wich seems the exact thing I am looking for, but I was not able to find some property like a "hit counter" for them. The gdb version I am currently using is 7.5.
Thanks a lot in advance.
If your program execution must not be slowed down, you will probably need some HW tool. See these:
Keil real time trace
Lauterbach PowerDebug
(probably other similar solutions)
To learn a bit more about FreeBSD and *nix systems in general, I'm starting to look at the binaries from the DEFCON 17 Capture The Flag game. Right now, I'm reversing the tucod binary. Here's some possibly useful information on tucod:
tucod: ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), for FreeBSD 7.2, dynamically linked (uses shared libs), FreeBSD-style, stripped
Some other possibly useful information gained from some brief static analysis is that tucod binds on port 0xDEAD (cute, eh?) and if you give it a specific password ("HANGEMHIGH!") it will play a game of hang-man with you.
The problem that I'm encountering is that I'm not hitting my breakpoints in gdb. Specifically, the breakpoint that I'm trying to reach is in the code that handles the client connection. Without breakpoints, the code executes as expected. When I set a breakpoint on that code, the child exits (instead of breaking into gdb, as expected). If I set breakpoints before the server forks off the child, I can hit those fine but after hitting "continue" the child does not continue to process my connection (that is, it won't ask me for a password or play hang-man).
Since the daemon forks when it receives a new connection, I try to tell gdb to follow the child with this command:
(gdb) set follow-fork-mode child
But after single-stepping the instructions after the fork, it appears that this isn't working.
I've tried looking for calls to signal, thinking they implemented a custom SIGINT handler (or similar), but the only call to signal that I can see handles SIGCHLD.
My breakpoint in gdb currently looks like this:
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y 0x080497d0
And 0x080497d0 is the address I want to break on in the client processing code.
I'm sort of new to analyzing software on *nix systems and could use some pointers. How else should I go about troubleshooting why GDB will not hit my breakpoints? Or is there something major I'm just overlooking?
There's a torrent available with all of the game binaries for those interested in seeing the binary first-hand.
Look here for the answer. In short, it looks like GDB supports child debug mode only on HP-UX and Linux.