I am looking for a way to do things such as attach to a process, set breakpoints, view memory, and other things that gdb/lldb can do. I cannot, however, find a way to do these things.
This question is similar to this one, but for MacOS instead of Windows. Any help is appreciated!
Note: I want to make a debugger, not use one.
Another thing is that i dont want this debugger to be super complicated, all i need is just reading/writing memory, breakpoint handling, and viewing the GPR
If you really want to make your own debugger, another way to start would be to figure out how to cons up and parse the gdb-remote protocol packets (e.g. https://sourceware.org/gdb/onlinedocs/gdb/Remote-Protocol.html). That's the protocol gdb uses when remote debugging and lldb uses for everything but Windows debugging. On MacOS, lldb spawns a debugserver instance which does the actual debugging and controls it with gdb-remote protocol packets. On Linux, it uses the lldb-server tool that's part of the Linux lldb distribution for the same purpose.
The gdb-remote protocol has primitives for most of the operations you want to perform, launch a process, attach to a process, set breakpoints, read memory & registers and isolates you from a lot of the low-level details of controlling processes.
You can help yourself out by observing how lldb uses this protocol by running an lldb debug session with:
(lldb) log enable gdb-remote packets
But you might also have a look at the SB API's in lldb. The documentation is not as advanced as it should be but there are a bunch of examples in the examples/python directory of the lldb sources to get you started, and in general the API's are pretty straightforward and self-explanatory.
LLDB has an API that can be consumed from C++ or Python. Maybe this is what you’re looking for.
Unfortunately the documentation is fairly threadbare, and there don’t seem to be usage examples. This will therefore entail some reading of the source and a lot of trial and error.
If you want to write your own debugger, you'll need to obtain a task port to the process (task_for_pid), then you can read/write/iterate to virtual memory (mach_vm_read, mach_vm_write, mach_vm_region). To do breakpoints, you need to first set up an exception handler then you can manipulate the debug registers on threads (task_threads, thread_get_state, thread_set_state) and handle them in the exception handler.
Reference to some not all that correct debugger code I've written because breakpoints (especially instruction ones) are a bit involved.
MacDBG may be another lightweight reference but I haven't used it myself.
Well, if you want to write a debugger, take a look at the gdb/lldb source code. I'd suggest the latter, due to historical legacy in gdb that might cloud whatever is actually going on.
Use a debugger. Such as gdb or lldb for example. They have plenty of documentation to teach you the how bit, but for example gdb -p <pid of process> will attach gdb to a running process.
If you want to drive gdb for example from a C++ program, then launch it in a separate process (see fork and exec) with the aguments it needs (probably including the one to enable its machine parsable interface). Make sure you set up pipes to its stdin/stdout so you can read its output and send it commands.
If instead you want to write your own debugger from scratch then that is a huge undertaking. I suggest starting by reading the source of an existing open source debugger.
Whilst you could look at source code of another debugger, you may find it difficult to understand without the knowledge of the underlying concepts. Therefore, I recommend you start by obtaining a good grounding of the concepts with the following resources:
Mac OS X Sys Internals
Rather outdated now, but the original bible for the internals of Mac
Mac OS X and iOS Internals
Again, outdated but useful. This link is Jonathan Levin's (the author's) own site and he's now providing it for free, due to issues he had with the publisher. He's since purchased back the rights, making it available to all.
*OS Internals
The current bible of Mac Internals, also by Jonathan Levin. Books III and I have been published, with book II to follow shortly!
Related
I've built OpenWrt for x86 and I'm using QEMU to run it virtually.I'm trying to debug this system in real time. I need to see things like network traffic flowing etc.
I can attach gdb remotely and execute (mostly) step by step with break points. I really want trace points though. I don't want to pause execution and loose network flow. When I tried setting trace points using tstart, I see the message "Target does not support this command". I did a bit of reading of the gdb documentation and from what I can tell the gdb stub that runs to intercept normal execution in QEMU does not support trace points.
From here I started looking at other tools and ran across PANDA (https://github.com/panda-re/panda). As I understand PANDA will capture a complete system trace in a log and allow for replay. I think this tool is supposed to do what I need, but I cannot seem to replay the results. I see the logs, I just can't replay them.
Now, I'm a bit stuck on what other tools/options I might have to actually trace a running embedded system. Are there any good tools you can recommend? Or perhaps another method I've missed?
If you want to see the system calls and signals use strace.
Strace can also be used with running process and it can put the output in a log file if required.
In OpenWrt it is possible to build with ftrace. Ftrace has much of the functionality I required but not all.
To build with ftrace, the option for ftrace must be selected in the build menu. Additionally there are a variety of tracer options that must also be enabled.
The trace-cmd (ftrace) is located in menuconfig/Development
Tracing support is under menuconfig/Global build settings/Compile the kernel with tracing support and includes: Trace system calls, Trace process context switches and events, and Function tracer (Function graph tracer, Enable/disable function tracing dynamically, and Function profiler)
I'm also planning to build a custom GDB stub to do this a little bit better as I also want to see the data passed to the functions not just the function calls.
I am working on a big code base. It is heavily multithreaded.
After running the linux based application for a few hours, in the end, right before reporting, the application silences. It doesn't die, it doesn't crash, it just waits there. Joins, mutexes, condition variables ... any of these can be the culprit.
If it had crashed, I would at least have a chance to find the source using debugger. But this way, I have no clue how to use what tool to find the bug. I can't even post a code sample for you. The only thing that can possibly help is to tap MANY places with cout to get a visual where the application is.
Have you been in such a situation? What do you recommend?
If you're running under Linux then just use gdb to run the program. When the application 'silences', interrupt it with CTRL+C, then type backtrace to see the call stack. With this you will find out the function where your application was blocked.
Incase of linux, gdb will be great help. Another tool that can be of great help is strace (This can also be used where there are problems with program for with source is not readily available because strace does not need recompilation to trace them.)
strace shall intercept/record system calls that are called by a process and also the signals that are received by a process. It will be able to show the order of events and all the return/resumption paths of calls. This can take you almost closer to the area of problem.
iotop, LTTng and Ftrace are few of other tools that be helpful to you in this scenario.
My application is deployed at customer sites, that I can not access, and has no internet connection.
There are complains that in several sites, once in a week or so, the application become unresponsive, so that the operators need to kill and restart it.
We were unable to observe it in our site.
Is there something I can do that may help me find the problem?
It is a VC2008 Win32 MFC applications.
The application is quite complex, and includes many threads, synchronization mechanisms, database access, HMI, communication channels...
Note: The custmer can send us log files.
Note: The application does not crash. It just hangs. Since I don't know what is the nature of the problem, I have no way to know programmatically that something went wrong (or do I?)
I have had great success with ADplus and WinDBG in the past. You may check it out. Especially check out the Hang mode in ADplus.
I would start with some questions - is the CPU hogged during these unresponsive times? Is there a specific process that's hogging it? (You can use PerfMon to get the answers). Depending on the answers I would probably proceed by taking a dump of the process at this stage (ProcDump by sysinternals is great for these purposes) and investigate it offline.
In similar situation on a non-windows platform we have the capability to gather system dumps. Get a thread dump of the entire system for off-site analysis. This enables us to find deadlocks quite easily. For slow problems rather than stop a single dump is not enough. Then we need a sequence of dumps, and some good luck.
Another, rather messier technique is to have enough trace, and enough fine-grained control of trace in the app. Then turn on some trace and hope to spot where the delays are happening.
My experience with finding bugs in installations on the other side of the planet shows three helpful techniques: Logging, logging, and logging.
What do those log files say your customers sent you? If they aren't detailed enough, send them a version that logs more. Use binary approximation to home in on the error.
To know where the process is hung is better to start with the stack trace at that instant.
Now since your program is installed remotely and you can't access it, you can write a monitoring program which can periodically check the stack of your program and log it. This information along with your logging mechanism will make things easier to identify and debug.
Since I am not a windows programmer, i don't know much about such tools availability in windows, however i think you need something similar to this http://www.codeproject.com/KB/threads/StackWalker.aspx
I am using Xcode in order to debug C++ programs. The main problem for me is that it takes around 10 munutes till the program gets to the point of the program that I need to debug. Then I realize about something inspecting the variables and some other stuff, and modify the code. Then 15 minutes again and so ...
I wonder if there is possible in some way in Xcode or in another IDE or compiler/debugger for C++, to "save" in some way a desired debugging state of the program. So if my compouter crashes or I modify the code and make some mistakes, one can open this saved state instantly and get fast to the point where one left before.
I also wonder if at this moment Xcode can "backwards debugging". GDB can for sure, as for september 2009. Or what do you think is the best IDE to do this.
Thanks a lot
GDB has "backwards debugging" (or more correctly "Reverse Debugging") for a limited number of platforms (list of native supported ones):
i386-linux
amd64-linux
moxie-elf ( http://moxielogic.org/blog/ )
So it is impossible for now to use this functionality on Mac OS X, with Xcode or without it.
Saving of program state in offline is very hard task. It is almost impossible to restore state of file descriptors, network connections, memory state (randomization of layout), even pid.
Such task is related to "Live migration" problem in openvz.
"Edit and Continue" feature from MSVS allow you to continue running after breakpoint with new version of code. It is supported for C#, C++ and Basic.
http://msdn.microsoft.com/en-us/library/esaeyddf(VS.80).aspx
we have a problem with an application we're developing. Very seldom, like once in a hundred, the application crashes at start up. When the crash happens it brings down the whole system, the computer starts to beep and freezes up completely, the only way to recover is to turn off the power (we're using Windows XP). The rarity of the crash combined with the fact that we can't break into the debugger or even generate a stackdump when it occurs makes it extremely hard to debug.
I'm looking for something that logs all function calls to a file. Does such a tool exist? It shouldn't be impossible to implement, profilers like VTune does something very similar.
We're using visual studio 2008 (C++).
Thanks
A.B.
Logging function entries/exits is a low-level approach to your problem. I would suggest using automatic debugger instrumentation (using Debugger key under Image File Execution Options with regedit or using gflags from the package I provide a link to below) and trying to repro the problem until it crashes. Additionally, you can have the debugger log function call history of suspected module(s) using a script or have collect any other information.
But not knowing the details of your application it is very hard to suggest a solution. Is it a user app, service or a driver? What does "crashes at startup" mean - at windows startup or app's startup?
Use this debugger package to troubleshoot.
The only problem with the logging idea is that when the system crashes, the latest log entries might still be in the cache and have no chance to be written to disk...
If it was me I would try running the program on a different PC - it might be flaky hardware or drivers causing the problem. An application program "shouldn't" be able to bring down the system.
A few Ideas-
There is a good chance that just prior to your crash there is some sort of exception in the application. if you set you handler for all unhandled exceptions using SetUnhandledExceptionFilter() and write a stack trace to your log file, you might have a chance to catch the crash in action.
Just remember to flush the file after every write.
Another option is to use a tool such as strace which logs all of the system calls into the kernel (there are multiple flavors and implementations for that so pick your favorite). if you look at the log just before the crash you might find the culprit
Have you considered using a second machine as a remote debugger (via the network)? When the application (and system) crashes, the second machine should still show some useful information, if not the actual point of the problem. I believe VC++ has that ability, at least in some versions.
For Visual C++ _penter() and _pexit() can be used to instrument your code.
See also Method Call Interception in C++.
GCC (including the version MingGW for Windows development) has a code generation switch called -finstrument-functions that tells the compiler to emit special calls to functions called __cyg_profile_func_enter and __cyg_profile_func_exit around every function call. For Visual C++, there are similar options called /GH and /Gh. These cause the compiler to emit calls to __penter and __pexit around function calls.
These instrumentation modes can be used to implement a logging system, with you implementing the calls that the compiler generates to output to your local filesystem or to another computer on your network.
If possible, I'd also try running your system using valgrind or a similar checking tool. This might catch your problem before it gets out-of-hand.