Extract execution log from gdb record in a VirtualBox VM - gdb

I am attempting to use gdb's record feature to generate a list of the instructions executed for the tutorial example
I can use gdb record to step forward and back successfully and save the execution log to a file using "record save".
I think what I want to do is "record instruction-history" which from docs
Disassembles instructions from the recorded execution log
But when I attempt this i get the error:
You can't do that when your target is 'record-full'
Attempting to set the record target to btrace returns the error:
Target does not support branch tracing.
I am running gdb 7.6 in a VirtualBox VM, do i need to be running natively or is there some other magic i'm missing.

Your problem comes from a problem on VirtualBox itself to perform this operation. As you can see in this link, more specifically in this lines:
if (packet->support != PACKET_ENABLE)
error (_("Target does not support branch tracing."));
This problem is explained here.
But VirtualBox does NOT
emulate certain debugging features of modern x86 CPUs like branch target
store or performance counters.
My best guess is to install some other VirtualBox features that allow you to perform such operations, or switch to a new virtual environment.
I'll keep searching for information.

Related

My OS boots on Qemu but triple faults on VMWare

I built an operating system and it successfully booted with qemu. However when I try to boot in on VMWare, a triple fault occurs. I used a floppy to boot it in both cases. How can I find the error code of the triple fault from VMWare? I tried looking in the virtual machine logs but I couldn't find any more details.
As you know,qemu runs programs by its virtual cpu and full virtual hardware...
but vmware will use VT(virt-tech) to get better performance and all device is running on mix-virtual mode. so it will takes you more difficulty such as running status checking will be restrict ,random config address...or whatever things.they all make your kernel into fault
To find this problem.make lots of breakpoints before where will be problem(such as running level switching,write MSR regs,config ACPI..)and run,if it still running correctly,remove next breakpoint. do debug one by one
could you give me your repo address if possible then i will answer this question better(upload your logs is also can help me)

How to understand and debug from a VirtualBox log file?

I have followed this tutorial for developing an operating system. I am using Windows 10 as my host sytem and used wsl for compiling. But my VM fails as soon as I enable interrupts.
This is the log file of the VM that is output, but I cannot understand it. I am pretty naive with VirtualBox. Can someone explain any possible error you see?
Here is the code of the Os. I just have changed the structure I believe. Rest code in execution point of view is same as shown in video series.
That is a lot of log to scroll through and it's hard to be sure on the face of it that just looking at that would be able to tell us what about your startup code (not visible to us as part of the question) would trigger it. However, I can speak to some general strategies about approaching a log file like this.
We can see some general state transitions in there. The log ends with:
00:00:15.712045 Changing the VM state from 'DESTROYING' to 'TERMINATED'
So I can go back through and look at where the first instance of DESTROYING showed up, which was:
00:00:15.698320 Changing the VM state from 'POWERING_OFF' to 'OFF'
00:00:15.701802 Changing the VM state from 'OFF' to 'DESTROYING'
Following the same process backwards to POWERING_OFF, I see:
00:00:08.577363 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
00:00:14.342287 ERROR [COM]: aRC=VBOX_E_INVALID_VM_STATE (0x80bb0002) aIID={872da645-4a9b-1727-bee2-5585105b9eed} aComponent={ConsoleWrap} aText={Invalid machine state GuruMeditation when checking if the guest entered the ACPI mode)}, preserve=false aResultDetail=0
00:00:15.643579 GUI: Request for close-action to power VM off.
00:00:15.643599 GUI: Passing request to power VM off from machine-logic to UI session.
00:00:15.643606 GUI: Powering VM down on UI session power off request...
00:00:15.644257 Console: Machine state changed to 'Stopping'
00:00:15.644763 Console::powerDown(): A request to power off the VM has been issued (mMachineState=Stopping, InUninit=0)
00:00:15.645075 Changing the VM state from 'GURU_MEDITATION' to 'POWERING_OFF'
That error line at the top of that block may point to something searchable that would turn up other instances of people having the same or a similar problem. If you scroll up a bit, you can also see that something VGA-related was happening right before the error, which may help narrow it down if it's directly related to the error, or may be another step to backtrack through on the way to the real issue.

How can you trace execution of an embedded system emulted in QEMU?

I've built OpenWrt for x86 and I'm using QEMU to run it virtually.I'm trying to debug this system in real time. I need to see things like network traffic flowing etc.
I can attach gdb remotely and execute (mostly) step by step with break points. I really want trace points though. I don't want to pause execution and loose network flow. When I tried setting trace points using tstart, I see the message "Target does not support this command". I did a bit of reading of the gdb documentation and from what I can tell the gdb stub that runs to intercept normal execution in QEMU does not support trace points.
From here I started looking at other tools and ran across PANDA (https://github.com/panda-re/panda). As I understand PANDA will capture a complete system trace in a log and allow for replay. I think this tool is supposed to do what I need, but I cannot seem to replay the results. I see the logs, I just can't replay them.
Now, I'm a bit stuck on what other tools/options I might have to actually trace a running embedded system. Are there any good tools you can recommend? Or perhaps another method I've missed?
If you want to see the system calls and signals use strace.
Strace can also be used with running process and it can put the output in a log file if required.
In OpenWrt it is possible to build with ftrace. Ftrace has much of the functionality I required but not all.
To build with ftrace, the option for ftrace must be selected in the build menu. Additionally there are a variety of tracer options that must also be enabled.
The trace-cmd (ftrace) is located in menuconfig/Development
Tracing support is under menuconfig/Global build settings/Compile the kernel with tracing support and includes: Trace system calls, Trace process context switches and events, and Function tracer (Function graph tracer, Enable/disable function tracing dynamically, and Function profiler)
I'm also planning to build a custom GDB stub to do this a little bit better as I also want to see the data passed to the functions not just the function calls.

Distributed software debug with gdb

I am currently developing a distributed software in C++ using linux which is executed in more than 20 nodes simultaneously. So one of the most challenging issue that I found is how to debug it.
I heard that is possible to manage in a single gdb session multiple remote sessions (e.g. in the master node I create the gdb session and in every other node I launch the program using gdbserver), is it possible? If so can you give an example? Do you know any other way to do it?
Thanks
You can try to do it like this:
First start nodes with gdbserver on remote hosts. It is even possible to start it without a program to debug, if you start it with --multi flag. When server is in multi mode, you can control it from your local session, I mean that you can make it start a program you want to debug.
Then, start multiple inferiors in your gdb session
gdb> add-inferior -copies <number of servers>
switch them to a remote target and connect them to remote servers
gdb> inferior 1
gdb> target extended-remote host:port // use extended to switch gdbserver to multi mode
// start a program if gdbserver was started in multi mode
gdb> inferior 2
...
Now you have them all attached to one gdb session. The problem is that, AFAIK, it is not much better than to start multiple gdb's from different console tabs. On the other hand you can write some scripts or auto tests this way. See the gdb tutorial: server and inferiors.
I don't believe there is one, simple, answer to debugging "many remote applications". Yes, you can attach to a process on another machine, and step through it in GDB. But it's quite awkward to debug a large number of interdependent processes, especially when the problem is complicated.
I believe a good set of logging capabilities in the code, supplemented with additional logs for specific debugging as needed, is more likely to give you a good/fast result.
Another option might be to run the processes on one machine, rather than on multiple machines. Perhaps even use threads within one process, to simulate the behaviour of multiple machines, simplifying the debugging process. Of course, this doesn't prevent bugs that appear ONLY when you run 20 processes on 20 different machines. But the basic idea is to reduce the number of those bugs to a minimum, and debug most things in a "simpler environment".
Aggressive use of defensive programming paradigms, such as liberal use of assert is clearly a good idea (perhaps with a macro to turn it off for the production runs, but make sure that you don't just leave error paths completely unchecked - it is MUCH harder to detect that the reason something crashes is that a memory allocation failed than to track down where that NULL pointer came from some 20 function calls away from a failed allocation.

oprofile on Linux running in a virtual machine

I'm running a Linux Ubuntu 10.4 VM using VirtualBox. I'm trying to use oprofile to profile some application in the virtual machine. I've installed oprofile 0.9.6 but I cannot get it to work. When I try to start I get the following error:
opcontrol --start
/usr/local/bin/opcontrol: line 323: /usr/local/bin/ophelp: cannot execute binary file
/usr/local/bin/opcontrol: line 1483: /usr/local/bin/oprofiled: cannot execute binary file
Couldn't start oprofiled.
Check the log file "/var/lib/oprofile/samples/oprofiled.log" and kernel syslog
As I'm not sure if VirtualBox could provide access to the performance counters (I'm in doubt here so if you have any pointers it would be great) I defaulted oprofile to the timer interrupt like so:
opcontrol --deinit
/usr/local/bin/opcontrol: line 323: /usr/local/bin/ophelp: cannot execute binary file
Unloading oprofile module
root#dev-ubuntu-10:/usr/local/bin# /sbin/modprobe oprofile timer=1
root#dev-ubuntu-10:/usr/local/bin# opcontrol --init
But still not working and I'm getting the same error. Is it even possible to run oprofile in a VM?
Thanks
I've tried something similar in the past, only with VMware Fusion and a different profiler, and run into the same problem. It seems that access to the performance registers and other low level stuff that profilers need is just not feasible in a VM. You'll need a real machine for profiling, I'm afraid.
This error:
/usr/local/bin/ophelp: cannot execute binary file
usually means that you are attempting to execute an x86_64 binary on a 32-bit kernel.
What do file usr/local/bin/ophelp and uname -a print?
A couple of years ago I had some problem running oprofile inside vmware. I wrote my little experience on this post http://blogs.epfl.ch/category/3239
You could try installing older versions like oprofile-0.9.7
extract it anywhere then follow steps:
install it by > 1 ./configure 2. make 3. make install
Then try using it it works fine you might want to turn on virtual CPU counters in VMWARE and disable nmi_watchdog registers in linux as they might be used by other profilers.
use of HPC(hardware performance counters) requires hardware supprot, try to install cpuid in vbox, you will see
Architecture Performance Monitoring Features (0xa/ebx):
core cycle event not available = false
instruction retired event not available = false
reference cycles event not available = false
last-level cache ref event not available = false
last-level cache miss event not avail = false
branch inst retired event not available = false
branch mispred retired event not avail = false
Architecture Performance Monitoring Features (0xa/edx):
number of fixed counters = 0x0 (0)
bit width of fixed counters = 0x0 (0)
It seems that just Vmware and KVM can emulate PMU unit, and not the VBOX