I'm trying to generate a callgraph using oprofile and for some reason it fails.
I'm using the below command to config it:
opcontrol --shutdown
opcontrol --reset
opcontrol --no-vmlinux
opcontrol --separate=library
opcontrol --event=default
opcontrol --callgraph=20
opcontrol --status
Here I get:
Daemon not running
Event 0: CPU_CLK_UNHALTED:100000:0:1:1
Separate options: library
vmlinux file: none
Image filter: none
Call-graph depth: 20
Buffer size: 10000000
CPU buffer watershed: 2560000
CPU buffer size: 160000
Then when trying to generate callgraph (for example using opreport pdpd -l --callgraph -o profile_pdp.txt)
I get:
30 0.7659 libpthread-2.5.so pthread_mutex_lock
30 100.000 libpthread-2.5.so pthread_mutex_lock [self]
My linux kernel version is 2.6.18
I do get the following error when running opreport (don't know if relevant):
opreport: /usr/lib64/libstdc++.so.6: no version information available (required by opreport)
Any idea why I can't get the full callgraph?
Found the issue, it was working with a 64bit kernel while debugging 32bit exe, don't know whay it is an issue for oprofile.
Related
My system is V100 with the following information:
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.6 |
NVIDIA Nsight Systems version 2021.5.2.53-28d0e6e
sudo sh -c “echo 2 >/proc/sys/kernel/perf_event_paranoid”
/bin/bash: /proc/sys/kernel/perf_event_paranoid: Read-only file system
Note that perf_event_paranoid is 3.
Output:
Generated:
/home/build/Baseline.nsys-rep
That’s my command prefix:
nsys profile --capture-range=cudaProfilerApi --trace-fork-before-exec true --force-overwrite true -s cpu --cudabacktrace=all --stats=true -t cuda,nvtx,osrt,cudnn,cublas -o Baseline -w true
That's when I check nsys status:
nsys status -e
Timestamp counter supported: No
Sampling Environment Check
Linux Kernel Paranoid Level = -1: OK
Linux Distribution = Ubuntu
Linux Kernel Version = 5.0.0-1032-azure: OK
Linux perf_event_open syscall available: OK
Sampling trigger event available: OK
Intel(c) Last Branch Record support: Not Available
Sampling Environment: OK
That's the output from the Nsight viewer: (No Kernel data)
Profile Output
That's the diagnostics view:
Diagnostics View
I tried CUDA Version 11.0 and that only made Nsight produce profiles with my device driver. Other Cuda versions were not getting me the NSight Profiles.
Please check the following post for more details:
https://forums.developer.nvidia.com/t/nsys-does-not-show-the-kernels-output/229526/17
I am trying to start with BPF CO:RE Development.
Using Ubuntu 20.04 LTS in a VM, I needed to recompile the kernel and install pahole (from apt install dwarves) so that BTF is enabled (I set CONFIG_DEBUG_FS=y and CONFIG_DEBUG_INFO_BTF=y).
So my setup is:
Ubuntu 20.04
Kernel 5.4.0-90-generic
bpftool --version: /usr/lib/linux-tools/5.4.0-90-generic/bpftool v5.4.148
/sys/kernel/btf/vmlinux exists and can be read out with cat.
But bpftool shows the following error:
$ sudo bpftool btf dump file /sys/kernel/btf/vmlinux format c
libbpf: failed to get EHDR from /sys/kernel/btf/vmlinux
Error: failed to load BTF from /sys/kernel/btf/vmlinux: Unknown error -4001
From https://github.com/libbpf/libbpf/blob/master/src/libbpf.h
it looks like it is LIBBPF_ERRNO__FORMAT, /* BPF object format invalid */
but I can not find out what's wrong.
Does anybody know where the mistake might be?
Thanks in advance!
EDIT: Added bpftool version
You need to update bpftool to support a fallback to reading BTF as raw data if the input file is not an object file. The minimum bpftool version required is v5.5 as that's the Linux release where the patch landed. In general, I would recommend to always use the latest bpftool version as there are no backports.
Update:
It looks like bpftool only accepts a ELF-file with the compiled runnning kernel in it, but my /sys/kernel/btf/vmlinux is not:
$ file /sys/kernel/btf/vmlinux
/sys/kernel/btf/vmlinux: data
Same for /boot/vmlinuz:
$ sudo file /boot/vmlinuz-5.4.0-90-generic
/boot/vmlinuz-5.4.0-90-generic: Linux kernel x86 boot executable bzImage, version 5.4.0-90-generic (root#elde-dev) #101+test1 SMP Tue Nov 23 16:38:41 UTC 2021, RO-rootFS, swap_dev 0xD, Normal VGA
Does anybody know why my /sys/kernel/btf/vmlinux does not show the right format?
I found this workaround:
Using this script (https://elixir.bootlin.com/linux/latest/source/scripts/extract-vmlinux) as suggested here (https://unix.stackexchange.com/questions/610672/where-is-the-linux-kernel-elf-file-located) I could get the "working" vmlinux-file which then could be read by bpftool. But this can not really be the right way for BPF CO:RE I guess... Also, in all the tutorials, bpftool is used directly with /sys/kernel/btf/vmlinux.
So why do I get the wrong format?
EDIT: As suggested above, just downoad the newest linux kernel, compile bpftool from there and use that.
I'm having issues when trying to debug a cross compiled binary on my WSL2 host and only end up with backtraces with addresses in ?? (), any hint's on what to verify and change are welcome!
file mybin shows:
ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=..., with debug_info, not stripped
The application is started in WSL2 via qemu-i386(based on output from ps)
NOTE: I was wondering a bit about this because in my prev dev env using vm-ware and ubuntu 18.04 i was not seeing qemu-i386 used but did not think more about it based on WSL2 issues regarding 32bit application support referring to qemu and binfmt solving it.
I'm running gdb-multiarch (GNU gdb (Ubuntu 9.2-0ubuntu1~20.04) 9.2)
Loading the executable and listing symbols with info functions <a_regex> works fine but when attaching and breaking i get bt's like this (NOTE output below is taken from VSCode with a few logging flags enabled, hence the -exec bt thing for example):
-exec bt
1: (777701) <-1183-interpreter-exec console "bt"
1: (777704) ->~"#0 0x000000000047a4ea in ?? ()\n"
1: (777707) ->~"#1 0x00007ffd2dcdb1c0 in ?? ()\n"
1: (777709) ->~"#2 0x0000000000467efc in ?? ()\n"
1: (777711) ->~"#3 0x0000000000000000 in ?? ()\n"
NOTE: When attaching i get the following warning:
warning: Selected architecture i386 is not compatible with reported target architecture i386:x86-64
setting architecture to i386:x86-64 is accepted by gdb but makes no difference
Setting a breakpoint gives the following error:
1: (40020) ->&"Cannot insert breakpoint 1.\n"
1: (40020) ->&"Cannot access memory at address 0xbce346f\n"
1: (40020) ->&"\n"
1: (40023) ->^error,msg="Command aborted."
UPDATE: SOLVED
Thought installing gcc-multilib solved it but it seems more likely the issue was because of a bug in Docker Desktop which has been fixed in v3.2.2. See description in my own answer below.
The qemu-i386 thing was bugging me so I decided to try compiling a simple .c with -m32 flag to check if it also would trigger being run via qemu, got errors because I was missing gcc-multilib so I installed it.
Started the buildt binary and noticed that it did not run via qemu-i386.
Started my original application again and this time it did also not start via qemu.
Started gdb-multiarch, loaded the bin and attached to the process and now suddenly everything worked fine, got a nice proper backtrace!
I'm trying to exploit the binaries from Damn vulnerable Router Firmware but I have issues with debuggging with gdb.
to run the program i use this command :
sudo chroot . ./qemu-mipsel-static ./pwnable/Intro/stack_bof_01
and it works but when i try to run gdb with :
sudo chroot . ./qemu-mipsel-static gdb ./pwnable/Intro/stack_bof_01
I have that :
(gdb) r
Starting program: /pwnable/Intro/stack_bof_01
qemu: Unsupported syscall: 4026
Cannot exec /bin/bash: No such file or directory.
qemu: Unsupported syscall: 4026
Could not open /proc/12532/status
I tried to copy the binary in a qemu VM but I don't have the whole system so it don't work.
So , please , what's is the best way to debug a program from a firmware on a different architecture than x86 ?
In qemu user mode, run the program using the command with the option -g:
sudo chroot . ./qemu-mipsel-static -g 1234 ./pwnable/Intro/stack_bof_01
then start the gdb-multiarch (or gdb that corresponds to that architecture), and attach to it like this:
target remote 127.0.0.1:1234
then you can debug it happily.
Environment: OS X 10.9, XCode 4.6.3,
tweejump git:(master) ✗ apportable --version
Apportable SDK version release_1.0.31 (53ea42fec9b094b91c988f3bfde6dff8ba683a4d starter)
clang version 7fc8b05e4f57f61dbbbe5c8e62581b0e0c42941e
gdb version ff0611b8b721b3bf393c655c7d147de52cc850ac
android sdk version r21.0.1.1
android ndk version r8d.1
unknown ninja
I downloaded tweetjump built it and install this game.
Then I want to check if I can debug with gdb using
apportable just_debug
and
ROOTED=yes apportable just_debug
all these two commands gave me same information;
building with TARGET_ARCH_ABI:armeabi ARM_NEON:False
Building to /Users/xxx/.apportable/SDK/Build/android-armeabi-debug
Loading configuration.
Finished parsing configuration.
scons: Building targets ...
Debugging...
Starting: Intent { cmp=com.iplayful.tweejump/com.apportable.activity.VerdeActivity (has extras) }
Warning: Activity not started, its current task has been brought to the front
Failed to load one the Breakpoints files:
/Users/xxx/workspace/tweejump/tweejump.xcodeproj/xcuserdata/xxx.xcuserdatad/xcdebugger/Breakpoints.xcbkptlist
/Users/xxx/workspace/tweejump/tweejump.xcodeproj/xcuserdata/xxx.xcuserdatad/xcdebugger/Breakpoints_v2.xcbkptlist
Attaching to pid 8085
Cannot attach to lwp 8085: Operation not permitted (1)
Exiting
I saw some run-as answer, but how can an android newbie work it out. Can I have a step by step tutorial.
Edit1:
device: SAMSUNG SCH-I739
Android version: 4.1.2
Edit2:
I searched and found a debug solution:
$ adb shell
$ su
$ cd /data/data/com.iplayful.tweejump/lib/gdbserver :1111 --attach 26337
in my Mac:
$ ~/.apportable/toolchain/macosx/gdb/bin/arm-elf-linux-gdb
(gdb) file ./gdb/app_process
(gdb) shell adb forward tcp:1111 tcp:1111
(gdb) target remote :1111
(gdb) continue
then, gdb attached to gdbserver.
But gdb can't find the symbol, so this is the second question.
If I use this method to debug game, where to find game's symbol and libraries?
It looks like there is a gdbserver running on the device in a bad state.
Try rebooting the device and then apportable just_debug
If there are still issues, add the Android device and Android version to the question.