Visual Studio 2017 Linux remote debugging (gdbserver) - c++

Question

After I installed gdbserver on my remote machine (Ubuntu 16.04.4 LTS), I tested the following c++ code by making a "cross-platform console application (linux)" project in Visual Studio 2017:
#include <cstdio>
int main()
{
printf("hello from testLinuxDebug!\n");
return 0;
}
I added the connection information (ip address, id, password) of my Ubuntu machine to connection manager and selected "gdbserver" for debugging mode.
Then I started debugging and got the following message:
Unable to start debugging. Unexpected GDB output from command
"-interpreter-exec console "target remote localhost63361"". Remote
connection closed.
Message after starting to debugging (Image captured from VS2017)
By the way, I got the following message from the Linux Console Window of debug menu of Visual Studio 2017:
Process /home/.../projects/testLinuxDebug/bin/x64/Debug/testLinuxDebug.out
created; pid = 29277 Listening on port 4444 Remote debugging from host
127.0.0.1 /build/gdb-9un5Xp/gdb-7.11.1/gdb/gdbserver/regcache.c:264: A problem > internal to GDBserver has been detected. Unknown register ymm0h requested
Message from Linux Console Window (Image captured from VS2017)
Could anybody help me with this problem?

Answer 1

I was having this exact same problem on my Ubuntu 16.04 machine.
I went through the source code on gdbserver, and it appears to be a problem with a processor register (ymm0h) which is only available to i386 processors.
The thing is, I don't know how to fix gdbserver to not use this register in particular, but you can solve the error by upgrading your gdb & gdbserver to version 8.3 on the Ubuntu machine. If my guess is correct, version 8.1 was the one that fixed this issue, but by default, Ubuntu 16.04 has version 7.11.1 for both gdb & gdbserver.
You can run these commands on the linux machine to do that:
wget "http://ftp.gnu.org/gnu/gdb/gdb-8.3.tar.gz"
tar xzf gdb-8.3.tar.gz
cd gdb-8.3
./configure --prefix=/usr --with-system-readline
make
sudo make install
You may need to install some extra packages in order to compile the gdb:
sudo apt-get install libreadline6-dev texinfo
If you still have problems trying to install gdb, try this example from Linux from Scratch.
Good luck!

Answer 2

I encountered the same problem as you, although for me the issue was on Microsoft's Visual Studio Online platform - I couldn't use pwntools' gdb.debug/gdb.attach functionality (that relies on gdbserver) due to this very same error in VSOnline's docker environment.
Leaving this answer here for people who might encounter the same issues that I have.
The issue and the steps are pretty similar to what Canella described - the root of the problem is indeed the old GDB version in the environment (7.12 at the time of writing).
I've compiled my GDB with python3 bindings, and you might want to do that too if you're planning to use GEF or similar GDB extensions.
Here's how to do that in your VSOnline environment's terminal:
# I needed to get rid of the apt-provided gdb for it to work properly
sudo apt remove gdb
# VSOnline has a weird way of handling python versions
# you're better off installing python3 from apt, if it's not installed yet
sudo apt update
sudo apt install libreadline6-dev texinfo python3 python3-dev
wget "http://ftp.gnu.org/gnu/gdb/gdb-8.3.tar.gz"
tar xzf gdb-8.3.tar.gz
cd gdb-8.3
mkdir build && cd build
../configure --prefix=/usr --with-system-readline --with-python=/usr/bin/python3
make
sudo make install