Unable to login GCP Virtual Machine (Reason: failed to connect to backend) - google-cloud-platform

My GPC Linux VM is working fine and one day naturally can't log in, even though it is still running,
I used the nomachine to login and got error "authentication failed, please try again."
when SSH:
Connection via Cloud Identity-Aware Proxy Failed
Code: 4003
Reason: failed to connect to backend
You may be able to connect without using the Cloud Identity-Aware Proxy.
I also try add:
#! /bin/bash
useradd -G sudo USERNAME
echo 'USERNAME:PASSWORD' | chpasswd
and
route add default gw <gateway ip> eth0
I created another virtual machine and its image, but after a day I couldn't connect

Ensure you have a firewall rule to allow Cloud Identity-Aware Proxy (IAP) to connect to port 22 on the instance. Full instructions can be found at [1].
[1] https://cloud.google.com/iap/docs/using-tcp-forwarding?authuser=0#create-firewall-rule
Cause:
When an instance does not have a public IP address, SSH in a Browser needs to forward the SSH connection through IAP. The error "failed to connect to backend" indicates that the IAP proxy service was unable to open a TCP connection to the instance.
This is most often due to the VPC firewall not having a rule which allows the proxy to connect to the instance. It coulld also be due to the OS firewall or other VM network connectivity issue.

Related

Google Compute Engine Unable to Access VM

I am unable to ssh into a VM on GCP Compute Engine
However, when I run the command with the --troubleshoot flag, it seems like everything is okay.
When I connect through the console, I get an error message saying "You cannot connect to the VM instance because of an unexpected error".
Also, other people from my organization are able to connect.
I am unable to figure out what the error is. Any help would be appreciated.
From the error message that you got "Permission denied (publickey)" you can check this documentation for further troubleshooting.
Further more you can investigate also the Identity-Aware Proxy (IAP).
If you use Identity-Aware Proxy (IAP) for TCP forwarding, update your custom firewall rule to accept traffic from IAP, then check your IAM permissions.
Update your custom firewall rule to allow traffic from 35.235.240.0/20, the IP address range that IAP uses for TCP forwarding. For more information, see Create a firewall rule.
Grant permissions to use IAP TCP forwarding, if you haven't already done so.
For the error message "You cannot connect to the VM instance because of an unexpected error".
The VM is booting up and sshd is not running yet. You can't connect to a VM before it is running.
To resolve this issue, wait until the VM has finished booting and try to connect again.
The firewall rule allowing SSH is missing or misconfigured. By default, Compute Engine VMs allow SSH access on port 22. If the default-allow-ssh rule is missing or misconfigured, you won't be able to connect to VMs.
To resolve this issue, Check your firewall rules and re-add or reconfigure default-allow-ssh.
sshd is running on a custom port. If you configured sshd to run on a port other than port 22, you won't be able to connect to your VM.
To resolve this issue, create a custom firewall rule allowing tcp traffic on the port that your sshd is running on using the following command:
gcloud compute firewall-rules create FIREWALL_NAME \
--allow tcp:PORT_NUMBER
For further troubleshooting on SSH you see this documentation on Common SSH errors.

Unable to login to a GCP VM machine: kex_exchange_identification: Connection closed by remote host

I am trying to login to a VM machine but unable to get in there as the below error pops up:
ERROR: (gcloud.beta.compute.start-iap-tunnel) Error while connecting [[Errno 1] Operation not permitted].
kex_exchange_identification: Connection closed by remote host
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Command used to log in:
gcloud beta compute ssh --zone "us-east1-b" "user1#test" --project "my-test-project"
When I use the same command from a different machine then it allows me to login to the test VM.
I have been looking at possible solutions related to the firewall settings but unable to get success. Can anyone help here ?
I try to connect with an instance with the command that you post and worked for me when I use it from the cloud shell and from another instances. The error that is given to you could happen for different reason, which are:
The VM is booting up and sshd is not running yet. You can't connect to a VM before it is running.
To resolve this issue, wait until the VM has finished booting and try to connect again.
The firewall rule allowing SSH is missing or misconfigured. By default, Compute Engine VMs allow SSH access on port 22. If the default-allow-ssh rule is missing or misconfigured, you won't be able to connect to VMs.
To resolve this issue, Check your firewall rules and re-add or reconfigure default-allow-ssh.
sshd is running on a custom port. If you configured sshd to run on a port other than port 22, you won't be able to connect to your VM. To resolve this issue, create a custom firewall rule allowing tcp traffic on the port that your sshd is running on using the following command:
gcloud compute firewall-rules create FIREWALL_NAME --allow tcp:PORT_NUMBER
Your custom SSH firewall rule doesn't allow traffic from Google services. SSH connections from the Cloud Console are refused if custom firewall rules do not allow connections from Google's IP address range.
To solve this issue visit this link
The sshd daemon isn't running or isn't configured properly. The sshd daemon enables SSH connections. If it's misconfigured or not running, you can't connect to your VM. To resolve this issue, review the user guide for your operating system to ensure that your sshd_config is set up correctly.

Code: 4010 - Connection via Cloud Identity-Aware Proxy Failed

I have seen a similar error on stackoverflow, but with a different code (so maybe it is not the same?). Any how I have been thrown this error spontaneously. Sometimes after 20 seconds of starting my instance (and launching SSH in browser), sometimes after 30 minutes, but it completely shuts down my instance.
Connection via Cloud Identity-Aware Proxy Failed
Code: 4010
Reason: destination read failed
You may be able to connect without using the Cloud Identity-Aware Proxy.
If I click the "Cloud Identity-Aware Proxy" button I am getting:
Connection Failed
We are unable to connect to the VM on port 22. Learn more about possible causes of this issue.
Any idea what is happening? I havn't done any changes in my instance settings for a long time.
The issue regarding the Identity-Aware Proxy(IAP) connection to the instance is due to the lack of a firewall rule allow-ingress-from-iap with this IP ranges 35.235.240.0/20 that needs to be configured when using IAP.
To Allow SSH access to all VM instances in your network, do the following:
1- Open the Firewall Rules page (Navigation menu > VPC network > Firewall) and click Create firewall rule
2- Configure the following settings:
Name: allow-ingress-from-iap
Direction of traffic: Ingress
Target: All instances in the network
Source filter: IP ranges
Source IP ranges: 35.235.240.0/20
Protocols and ports: Select TCP and enter 22 to allow SSH
3- Click Create

Cannot Start mlflow ui on google cloud platform virtual machine instance

after running mlflow ui on command line
and clicking http://127.0.0.1:5000/
i get site cannot be reached
127.0.0.1 refused to connect.
I have already updated firewall rules on VPC network in GCP and on my local machine and activated the ports
This site can’t be reached127.0.0.1 refused to connect.
Try:
Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED
You can connect to your MlFlow UI by using the IP address of your Google Cloud VM instead of 127.0.0.1, which is equal to localhost, meaning it is only available inside the Google Cloud VM.

Cannot Connect by Cloud SQL Proxy from Cloud Shell By Proxy

I am following the Django sample for GAE and have problem to connect to Cloud SQL instance by Proxy from Google Cloud Shell. Possibly related to permission setting since I see the request not authorized,
Other context,
"gcloud beta sql connect auth-instance --user=root" has no problem to connect.
I have a service account for SQL Proxy Client.
I possibly miss something. Could someone please shed some light? Thanks in advance.
Thanks in advance.
Proxy log:
./cloud_sql_proxy -instances=auth-158903:asia-east1:auth-instance=tcp:3306
2017/02/17 14:00:59 Listening on 127.0.0.1:3306 for auth-158903:asia-east1:auth-instance
2017/02/17 14:00:59 Ready for new connections
2017/02/17 14:01:07 New connection for "auth-158903:asia-east1:auth-instance"
2017/02/17 14:03:16 couldn't connect to "auth-158903:asia-east1:auth-instance": dial tcp 107.167.191.26:3307: getsockopt: connection timed out
Client Log:
mysql -u root -p --host 127.0.0.1
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
I also try with credential file but still no luck,
./cloud_sql_proxy -instances=auth-158903:asia-east1:auth-instance=tcp:3306 -credential_file=Auth-2eede8ae0d0b.jason
2017/02/17 14:21:36 using credential file for authentication; email=sql-proxy-client#auth-158903.iam.gserviceaccount.com
2017/02/17 14:21:36 Listening on 127.0.0.1:3306 for auth-158903:asia-east1:auth-instance
2017/02/17 14:21:36 Ready for new connections
2017/02/17 14:21:46 New connection for "auth-158903:asia-east1:auth-instance"
2017/02/17 14:21:48 couldn't connect to "auth-158903:asia-east1:auth-instance": ensure that the account has access to "auth-158903:asia-east1:auth-instance" (and make sure there's no typo in that name). Error during get instance auth-158903:asia-east1:auth-instance: googleapi: **Error 403: The client is not authorized to make this request., notAuthorized**
I can reproduce this issue exactly if I only give my service account "Cloud SQL Client" IAM role. When I give my service account the "Cloud SQL Viewer" role as well, it can then connect. I suggest you try this and see if it helps.
It looks like a network connectivity issue.
Read this carefully if you use a private IP :
https://cloud.google.com/sql/docs/mysql/private-ip
Note that the Cloud SQL instance is in a Google managed network and the proxy is meant to be used to simplify connections to the DB within the VPC network.
In short: running cloud-sql-proxy from a local machine will not work, because it's not in the VPC network. It should work from a Compute Engine VM that is connected to the same VPC as the DB.
What I usually do as a workaround is use gcloud ssh from a local machine and port forward over a small VM in compute engine, like:
gcloud beta compute ssh --zone "europe-north1-b" "instance-1" --project "my-project" -- -L 3306:cloud_sql_server_ip:3306
Then you can connect to localhost:3306 (make sure nothing else is running or change first port number to one that is free locally)
The Cloud SQL proxy uses port 3307 instead of the more usual MySQL port 3306. This is because it uses TLS in a different way and has different IP ACLs. As a consequence, firewalls that allow MySQL traffic won't allow Cloud SQL proxy by default.
Take a look and see if you have a firewall on your network that blocks port 3307. To use Cloud SQL proxy, authorize this port for outbound connections.