I am testing the following configuration.
Cloud SQL (tetsql-1) in Region X Zone A
A Compute Engine VM (TestVM-1) in the same Region X Zone A. OS is Centos 7
Compute Engine VM is running cloud SQL proxy on non default port (9090)
With the above configuration I am able to logon to testsql-1 from TestVM-1 with below command:
`mysql -h 127.0.0.1 --port 9090 -u testuser -D testDB -p`
However I am not able use the internal IP of TestVM-1 in the above command. It gives an error.
Another observation is I am able to do telnet 127.0.0.1 9090 but when I try telnet <VM -Internal-IP> 9090 returns a connection refused error.
Does anyone know if this is expected behaviour? If this is expected, why is it so?
The cloud proxy uses 127.0.0.1 by default, where it accepts connections.
To configure another IP Address, you have to set it in the instances parameter:
./cloud_sql_proxy -instances=<myCloudSQLproject:myCloudSQLzone:mycloudsqlinstance>=tcp:<IP_Address>:<PORT>
Something like this:
./cloud_sql_proxy -instances=project_xxx:us-central1:database_yyy=tcp:10.203.23.12:9090
This configuration allows connecting to this cloud proxy from others hosts as well.
The reason that you can connect to 127.0.0.1 but you cannot connect using the VM's private IP address is that the Proxy is NOT listening on the private IP address.
The Cloud SQL Proxy listens on the loopback adapter's internal address which is 127.0.0.1. This address only exists inside the computer.
You're able to connect from your VM to Cloud SQL because you're using the proxy. If you would like to connect to your Cloud SQL then you have whitelist the IP address of your VM in Cloud SQL's connections tab, please refer to this documentation.
This is expected behavior. Private IPs are only accessible from a Virtual Private Cloud (VPC). In order for a resource (such as a GCE instance) to connect, it must also be on that VPC.
See this page for instructions on how to add a GCE instance to a VPC, and see this page for more on the environment requirements for Private IP.
Related
Can DBeaver create two SSH tunnels and then connect to a database?
I have successfully created one SSH tunnel but not two.
I am trying to connect to an AWS RDS database via Bastion host. Bastion host only allows SSH access from my corporate IP range.
This means that when I am in the office I can connect to the RDS from DBeaver just fine:
My computer is in the allowed IP range
DBeaver creates an SSH tunnel to a Bastion host in my VPC inside the AWS cloud
DBeaver connects to the RDS database
The issue arises when I work from home.
I would have to add "zero" step to have an allowed IP address for the Bastion host connection:
0) Connect to the machine inside the office
I have not yet managed to achieve this. Has anyone got an idea of how to do this?
Kudos to #erik258 for pointing me in the right direction.
I have created an SSH tunnel between an office machine and the Bastion host. When in the office machine, when I access http://localhost:<local_port> I am in effect communicating with the <RDS_endpoint> on port <remote_port>.
Steps:
Create an SSH tunnel from the office machine to the Bastion host. Source
$ ssh -L <local_port>:<RDS_endpoint>:<remote_port> -i <path_to_ssh_key> ec2-user#<Bastion_host public IP>
<local_port> - random port
<remote_port> - port RDS endpoint listens to (5432 for PostgreSQL)
<RDS_endpoint> - endpoint specified on the AWS RDS page
Create DBeaver connection. In the "SSH" section specify your office machine. In "Main" section, set "Host" as localhost, and "Port" as <local_port>.
I successfully created a Compute Engine VM instance, and installed MySQL on it using this guide:
https://cloud.google.com/solutions/setup-mysql
Now I want to connect to it from App Engine and from my home too if possible.
Using this guide: https://cloud.google.com/appengine/docs/standard/python/connecting-vpc
I created the connector, giving it the suggested 10.x.x.x/28 IP address.
In my app engine's app.yaml I inserted:
vpc_access_connector:
name: projects/xxxxxxxxxx/locations/europe-west3/conectors/xxxxxx
The VM instance shows an internal IP of 10.x.x.x and an external IP of 34.x.x.x
I am trying to connect from PHP using this line:
new mysqli($servername, $username, $password, $dbname, 3306, null);
but I get different errors.
When connecting to 34.x.x.x from my home:
mysqli::__construct(): (HY000/2002): No connection could be made because the target machine actively refused it.
(I even created a Firewall rule to allow all traffic from my home IP)
When connecting to10.x.x.x from app engine:
2002: Connection timed out
When connecting to 10.x.x.x from app engine:
2002: Connection refused
How can I make a connection?
My first guess is that the service inside the instance is not up and running, check if the service of MySQL is running and listening, you can try this by doing a nmap test vs. the public IP of your MySQL VM instance nmap 34.0.0.0. You should see something like this (if you are using the standard port):
PORT STATE SERVICE
22/tcp open ssh
80/tcp closed http
443/tcp closed https
3306/tcp open mysql
Remember that there are two firewalls you need to take care of on GCP, (VPC firewall & OS firewall). So if you are not able to see this port start the SQL service by connecting into your VM instance and typing sudo systemctl start mysql, now run again nmap 34.0.0.0 and you should see the service.
To connect from App Engine to MySQL on Compute Engine VM instance just follow this other guide .
To connect from your home It will be better for you to connect to the instance via SSH and then to access your database, or you can keep connecting by a VPN to access it with the internal IP address if you need to connect directly to the socket, just take care to avoid hitting the limitations or missing something like IAM roles and permission needed for this connection.
I'm trying to put use an RDS proxy to pool and share connections established with an RDS database with a PostgreSQL engine. The problem is I'm able to open a connection to the DB, both through an ECS instance or in PgAdmin, however, I'm not able to connect through the proxy. To attempt a connection through the proxy in PgAdmin, I'm using the proxy endpoint as opposed to the DB endpoint but the connection attempt times out.
I've successfully created the proxy and associated with my DB, both proxy and DB status is available. I've followed the example proxy setup and the DB and the proxy are using the same VPC security group.
Any ideas?
It seems to me that you are connecting to the proxy from outside of AWS. If this is the case, then its not possible to do this directly:
Your RDS Proxy must be in the same VPC as the database. The proxy can't be publicly accessible, although the database can be.
Your may be able to connect to RDS since it publicly accessible. RDS proxy on the other hand, can only be access from within the same VPC, e.g., from an instance.
Therefor, the solution is to setup an instance in the same VPC as your RDS and proxy. The instance must be accessible using SSH.
On the instance, you can run pgadmin4, in docker:
docker run --rm -p 8080:80 \
-e 'PGADMIN_DEFAULT_EMAIL=user#domain.com' \
-e 'PGADMIN_DEFAULT_PASSWORD=Fz77T8clJqJ4XQrQunGA' \
-d dpage/pgadmin4
The command, after setting up the docker, will server pgadmin4 on port 8080 on the instance.
You can check on the instance if its working:
curl localhost:8080
which can give the following indicating that its working:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: /login?next=%2F. If not click the link
However, since your instance is only accessible through ssh (port 22), to access it from your local workstation, you have to establish an ssh tunnel:
ssh -i <private-key> -L 8080:localhost:8080 -N ubuntu#<public-instance-ip> -v
In the above, my instance was Ubuntu. For Amazon Linux 2, the user would be ec2-user.
The tunnel will forward port 8080 from the instance (i.e. pgadmin4) to your local workstation on port 8080.
Then you just point your browser to localhost:8080 and you should see the pgadmin4 welcome screen.
P.S. My RDS and proxy settings used for the verification:
There are 8 failed tasks in a particular executor. I want to connect to it via ssh to view the yarn logs.
The executor address is: ip-123-45-6-78.us-west-2.compute.internal:34265
I've tried both:
ssh ip-123-45-6-78.us-west-2.compute.internal:34265
and
ssh ip-123-45-6-78.us-west-2.compute.internal
But both produce the following error:
Could not resolve hostname ip-123-45-6-78.us-west-2.compute.internal:
Name or service not known
I've also added to the .ssh/config file the same key-pair I use to connect to the master:
Host master
HostName ec2-09-876-543-21.us-west-2.compute.amazonaws.com
User hadoop
IdentityFile ~/keypair.pem
Host worker
HostName ip-123-45-6-78.us-west-2.compute.internal
User hadoop
IdentityFile ~/keypair.pem
And also both ssh worker and ssh worker:34265 don't work.
Just to be clear: ssh master does work!
The Spark application is running on an EMR cluster.
From the hostname *.compute.internal these are internal IP address (private IP) and you can not ssh from your local system.
You are able to SSH to master because you are using public IP address of the master instance. try to use the public IP address for the worker too and it should work.
Or the option is to create ssh-tunnel through the master server, you can try something like
Host worker
HostName ip-123-45-6-78.us-west-2.compute.internal
User hadoop
IdentityFile ~/keypair.pem
ProxyCommand ssh master -W %h:%p
The hostname you're trying to connect to will not resolve as you're outside of the AWS VPC. Private records (those as part of the compute.internal domain) only resolve if the DNS of the network goes through the Route 53 Private Resolver.
If you're not to worried about resolving the DNS hostnames you can instead attempt connecting via the private IP directly (assuming you have access via either a VPN connection or Direct Connect). Alternatively connect via an instance that has public ingress i.e. Client -> Jump Server -> Private Host.
If you do want to resolve via private domain name the following are the best options:
Inbound Resolver
Simple AD
Setup an EC2 based DNS server in your VPC.
I have a AWS EC2 instance running and I am supposed to access the localhost URL of the instance. Whenever I try the localhost:port/index.html URL I get a server not responding error. I tried using the public IP of the instance instead, but that failed. I configured AWS by exposing the particular port number for the inbound traffic (IP : 0.0.0.0/0) that did not work either. How should I configure so that I can access the URL?
If a service is running at localhost:45984 on an EC2 instance, you cannot access that server from your browser on your local machine unless you employ port forwarding.
Here's a good article that explains the different concepts:
https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding
I believe what you want is local port forwarding, where you set up a tunnel so that you can access "localhost:45984" on your EC2 instance from some port you specify on your Mac.