Followed instructions to created Google Cloud SQL instance with PostgreSQL db. I can examine it from glcoud command line with the instance name:
gcloud sql instances describe my-fancy-instance-name
I get back oodles of details!
When I created the SSL certs, it gave me the psql command to connect, which matches the instructions from the instructions given by Google documentation on how to connect via public IP (I left that enabled for now).
Now, with the discovered IP and port, which was also verified by the gcloud command, I try to call psql to connect:
psql "sslmode=verify-ca sslrootcert=server-ca.pem \
sslcert=client-cert.pem sslkey=client-key.pem \
hostaddr=my_INSTANCE_IP \
user=my_USER_NAME dbname=my_DB_NAME"
It takes a bit but it comes back and reports:
psql: could not connect to server: Operation timed out
Is the server running on host “nnn.ooo.ppp.qqq" and accepting
TCP/IP connections on port xxxxx?
and I verified the ip, but the port was NOT returned by the gcloud command...
Now, I have my SSL (.pem) files in ~/.postgress folder and added that path in my command as well, but no joy.
Any ideas why the psql can't connect (connection times out)? Especially since the gcloud command works?
The error message suggests that the CloudSQL instance is not exposed to the local host IP. As in the documentation step 7, you need to add your local psql client IP to the Authorized network. Can you confirm if you did that? If your local host IP is dynamic, then you have to regularly re-add the IP to Authorized network once your local IP changed.
Following the Google documentation you provided, I successfully connected to my Cloud Postgresql database from my local psql client.
Related
I am trying to explore a way to connect postgres 13 cloud sql instance with only private IP from my local windows machine ..
I am able to connect through a compute instance tied with the same subnet as a default GCP behavior..
I want it to be secure my cloud instance to be accessible only through a VPN setup.. I have a Open VPN CE installed and whitelisted the Open VPN ip in the GCP firewall rule. Still getting the error message.
psql: error: could not connect to server: Connection timed out
Is the server running on host "{ip_address}" and accepting
TCP/IP connections on port 5432?
There are methods though to connect via private IP having enabled cloud proxy. But is there a way that i can make it happen via other VPNs.?
When you receive this error most of the time it is due to not having
PostgreSQL configured to allow TCP/IP connections or at least no
connections from your particular workstation. Here is a list of
common causes to this problem:
a) postgresql.conf is not set up to allow TCP/IP connections.
You'll want to look at the listen_address configuration parameter.
b) postgresql.conf is not set up to allow connections on a
non-standard port number. To determine this look at the port
configuration option.
c) Authentication rules in PostgreSQL's access configuration file
(pg_hba.conf) are not set up to allow either your user or IP
address to connect to that database. See the
official-documentation for more information on setting up your
pg_hba.conf properly.
d) Ensure that there are no firewalls, such as iptables that are
keeping your local system from even establishing a connection to the
remote host. For common PostgreSQL problems and possible solutions,
check here.
You have to edit the postgresql.conf file and change the line
with 'listen_addresses'. This file you can find in the
/etc/postgresql/13/main directory. To connect PostgreSQL server from other computers, you have change this config line in this way:
listen_addresses = '*'
Then you have to edit the pg_hba.conf file, too. In this file you
have set, from which computers you can connect to this server and
what method of authentication you can use. Usually you will need
similar line:
host all all <IP address> md5
For detailed steps, you can check here.
Finally i had to set the cloud SQL proxy on a f1-micro VM instance having only private IP ..
Whitelisted the port 5432 on the firewall rule.
From table plus i use the private IP of the vm instance to connect to my cloud postgress instance.
Very well If somebody has other alternatives please do let me know.
I have a database running in a private VPC. The database isn't publicly accessible. Does superset support Connecting to databases via an SSH Tunnel? If so any link to the docs?
I couldn't find it in the docs, but with the SSH tunnel you basically need to change the DB host to 127.0.0.1 (not localhost, since that's a keyword reserved for connection using the MySQL socket), and everything else should be the same.
For example, I tested with a MySQL database that I have running on host.example.com. I first created a tunnel redirecting my local port 3336 to MySQL's port 3306 (I did that because I already have MySQL running locally on 3306):
ssh -N -L 3336:127.0.0.1:3306 host.example.com
Then I was able to add it to Superset using this SQLAlchemy URI:
mysql://username:password#127.0.0.1:3336/dbname
This worked for me:
ssh -i /path/key-pair_instance1.pem username_of_instance1#i-0123456789abcdefa -L 9090:ec2-198-51-100-2.compute-1.amazonaws.com:3306
Here is the source with further explanation.
I'm trying to put use an RDS proxy to pool and share connections established with an RDS database with a PostgreSQL engine. The problem is I'm able to open a connection to the DB, both through an ECS instance or in PgAdmin, however, I'm not able to connect through the proxy. To attempt a connection through the proxy in PgAdmin, I'm using the proxy endpoint as opposed to the DB endpoint but the connection attempt times out.
I've successfully created the proxy and associated with my DB, both proxy and DB status is available. I've followed the example proxy setup and the DB and the proxy are using the same VPC security group.
Any ideas?
It seems to me that you are connecting to the proxy from outside of AWS. If this is the case, then its not possible to do this directly:
Your RDS Proxy must be in the same VPC as the database. The proxy can't be publicly accessible, although the database can be.
Your may be able to connect to RDS since it publicly accessible. RDS proxy on the other hand, can only be access from within the same VPC, e.g., from an instance.
Therefor, the solution is to setup an instance in the same VPC as your RDS and proxy. The instance must be accessible using SSH.
On the instance, you can run pgadmin4, in docker:
docker run --rm -p 8080:80 \
-e 'PGADMIN_DEFAULT_EMAIL=user#domain.com' \
-e 'PGADMIN_DEFAULT_PASSWORD=Fz77T8clJqJ4XQrQunGA' \
-d dpage/pgadmin4
The command, after setting up the docker, will server pgadmin4 on port 8080 on the instance.
You can check on the instance if its working:
curl localhost:8080
which can give the following indicating that its working:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: /login?next=%2F. If not click the link
However, since your instance is only accessible through ssh (port 22), to access it from your local workstation, you have to establish an ssh tunnel:
ssh -i <private-key> -L 8080:localhost:8080 -N ubuntu#<public-instance-ip> -v
In the above, my instance was Ubuntu. For Amazon Linux 2, the user would be ec2-user.
The tunnel will forward port 8080 from the instance (i.e. pgadmin4) to your local workstation on port 8080.
Then you just point your browser to localhost:8080 and you should see the pgadmin4 welcome screen.
P.S. My RDS and proxy settings used for the verification:
When creating my Postgres Cloud SQL instance I specified that would like to connect to it using private IP and chose my default network.
My VM sits in the same default network.
Now, I follow instructions as described here https://cloud.google.com/sql/docs/postgres/connect-compute-engine
and try executing
psql -h [CLOUD_SQL_PRIVATE_IP_ADDR] -U postgres
from my VM, but get this error:
psql: could not connect to server: Connection timed out Is the server
running on host "CLOUD_SQL_PRIVATE_IP_ADDR" and accepting TCP/IP connections on
port 5432?
Anything I am under-looking?
P.S. My Service Networking API (whatever that is) is enabled.
If you have ssh to a VM in the same network you can connect to Cloud SQL using cloud SQL proxy:
Open the ssh window (VM-instances in Computer engine and click on ssh), then download the proxy file with:
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
Execute, in the ssh shell
chmod +x cloud_sql_proxy
Create a service account with role Cloud SQL Client and create an api key. Download the json key in your local computer.
In the ssh vm shell click on the wheel and "upload", and upload the key file
5.
./cloud_sql_proxy -instances=<Instance connection name>=tcp:5432 -credential_file=<name of the json file>
where "Instance connection name" can be found in SQL-Overview -> Connect to this instance
Finally
psql "host=127.0.0.1 port=5432 sslmode=disable user=<your-user-name> dbname=<your-db-name>"
On the other hand, if you want to connect to cloud sql from your local computer and the cloud sql instance does not have a public ip you have to connect through a bastion host configuration.
https://cloud.google.com/solutions/connecting-securely
According to this document connect via private ip, you need to setup following item:
You must have enabled the Service Networking API for your project. If you are using shared VPC , you also need to enable this API for the host project.
Enabling APIs requires the servicemanagement.services.bind IAM permission.
Establishing private services access requires the Network Administrator IAM role.
After private services access is established for your network, you do not need the Network Administrator role to configure an instance to use private IP.
I am following the Django sample for GAE and have problem to connect to Cloud SQL instance by Proxy from Google Cloud Shell. Possibly related to permission setting since I see the request not authorized,
Other context,
"gcloud beta sql connect auth-instance --user=root" has no problem to connect.
I have a service account for SQL Proxy Client.
I possibly miss something. Could someone please shed some light? Thanks in advance.
Thanks in advance.
Proxy log:
./cloud_sql_proxy -instances=auth-158903:asia-east1:auth-instance=tcp:3306
2017/02/17 14:00:59 Listening on 127.0.0.1:3306 for auth-158903:asia-east1:auth-instance
2017/02/17 14:00:59 Ready for new connections
2017/02/17 14:01:07 New connection for "auth-158903:asia-east1:auth-instance"
2017/02/17 14:03:16 couldn't connect to "auth-158903:asia-east1:auth-instance": dial tcp 107.167.191.26:3307: getsockopt: connection timed out
Client Log:
mysql -u root -p --host 127.0.0.1
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0
I also try with credential file but still no luck,
./cloud_sql_proxy -instances=auth-158903:asia-east1:auth-instance=tcp:3306 -credential_file=Auth-2eede8ae0d0b.jason
2017/02/17 14:21:36 using credential file for authentication; email=sql-proxy-client#auth-158903.iam.gserviceaccount.com
2017/02/17 14:21:36 Listening on 127.0.0.1:3306 for auth-158903:asia-east1:auth-instance
2017/02/17 14:21:36 Ready for new connections
2017/02/17 14:21:46 New connection for "auth-158903:asia-east1:auth-instance"
2017/02/17 14:21:48 couldn't connect to "auth-158903:asia-east1:auth-instance": ensure that the account has access to "auth-158903:asia-east1:auth-instance" (and make sure there's no typo in that name). Error during get instance auth-158903:asia-east1:auth-instance: googleapi: **Error 403: The client is not authorized to make this request., notAuthorized**
I can reproduce this issue exactly if I only give my service account "Cloud SQL Client" IAM role. When I give my service account the "Cloud SQL Viewer" role as well, it can then connect. I suggest you try this and see if it helps.
It looks like a network connectivity issue.
Read this carefully if you use a private IP :
https://cloud.google.com/sql/docs/mysql/private-ip
Note that the Cloud SQL instance is in a Google managed network and the proxy is meant to be used to simplify connections to the DB within the VPC network.
In short: running cloud-sql-proxy from a local machine will not work, because it's not in the VPC network. It should work from a Compute Engine VM that is connected to the same VPC as the DB.
What I usually do as a workaround is use gcloud ssh from a local machine and port forward over a small VM in compute engine, like:
gcloud beta compute ssh --zone "europe-north1-b" "instance-1" --project "my-project" -- -L 3306:cloud_sql_server_ip:3306
Then you can connect to localhost:3306 (make sure nothing else is running or change first port number to one that is free locally)
The Cloud SQL proxy uses port 3307 instead of the more usual MySQL port 3306. This is because it uses TLS in a different way and has different IP ACLs. As a consequence, firewalls that allow MySQL traffic won't allow Cloud SQL proxy by default.
Take a look and see if you have a firewall on your network that blocks port 3307. To use Cloud SQL proxy, authorize this port for outbound connections.