I have a database running in a private VPC. The database isn't publicly accessible. Does superset support Connecting to databases via an SSH Tunnel? If so any link to the docs?
I couldn't find it in the docs, but with the SSH tunnel you basically need to change the DB host to 127.0.0.1 (not localhost, since that's a keyword reserved for connection using the MySQL socket), and everything else should be the same.
For example, I tested with a MySQL database that I have running on host.example.com. I first created a tunnel redirecting my local port 3336 to MySQL's port 3306 (I did that because I already have MySQL running locally on 3306):
ssh -N -L 3336:127.0.0.1:3306 host.example.com
Then I was able to add it to Superset using this SQLAlchemy URI:
mysql://username:password#127.0.0.1:3336/dbname
This worked for me:
ssh -i /path/key-pair_instance1.pem username_of_instance1#i-0123456789abcdefa -L 9090:ec2-198-51-100-2.compute-1.amazonaws.com:3306
Here is the source with further explanation.
Related
I'm trying to put use an RDS proxy to pool and share connections established with an RDS database with a PostgreSQL engine. The problem is I'm able to open a connection to the DB, both through an ECS instance or in PgAdmin, however, I'm not able to connect through the proxy. To attempt a connection through the proxy in PgAdmin, I'm using the proxy endpoint as opposed to the DB endpoint but the connection attempt times out.
I've successfully created the proxy and associated with my DB, both proxy and DB status is available. I've followed the example proxy setup and the DB and the proxy are using the same VPC security group.
Any ideas?
It seems to me that you are connecting to the proxy from outside of AWS. If this is the case, then its not possible to do this directly:
Your RDS Proxy must be in the same VPC as the database. The proxy can't be publicly accessible, although the database can be.
Your may be able to connect to RDS since it publicly accessible. RDS proxy on the other hand, can only be access from within the same VPC, e.g., from an instance.
Therefor, the solution is to setup an instance in the same VPC as your RDS and proxy. The instance must be accessible using SSH.
On the instance, you can run pgadmin4, in docker:
docker run --rm -p 8080:80 \
-e 'PGADMIN_DEFAULT_EMAIL=user#domain.com' \
-e 'PGADMIN_DEFAULT_PASSWORD=Fz77T8clJqJ4XQrQunGA' \
-d dpage/pgadmin4
The command, after setting up the docker, will server pgadmin4 on port 8080 on the instance.
You can check on the instance if its working:
curl localhost:8080
which can give the following indicating that its working:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: /login?next=%2F. If not click the link
However, since your instance is only accessible through ssh (port 22), to access it from your local workstation, you have to establish an ssh tunnel:
ssh -i <private-key> -L 8080:localhost:8080 -N ubuntu#<public-instance-ip> -v
In the above, my instance was Ubuntu. For Amazon Linux 2, the user would be ec2-user.
The tunnel will forward port 8080 from the instance (i.e. pgadmin4) to your local workstation on port 8080.
Then you just point your browser to localhost:8080 and you should see the pgadmin4 welcome screen.
P.S. My RDS and proxy settings used for the verification:
Followed instructions to created Google Cloud SQL instance with PostgreSQL db. I can examine it from glcoud command line with the instance name:
gcloud sql instances describe my-fancy-instance-name
I get back oodles of details!
When I created the SSL certs, it gave me the psql command to connect, which matches the instructions from the instructions given by Google documentation on how to connect via public IP (I left that enabled for now).
Now, with the discovered IP and port, which was also verified by the gcloud command, I try to call psql to connect:
psql "sslmode=verify-ca sslrootcert=server-ca.pem \
sslcert=client-cert.pem sslkey=client-key.pem \
hostaddr=my_INSTANCE_IP \
user=my_USER_NAME dbname=my_DB_NAME"
It takes a bit but it comes back and reports:
psql: could not connect to server: Operation timed out
Is the server running on host “nnn.ooo.ppp.qqq" and accepting
TCP/IP connections on port xxxxx?
and I verified the ip, but the port was NOT returned by the gcloud command...
Now, I have my SSL (.pem) files in ~/.postgress folder and added that path in my command as well, but no joy.
Any ideas why the psql can't connect (connection times out)? Especially since the gcloud command works?
The error message suggests that the CloudSQL instance is not exposed to the local host IP. As in the documentation step 7, you need to add your local psql client IP to the Authorized network. Can you confirm if you did that? If your local host IP is dynamic, then you have to regularly re-add the IP to Authorized network once your local IP changed.
Following the Google documentation you provided, I successfully connected to my Cloud Postgresql database from my local psql client.
When creating my Postgres Cloud SQL instance I specified that would like to connect to it using private IP and chose my default network.
My VM sits in the same default network.
Now, I follow instructions as described here https://cloud.google.com/sql/docs/postgres/connect-compute-engine
and try executing
psql -h [CLOUD_SQL_PRIVATE_IP_ADDR] -U postgres
from my VM, but get this error:
psql: could not connect to server: Connection timed out Is the server
running on host "CLOUD_SQL_PRIVATE_IP_ADDR" and accepting TCP/IP connections on
port 5432?
Anything I am under-looking?
P.S. My Service Networking API (whatever that is) is enabled.
If you have ssh to a VM in the same network you can connect to Cloud SQL using cloud SQL proxy:
Open the ssh window (VM-instances in Computer engine and click on ssh), then download the proxy file with:
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
Execute, in the ssh shell
chmod +x cloud_sql_proxy
Create a service account with role Cloud SQL Client and create an api key. Download the json key in your local computer.
In the ssh vm shell click on the wheel and "upload", and upload the key file
5.
./cloud_sql_proxy -instances=<Instance connection name>=tcp:5432 -credential_file=<name of the json file>
where "Instance connection name" can be found in SQL-Overview -> Connect to this instance
Finally
psql "host=127.0.0.1 port=5432 sslmode=disable user=<your-user-name> dbname=<your-db-name>"
On the other hand, if you want to connect to cloud sql from your local computer and the cloud sql instance does not have a public ip you have to connect through a bastion host configuration.
https://cloud.google.com/solutions/connecting-securely
According to this document connect via private ip, you need to setup following item:
You must have enabled the Service Networking API for your project. If you are using shared VPC , you also need to enable this API for the host project.
Enabling APIs requires the servicemanagement.services.bind IAM permission.
Establishing private services access requires the Network Administrator IAM role.
After private services access is established for your network, you do not need the Network Administrator role to configure an instance to use private IP.
AWS DocumentDB is a relatively new service we're trying to migrate to. To connect from outside of the VPC, you have to create a tunnel to an existing instance.
For example:
ssh -i "ec2Access.pem" -L 27017:sample-cluster.cluster-cu52jq5kfddg.us-east-1.docdb.amazonaws.com:27017 ubuntu#ec2-34-229-221-164.compute-1.amazonaws.com -N
And then you can connect from mongo shell with:
mongo --sslAllowInvalidHostnames --ssl --sslCAFile rds-combined-ca-bundle.pem --username <yourUsername> --password <yourPassword>
You can see this info at: https://docs.aws.amazon.com/documentdb/latest/developerguide/connect-from-outside-a-vpc.html
I'm actually able to get a shell into document db following the above instructions but I can not connect to it using MongoDB compass. We need to be able to do this for our non-developer team.
I have done this using only MongoDB Compass (Community, v. 1.16.4), without creation of a tunnel via external tools.
Firstly, download AWS's certificate from:
https://s3.amazonaws.com/rds-downloads/rds-combined-ca-bundle.pem.
I have got this link from:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html
Also make sure that you have an access to your DocDB cluster from the EC2 instance.
To do it look into security groups settings specified for your DocDB cluster.
Given that you are able to get the access using mongo shell, you should have it.
Then on the connection screen in MongoDB Compass specify the following (your credentials used as an example):
Hostname: sample-cluster.cluster-cu52jq5kfddg.us-east-1.docdb.amazonaws.com
Port: 27017
Authentication: Username/Password
Username: YourDocDBUsername
Password: YourDocDBUserPassword
SSL: Server validation
Certificate Authority: (select downloaded rds-combined-ca-bundle.pem)
SSH Tunnel: Use identity file
SSH Hostname: ec2-34-229-221-164.compute-1.amazonaws.com
SSH Tunnel Port: 22
SSH Username: ubuntu
SSH Identity File: ec2Access.pem
I similarly was able to connect to my tunneled DocumentDB instance via mongosh but not with MongoDB Compass. I noticed that mongosh outputs the connection string so I copied that in Compass and it worked.
mongodb://<credentials>#localhost:27017/?directConnection=true&tls=true&tlsAllowInvalidHostnames=true&tlsCAFile=<path_to_pem>
It looks like the part that I was missing was
directConnection=true
My postgres DB is in AWS instance running windows server 2012.
My tableau desktop is in local machine.
While connecting to the DB using external instance IP and default postgres PORT 5432 , it shows the error
Unable to connect to the ODBC Data Source.
Check that the necessary drivers are installed and that the connection properties are valid.
Unable to connect to the server "ec2-xx.xx.xx.xx.ap-south-1.compute.amazonaws.com".
Check that the server is running and that you have access privileges to the requested database.
I have made sure that:
AWS security group is configured for all incoming ports
Windows server 2012 firewall is accepting all inbound traffic.
My local desktop is allowing to send traffic to remote server.And I can telnet hostIP/hostname PORT successfully.
Postgres service is running on 5432 port.
Going by the points you mentioned, I would suggest to see if you have the required drivers installed for postgres. Also, if possible for you for the time being you can install any client temporarily like sqldeveloper(later you can remove this) on the tableau desktop.
Now because you are able to telnet Ip 5432, you can test your connection from the client, If this doesnt work u know that the problem is in configuration else we need to dig more into that.
It seems like I have not set the pg_hba.conf file correctly for postgres. Allowing the connection request from all external IPs let me resolve the issue.
Adding the line at the end of the file,
host all postgres 0.0.0.0 md5