I'm trying to put use an RDS proxy to pool and share connections established with an RDS database with a PostgreSQL engine. The problem is I'm able to open a connection to the DB, both through an ECS instance or in PgAdmin, however, I'm not able to connect through the proxy. To attempt a connection through the proxy in PgAdmin, I'm using the proxy endpoint as opposed to the DB endpoint but the connection attempt times out.
I've successfully created the proxy and associated with my DB, both proxy and DB status is available. I've followed the example proxy setup and the DB and the proxy are using the same VPC security group.
Any ideas?
It seems to me that you are connecting to the proxy from outside of AWS. If this is the case, then its not possible to do this directly:
Your RDS Proxy must be in the same VPC as the database. The proxy can't be publicly accessible, although the database can be.
Your may be able to connect to RDS since it publicly accessible. RDS proxy on the other hand, can only be access from within the same VPC, e.g., from an instance.
Therefor, the solution is to setup an instance in the same VPC as your RDS and proxy. The instance must be accessible using SSH.
On the instance, you can run pgadmin4, in docker:
docker run --rm -p 8080:80 \
-e 'PGADMIN_DEFAULT_EMAIL=user#domain.com' \
-e 'PGADMIN_DEFAULT_PASSWORD=Fz77T8clJqJ4XQrQunGA' \
-d dpage/pgadmin4
The command, after setting up the docker, will server pgadmin4 on port 8080 on the instance.
You can check on the instance if its working:
curl localhost:8080
which can give the following indicating that its working:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: /login?next=%2F. If not click the link
However, since your instance is only accessible through ssh (port 22), to access it from your local workstation, you have to establish an ssh tunnel:
ssh -i <private-key> -L 8080:localhost:8080 -N ubuntu#<public-instance-ip> -v
In the above, my instance was Ubuntu. For Amazon Linux 2, the user would be ec2-user.
The tunnel will forward port 8080 from the instance (i.e. pgadmin4) to your local workstation on port 8080.
Then you just point your browser to localhost:8080 and you should see the pgadmin4 welcome screen.
P.S. My RDS and proxy settings used for the verification:
Related
I created a private endpoint in AWS API Gateway to make it private. Only resources in the private vpc can access to this endpoint. This is my design and it works very well. The problem is that it makes me a bit hard to connect it from my local computer.
As a workaround, I can launch a EC2 instance which in the same VPC and I can connect to this EC2 to access the endpoint. But it is not easy to do. I'd like to run postman from my local to connect to the API endpoint. I am looking for a better way to allow me to access it from my local.
Can anyone help me on that?
There are few ways. One one would be VPN, but I personally use proxy capability of postman. What I do has five stages.
1. Bastion host - Amazon Linux 2
with ssh port open in a public subnet.
2. ssh dynamic tunnel
ssh -D 1080 -C -q -N -oStrictHostKeyChecking=no -l ec2-user <public-ip-of-your-bastion> -v
The command will create SOCKS5 proxy from your local workstation to the bastion on port 1080.
3. HTTP -> SOCKS5 conversion
Since postman does not support SOCK5, a conversion is needed. I use http-proxy-to-socks:
hpts -s 127.0.0.1:1080 -p 8080
which will create HTTP proxy on port 8080 forwarded to SOCKS5 proxy on port 1080.
4. Setup HTTP proxy on postman
5. Query private API from postman
Query the private API from your local workstation using its https endpoint, just like if it was a public API.
I created an EC2 linux1 instance and I SSH into it. I installed NodeJS and cloned a git repository to the instance. The app is successfully running and connected to the MySQL database instance I created from the RDS. Assume the app name is my-app. What I want is to be able to access the the app on the server.
I tried
- ec2-{Public DNS (IPv4)}.compute-1.amazonaws.com/my-app/{endpoint} (not working)
- {Public DNS (IPv4)}/my-app/{endpoint} (not working) (not working)
The security group of the instance is set like below:
Any help is appreciated
If your app is running on port 4000, you need to either open that port in your security group, or add a firewall/reverse proxy to forward from 80/443 to 4000.
You can use iptables to forward the port:
Forwarding traffic from 80 to 8080
or apache as a reverse proxy:
Apache redirect to another port
I am testing the following configuration.
Cloud SQL (tetsql-1) in Region X Zone A
A Compute Engine VM (TestVM-1) in the same Region X Zone A. OS is Centos 7
Compute Engine VM is running cloud SQL proxy on non default port (9090)
With the above configuration I am able to logon to testsql-1 from TestVM-1 with below command:
`mysql -h 127.0.0.1 --port 9090 -u testuser -D testDB -p`
However I am not able use the internal IP of TestVM-1 in the above command. It gives an error.
Another observation is I am able to do telnet 127.0.0.1 9090 but when I try telnet <VM -Internal-IP> 9090 returns a connection refused error.
Does anyone know if this is expected behaviour? If this is expected, why is it so?
The cloud proxy uses 127.0.0.1 by default, where it accepts connections.
To configure another IP Address, you have to set it in the instances parameter:
./cloud_sql_proxy -instances=<myCloudSQLproject:myCloudSQLzone:mycloudsqlinstance>=tcp:<IP_Address>:<PORT>
Something like this:
./cloud_sql_proxy -instances=project_xxx:us-central1:database_yyy=tcp:10.203.23.12:9090
This configuration allows connecting to this cloud proxy from others hosts as well.
The reason that you can connect to 127.0.0.1 but you cannot connect using the VM's private IP address is that the Proxy is NOT listening on the private IP address.
The Cloud SQL Proxy listens on the loopback adapter's internal address which is 127.0.0.1. This address only exists inside the computer.
You're able to connect from your VM to Cloud SQL because you're using the proxy. If you would like to connect to your Cloud SQL then you have whitelist the IP address of your VM in Cloud SQL's connections tab, please refer to this documentation.
This is expected behavior. Private IPs are only accessible from a Virtual Private Cloud (VPC). In order for a resource (such as a GCE instance) to connect, it must also be on that VPC.
See this page for instructions on how to add a GCE instance to a VPC, and see this page for more on the environment requirements for Private IP.
When creating my Postgres Cloud SQL instance I specified that would like to connect to it using private IP and chose my default network.
My VM sits in the same default network.
Now, I follow instructions as described here https://cloud.google.com/sql/docs/postgres/connect-compute-engine
and try executing
psql -h [CLOUD_SQL_PRIVATE_IP_ADDR] -U postgres
from my VM, but get this error:
psql: could not connect to server: Connection timed out Is the server
running on host "CLOUD_SQL_PRIVATE_IP_ADDR" and accepting TCP/IP connections on
port 5432?
Anything I am under-looking?
P.S. My Service Networking API (whatever that is) is enabled.
If you have ssh to a VM in the same network you can connect to Cloud SQL using cloud SQL proxy:
Open the ssh window (VM-instances in Computer engine and click on ssh), then download the proxy file with:
wget https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 -O cloud_sql_proxy
Execute, in the ssh shell
chmod +x cloud_sql_proxy
Create a service account with role Cloud SQL Client and create an api key. Download the json key in your local computer.
In the ssh vm shell click on the wheel and "upload", and upload the key file
5.
./cloud_sql_proxy -instances=<Instance connection name>=tcp:5432 -credential_file=<name of the json file>
where "Instance connection name" can be found in SQL-Overview -> Connect to this instance
Finally
psql "host=127.0.0.1 port=5432 sslmode=disable user=<your-user-name> dbname=<your-db-name>"
On the other hand, if you want to connect to cloud sql from your local computer and the cloud sql instance does not have a public ip you have to connect through a bastion host configuration.
https://cloud.google.com/solutions/connecting-securely
According to this document connect via private ip, you need to setup following item:
You must have enabled the Service Networking API for your project. If you are using shared VPC , you also need to enable this API for the host project.
Enabling APIs requires the servicemanagement.services.bind IAM permission.
Establishing private services access requires the Network Administrator IAM role.
After private services access is established for your network, you do not need the Network Administrator role to configure an instance to use private IP.
I have an aws instance which hold Gitlab app and I have another for holding database, how can I achieve the connection between with two instances?
If the second instance is only there to store a database (and it's something like postgres or mysql) I'd recommend using RDS for it instead. It sets up the database in a way where you can whitelist the security group of the EC2 instance (your gitlab app) and provides dns and automatic backups/replication (if you enable multi-AZ).
This guide is a good place to start:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html
Make sure your security group allows your instances to reach one another by modifying the inbound connection to reach the host on the required ports.
Start your database on the one instance
Your app should be able to connect to the database onces its up
For psql for example you would do something like
psql -h <DATABASE AWS INSTANCE IP> -p <port> -U <username> -W <password> <database>