I have an aws instance which hold Gitlab app and I have another for holding database, how can I achieve the connection between with two instances?
If the second instance is only there to store a database (and it's something like postgres or mysql) I'd recommend using RDS for it instead. It sets up the database in a way where you can whitelist the security group of the EC2 instance (your gitlab app) and provides dns and automatic backups/replication (if you enable multi-AZ).
This guide is a good place to start:
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_GettingStarted.CreatingConnecting.PostgreSQL.html
Make sure your security group allows your instances to reach one another by modifying the inbound connection to reach the host on the required ports.
Start your database on the one instance
Your app should be able to connect to the database onces its up
For psql for example you would do something like
psql -h <DATABASE AWS INSTANCE IP> -p <port> -U <username> -W <password> <database>
Related
I'm trying to put use an RDS proxy to pool and share connections established with an RDS database with a PostgreSQL engine. The problem is I'm able to open a connection to the DB, both through an ECS instance or in PgAdmin, however, I'm not able to connect through the proxy. To attempt a connection through the proxy in PgAdmin, I'm using the proxy endpoint as opposed to the DB endpoint but the connection attempt times out.
I've successfully created the proxy and associated with my DB, both proxy and DB status is available. I've followed the example proxy setup and the DB and the proxy are using the same VPC security group.
Any ideas?
It seems to me that you are connecting to the proxy from outside of AWS. If this is the case, then its not possible to do this directly:
Your RDS Proxy must be in the same VPC as the database. The proxy can't be publicly accessible, although the database can be.
Your may be able to connect to RDS since it publicly accessible. RDS proxy on the other hand, can only be access from within the same VPC, e.g., from an instance.
Therefor, the solution is to setup an instance in the same VPC as your RDS and proxy. The instance must be accessible using SSH.
On the instance, you can run pgadmin4, in docker:
docker run --rm -p 8080:80 \
-e 'PGADMIN_DEFAULT_EMAIL=user#domain.com' \
-e 'PGADMIN_DEFAULT_PASSWORD=Fz77T8clJqJ4XQrQunGA' \
-d dpage/pgadmin4
The command, after setting up the docker, will server pgadmin4 on port 8080 on the instance.
You can check on the instance if its working:
curl localhost:8080
which can give the following indicating that its working:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: /login?next=%2F. If not click the link
However, since your instance is only accessible through ssh (port 22), to access it from your local workstation, you have to establish an ssh tunnel:
ssh -i <private-key> -L 8080:localhost:8080 -N ubuntu#<public-instance-ip> -v
In the above, my instance was Ubuntu. For Amazon Linux 2, the user would be ec2-user.
The tunnel will forward port 8080 from the instance (i.e. pgadmin4) to your local workstation on port 8080.
Then you just point your browser to localhost:8080 and you should see the pgadmin4 welcome screen.
P.S. My RDS and proxy settings used for the verification:
Is it possible that redis cli is given less priority to connect when memory consumption is high but application is allowed to communicate?
I am unable to connect via cli so can't check anything. Also, don't have the redis server access.
We connect without authentication -
redis-cli -h <hostname>
I ran a process which inserted too many redis keys and that caused this situation. Now, I am not able to delete those keys. I am afraid, the other necessary keys would get evicted as old and system would start doing processing for things not available in redis.
Not able to connect via telnet as well.
Is it possible to connect via a Python script at this point?
If I restart the Java application, will it be able to connect anymore?
Will redis server access via AWS console be able to delete any of the key patterns? I don't have the access currently, so not able to confirm myself. Never used via it also.
Update
Following are graphs taken from AWS console, over the last 1 day since this issue happened -
Update
I went through the FAQ of elasticache, but did not find any mention of being able to manage the data at key value pair level or presence of some special privilege users like root in case of MySql which is able to connect when no other users are able to connect.
All I found is cluster level management capabilities.
From the question it is not clear whether the redis-cli -h <host> command you're running is from within the EC2 or it is from you local machine (Outside AWS VPC).
Accessing from EC2
You will have to ensure following points:
Both the EC2 instance and Redis Instance are on the same VPC.
The security group on EC2 should be allowing port 6379 (It should already be if an application is able to access Redis on the same EC2)
Accessing from outside Amazon VPC
This is not something that's preconfigured and I will suggest that you go through the Accessing Your Cluster docs under the heading "How to Access ElastiCache Resources from Outside AWS".
First, check the connectivity from source(Generally Ec2 instance) to Target (Redis Host).
We can use simple command for that like
#curl -v hostIP(or dnsName):Port
#curl -v myredis.com:6379 or curl -v 192.17.37.42:6379
If you see "Connected" then there is no issue with the network otherwise you have to look into network configurations like firewalls.
Next, you can connect to Redis using redis-cli with the below command:
#redis-cli -h myredis.com -p 6379
I'd like to configure my coldfusion instance to connect to a MySQL database over SSH but I'm not really sure how.
Basically, I have an EC2 instance in the same region as an RDS instance for the purposes of a development environment. I want to hook into my Production RDS instance so that I can do some tests with production data for a specific feature I'm working on but it's turning out to be quite a bit of trouble since it's in a different region.
I'd rather not alter AWS in any way to achieve this. So far the only thing I could think to try was to SSH into my EC2 instance and setup a tunnel like this
ssh -i ./mykey.pem -N -L 3306:localhost:3306 username#host_ip
When I enter this command I don't see any output but I assume it is running, however when I try to access my EC2 instance via the web I see this error: Timed out trying to establish connection
Is there something wrong with my setup? I know I have the correct key, credentials, and host but I am a bit confused on the ports. I figured my coldfusion admin panel is looking on port 3306 and my database is served on port 3306 so 3306:localhost:3306 seems correct to me but obviously I am doing something wrong.
I'm attempting to run a webserver that uses an RDS database with EC2 inside a docker container.
I've setup the security groups so the EC2 host's role is allowed to access the RDS and if I try to access it from the host machine directly everything works correctly.
However, when I run a simple container on the host and attempt to access the RDS, it get's blocked as if the security group weren't letting it through. After a bunch of trial and error it seemed that indeed the containers requests aren't appearing to come from the EC2 host so the firewall says no.
I was able to work around this in the short-run by setting --net=host on the docker container, however this breaks a lot of great docker networking functionality like being able to map ports (ie, now I need to make sure each instance of the container listens on a different port by hand).
Has anyone found a way around this? It seems like a pretty big limitation to running containers in AWS if you're actually using any AWS resources.
Yes, containers do hit the public IPs of RDS. But you do not need to tune low-level Docker options to allow your containers to talk to RDS. The ECS cluster and the RDS instance have to be in the same VPC and then access can be configured through security groups. The easiest way to do this is to:
Navigate to the RDS instances page
Select the DB instance and drill in to see details
Click on the security group id
Navigate over to the Inbound tab and choose Edit
And ensure there is a rule of type MySQL/Aurora with source Custom
When entering the custom source, just start typing in the name of the ECS cluster and the security group name will be auto-completed for you
This tutorial has screenshots that illustrate where to go.
Full disclosure: This tutorial features containers from Bitnami and I work for Bitnami. However the thoughts expressed here are my own and not the opinion of Bitnami.
Figured out what was happening, posting here in case it helps anyone else.
Requests from within the container were hitting the public ip of the RDS rather than the private (which is how the security groups work). It looks like the DNS inside the docker container was using the 8.8.8.8 google dns and that wouldn't do the AWS black magic of turning the rds endpoint into the private ip.
So for instance:
DOCKER_OPTS="--dns 10.0.0.2 -H tcp://127.0.0.1:4243 -H unix:///var/run/docker.sock -g /mnt/docker"
The inbound rule for the RDS should be set to the private IP of the EC2 instance rather than the public IPv4.
As #adamneilson mentions, setting the Docker options are your best bet. Here is how to discover your Amazon DNS server on the VPC. Also the section Enabling Docker Debug Output in the Amazon EC2 Container Service Developer Guide Troubleshooting mentions where the Docker options file is.
Assuming you are running a VPC block of 10.0.0.0/24 the DNS would be 10.0.0.2.
For CentOS, Red Hat and Amazon:
sed -i -r 's/(^OPTIONS=\")/\1--dns 10.0.0.2 /g' /etc/sysconfig/docker
For Ubuntu and Debian:
sed -i -r 's/(^OPTIONS=\")/\1--dns 10.0.0.2 /g' /etc/default/docker
When I tried to connect to AWS RDS in inside of docker container, I got "Access denied for user 'username'#'xxx.xx.xxx.x' (using password: YES)" error.
To solve this issue, I did below two ways:
I created new user and assigned grant.
$ CREATE USER 'newuser'#'%' IDENTIFIED BY 'password';
$ GRANT ALL ON newuser#'%' IDENTIFIED BY 'password';
$ FLUSH PRIVILEGES;
Added global DNS address 8.8.8.8 into docker container when run docker, so that the docker container can resolve IP address of AWS RDS from domain name.
$ docker run --name backend-app --dns=8.8.8.8 -p 8000:8000 -d backend-app
Then I connected from inside of docker container to AWS RDS, successfully.
Note: Firstly, I tried second way. But I didn't solve the connection problem. When I tried both two ways, I was success.
I am new to AWS , I can access EC2 user by putty. Like this how can i connect AWS RDS by PUTTY ?
I have AWS RDS Endpoint , If i try like the below
mysql -h testing.cx2wamr3cbzm.us-west-2.rds.amazonaws.com -P 3306 -u Username -p
I am getting some error like
ERROR 2003 (HY000): Can't connect to MySQL server on 'testing.cx2wamr3cbzm.us-west-2.rds.amazonaws.com' (110)
I think there is a problem with security group.I searched more, but its difficult to understand for me.
I want to create database and assign rights ?
Thanks
RDS is not available to the world by default. It's also generally a bad idea to allow access to the RDS from anywhere except from inside your VPC. I recommend you do the following:
Create a security group that allows access to the RDS over port 3306 from your EC2 instance
Visit https://console.aws.amazon.com/ec2/home#s=SecurityGroups and create a new security group.
Switch to the inbound tab and choose MYSQL from the dropdown.
Erase the 0.0.0.0/0 in the source field then click the input field. It will present you with a list of existing security groups. Choose the one that your EC2 instance belongs to.
Click the apply rule changes button
Assign the security group to your RDS
Visit https://console.aws.amazon.com/rds/home#dbinstances: and select your RDS instance and under the Instance Actions menu select Modify
Change the RDS security group to the one you just created
Make sure to select the Apply immediately option at the bottom of this page
Click Continue and apply the new changes. (the change can sometimes take a couple of minutes)
SSH into your EC2 instance then run the mysql command in your question