Docker container deployed via Beanstalk cannot connect to the database on RDS - amazon-web-services

I'm new to both docker and AWS. I just created my very first docker image. The application is a backend microservice with rest controllers persisting data in a MySQL database. I've manually created the database in RDS and after running the container locally, the rest APIs work fine in Postman.
Here is the Dockerfile:
FROM openjdk:8-jre-alpine
MAINTAINER alireza.online
COPY ./target/Practice-1-1.0-SNAPSHOT.jar /myApplication/
COPY ./target/libs/ /myApplication/libs/
EXPOSE 8080
CMD ["java", "-jar", "./myApplication/Practice-1-1.0-SNAPSHOT.jar"]
Then I deployed the docker image via AWS Beanstalk. Here is the Dockerrun.aws.json:
{
"AWSEBDockerrunVersion": "1",
"Image": {
"Name": "aliam/backend",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "8080"
}
],
"Logging": "/var/log/nginx"
}
And everything went well:
But now, I'm getting "502 Bad Gateway" in postman when trying to run "backend.us-east-2.elasticbeanstalk.com/health".
I checked the log on Beanstalk and realized that the application has problem connecting to the RDS database:
"Could not create connection to database server. Attempted reconnect 3 times. Giving up."
What I tried to do to solve the problem:
1- I tried to assign the same security group the EC2 instance is using to my RDS instance, but it didn't work.
2- I tried to make more inbound rules on the security group to add public and private IPs of the EC2 instance but I was not sure about the port and the CIDR I should define and couldn't make it.
Any comment would be highly appreciated.

Here are resources in your stack:
LoadBalancer -> EC2 instance(s) -> MySQL database
All of them need to have SecurityGroups assigned to them, allowing connections on the right ports to the upstream resources.
So, if you assign sg-1234 security group to your EC2 instances, and sg-5678 to your RDS database, there must be a rule existing in the sg-5678 allowing inbound connections from sg-1234 (no need for CIDRs, you can open a connection from SG to SG). The typical MySQL port is 3306.
Similarly, the LoadBalancer (which is automatically created for you by ElasticBeanstalk) must have access to your EC2 instance's 8080 port. Furthermore, if you want to access your instances with the "backend.us-east-2.elasticbeanstalk.com/health" domain name, the loadbalancer would have to listen on port 80 and have a target group of your instances on 8080 port.
Hope this helps!

Related

Why does AWS ECS allows inbound traffic to ALL ports by default?

I am deploying the following relatively simple docker-compose.yml file on AWS ECS via the Docker CLI.
It uses tomcat server image which can be also replaced by any other container which does not exits of startup.
services:
tomcat:
image: tomcat:9.0
command: catalina.sh run
ports:
- target: 8080
published: 8080
x-aws-protocol: http
Commands used
docker context use mycontextforecs
docker compose up
The cluster, services, task, target, security groups and application load balancer are automatically created as expected.
But, the security group created by AWS ECS allows inbound traffic on ALL ports by default instead of only the exposed 8080.
Following is a screenshot of the security group, which also has a comment -
"tomcat:8080/ on default network"
But port range is "All" instead of 8080
I've read the following and some other stackoverflow links but could not get an answer.
https://docs.docker.com/cloud/ecs-compose-features/
https://docs.docker.com/cloud/ecs-architecture/
https://docs.docker.com/cloud/ecs-integration/
I understand that the default "Fargate" instance type gets a public ip assigned.
But why does ECS allow traffic on all ports?
If I add another service in the docker-compose file, the default security group gets shared between both of them.
As a result, anyone can telnet into the port exposed by the service due to this security group rule.

My docker container which is running inside AWS ElasticBeanstalk is not able to connect with the host

My application runs on port 5000 and I have exposed the 5000 port in the docker file.
This is my docker-compose.yml file
"services":
"backend":
"image": "<imageURL>"
"ports":
- "5000:8080"
Container port and application port: 5000
Server port: 8080
The security group of have also been configured properly and the application is able to connect with the database but not working as I try to ping that IP of the server.
My application has an ping API.
Not sure what are you referring to with EBS server as "EBS" is Elastic Block Storage, not the computing
If you're using AWS ECS you need to configure "PortMapping" to map external ports with the container ports
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_PortMapping.html
If you're using EC2 make sure that your have your service listening to all IPs using the netstat command
netstat -anlp | grep [your port]
and security group inbound and outbound rules are configured properly

ssh tunneling/port forwarding not working through EC2 instance to an Elasticsearch cluster in a VPC

I have my Elasticsearch cluster in a VPC, I'd like to access this EC cluster from my local Macbook.
I have set up a bastion host that uses the same VPC and the same security group, and I was able to ssh into this bastion host from my Macbook.
But somehow, my code just cannot connect to my ES cluster through this bastion host, here's my command to run port forwarding:
ssh -i ~/Downloads/keypairs/20210402-02.pem ubuntu#ec2-123-456.us-west-2.compute.amazonaws.com -N -L 9200:vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com:443
Here's my timeout exception when accessing the ES cluster in the VPC:
java.net.ConnectException: Timeout connecting to [vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com/10.0.47.182:443]
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:823) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:248) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.6.1.jar:7.6.1]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1514) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1484) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1454) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]
at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:497) ~[elasticsearch-rest-high-level-client-7.6.1.jar:7.6.1]
Here are the rules of my SG:
Inbound:
All TCP TCP 0 - 65535 0.0.0.0/0
All traffic All All sg-abc123 / default
SSH TCP 22 0.0.0.0/0
Outbound:
All traffic All All 0.0.0.0/0
When I've ssh'ed into my bastion host, and run curl vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com, I got this response:
{
"name" : "abc123",
"cluster_name" : "abc123097:es-domain-beta-20210331",
"cluster_uuid" : "abc123def",
"version" : {
"number" : "7.8.0",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "unknown",
"build_date" : "2021-01-15T06:15:47.944536Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
I'm suspecting my command to do port forwarding is not correct? But after research, this looks the most legit option to me.
Any insight would be greatly appreciated!
The code running on your local computer is trying to connect directly to the Elasticsearch server without going through the SSH tunnel. The SSH command is opening a tunnel from your local port 9200 to the remote server. The local software trying to connect to Elasticsearch should be connecting to localhost:9200 not vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com/10.0.47.182:443.
The endpoint vpc-es-domain-20210331-abc123def.us-west-2.es.amazonaws.com/10.0.47.182:443 doesn't look valid anyway. It has a hostname and an IP address in there.
You mentioned in the comments:
"I actually launched my bastion host using the same SG as my ES"
However just placing two resources in the same security group does nothing unless that security group also has a rule specifically allowing traffic between the resources within it. Security groups do not have this rule by default, except for the default security group in the default VPC that is created automatically when you first create your AWS account.
So please make sure that the security group has a rule that will allow the bastion host to connect to the Elasticsearch server over port 443.

Deploying Docker to AWS Elastic Beanstalk -- how to forward port to host? (port binding)

I have a project set up with CircleCI that I am using to auto-deploy to Elastic Beanstalk. My EBS environment is a single container, auto-scaling, web environment. I am trying to run a service that listens on raw socket port 8080.
My Dockerfile:
FROM golang:1.4.2
...
EXPOSE 8080
My Dockerrun.aws.json.template:
{
"AWSEBDockerrunVersion": "1",
"Authentication": {
"Bucket": "<bucket>",
"Key": "<key>"
},
"Image": {
"Name": "project/hello:<TAG>",
"Update": "true"
},
"Ports": [
{
"ContainerPort": "8080"
}
]
}
I have made sure to expose port 8080 on the "role" assigned to my project environment.
I used the exact deployment script from the CircleCI tutorial linked above (except with changed names).
Within the EC2 instance that is running my EBS application, I can see that the Docker container has run successfully, except that Docker did not forward the exposed port to the host container. I have encountered this in the past when I ran docker run .... without the -P flag.
Here is an example session after SSH-ing into the machine:
[ec2-user#ip-xxx-xx-xx-xx ~]$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a036bb061aea aws_beanstalk/staging-app:latest "/bin/sh -c 'go run 3 days ago Up 3 days 8080/tcp boring_hoover
[ec2-user#ip-xxx-xx-xx-xx ~]$ curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused
What I expect to see is the ->8080 or whatever in the container that forwards it onto the host.
When I do docker inspect on my container, I also see that these two configurations are not what I want:
"PortBindings": {},
"PublishAllPorts": false,
How can I trigger a port binding in my application?
Thanks in advance.
It turns out I made a misunderstanding in how Docker's networking stack works. When a port is exposed but not published, it is still available to the local network interface through the Docker container's private IP address. You can obtain this IP address by checking docker inspect <container>.
Rather than doing curl localhost:8080 I could do curl <containerIP>:8080.
In my EBS deploy, nginx was automatically setup to forward (HTTP) traffic from Port 80 to this internal private port as well.
I had the same problem in a rails container (port 3000 using puma) by default rails server only binds localhost to the listening interface, I had to use -b option to bind 0.0.0.0 and that solved the problem.
In react I have no the same problem cause npm serve package binds all interfaces by default

App running in Docker on EB refuses connecting to self

I have a Play 2 web application, which I deploy to Elastic Beanstalk using Docker. In this web app, I start an Akka cluster. The starting procedure involves adding all nodes in the autoscaling group as seed nodes (including itself). On the first deploy to EB I specify to deploy to a VPC (I only select one availability zone).
When I run the app and start the cluster, I get the following message:
AssociationError [akka.tcp://cluster#localhost:2551] -> [akka.tcp://cluster#172.31.13.25:2551]: Error [Invalid address: akka.tcp://cluster#172.31.13.25:2551] [
akka.remote.InvalidAssociation: Invalid address: akka.tcp://cluster#172.31.13.25:2551
Caused by: akka.remote.transport.Transport$InvalidAssociationException: Connection refused: /172.31.13.25:2551
Where 172.31.13.25 is the IP of the EC2 instance, and 2551 is the port.
In my Dockerfile I have "EXPOSE 9000 2551". In the EC2 Security Group I have enabled all inbound traffic to
0.0.0.0/0 (and all outbound traffic). In the VPC Network ACLs (and security groups) I've also opened for all traffic.
This is my Dockerfile
FROM dockerfile/java:latest
MAINTAINER a <a#b.de>
EXPOSE 9000 2551
ADD files /
WORKDIR /opt/docker
RUN ["chown", "-R", "daemon", "."]
USER daemon
ENTRYPOINT ["bin/myapp"]
CMD []
Why does my EC2 instance refuse a connection to itself on port 2551?
Turns out this is not possible as of now using Docker on Elastic Beanstalk.
It is, however, possible using Tomcat.
Using play/activator, you can deploy a WAR file. By injecting the following .ebextensions config file into the war file, I was able to get an extra port open between the EC2 instances:
Resources:
ExtraPortsSGIngress:
Type: AWS::EC2::SecurityGroupIngress
Properties:
GroupId: { "Ref" : "AWSEBSecurityGroup" }
IpProtocol: "tcp"
FromPort: "2551"
ToPort: "2551"
SourceSecurityGroupId: { "Ref" : "AWSEBSecurityGroup" }