I want to deploy my nodejs webapp continuously using just One EC2 instance with ECS. I cannot create multiple instances for this app.
My current continuous integration process:
Travis build the code from github, build tag and push docker image and deployed to ECS via ECS Deploy shell script.
Everytime the deployment happen, following error occurs. Because the port 80 is always used by my webapp.
The closest matching container-instance ffa4ec4ccae9
is already using a port required by your task
Is it actually possible to use ECS with one instance? (documentation not clear)
How to get rid of this port issue on ECS? (stop the running container)
What is the way to get this done without using a Load Balancer?
Anything I missed or doing apart from the best practises?
The main issue is the port conflict, which occurs when deploying a second instance of the task on the same node in the cluster. Nothing should stop you from having multiple container instances apart from that (e.g. when not using a load balancer; binding to any ports at all).
To solve this issue, Amazon introduced a dynamic ports feature in a recent update:
Dynamic ports makes it easier to start tasks in your cluster without having to worry about port conflicts. Previously, to use Elastic Load Balancing to route traffic to your applications, you had to define a fixed host port in the ECS task. This added operational complexity, as you had to track the ports each application used, and it reduced cluster efficiency, as only one task could be placed per instance. Now, you can specify a dynamic port in the ECS task definition, which gives the container an unused port when it is scheduled on the EC2 instance. The ECS scheduler automatically adds the task to the application load balancer’s target group using this port. To get started, you can create an application load balancer from the EC2 Console or using the AWS Command Line Interface (CLI). Create a task definition in the ECS console with a container that sets the host port to 0. This container automatically receives a port in the ephemeral port range when it is scheduled.
Here's a way to do it using the green/blue deployment pattern:
Host your containers on port 8080 & 8081 (or whatever port you want). Let's call 8080 green and 8081 blue. (You may have to switch the networking mode from bridge to host to get this to work on a single instance).
Use Elastic Load Balancing to redirect the traffic from 80/443 to green or blue.
When you deploy, use a script to swap the active listener on the ELB to the other color/container.
This also allows you to roll back to a 'last known good' state.
See http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html for more information.
Related
I have 2 ECS Services running with EC2 launch type and bridge networking mode. each service is having 1 task and both tasks are running on same EC2 container host.
on the Same ECS host, API container in 1st task is trying to communicate DB container in 2nd task by host name and port number( ex: http://abc-def:5000). what are the ways to achieve this ?
Your options are:
Put everything in a single task definition. Then you can use the link attribute to enable container A to communicate with container B like so B:port. Notice that link does not support circular relations meaning if A can talk with B, B will not be able to do that as well.
Switch to network mode host. This way you can communicate with localhost.
Proper service discovery with a tool like Consul or AWS ECS Service Discovery. I have no experience with the latter. See here.
Put your tasks behind ALB and use this load balancer to communicate between tasks.
We are using Eureka with AWS ECS service that can scale docker containers.
In ECS if you leave out the host port, or specify it as being '0', in your task definition, then the port will be chosen automatically and reported back to the service. After the task is running, describing it should show what port(s) it bound to.
How does Eureka can resolve what port to use if we have several EC2 instance. For example Service A from EC2-A try to call Service B from EC2-B. So Eureka can resolve hostname , but cannot identify exposed port
Hi #Aleksandr Filichkin,
I don't think Application Load Balancer and service registry does the same.
The main difference traffic flows over the (application) load balancer whereas the service registry just gives you a healthy endpoint that your client directly can address (so the network traffic does not flow over the service registry).
Cheap is a very relative term, maybe it's cheap for some, maybe it's an unnecessary overhead for others.
The issue was resolved
https://github.com/Netflix/eureka/issues/937
Currently ECS agent knows about running port.
But I don't recommend to use Eureka with ECS, because Application Load Balancer does the same. It works as service registry and discovery. You don't need to run addition service(Eureka), ALB is cheap.
There is another solution.
You can create an application loadbalancer and a target group, in which the docker containers can be launched.
Every docker container has set their hostname to the hostname of the loadbalancer. If you need a pretty url, then you can utilize Route53 for DNS-Routing.
It looks like this:
Service Discovery with Loadbalancer-Hostname
Request Flow
If you have two containers of the same task on different hosts, both will communicate the same loadbalancer hostname to eureka.
With this solution you can use eureka with docker on AWS ECS without loosing the advantages and flexibility of dynamic port mapping.
For performance reasons, we need to use docker networkMode=host in ECS. Under this setting, is it possible to have ECS manage registration/de-registration of container against the ALB/ELB? If not what are some of the typical options use to manage this process?
No! in my experiences it was not possible to have ALB and network mode host and dynamic ports. I'm trying to find documentation that specifies it, but I found out by trying to create a service with networkmode = "host", and a dynamic port (0 ) with an ALB and received a cloudformation error on creation.
My use case was that statsd is running bound to ec2 machine, and I was hoping to be able to deploy ALB service in neworkMode host so it would be easy to reference statsd from the container using localhost.
To get around this with the ALB and bridge networking each ECS container instance has a configuration file put on it with it's IP so the container can avoid having to hit the metadata API to get the ECS container instance IP.
I have three EC2 instances, with a classic load balancer. Ideally I should have two tasks running in two instances. So when creating the service I made the desired count of the tasks to 2.
My problem arises when I try to run new version of the task definition. I update the service to run the new task definition. So it should theoretically run two updated tasks replacing the old ones, since i have three ec2 running.
What happens actually is only one updated task is running together with the old tasks. So altogether 3 tasks running even though the desired count is set to 2, as you are able to see in the given image.
Does anyone know a solution for this ?
When using a classic load balancer, you can only map static ports on the ec2 instance.
Your deployment settings are:
min-health: 100%
max-healthy: 200%
The new version of the service would require two more hosts available with the free tcp port you requested. Since you only have 3 servers in the cluster, this condition will not be satisfied. You can either add more servers to your cluster, or use the Application Load Balancer (ALB) which will integrate with docker dynamic port mapping.
Update regarding security groups:
To manage security groups, you can tag a security group with another. For example, tag your ALB with 'app-gateway-alb' which allows specific ports from outside your network, then on the container have a security group which allows ANY TCP from 'app-gateway-alb' this is achieved by putting the security group ID in the text box where you would generally put the CIDR rule.
I have one ecs task definition. Can multiple tasks of the above task definition can run on one EC2 instance on several ports ?
I have already have running several tasks running on several ec2 instances. I want to reduce the consumption of resources. As one EC2 instance has more than enough resources to run several tasks. Is it possible to run same task several times on different ports on one EC2 instance ?
Yes, ECS has very good support for this since 2016. You can leave the host port empty in the container definition, this will result in a random port to be chosen for your container. As a result, more instances of the same task definition can run on one ECS instance.
You can configure your ECS service in combination with an Application Load Balancer so that when it starts a new task, it will register the port number in the associated target group. This way you never have to deal with the random port.
If you setup your service via the AWS console, configuration is pretty straightforward.
This can be configured by setting Host Port to be 0 in Port Mappings setting of Container Definitions while defining Task.
Following setting is available in Container Definition of Task.
It allows ECS to assign random ports to Tasks running in same EC2.
For more details please check - Setup Dynamic Port Mapping for ECS