I have three EC2 instances, with a classic load balancer. Ideally I should have two tasks running in two instances. So when creating the service I made the desired count of the tasks to 2.
My problem arises when I try to run new version of the task definition. I update the service to run the new task definition. So it should theoretically run two updated tasks replacing the old ones, since i have three ec2 running.
What happens actually is only one updated task is running together with the old tasks. So altogether 3 tasks running even though the desired count is set to 2, as you are able to see in the given image.
Does anyone know a solution for this ?
When using a classic load balancer, you can only map static ports on the ec2 instance.
Your deployment settings are:
min-health: 100%
max-healthy: 200%
The new version of the service would require two more hosts available with the free tcp port you requested. Since you only have 3 servers in the cluster, this condition will not be satisfied. You can either add more servers to your cluster, or use the Application Load Balancer (ALB) which will integrate with docker dynamic port mapping.
Update regarding security groups:
To manage security groups, you can tag a security group with another. For example, tag your ALB with 'app-gateway-alb' which allows specific ports from outside your network, then on the container have a security group which allows ANY TCP from 'app-gateway-alb' this is achieved by putting the security group ID in the text box where you would generally put the CIDR rule.
Related
I use AWS at work and I am fairly new to this.
I have multiple Services with one Task/Container running. Each Container is fundamentally the same with a few changes, it's basically for different stages/deployments. I have one target group for each, so my load balancer routes requests from specific domains to each.
For example: if host is example1.com then forward to exampleTargetGroup1 and so on.
The Problem
As you may know each time a container is updated, its IP changes, hence I have to re-register the new IP to the target group
I have found several approaches to this problem. Most of them suggest to use a Network Load Balancer for a static IP, but this doesn't work because, as I understand it, it registers the containers automatically on updates.
Another solution is to trigger a Lambda function on a cloud watch events when the Task is being updated. The function grabs the IP and updates the Route53 record. My Idea was to take this approach and deregister the old IP in the target group and register the new one.
My Questions
Is there a better solution to this or did I understand the first solutions wrong? If the last solution is optimal for my problem is there maybe a code sample so I won't need to figure it out?
EDIT:
Thanks to Mark B I now know, you should preferably use the AWS API or a tool like Terraform to create an ECS Service and associating a target group to it.
"but this doesn't work because, as I understand it, it registers the
containers automatically on updates."
I think you are misunderstanding something here. Each ECS service should be associated with a load balancer Target Group. Whenever the service creates a task, the service will automatically add that task's IP to the target group. Whenever the service removes a task, it will also remove that task's IP from the target group. This works with both Network Load Balancers and Application Load Balancers.
You stated the following:
"I have multiple Services with one Task/Container running"
So you have one task per service, and one service per target group. From your description, your architecture should look like this:
One load Balancer with multiple domains pointing at it.
In the Load Balancer listener configuration, you have each domain configured to route to a different target group.
Each ECS service configured with a task count of 1
Load balancer -> domain name 1 -> target group 1 -> ECS service 1 -> ECS task 1
Load balancer -> domain name 2 -> target group 2 -> ECS service 2 -> ECS task 2
Load balancer -> domain name 3 -> target group 3 -> ECS service 3 -> ECS task 3
etc...
In the above scenario, as long as you have each ECS service configured with the appropriate target group, each time that service redeploys a task it will automatically update the target group to point to the updated task.
In other words ECS will "dynamically register the IP to target group", exactly like you are wanting.
So I am trying to get my AWS setup working with DNS.
I have 2 instances (currently).
I have 4 task definitions. 3 of these need to run on port 80/443, however all on separate subdomains.
Currently if I stop/start a task, it can end up on either of my instances. This causes issues with the subdomain DNS potentially being pointed in the wrong places.
I imagine I need to setup some kind of load balancer to point the DNS at, but unsure how to get that to route through to the correct tasks.
So my questions:
Do I need a single load balancer, or one per 'task / subdomain'?
How do I handle the ports to go from a set source port, to one of any number of destination ports (if I end up having multiple containers running the same task)
Am I over complicating this massively, or is there a simpler way to achieve this?
Do I need a single load balancer, or one per 'task / subdomain'?
You can have a single application load balancer and three target groups for Api, Site and Web App. Then you can do a rule base routing in the load balancer listener as shown in the following screenshot.
Ref: http://docs.aws.amazon.com/elasticloadbalancing/latest/application/listener-update-rules.html
You can then map your domains www.domain.com and app.domain.com to the load balancer
How do I handle the ports to go from a set source port, to one of any number of destination ports (if I end up having multiple containers running the same task)
When you create services for your task definitions in ECS you can configure load balancing using the target groups you created.
Ref: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-service.html (Check on "Configuring Your Service to Use a Load Balancer")
I have one ecs task definition. Can multiple tasks of the above task definition can run on one EC2 instance on several ports ?
I have already have running several tasks running on several ec2 instances. I want to reduce the consumption of resources. As one EC2 instance has more than enough resources to run several tasks. Is it possible to run same task several times on different ports on one EC2 instance ?
Yes, ECS has very good support for this since 2016. You can leave the host port empty in the container definition, this will result in a random port to be chosen for your container. As a result, more instances of the same task definition can run on one ECS instance.
You can configure your ECS service in combination with an Application Load Balancer so that when it starts a new task, it will register the port number in the associated target group. This way you never have to deal with the random port.
If you setup your service via the AWS console, configuration is pretty straightforward.
This can be configured by setting Host Port to be 0 in Port Mappings setting of Container Definitions while defining Task.
Following setting is available in Container Definition of Task.
It allows ECS to assign random ports to Tasks running in same EC2.
For more details please check - Setup Dynamic Port Mapping for ECS
I want to deploy my nodejs webapp continuously using just One EC2 instance with ECS. I cannot create multiple instances for this app.
My current continuous integration process:
Travis build the code from github, build tag and push docker image and deployed to ECS via ECS Deploy shell script.
Everytime the deployment happen, following error occurs. Because the port 80 is always used by my webapp.
The closest matching container-instance ffa4ec4ccae9
is already using a port required by your task
Is it actually possible to use ECS with one instance? (documentation not clear)
How to get rid of this port issue on ECS? (stop the running container)
What is the way to get this done without using a Load Balancer?
Anything I missed or doing apart from the best practises?
The main issue is the port conflict, which occurs when deploying a second instance of the task on the same node in the cluster. Nothing should stop you from having multiple container instances apart from that (e.g. when not using a load balancer; binding to any ports at all).
To solve this issue, Amazon introduced a dynamic ports feature in a recent update:
Dynamic ports makes it easier to start tasks in your cluster without having to worry about port conflicts. Previously, to use Elastic Load Balancing to route traffic to your applications, you had to define a fixed host port in the ECS task. This added operational complexity, as you had to track the ports each application used, and it reduced cluster efficiency, as only one task could be placed per instance. Now, you can specify a dynamic port in the ECS task definition, which gives the container an unused port when it is scheduled on the EC2 instance. The ECS scheduler automatically adds the task to the application load balancer’s target group using this port. To get started, you can create an application load balancer from the EC2 Console or using the AWS Command Line Interface (CLI). Create a task definition in the ECS console with a container that sets the host port to 0. This container automatically receives a port in the ephemeral port range when it is scheduled.
Here's a way to do it using the green/blue deployment pattern:
Host your containers on port 8080 & 8081 (or whatever port you want). Let's call 8080 green and 8081 blue. (You may have to switch the networking mode from bridge to host to get this to work on a single instance).
Use Elastic Load Balancing to redirect the traffic from 80/443 to green or blue.
When you deploy, use a script to swap the active listener on the ELB to the other color/container.
This also allows you to roll back to a 'last known good' state.
See http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html for more information.
I want to be able to use an ALB (ELBv2) to route traffic to multiple port mappings that are exposed by a task of a given service.
Example --
Service A is composed of 1 Task running with Task Definition B.
Task Definition B has one 'Container' which internally runs two daemons on two different port numbers (port 8000 and port 9000, both TCP). Thus, Task Definition B has two ports that need to be mapped to the ALB.
I'm not too worried about the ports that the ALB exposes (they don't have to be 8000 and 9000, but will help if they were).
my-lb-dns.com:8000 -> myservice:8000
my-lb-dns.com:9000 -> myservice:9000
Any ideas on how to create multiple listeners and target groups to achieve this? Nothing in the Console UI is allowing me to do this, and the API has not been very helpful either.
After speaking with AWS support, it appears that the ECS service is geared toward micro-services that are expected to expose only one port.
Having an ECS Service use an Application Load Balancer to map two or more ports isn't supported.
Of course, an additional Load Balancer can be manually added by configuring the appropriate target groups etc., but ECS will not automatically update the configuration when services are updated or scaled up, and also when the underlying container instances change.