How to register containers using networkMode=host automatically in ECS? - amazon-web-services

For performance reasons, we need to use docker networkMode=host in ECS. Under this setting, is it possible to have ECS manage registration/de-registration of container against the ALB/ELB? If not what are some of the typical options use to manage this process?

No! in my experiences it was not possible to have ALB and network mode host and dynamic ports. I'm trying to find documentation that specifies it, but I found out by trying to create a service with networkmode = "host", and a dynamic port (0 ) with an ALB and received a cloudformation error on creation.
My use case was that statsd is running bound to ec2 machine, and I was hoping to be able to deploy ALB service in neworkMode host so it would be easy to reference statsd from the container using localhost.
To get around this with the ALB and bridge networking each ECS container instance has a configuration file put on it with it's IP so the container can avoid having to hit the metadata API to get the ECS container instance IP.

Related

Two ECS tasks connectivity in one EC2 Host

I have 2 ECS Services running with EC2 launch type and bridge networking mode. each service is having 1 task and both tasks are running on same EC2 container host.
on the Same ECS host, API container in 1st task is trying to communicate DB container in 2nd task by host name and port number( ex: http://abc-def:5000). what are the ways to achieve this ?
Your options are:
Put everything in a single task definition. Then you can use the link attribute to enable container A to communicate with container B like so B:port. Notice that link does not support circular relations meaning if A can talk with B, B will not be able to do that as well.
Switch to network mode host. This way you can communicate with localhost.
Proper service discovery with a tool like Consul or AWS ECS Service Discovery. I have no experience with the latter. See here.
Put your tasks behind ALB and use this load balancer to communicate between tasks.

Exposing to public more than 1 port with AWS ECS service and Elastic LoadBalancer

I have service that exposes multiple ports and it worked fine with kubernetes but now we move it to AWS ECS. It seems I can only expose ports via Load Balancer and I am limited to 1 port per service/tasks even when docker defines multiple ports I have to choose one port
Add to load balancer button allows to add one port. Once added there is no button to add second port.
Is there any nicer workarround than making second proxy service to expose second port?
UPDATE: I use fargate based service.
You don't need any workaround, AWS ECS now supports multiple target groups within the same ECS service. This will be helpful for the use-cases where you wanted to expose multiple ports of the containers.
Currently, if you want to create a service specifying multiple target groups, you must create the service using the Amazon ECS API, SDK, AWS CLI, or an AWS CloudFormation template. After the service is created, you can view the service and the target groups registered to it with the AWS Management Console.
For example, A Jenkins container might expose port 8080 for the
Jenkins web interface and port 50000 for the API.
Ref:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/register-multiple-targetgroups.html
https://aws.amazon.com/about-aws/whats-new/2019/07/amazon-ecs-services-now-support-multiple-load-balancer-target-groups/
Update:
I was able to configure target group using Terraform but did not find so far this option on AWS console.
resource "aws_ecs_service" "multiple_target_example" {
name = "multiple_target_example1"
cluster = "${aws_ecs_cluster.main.id}"
task_definition = "${aws_ecs_task_definition.with_lb_changes.arn}"
desired_count = 1
iam_role = "${aws_iam_role.ecs_service.name}"
load_balancer {
target_group_arn = "${aws_lb_target_group.target2.id}"
container_name = "ghost"
container_port = "3000"
}
load_balancer {
target_group_arn = "${aws_lb_target_group.target2.id}"
container_name = "ghost"
container_port = "3001"
}
depends_on = [
"aws_iam_role_policy.ecs_service",
]
}
Version note:
Multiple load_balancer configuration block support was added in Terraform AWS Provider version 2.22.0.
ecs_service_terraform
I can't say that this will be a nice workaround but I was working on a project where I need to run Ejabberd using AWS ECS but the same issue happened when its come to bind port of the service to the load balancer.
I was working with terraform and due to this limitation of AWS ECS, we agree to run one container per instance to resolve the port issue as we were supposed to expose two port.
If you do not want to assign a dynamic port to your container and you want to run one container per instance then the solution will definitely work.
Create a target group and specify the second port of the container.
Go to the AutoScalingGroups of your ECS cluster
Edit and add the newly created target group of in the Autoscaling group of the ECS cluster
So if you scale to two containers it's mean there will be two instances so the newly launch instance will register to the second target group and Autoscaling group take care of it.
This approach working fine in my case, but few things need to be consider.
Do not bind the primary port in target, better to bind primary port in
ALB service. The main advantage of this approach will be that if your
container failed to respond to AWS health check the container will be
restart automatically. As the target groupe health check will not recreate your container.
This approach will not work when there is dynamic port expose in Docker container.
AWS should update its ECS agent to handle such scenario.
I have faced this issue while creating more than one container per instances and second container was not coming up because it was using the same port defined in the taskdefinition.
What we did was, Created an Application Load balancer on top of these containers and removed hardcoded ports. What application load balancer does when it doesn't get predefined ports under it is, Use the functionality of dynamic port mapping. Containers will come up on random ports and reside in one target group and the load balancer will automatically send the request to these ports.
More details can be found here
Thanks to mohit's answer, I used AWS CLI to register multiple target groups (multiple ports) into one ECS service:
ecs-sample-service.json
{
"serviceName": "sample-service",
"taskDefinition": "sample-task",
"loadBalancers":[
{
"targetGroupArn":"arn:aws:elasticloadbalancing:us-west-2:0000000000:targetgroup/sample-target-group/00000000000000",
"containerName":"faktory",
"containerPort":7419
},
{
"targetGroupArn":"arn:aws:elasticloadbalancing:us-west-2:0000000000:targetgroup/sample-target-group-web/11111111111111",
"containerName":"faktory",
"containerPort":7420
}
],
"desiredCount": 1
}
aws ecs create-service --cluster sample-cluster --service-name sample-service --cli-input-json file://ecs-sample-service.json --network-configuration "awsvpcConfiguration={subnets=[subnet-0000000000000],securityGroups=[sg-00000000000000],assignPublicIp=ENABLED}" --launch-type FARGATE
If the task needs internet access to pull image, make sure subnet-0000000000000 has internet access.
Security group sg-00000000000000 needs to give relevant ports inbound access. (in this case, 7419 and 7420).
If the traffic only comes from ALB, the task does not need public IP. Then assignPublicIp can be false.
Usually, I use the AWS CLI method itself by creating task def, target groups and attaching them to the application load balancer. But the issue is that when there is multiple services to be done this is a time-consuming task so I would use terraform to create such services
terraform module link This is a multiport ECS service with Fargate deployment. currently, this supports only 2 ports. when using multiports with sockets this socket won't be sending any response so the health check might fail.
so to fix that I would override the port in the target group to other ports and fix that.
hope this helps

Communication Between Microservices in Aws Ecs

I'm having troubles with the communication between microservices. I have many spring boot applications, and many requests HTTP and AMQP (RabbitMQ) between them. Locally (in dev) I use Eureka (Netflix Oss) without Docker Images.
The question is: in the Amazon ECS Infraestructure how i can work with the same behavior? Which is the common pratice for the communication between microservices using Docker? I can still use Eureka for Service Discovery? Besides that, how this comunication will works between container instances?
I'd suggest reading up on ECS Service Load Balancing, in particular two points:
Your ECS service configuration may say that you let ECS or the EC2 instance essentially pick what port number the service runs on externally. (ie for Spring Boot applications inside the Docker container your application thinks it's running on port 8080, but in reality to anything outside the Docker container it may be running on port 1234
2 ECS clusters will check the health endpoint you defined in the load balancer, and kill / respawn instances of your service that have died
A load balancer gives you different ways to specify which application in the cluster you are talking to. This can be route based or DNS name based (and maybe some others). Thus http://myservice.example.com/api could point to a different ECS service than http://myservice.exaple.com/app... or http://app.myservice.example.com vs http://api.myservice.example.com.
You can configure ECS without a load balancer, I'm unsure how well that will work in this situation.
Now, you're talking service discovery. You can still use Eureka for service discovery, having Spring Boot take care of that. You may need to be clever about how you tell Eureka where your service lives (as the hostname inside the Docker container may be useless, and the port number inside the container totally will be too.) You may need to do something clever here to correctly derive that number, like introspecting using AWS APIs. I think this SO answer describes it correctly, or at least close enough to get started.
Additionally apparently ECS has service discovery built in now. This is either new since I last used ECS, or we didn't use it because we had other solutions. If you aren't completely tied to Eureka for other reasons.
Thanks for you reply. For now I'm using Eureka because I'm using Feign too for comunication between microservices.
My case is this: I have microservices (Example A,B,C). A communicates with B and C by way of Feign (Rest).
Microservices Example
Example of Code on Microservice A:
#FeignClient("b-service")
public interface BFeign {
}
#FeignClient("c-service")
public interface CFeign {
}
Using ECS and ALB it's possible still using Feign? If yes or not, how you suggest I do this?

Eureka with AWS ECS

We are using Eureka with AWS ECS service that can scale docker containers.
In ECS if you leave out the host port, or specify it as being '0', in your task definition, then the port will be chosen automatically and reported back to the service. After the task is running, describing it should show what port(s) it bound to.
How does Eureka can resolve what port to use if we have several EC2 instance. For example Service A from EC2-A try to call Service B from EC2-B. So Eureka can resolve hostname , but cannot identify exposed port
Hi #Aleksandr Filichkin,
I don't think Application Load Balancer and service registry does the same.
The main difference traffic flows over the (application) load balancer whereas the service registry just gives you a healthy endpoint that your client directly can address (so the network traffic does not flow over the service registry).
Cheap is a very relative term, maybe it's cheap for some, maybe it's an unnecessary overhead for others.
The issue was resolved
https://github.com/Netflix/eureka/issues/937
Currently ECS agent knows about running port.
But I don't recommend to use Eureka with ECS, because Application Load Balancer does the same. It works as service registry and discovery. You don't need to run addition service(Eureka), ALB is cheap.
There is another solution.
You can create an application loadbalancer and a target group, in which the docker containers can be launched.
Every docker container has set their hostname to the hostname of the loadbalancer. If you need a pretty url, then you can utilize Route53 for DNS-Routing.
It looks like this:
Service Discovery with Loadbalancer-Hostname
Request Flow
If you have two containers of the same task on different hosts, both will communicate the same loadbalancer hostname to eureka.
With this solution you can use eureka with docker on AWS ECS without loosing the advantages and flexibility of dynamic port mapping.

How to deploy continuously using just One EC2 instance with ECS

I want to deploy my nodejs webapp continuously using just One EC2 instance with ECS. I cannot create multiple instances for this app.
My current continuous integration process:
Travis build the code from github, build tag and push docker image and deployed to ECS via ECS Deploy shell script.
Everytime the deployment happen, following error occurs. Because the port 80 is always used by my webapp.
The closest matching container-instance ffa4ec4ccae9
is already using a port required by your task
Is it actually possible to use ECS with one instance? (documentation not clear)
How to get rid of this port issue on ECS? (stop the running container)
What is the way to get this done without using a Load Balancer?
Anything I missed or doing apart from the best practises?
The main issue is the port conflict, which occurs when deploying a second instance of the task on the same node in the cluster. Nothing should stop you from having multiple container instances apart from that (e.g. when not using a load balancer; binding to any ports at all).
To solve this issue, Amazon introduced a dynamic ports feature in a recent update:
Dynamic ports makes it easier to start tasks in your cluster without having to worry about port conflicts. Previously, to use Elastic Load Balancing to route traffic to your applications, you had to define a fixed host port in the ECS task. This added operational complexity, as you had to track the ports each application used, and it reduced cluster efficiency, as only one task could be placed per instance. Now, you can specify a dynamic port in the ECS task definition, which gives the container an unused port when it is scheduled on the EC2 instance. The ECS scheduler automatically adds the task to the application load balancer’s target group using this port. To get started, you can create an application load balancer from the EC2 Console or using the AWS Command Line Interface (CLI). Create a task definition in the ECS console with a container that sets the host port to 0. This container automatically receives a port in the ephemeral port range when it is scheduled.
Here's a way to do it using the green/blue deployment pattern:
Host your containers on port 8080 & 8081 (or whatever port you want). Let's call 8080 green and 8081 blue. (You may have to switch the networking mode from bridge to host to get this to work on a single instance).
Use Elastic Load Balancing to redirect the traffic from 80/443 to green or blue.
When you deploy, use a script to swap the active listener on the ELB to the other color/container.
This also allows you to roll back to a 'last known good' state.
See http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-load-balancing.html for more information.