When developing locally, I run docker-compose, where I have two services Service1 and Service2. Service2 depends on Service1. When I deploy them to ECS, I create them within one task definition and provide JSON array of container definitions to spin them up.
When I run them locally, within docker-compose, from Service2 I can call http://Service1:8080/v1/graphql (since they're in docker-compose together I can call it by the service name) ... however, when I deploy to ECS and I make that same API call, I get a 404.
Based on this: Docker links with awsvpc network mode I've also tried http://localhost:8080/v1/graphql ... I'd appreciate any help!
I'd try service discovery as mentioned here:
Amazon ECS now includes integrated service discovery. This makes it possible for an ECS service to automatically register itself with a predictable and friendly DNS name in Amazon Route 53. As your services scale up or down in response to load or container health, the Route 53 hosted zone is kept up to date, allowing other services to lookup where they need to make connections based on the state of each service.
See an example here.
Related
I have three docker containers all running in their own Tasks (3 tasks), and each task running in a separate Fargate service (3 services) on ECS Fargate. I have all the services behind an application load balancer (ALB) with path based routing setup. Below is how the path based routing works:
example.com/app_one forwards traffic to service_one_target_group
example.com/app_two forwards traffic to service_two_target_group
example.com/app_three forwards traffic to service_three_target_group
One thing to note. app_one & app_two are Node JS apps and app_three is a middleware, GraphQL server used to connect to a database.
I need the the services for app_one & app_two to be able to discover the app_three service.
I know Service Discovery is an option but I am unsure how to implement the Service Discovery in a scenario with path based routing. Any advice would be helpful.
I have read other questions about this that all mention enabling service discovery, but my issue is a little different as to how to go about setting this up for my current Fargate deployments.
I have four spring boot api containers built via Gradle, pushed to ECR, and deployed in ECS Fargate with Terraform IaC setting up the appropriate resources. Three of these containerized apis have environment variables set within them to reference the fourth container, thus making an external api call outside of the container to that one service. DNS and 443 load balancer is setup for these deployments.
I have created a new service in the cluster containing the api that needs to be discovered. I have enabled service discover and created a local CloudMap A record for the api and then set each environment variable in the other containzers to use that local A record url, e.g., ecsservicename.local. Additionally I have tried to dig the service that I am connecting to in the other apis and that returns an IP so I am sure that that is working.
My questions are as follows:
(1) Since really only one services should be picked up by the others, was it correct to set service discovery on that one api and not the others or should I set up service discovery on all the other apis?
(2) Even if route53 is setup should this be an A record or SRV? I was confused by the documentation as to when to use which on aws.
(3) Is there a better or easier approach to use for inter-container communication that I am missing?
That's correct. Discovery should be set only for the one service. Other discoveries are not needed, as you are not inter-connection to those other services.
SRV also gives port, so from docs:
if the task definition that your service task specifies uses the bridge or host network mode, an SRV record is the only supported DNS record type.
I think your architecture is well thought and can't think of anything "easier" or better.
I have dockersied our webapp into two different docker files, such as:
1.Frontend
2.Backend
Both docker apps have their own .env files , i have to defined deployed server Ip address to connect them.
In frontend-env : configure backend ip
but deploying in ECS service, each container will be different ip ,how to solve this issue scale out and still connect the service each other.
so far :
Create spreate ecs cluster for both frontend and backend, with ALB.
Give the ALB in env files to connect them or hit the api.
any other solutions for this deployment?
You should be using Service Discovery to achieve this. You can read the announcement blog here. In a nutshell the way it works is that if you have two ECS services frontend and backend you want to expose frontend with an ALB for public access but enabling service discovery you will be able to have all tasks that belong to frontend to be able to connect to the tasks that belong to backend by calling backend.<domain> (where is the SD namespace you defined). When you do so ECS Service Discovery will resolve backend.<domain> to the private IP addresses of the tasks in backend thus eliminating the need for a load balancer in front of it.
If you want a practical example of how this works you can investigate this basic demo app:
We are using Eureka with AWS ECS service that can scale docker containers.
In ECS if you leave out the host port, or specify it as being '0', in your task definition, then the port will be chosen automatically and reported back to the service. After the task is running, describing it should show what port(s) it bound to.
How does Eureka can resolve what port to use if we have several EC2 instance. For example Service A from EC2-A try to call Service B from EC2-B. So Eureka can resolve hostname , but cannot identify exposed port
Hi #Aleksandr Filichkin,
I don't think Application Load Balancer and service registry does the same.
The main difference traffic flows over the (application) load balancer whereas the service registry just gives you a healthy endpoint that your client directly can address (so the network traffic does not flow over the service registry).
Cheap is a very relative term, maybe it's cheap for some, maybe it's an unnecessary overhead for others.
The issue was resolved
https://github.com/Netflix/eureka/issues/937
Currently ECS agent knows about running port.
But I don't recommend to use Eureka with ECS, because Application Load Balancer does the same. It works as service registry and discovery. You don't need to run addition service(Eureka), ALB is cheap.
There is another solution.
You can create an application loadbalancer and a target group, in which the docker containers can be launched.
Every docker container has set their hostname to the hostname of the loadbalancer. If you need a pretty url, then you can utilize Route53 for DNS-Routing.
It looks like this:
Service Discovery with Loadbalancer-Hostname
Request Flow
If you have two containers of the same task on different hosts, both will communicate the same loadbalancer hostname to eureka.
With this solution you can use eureka with docker on AWS ECS without loosing the advantages and flexibility of dynamic port mapping.
I'm trying to setup a basic web application, which has an associated database, in AWS ECS. Locally I have these setup in different containers, and on ECS, I'd like to have separate task definitions so that I may scale the two separately.
I registered my first task definition as david_mongodb successfully in ECS. It has a container named david_mongodb in it.
Then I attempted to register my second task definition as david_web, which has a container named david_web that links the database via david_mongodb:db.
When I click 'Create', it returns an error:
Unable to create Task Definition
Linked container 'david_mongodb:db' doesn't exist.
It seems like task definitions can't see container names in other task definitions? I'm thinking putting both david_web and david_mongodb containers in the same task definition would work, but I don't want to do that: it would prevent me from scaling either web app or database separately. This overview seems to confirm that my architecture is recommended...
So how do I link containers that live in different task definitions? Or is there another clever way of handling this?
Links in an ECS task definition are analogous to Docker links and only work when the containers are part of the same task definition (containers that are part of a single task definition are placed together on the same host). In order to communicate between containers in different task definitions, you'll need a mechanism for discovering where the containers are located (what host) as well as the port for communication.
ECS has integration with Elastic Load Balancing (Application Load Balancers, Network Load Balancers, and Classic Load Balancers) through the service feature, where tasks will be automatically registered in the ELB and deregistered in the ELB appropriately.
ECS also has integration with Route 53 Auto Naming for DNS-based service discovery using A and SRV records. Your service's tasks can be automatically entered into and removed from DNS records.
Service Discovery for Amazon ECS Using DNS describes a different approach where a Lambda function listens to the ECS event stream through CloudWatch Events and updates Route 53 DNS records. This method has been superceded by the Route 53 Auto Naming feature described above.
If you want to avoid load balancers and DNS, another pattern might be an ambassador container (there's a sample called the ecs-task-kite that uses the ECS API) or you might be interested in an overlay network (Weave has a fairly detailed getting started guide for their solution).
Nathan Peck is keeping track of a number of different subjects related to ECS, including service discovery, here.
you could now refer to this official best practices guide on networking between Amazon ECS services in a VPC, discussing on the considerations when adopting service discovery, ELB or service mesh for service-to-service communication with ECS.