I have read other questions about this that all mention enabling service discovery, but my issue is a little different as to how to go about setting this up for my current Fargate deployments.
I have four spring boot api containers built via Gradle, pushed to ECR, and deployed in ECS Fargate with Terraform IaC setting up the appropriate resources. Three of these containerized apis have environment variables set within them to reference the fourth container, thus making an external api call outside of the container to that one service. DNS and 443 load balancer is setup for these deployments.
I have created a new service in the cluster containing the api that needs to be discovered. I have enabled service discover and created a local CloudMap A record for the api and then set each environment variable in the other containzers to use that local A record url, e.g., ecsservicename.local. Additionally I have tried to dig the service that I am connecting to in the other apis and that returns an IP so I am sure that that is working.
My questions are as follows:
(1) Since really only one services should be picked up by the others, was it correct to set service discovery on that one api and not the others or should I set up service discovery on all the other apis?
(2) Even if route53 is setup should this be an A record or SRV? I was confused by the documentation as to when to use which on aws.
(3) Is there a better or easier approach to use for inter-container communication that I am missing?
That's correct. Discovery should be set only for the one service. Other discoveries are not needed, as you are not inter-connection to those other services.
SRV also gives port, so from docs:
if the task definition that your service task specifies uses the bridge or host network mode, an SRV record is the only supported DNS record type.
I think your architecture is well thought and can't think of anything "easier" or better.
Related
Consider a scenario with two identical Cloud Run services ("Real Time App") deployed in two different regions, let's say EU and US. These services use Firestore for real time communication, and minimizing latency is important. Since Firestore only allows specifying one region per project, each Cloud Run service is deployed in its own project and uses its regional Firestore instance. It is not needed for a service in the US to access Firestore data in the EU and vice versa.
Is there a way to deploy a global HTTPS load balancer to route requests to Cloud Run service closest to the user when services are defined in different projects?
I attempted a setup with a shared VPC between a Host "Global" project (in the US) and 2 service projects (EU and US). I created a Cloud Run Service, Network Endpoint Group (NEG), and Backend Service in each regional project. I then attempted to create a Global forwarding rule, Target HTTPs proxy, and URL Map in the host project. However, the URL Map cannot be fed a backend service in another project, complaining that:
Cross-project references for this resource are not allowed.
Indeed, per the Shared VPC Architecture and Cross-project service referencing section of the documentation it seems that:
Cross-project service referencing is not supported for the global external HTTP(S) load balancer
and that, if I understood correctly, the following rules apply:
The NEG must be defined in the same project as the Cloud Run Service
The Backend Service must be in the same project as the NEG
The Target HTTP(s) Proxy and associating URL map must be in the same project as the Backend Service
The Forwarding Rule must be in the same project as the Backend Service
essentially requiring the entire chain to be defined in one project.
Are there recommended workarounds for this scenario?
One solution I can think of is to create a "Router" Cloud Run Service in the Host global project behind a load balancer, with multi region deployment. Its sole purpose is to respond to the client with the regional URL endpoint of the closest "Real Time App" Cloud Run service.
I am wondering whether there is a more elegant solution, though.
I'd like to have my Google Cloud Run services privately communicate with one another over non-HTTP and/or without having to add bearer authentication in my code.
I'm aware of this documentation from Google which describes how you can do authenticated access between services, although it's obviously only for HTTP.
I think I have a general idea of what's necessary:
Create a custom VPC for my project
Enable the Serverless VPC Connector
What I'm not totally clear on is:
Is any of this necessary? Can Cloud Run services within the same project already see each other?
How do services address one another after this?
Do I gain the ability to use simpler by-convention DNS names? For example, could I have each service in Cloud Run manifest on my VPC as a single first level DNS name like apione and apitwo rather than a larger DNS name that I'd then have to hint in through my deployments?
If not, is there any kind of mechanism for services to discover names?
If I put my managed Cloud SQL postgres database on this network, can I control its DNS name?
Finally, are there any other gotchas I might want to be aware of? You can assume my use case is very simple, two or more long lived services on Cloud Run, doing non-HTTP TCP/UDP communications.
I also found a potentially related Google Cloud Run feature request that is worth upvoting if this isn't currently possible.
Cloud Run services are only reachable through HTTP request. you can't use other network protocol (SSH to log into instances for example, or TCP/UDP communication).
However, Cloud Run can initiate these kind of connection to external services (for instance Compute Engine instances deployed in your VPC, thanks to the serverless VPC Connector).
the serverless VPC connector allow you to make a bridge between the Google Cloud managed environment (where live the Cloud Run (and Cloud Functions/App Engine) instances) and the VPC of your project where you have your own instances (Compute Engine, GKE node pools,...)
Thus you can have a Cloud Run service that reach a Kubernetes pods on GKE through a TCP connection, if it's your requirement.
About service discovery, it's not yet the case but Google work actively on that and Ahmet (Google Cloud Dev Advocate on Cloud Run) has released recently a tool for that. But nothing really build in.
When developing locally, I run docker-compose, where I have two services Service1 and Service2. Service2 depends on Service1. When I deploy them to ECS, I create them within one task definition and provide JSON array of container definitions to spin them up.
When I run them locally, within docker-compose, from Service2 I can call http://Service1:8080/v1/graphql (since they're in docker-compose together I can call it by the service name) ... however, when I deploy to ECS and I make that same API call, I get a 404.
Based on this: Docker links with awsvpc network mode I've also tried http://localhost:8080/v1/graphql ... I'd appreciate any help!
I'd try service discovery as mentioned here:
Amazon ECS now includes integrated service discovery. This makes it possible for an ECS service to automatically register itself with a predictable and friendly DNS name in Amazon Route 53. As your services scale up or down in response to load or container health, the Route 53 hosted zone is kept up to date, allowing other services to lookup where they need to make connections based on the state of each service.
See an example here.
I would like to ask you if you have an microservice architecture (based on Spring Boot) involving Amazon Elastic Container Service (ECS) with Application Load Balancer(ALB), service discovery is performed automatically by the platform, or do you need a special mechanism (such as Eureka or Consul)?
From the documentation (ECS and ALB) is not clear you have this feature provided.
I have talked this with the Amazon support team and they respond the following:
"...using Service Discovery on AWS ECS[..] just with ALBs.
So, there could be three options here:
1) Using ALB/ELB as service endpoints (Target groups for ALBs, separate ELBs if using ELBs)
2) Using Route53 and DNS for Service Discovery
3) Using a 3rd Party product like Consul.io in combination with Nginx.
Let me speak about each of these options.
Using ALBs/ELBs
For this option the idea is to use the ELBs or ALB Target groups in front of each service.
We define an Amazon CloudWatch Events filter which listens to all ECS service creation messages from AWS CloudTrail and triggers an Amazon Lambda function.
This function identifies which Elastic Load Balancing load balancer (or an ALB Target group) is used by the new service and inserts a DNS resource record (CNAME) pointing to it, using Amazon Route 53.
The Lambda function also handles service deletion to make sure that the DNS records reflect the current state of applications running in your cluster.
The down side here is that it can incur higher costs if you are using ELBs - as you need an ELB for each service. And it might not be the simplest solution out there.
If you wish to read more on this you can do so here[1]
Using Route53
This approach involves the use of Route53 and running a simple agent[2] on your ECS container instances.
As your containers stop/start the agent will update the Route53 DNS records. It creates a SRV record. Likewise it will delete said records once the container is stopped.
Another part of this method is a Lambda function that performs health checks on ECS container instances - and removes them from R53 in case of a failure.
You can read up more on this method, on our blog post here[3].
Using a 3rd Party tool like Consul.io Using tools like Consul.io on ECS, will work - but is not supported by AWS. So you are free to use it, but we - unfortunately - do not offer support for it.
So, in conclusion - there are a few ways of implementing service discovery on AWS ECS - the two ways I showed here that use AWS resources, and of course the way of using 3rd party applications.
"
you dont have an out-of-the-box solution in AWS, although it is possible with some effort as described in https://aws.amazon.com/es/blogs/compute/service-discovery-an-amazon-ecs-reference-architecture/
You may also install Zuul + Ribbon + Eureka or Nginx + Consul and use ALB to distribute traffic among Zuul or Nginx
I'm trying to setup a basic web application, which has an associated database, in AWS ECS. Locally I have these setup in different containers, and on ECS, I'd like to have separate task definitions so that I may scale the two separately.
I registered my first task definition as david_mongodb successfully in ECS. It has a container named david_mongodb in it.
Then I attempted to register my second task definition as david_web, which has a container named david_web that links the database via david_mongodb:db.
When I click 'Create', it returns an error:
Unable to create Task Definition
Linked container 'david_mongodb:db' doesn't exist.
It seems like task definitions can't see container names in other task definitions? I'm thinking putting both david_web and david_mongodb containers in the same task definition would work, but I don't want to do that: it would prevent me from scaling either web app or database separately. This overview seems to confirm that my architecture is recommended...
So how do I link containers that live in different task definitions? Or is there another clever way of handling this?
Links in an ECS task definition are analogous to Docker links and only work when the containers are part of the same task definition (containers that are part of a single task definition are placed together on the same host). In order to communicate between containers in different task definitions, you'll need a mechanism for discovering where the containers are located (what host) as well as the port for communication.
ECS has integration with Elastic Load Balancing (Application Load Balancers, Network Load Balancers, and Classic Load Balancers) through the service feature, where tasks will be automatically registered in the ELB and deregistered in the ELB appropriately.
ECS also has integration with Route 53 Auto Naming for DNS-based service discovery using A and SRV records. Your service's tasks can be automatically entered into and removed from DNS records.
Service Discovery for Amazon ECS Using DNS describes a different approach where a Lambda function listens to the ECS event stream through CloudWatch Events and updates Route 53 DNS records. This method has been superceded by the Route 53 Auto Naming feature described above.
If you want to avoid load balancers and DNS, another pattern might be an ambassador container (there's a sample called the ecs-task-kite that uses the ECS API) or you might be interested in an overlay network (Weave has a fairly detailed getting started guide for their solution).
Nathan Peck is keeping track of a number of different subjects related to ECS, including service discovery, here.
you could now refer to this official best practices guide on networking between Amazon ECS services in a VPC, discussing on the considerations when adopting service discovery, ELB or service mesh for service-to-service communication with ECS.