I have an ECS cluster which has an Auto Scaling group with 2 EC2 instances. Also I have 3 services, and each service has its own task definition. Each EC2 instance run one docker container per service. Therefore I have 3 docker containers in each EC2 instance.
Each docker container simply run a spring boot app. Since I have 3 services, then I have 3 spring boot apps. Again, one container only runs one of these 3 spring boot apps. Each app exposes a rest ful API, with services like POST, GET etc, under URLs like /service1 or /service1/resource1. One important point here is I'm using dynamic port mapping in the container's host.
I have an external (internet-facing) ALB on port 443, which has 3 target group. Depending on the URL, the request will go to one of the 3 apps (or containers).
My problem is sometimes app A needs to make an http request to app B. My EC2 instances live in private subnet, while my ALB lives in a public subnet. So if I use my ALB to make http requests from inside a container to another, what's going to happen is the request will go through a NAT, and since the public ip of the NAT is not part of the security group of the ALB then it can't connect on port 443. I have 2 ways to make this work:
In the security groups of the ALB whitelist 0.0.0.0/0. I don't want to do that since the entire world will have access.
In the security group of the ALB whitelist the public IP of the NAT. I'm not sure about this approach. Is it recommendable?
A third option I'm trying to implement is to have a third load balancer, an internal one. But I'm lost here, as per AWS docs you can only assign 1 load balancer to your service. And since we are using dynamic port mapping I don't see a way to create manually an ALB and use the dynamic automatically assigned port.
How do you guys make this kind of connectivity between containers, where one container consumes a service that other provides?
As last comment, I use cloud formation for all. No manual setup from console.
Thanks,
You may try to whitelist you nat gateways public IP as a host with /32 mask. It is a quite normal approach because you already have exposed endpoints to the public internet via ALB. You only need to care about security rules updates in case of destroying or changing the NAT gateway, because it's IP may change.
Related
I have two fargate tasks running in two different clusters, the first one is running on port 3000 and can receive requests from anyone, the second one is running on port 8080 and can be accessed only by the first one. Both are in the same Security Group and VPC.
I created an inbound rule to allow public access for the first one, then I tried to create other inbound rule to enable the access for the second through security group ingress. But when the first service tries to access the second, I receive an Timeout Error.
When I allow the public access to the second service, the communication works properly, but I cannot allow it to run forever.
Each service has a load balancer configured, but I've already tried (unsuccessfully) to access the service by its task's public IP.
Anyone has any idea what I am doing wrong??
The inbound rules for the security group can be checked in this image
If the first service tries to access the second service by the second service's public IP, then the traffic will go out to the Internet and back, which will destroy the network traffic's association with the origin security group.
To keep the traffic inside the VPC, and to make sure the security group rules apply as intended, the first service needs to connect to the second service via the second service's private IP.
If you are using a load balancer for the second service, then it needs to be an internal load balancer, not an external load balancer.
I have been looking into different ways of connecting multiple miscorservices within their own services/tasks using ECS Fargate.
Normally, if all microservices are defined in the same task definition, we can just use the local ip with corresponding ports but this means we cannot scale individual microservices. From what I can tell there are two 'main' ways of enabling this communication when we break these out into multiple services:
Add a load balancer to each service and use the loadbalancers public ip as the single point of access from one service to another.
Questions I have on this is:
a. Do all the services that need to communicate need to sit in the same VPC and have the
service's incoming rules set to the security group of the load balancer?
b. Say we have now provisioned the entire set up and need to set one of the loadbalancer public DNS's in one microservices code base, whats the best way of attaining this, im guessing some sort of terraform script that 'assumes' the public DNS that will be added to it?
Making use of AWS Service Discovery, meaning we can query service to service with a simple built up identifier.
Question I have for this is:
a. Can we still attach load balancers to the services and STILL use service discovery? Or
does service discovery have an under the hood load balancer already configured?
Many thanks in advance for any help!
1.a All services in the same VPC and their security groups (SGs)
I assume that you are talking about case where each service will have its own load balancer (LB). Since the LBs are public, they can be in any VPC, region or account.
SGs are generally setup so that incoming rules to services allow only connections from the SG of the LB.
1.b DNS
Each task can have environmental variables. This is a good way to pass the DNS values. If you are taking about terraform (TF), then TF would provision the LBs and then create the tasks and set the env variables with DNS values of the LBs. Thus, you would know the DNS of LBs as they would have been created before your services.
2.a Service discovery (SD)
SD is only for private communication between services. No internet is involved, so everything must be in same VPC or peered-VPCs. So its basically oposite of the first solution with LBs.
I think you should be able to also use public LB along with SD.
SD does not use a LB. Instead when you query a DNS of a service through SD you will get private IP addresses of the tasks in random order. So the random order approximates load balancing of connections between tasks in a service.
I have a PHP + Apache application running in ECS with an Application Load Balance sitting in front of it. Everything works fine except when the application makes request to itself and the request times out.
Let's say the URL to reach the application is www.app.com and in PHP I use Guzzle to send requests to www.app.com but that request will always time out.
I suspect it is a networking issue with ALB but I do not know how I can go about fixing it. Any help please?
Thanks.
As you're using ECS I would recommend replacing calls to a public load balancer with a service mesh instead to allow your application to keep all HTTP(S) traffic internal to the network. This will improve both security and performance (latency is reduced). AWS has an existing product that integrates with ECS to allow this functionality named App Mesh/
Alternatively if you want to stick with what you currently have setup you will need to check the following functionality:
If the hosts are ECS hosts are private then they will need to connect outbound by using a NAT Gateway/NAT Instance in the routing table for the 0.0.0.0/0 route. For Fargate this will depend on if the container is public or private.
If the host/container is public it will need the internet gateway added to its route table for the 0.0.0.0/0 route. Even if inbound access from the ALB to the host is private the host will always speak outbound to the internet via an internet gateway.
Ensure that inbound/outbound security groups allow access to either HTTP or HTTPS
We are having several microservices on AWS ECS. We have single ALB which has different target group for different microservices. We want to expose some endpoints externally while some endpoints just for internal communication.
The problem is that if we put our load balancer in public VPC than it means that we are exposing all register endpoints externally. If we move load balancer to private VPC, we have to use some sort of proxy in public VPC, which required additional infra/cost and custom implementation of all security concerns like D-DOS etc.
What possible approaches we can have or does AWS provide some sort of out of the box solution for this ?
I would strongly recommend running 2 albs for this. Sure, it will cost you more (not double because the traffic costs won't be doubled), but it's much more straight forward to have an internal load balancer and an external load balancer. Work hours cost money too! Running 2 albs will be the least admin and probably the cheapest overall.
Checkout WAF. It stands for web application firewall and is available as AWS service. Follow these steps as guidance:
Create a WAF ACL.
Add "String and regex matching" condition for your private endpoints.
Add "IP addresses" condition for your IP list/range that are allowed to access private endpoints.
Create a rule in your ACL to Allow access if both conditions above are met.
Assign ALB to your WAF ACL.
UPDATE:
In this case you have to use external facing ALB in a public subnet as mentioned by Dan Farrell in comment below.
I would suggest doing it like this:
one internal ALB
one target group per microservice, as limited by ECS.
one Network load balancer(NLB), with one ip based target group.
The Ip based target group will have the internal ALB ip addresses,as the private ip addresses for ALB are not static, you will need to setup cloudwatch cron rule with this lambda function(forked from aws documentation and modified to work on public endpoints as well):
https://github.com/talal-shobaita/populate-nlb-tg-withalb/
Both ALB and NLB are scalable and protected from DDOS by AWS, AWS WAF is another great tool that can be attached directly to your ALB listener for extended protection.
Alternatively, you can wait for AWS to support multiple target group registration per service, it is already in their roadmap:
https://github.com/aws/containers-roadmap/issues/104
This how we eventually solved.
Two LB one in private and one in public subnet.
Some APIs meant to be public, so directly exposed through public LB.
For some private APIs endpoints need to be exposed, added a proxy in public LB and routed those particular paths from public LB to private LB through this proxy.
These days API Gateway is the best way to do this. You can have your API serve a number of different endpoints while serving only the public ones via API Gateway and proxying back to the API.
I don't see it mentioned yet so I'll note that we use a CloudMap for internal routing and an ALB for "external" (in our case simply intra/inter-VPC) communication. I didn't read in depth, but I think this article describes it.
AWS Cloud Map is a managed solution that lets you map logical names to the components/resources for an application. It allows applications to discover the resources using one of the AWS SDKs, RESTful API calls, or DNS queries. AWS Cloud Map serves registered resources, which can be Amazon DynamoDB tables, Amazon Simple Queue Service (SQS) queues, any higher-level application services that are built using EC2 instances or ECS tasks, or using a serverless stack.
...
Amazon ECS is tightly integrated with AWS Cloud Map to enable service discovery for compute workloads running in ECS. When you enable service discovery for ECS services, it automatically keeps track of all task instances in AWS Cloud Map.
You want to look at AWS Security Groups.
A security group acts as a virtual firewall for your instance to control inbound and outbound traffic.
For each security group, you add rules that control the inbound traffic to instances, and a separate set of rules that control the outbound traffic.
Even more specific to your use-case though might be their doc on ELB Security Groups. These are, as you may expect, security groups that are applied at the ELB level rather than the Instance level.
Using security groups, you can specify who has access to which endpoints.
I'm trying to work out how I can tighten up the security group which I have assigned to container instances in AWS ECS.
I have a fairly simple Express.js web service running in a service on ECS
I'm running an Application Load Balancer in front of ECS
My Express.js container exposes and listens on port 8080
I'm using dynamic port mapping to allow container instances to run multiple containers on the same port (8080)
I have found that in order for ALB to forward requests on to containers running in the ECS cluster, I need the security group which is assigned to the container instances to allow all traffic. Bearing in mind that these instances (for reasons I don't know) are assigned public IPv4 addresses - regardless of the fact that I've configured the cluster to place instances in my private subnets - so I'm not comfortable with these instances essentially being wide open, just to ALB can pass requests so them inside the VPC.
I understand that with dynamic port mapping, my containers or not running on one single port on the underlying Docker host that's running them. I also understand that there's no single IP that requests may arrive at the EC2 instances from the ALB, so it seems to me that I can't lock this down if I'm using dynamic port mapping, because there's no single point of origin or destination for the traffic that's coming into the EC2 instances. I feel like I'm missing something here, but I can't for the life of me work out how to do this.
How should I configure ECS or my EC2 security group to allow me to only allow access to the container instances from ALB and not from the rest of the internet?
I've tried to include as much info as is necessary without swamping the question with unnecessary details. If there's any details that would be useful that I've not included, please leave a comment and I'll be happy to provide them.
1) There is no reason why you have to have public ip addresses on your container instances. Just don't set the option at launch, see this page particularly step "e"
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_container_instance.html?shortFooter=true
If the instances are in a private subnet, then the routing should not allow ingres anyway...
2) It is possible to lock down the security using security groups. Using the "security group id" instead of the IP address means that you do not have to know the exact address of the ALB. See this page for instructions on ALB configuration in this way
http://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-update-security-groups.html