504 Gateway Time-out on ALB - amazon-web-services

I created an EC2 instance with apache installed on it and allowed HTTP/SSH traffic for my system only. I was able to access Web page using public IP of EC2. Then I configured ALB with same SG, registered same EC2 instance with ALB. Tried to access Web page using DNS name of ALB, got error -504 Gateway Time-out.
Increased time out interval to see if this resolves issue, didn't work. Then I revisited lesson and thought lets allow HTTP traffic to everyone(since it was allowed in lecture) in SG to see if it works and YES, it worked. I again changed SG to allow traffic only for my system and it failed again.

In this configuration, your security group needs to allow traffic from itself -- create rules that allow the appropriate ports, but use the security group sg-xxxx in place of an IP address, as the source. Merely being members of the same security group does not allow two systems to communicate with each other.
A better configuration would be for the balancer to have its own security group, and your instance's group would allow traffic from the balancer's group.
Note also that without the security group configuration being correct, you should also find that the health checks on the balancer are failing.

Related

how make instance group members work behind behind loadbalancer only

I am able to make work the backend service as an instance group - if i enable the "Allow http access" enabled while creating the members in the instance group.
However i want to disable this and make the network work only from the loadbalancer(external ip). However it is not working. The way I did was to define a firewall rule in the subnet where the instance group is there, such that the destination is the network tags defined for the instance group members
settings link as iamge
the source is dfined as the ip of the load balancer as a range.
Create VPC Firewall rules that only allow traffic from the load balancer and health check service.
130.211.0.0/22
35.191.0.0/16
34.96.0.0/20
34.127.192.0/18
References:
Firewall rules
Configuring a firewall rule
Where you looking is fine, you can do it.
The steps are, as I suggested in my comment a little bit more, I will resume them in this list, and let you the link of a qwiklab, you can check the steps there with the code to do it by yourself.
Basically:
Create the instances or instance group with the corresponding healtcheck.
Configure the Load balancer
Set the traffic to the new loadbalancer and build the proxy.
Create HTTPS Load Balancer and send the traffic to the Proxy.
https://www.qwiklabs.com/focuses/12007?catalog_rank=%7B%22rank%22%3A1%2C%22num_filters%22%3A0%2C%22has_search%22%3Atrue%7D&parent=catalog&search_id=15082883
I think that the link is creating instance by instance, but the steps should be the same for an instance group.

reference AWS Security Groups from other Security Groups

I am trying to understand this statement:
"You can reference AWS Security Groups from other Security Groups."
What exactly does this mean?
This is how I understood this. I created a security group and call it "SG-10" and attached it to an instance "EC2-10". The SG "SG-10" has allow port 443 defined inside it.
Now, I create a security group and call it "SG-20" and attached this to an instance "EC2-20". This also has port 443 allowed. Now, if I call "SG-10" inside "SG-20" does this mean that "EC2-10" will be able to connect to "EC2-20" on port 443.?
Regards,
Nik.
if I call "SG-10" inside "SG-20"
No, you can not call a security group, this does not make any sense.
If a security group A references security group B, it does mean that the instance to which the security group A is attached allows inbound or outbound traffic to another instance to which security group B is attached. But if you want to send traffic form instance with security group A to the instance with security group B, you have to use the IP or the DNS of the instance B.
Usually the reason why we would want to reference a security group instead of an IP, is that the IP might change over time or it is not exposed at all. A fairly common example is having an application load balancer (ALB) and a group of EC2 instances which allow traffic only from the ALB. The IP address of the ALB changes over time, so in order to be able to receive traffic from the ALB, we can reference the security group attached to it.
In we want to reference a security group from another security group, we have to edit the rules of the initial security group:
As an example of referencing security groups, imagine a 3-tier architecture:
A Load Balancer receiving traffic from the Internet and sending requests to an Amazon EC2 instance
An Amazon EC2 instance receiving requests from the Load Balancer and sending queries to an Amazon RDS database
An Amazon RDS database receiving requests from the EC2 instance
This would involve 3 security groups:
A security group on the Load Balancer (LB-SG) that allows incoming traffic on port 80/443.
A security group on the EC2 instance (App-SG) that allows incoming traffic on port 80 only from the load balancer. It does this by setting the source to LB-SG.
A security group on the RDS database (DB-SG) that allows incoming traffic on the relevant port (eg 3306) only from the EC2 instance. It does this by setting the source to App-SG.
All security groups allow All Outbound Traffic.
By referencing other security groups, resources can be added/removed without having to change the security groups. For example, another EC2 instance could be launched and assigned the App-SG security group. This new instance would then be able to communicate with the database since DB-SG allows incoming traffic from App-SG, without being tied to any specific IP addresses.
If a resource is associated with multiple security groups, then all rules apply to the resource. Security Groups only say what is 'Allowed'. They do not include 'Deny' rules.
In your SG-10/SG-20 example, you do not mention the source of the traffic, so it is not possible to answer your question. If you want EC2-20 to accept connections from EC2-10, then the SG-20 security group should allow connections with the Source set to SG-10.
As an aside, I should mention that Network ACLs should normally be left at their default "Allow All" settings unless there is a specific networking requirement (eg creating a DMZ).
I've been burned on this before by trying to connect to an EC2 instance's public address... sounds very similar to your setup. Please check this question: Source Security group isnt working as expected in aws. Actually. When you wire up the inbound rule so that the source is a security group, you must communicate through the source instance's private address.

Blocking inbound traffic for AWS VPC except from load balancer and select addresses

I have a couple of HTTP services running in containers using AWS Fargate. These services are meant to be accessed using an Application Load Balancer, and from a list of selected IP addresses. However, with default settings, the services are open to all inbound traffic.
The containers are all in the same VPC, Security Group and Subnets. The same goes for the load balancer.
I tried restricting the inbound traffic using Network ACLs, but this also blocked connections from the load balancer. What is the correct way of achieving the desired behavior (blocking external traffic)?
Both ECS tasks and service supports security groups. So you update the "Inbound traffic" of the security group to allow traffic only from ALB, read more about it here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-application-load-balancer.html
I think for your case it is better to use AWS WAF, If you want to allow some web requests and block others based on the IP addresses that the requests originate from, create an IP match condition for the IP addresses that you want to allow and another IP match condition for the IP addresses that you want to block. see the following link: here

IP whitelisting for local machine on ec2 instance using inbound rules in security group

I have configured to use my ip in the security group on ec2 instance. But I am getting 504 gateway timeout error.
When I make it open to world i.e 0.0.0.0/0 then it works well.
I checked for my IP address on the ec2 instance using "who am i" and this is similar to the one in the security group.
Please suggest how to make it work only for my machine.
I have followed the steps mentioned on
possible to whitelist ip for inbound communication to an ec2 instance behind an aws load balancer?
This is how my inbound rule for the security group looks.
All traffic All All 123.201.54.223/32 Dev Security Rule
Security groups will not allow you to make it work on a machine-by-machine basis, only by IPs and security groups, eg if you limit ingress by IP, any other machine using that same IP address (usually on same network/access point etc) will also be allowed in, not just your machine.
If you are using a load balancer, then it is the load balancer that should have access to your instance via its security group, and your access via IP should be controlled in the load balancer's security group, so you should use the settings you have quoted (at least to begin with!) on your LB security group, not your instance security group.
With the instance or group of instances (ie those that are behind the load balancer) in their security groups you want to only allow ingress from the load balancer security group, there's no need to set an IP address ingress (unless you want to allow eg ssh access from specific IP addresses or want them to talk to a database instance).
504 gateway timeout error It's mean your LB not able to communicate with the desired instance and you are able to communicate with LB.
All traffic All All 123.201.54.223/32 Dev Security Rule This will only allow traffic from you IP not Load Balancer IP.
You do not need to mention your IP in the security group of EC2, You have to allow traffic from LB that is 10.0.0.0/16.
HTTP 504: Gateway Timeout
Description: Indicates that the load balancer closed a connection
because a request did not complete within the idle timeout period.
Cause 1: The application takes longer to respond than the configured
idle timeout.
Solution 1: Monitor the HTTPCode_ELB_5XX and Latency metrics. If there
is an increase in these metrics, it could be due to the application
not responding within the idle timeout period. For details about the
requests that are timing out, enable access logs on the load balancer
and review the 504 response codes in the logs that are generated by
Elastic Load Balancing. If necessary, you can increase your capacity
or increase the configured idle timeout so that lengthy operations
(such as uploading a large file) can complete. For more information,
see Configure the Idle Connection Timeout for Your Classic Load
Balancer and How do I troubleshoot Elastic Load Balancing high
latency.
Cause 2: Registered instances closing the connection to Elastic Load
Balancing.
Solution 2: Enable keep-alive settings on your EC2 instances and make
sure that the keep-alive timeout is greater than the idle timeout
settings of your load balancer.
ts-elb-errorcodes-http504

AWS NLB in public subnets with EC2 in private subnets

Has someone configured a NLB in the public subnets of your VPC to route traffic to EC2 instances that are in the private subnets?
When using an ELB, a good solution is to create a Security Group for the ELB and then create another SecurityGroup for the private EC2 Instances, allowing incoming traffic from that ELB Security Group, as explained here:
https://aws.amazon.com/premiumsupport/knowledge-center/public-load-balancer-private-ec2/
"You can also add a rule on the instance’s security group to allow traffic from the security group assigned to the load balancer. For example, if the security group on the load balancer is sg-1234567a, make the following changes on the security group associated with the private instances"
Since you cannot associate a Security Group to a NLB, how could you accomplish this with the same type of security?
Thanks!
Since you cannot associate a Security Group to a NLB, how could you
accomplish this with the same type of security?
The security aspect does not change.
NLB is a different beast, it not the same as classic Load Balancers. For Classic Load Balancers, from the point of view of your instances, traffic does appear to come from inside the VPC. From outside, traffic goes to a (random and mutating) list of IP addresses, resolved by the DNS record that AWS provides to you.
Network Load Balancers are completely different. From the point of view of your instances, they are completely invisible. If it is an external network load balancer, traffic appears to be coming from instances on the internet directly (even though this is an illusion). Therefore, if you want to talk to everyone on the internet, 0.0.0.0/0 is what you open it to.
This is, in fact, what the documentation says:
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/target-group-register-targets.html#target-security-groups
Recommended Rules
Inbound Source Port Range Comment
Client IP addresses instance listener Allow traffic from clients on the instance listener port
VPC CIDR health check Allow traffic from the load balancer on the health check port
Client IP addresses is whatever your client IPs are. If they are on the open internet, 0.0.0.0/0 it is. Adding the NLB private IP address, as I saw in other responses, accomplishes nothing. Traffic is not coming from there, as far as the instances are concerned.
On the security angle, nothing changes. Since your instances are in private subnets, traffic cannot flow directly to them, as there is a NAT gateway in the middle. It can only flow from them to the internet (through NAT gateway, then internet gateway). Even if you specify all traffic is allowed from everywhere, traffic still won't come. It will have to come through another way. In your case, that way is the NLB, which has a fixed number of ports it listens to, and only sends traffic to the destination ports on the instances you specify.
If you are moving from classic Load Balancers to NLBs, move the security group rules from the Load Balancer to your instances. Or better yet, since you can have multiple security groups, just add the SG you currently have for the classic LB to the instances(and update any ASGs as needed). Your security posture will be exactly the same. With the added benefit that now your applications won't need things like proxy protocol to figure out where traffic is coming from, it is no longer obfuscated by the load balancer.
That is indeed true as per AWS Documentation :
Network Load Balancers do not have associated security groups.
Therefore, the security groups for your targets must use IP addresses
to allow traffic from the load balancer.
So If you do not want to grant access to the entire VPC CIDR, you can grant access to the private IP addresses used by the load balancer nodes. There is one IP address per load balancer subnet.
On NLB Tab of there is one Network Interface per Load Balancer from there :
On the Details tab for each network interface, copy the address from
Primary private IPv4 IP.
You can use this private IP Address at add it SG of EC2 Instances.
Please Refer to AWS Documentation
Tail your http access logs and you will see there is no changing of source IP address from the network load balancer which means you need to allow 0.0.0.0/0 on the endpoints security group if the internet needs access to your endpoint.
This is only ok if you use a private subnet so be careful if you have this server on a public subnet as this solution would not be advisable. In this case just use an application load balancer. You can still setup the same listener and configure a target group by instance as well. The application load balancer will update the source IP address to it's own private address if you tail the access logs. The advantage of this is you only need to allow https traffic to the app load balancer and then you can accept http for the target group if you like from the load balancer.