Our setup is as follows:
VPC (with about 30-50 ec2 instances) -> EC2 Nat Instance -> Internet.
Since Dec 13, we have been seeing issues where randomly the connection were starting to refuse. No such issue was seen earlier. Only change is the processing of the urls via API has increased (In other words more TCP connections are getting initiated & worked on). Requesting an API Request (POST/GET/PUT doesn't matter) from an EC2 instance within VPC via NAT Instance to the Internet is failing at random.
I tried logging the Flow logs, but in these flow logs, I see the entry where it shows ACCEPT OK for the TCP log transmission ( pic attached - https://ibb.co/dwe3X6 ). However, the same capture on tcpdump (one specific ec2 instance within vpc), shows the TCP Retransmission failure (where traffic is going through the NAT instance) ( pic attached - https://ibb.co/npqozm ). They are of the same time and same ec2 instance.
Basically, the SYN packet gets initiated, but then the actual handshake doesn't go through. Note, that this doesn't happen all the time.
The tcp retransmission failures are random. Sometimes it works and sometimes it doesn't. So this is leading me to believe there is some sort of a queue or buffer in NAT instance which is hitting the limit and I am not sure how to get to root of this.
This suggests a problem out on the Internet or at the distant end.
ACCEPT means the connection (the instantion of the flow) was allowed by the security group and network ACLs, telling you nothing about whether the handshake succeeded. OK in the flow logs means the log entry itself is intact.
There is no reason to believe that the NAT instance is limiting this, because the SYN packets are indeed shown in wireshark leaving the instance headed for the Internet and the flow log suggests that it is indeed making its way out of the NAT instance successfully.
You used the term "refuse" but the wireshark entries are consistent with a Connection timed out error, rather than Connection refused, which is an active refusal by the far end (or, less commonly, by an intermediate firewall) due to the lack of a listening service on the destination port, which would cause the destination to respond with a TCP RST.
If you can duplicate the problem with a NAT Gateway, then you can be confident that it is not in any way related to the NAT instance itself, which is simply a Linux instance using iptables ... -j MASQUERADE.
The only thing the network infrastructure throttles is outbound connections to destination port 25, because of spam. Everything else is bounded only by the capabilities of the instance itself. With a t2.micro, you should have (iirc) in excess of 125 Mbits/sec of Ethernet bandwidth available, and the NAT capability is not particularly processor intensive, so unless you are exhausting the Ethernet bandwidth or the CPU credit balance of the instance, it seems unlikely that the NAT instance could be the cause.
Related
In my application:
ASP.NET Core 3.1 with Kestrel
Running in AWS ECS + Fargate
Services run in a public subnet in the VPC
Tasks listen only in the port 80
Public Network Load Balancer with SSL termination
I want to set the Security Group to allow inbound connections from anywhere (0.0.0.0/0) to port 80, and disallow any outbound connection from inside the task (except, of course, to respond to the allowed requests).
As Security Groups are stateful, the connection tracking should allow the egress of the response to the requests.
In my case, this connection tracking only works for responses without body (just headers). When the response has a body (in my case, >1MB file), they fail. If I allow outbound TCP connections from port 80, they also fail. But if I allow outbound TCP connections for the full range of ports (0-65535), it works fine.
I guess this is because when ASP.NET Core + Kestrel writes the response body it initiates a new connection which is not recognized by the Security Group connection tracking.
Is there any way I can allow only responses to requests, and no other type of outbound connection initiated by the application?
So we're talking about something like that?
Client 11.11.11.11 ----> AWS NLB/ELB public 22.22.22.22 ----> AWS ECS network router or whatever (kubernetes) --------> ECS server instance running a server application 10.3.3.3:8080 (kubernetes pod)
Do you configure the security group on the AWS NLB or on the AWS ECS? (I guess both?)
Security groups should allow incoming traffic if you allow 0.0.0.0/0 port 80.
They are indeed stateful. They will allow the connection to proceed both ways after it is established (meaning the application can send a response).
However firewall state is not kept for more than 60 seconds typically (not sure what technology AWS is using), so the connection can be "lost" if the server takes more than 1 minute to reply. Does the HTTP server take a while to generate the response? If it's a websocket or TCP server instead, does it spend whole minutes at times without sending or receiving any traffic?
The way I see it. We've got two stateful firewalls. The first with the NLB. The second with ECS.
ECS is an equivalent to kubernetes, it must be doing a ton of iptables magic to distribute traffic and track connections. (For reference, regular kubernetes works heavily with iptables and iptables have a bunch of -very important- settings like connection durations and timeouts).
Good news is. If it breaks when you open inbound 0.0.0.0:80, but it works when you open inbound 0.0.0.0:80 + outbound 0.0.0.0:*. This is definitely an issue due to the firewall dropping the connection, most likely due to losing state. (or it's not stateful in the first place but I'm pretty sure security groups are stateful).
The drop could happen on either of the two firewalls. I've never had an issue with a single bare NLB/ELB, so my guess is the problem is in the ECS or the interaction of the two together.
Unfortunately we can't debug that and we have very little information about how this works internally. Your only option will be to work with the AWS support to investigate.
I'm trying to get TCP timestamp from the packets for clock skewing purposes on my application which is hosted on EC2. In my network I have an ALB.
So my question is how do I get TCP level packet information in my app ? Since ALB filters out all the OSI Layers except application level (HTTP)
If the only reason to get access to TCP packet is to detect timestamp and correct clock drift, I would suggest to configure your EC2 instance to use NTP time server instead.
https://aws.amazon.com/blogs/aws/keeping-time-with-amazon-time-sync-service/
That being said, the ALB is not "removing" TCP information from network packets. HTTP connections made to your application are still transported over IP and TCP. If you need low level access to network packets from an app, I would suggest to look at the pCAP library which is used by TCPDUMP and many other tool to capture network traffic on an interface.
https://www.tcpdump.org/
[UPDATED to include comments]
It is important to understand the TCP connection between your client and the ALB is terminated at the ALB level. The ALB creates a second TCP connection to forward HTTP requests to your EC2 instance. The ALB does not remove information from TCP/IP, it just creates a second, independent and new connection. Usually the only information you want to propagate from the initial TCP connection is the source IP address. The ALB, like most load balancers and proxies, captures this information from the original connection (the one received from the client) and embed the information in an HTTP header called X-Forwarded-For.
This is documented at https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/x-forwarded-headers.html
If you want to capture other information from the original connection, I am afraid it will not be possible using ALB. (but I also would be very curious about the use case, i.e. WHAT you're trying to achieve)
In AWS console we must specify actual rules in order make it possible to access EC2 instances from remote localizations.
I mean rules like opening some port or access from allowed IP addresses.
And it is working for me now.
I consider following scenario:
Let's assume that we have application A which maitain long running connection and everything is working because
security rules are properly set. Now,
(a) someone remove rules allowing application A connect to EC2 instancs (so external IP address which is used by application A)
(b) at any point external IP address of machine used by application A change.
I consider if it is possible that connection established before occurence (a) or (b) keeps working? If yes, then how is it possible?
Here's a pretty basic explaination to your answers. Ofcourse, there's a lot more information on the matter, but I guess it is not of importance right now.
If you change a rule, let's assume it is a Firewall rule or AWS Security Group rule, the connection will terminate as the rule takes effect immediately.
Simply put, you are sending a stream of information packet by packet, so when the change is detected the packets will no longer be receieved and you will no longer receive a response, i.e. the connection will terminate.
If you change your IP and you are using TCP connections, which I assume you do, they will also terminate as TCP connections are based on IP:Port combinations, BUT if you are using DNS rather than just IP your traffic will be routed correctly, you might experience some downtime, but your service will get back working soon enough.
EDIT: As noted by Michael, the security group change doesn't cut off existing connections. The next time an attempt is made, it will block them.
I run a rabbit HA cluster with 3 nodes and a classic AWS load-balancer(LB) in front of them. There are two apps, one that publishes and the other one that consumes through the LB.
When publisher app starts sending 3 million messages, after short period of time its connection is put into Flow Control state. After the publishing is finished, in publisher app logs I can see that all 3 million messages are sent. On the other hand in consumer app log I can only see 500K - 1M messages (varies between runs), which means that the large number of messages is lost.
So what is happening is that in the middle of a run, classic LB decides to change its IP address or drop connections, thus loosing a lot of messages (see my update for more details).
The issue does not occur if I skip LB and hit the nodes directly, doing load-balancing on app side. Of course in this case I lose all the benefits of ELB.
My question are:
Why is LB changing IP addresses and dropping connections, is that related to high message rate from publisher or Flow Control state?
How to configure LB, so that this issue doesn't occur?
UPDATE:
This is my understanding what is happening:
I use AMQP 0-9-1 and publish without 'publish confirms', so message is considered sent as soon as it's put on a wire. Also, the connection on rabbitmq node is between LB and a node, not Publisher app and a node.
Before the communication enters Flow Control, messages are passed from LB to a node immediately
Then the connection between LB and a node enters Flow Control, Publisher App connection is not blocked and thus it continues to publish at the same rate. That causes messages to pile up on LB.
Then LB decides to change IP(s) or drop the connection for whatever reasons and create a new one, causing all the piled messages to be lost. This is clearly visible from the RabbitMQ logs:
=WARNING REPORT==== 6-Jan-2018::10:35:50 ===
closing AMQP connection <0.30342.375> (10.1.1.250:29564 -> 10.1.1.223:5672):
client unexpectedly closed TCP connection
=INFO REPORT==== 6-Jan-2018::10:35:51 ===
accepting AMQP connection <0.29123.375> (10.1.1.22:1886 -> 10.1.1.223:5672)
The solution is to use AWS network LB. The network LB is going to create a connection between Publisher App and rabbitmq node. So if the connection is blocked or dropped Publisher is going to be aware of that and act accordingly. I have run the same test with 3M messages and not the single message is lost.
In the AWS docs, there's this line which explains the behaviour:
Preserve source IP address Network Load Balancer preserves the client side source IP allowing the back-end to see the IP address of
the client. This can then be used by applications for further
processing.
From: https://aws.amazon.com/elasticloadbalancing/details/
ELBs will change their addresses when they scale in reaction to traffic. New nodes come up, and appear in DNS, and then old nodes may go away eventually, or they may stay online.
It increases capacity by utilizing either larger resources (resources with higher performance characteristics) or more individual resources. The Elastic Load Balancing service will update the Domain Name System (DNS) record of the load balancer when it scales so that the new resources have their respective IP addresses registered in DNS. The DNS record that is created includes a Time-to-Live (TTL) setting of 60 seconds, with the expectation that clients will re-lookup the DNS at least every 60 seconds. (emphasis added)
— from “Best Practices in Evaluating Elastic Load Balancing”
You may find more useful information in that "best practices" guide, including the concept of pre-warming a balancer with the help of AWS support, and how to ramp up your test traffic in a way that the balancer's scaling can keep up.
The behavior of a classic ELB is automatic, and not configurable by the user.
But it also sounds as if you have configuration issues with your queue, because it seems like it should be more resilient to dropped connections.
Note also that an AWS Network Load Balancer does not change its IP addresses and does not need to scale by replacing resources the way ELB does, because unlike ELB, it doesn't appear to run on hidden instances -- it's part of the network infrastructure, or at least appears that way. This might be a viable alternative.
AWS spot instances are started on pricing decrease, this can happen anytime of the day (or does not happen for many days, depending on the config), i'm looking for a similar way to start an instance when there's a user request, eg. http request/ssh connection ... etc
I'm willing to monitor inbound IP traffic to my elastic IPs (on my internet gateway), and if target instance is stopped then launch it, a kind of wake-on-inbound-traffic.
The monitoring can be in two ways:
Passive: mirror traffic, analyze it in offline with some libpcap-based tool and launch instances based on the capture analysis, the downside is the very early IP requests to that instance will timeout till it start and bootstrap.
Active: analyze inbound traffic inline (using libpcap-based tool as well) and start the target instance, redirect http requests to some "Please wait while warming your instance" till the instance get started.
Anyone thought of doing this ? any guidelines for doing it ?
thanks !