My company is using CloudFront to serve it's services for over a year. It is supposed to redirect trafic to a Load Balancer, which distributes load to the ECS.
Last night, all of a sudden certain IP addresses, like our office ip address, call center ip address, started to receive a CloudFront HTTP 504 Gateway Timout errors.
If I switch to mobile internet over my phone - everything seems to be fine. The execution time is not long at all - one of the services is a simple fornt-end only website.
The same happened without tinkering with a settings or anything. In addition, the same happened to our Production and Development environments, which are using different AWS accounts.
The WAF is turned off, so it should not be an issue here (dev environment never had it in a first place).
Most importantly some of our integrations stopped working for this very reason, therefore it is critical.
I would appreciate any help.
There is a auto-cloudfront security group, that whitelists CloudFront edge servers ip addresses, so Load Balancer would accept only connections from them.
There is a lambda function to automatically update Security Group with new ip addresses.
On the day of issue, AWS added 12 new CloudFront edge servers. In total there was 69 of them. The root cause of issue was that Security Group can only hold up to 60 ip addresses, while there are 69 servers/ip addresses. Therefore the Security Group did not get updated and some the edge servers were not whitelisted.
Related
I am using a public API hosted behind CloudFront. I want to get the least possible latency.
I have done the following things so far:
Hosted my code on EC2 in the same region as the API origin server. (I have a clue as to which region the origin server could be hosted in based on ping times)
Tried pinging the API domain name from various availability zones within that region and identified the zone with least ping time.
However during my testing, I came across instances where ping time from my desktop was smaller than the ping time from my EC2 instance. So I had a look at what the CNAME & IP, the original API domain name was getting resolved to in each case. I found them to be different.
Is there something I can do to ensure that DNS resolution of CloudFront leads me to the lowest latency link.
I can use the CNAME or IP that I found to be the fastest during my testing. But that CNAME/IP may change so that's not a proper solution.
I think the answer could lie in a smart DNS client which pings all the servers in DNS file periodically and resolves DNS requests to the the fastest IP. However I don't know if any such client exists and how to use it if it does.
I would like to hear your ideas.
Is there something I can do to ensure that DNS resolution of CloudFront leads me to the lowest latency link.
If you have your own domain name, you could try using Route53's latency record or Geoproximity routing
We have a load balancer that distributes traffic to a few servers.
In order to keep session integrity and some other bits we use a policy on the LB. Basically when a user hits the LB they receive a cookie that expires within 2 hours. This cookie tells the load balancer which server the user belongs to to stop them from switching server as they are progressing through the site.
This works fine in normal circumstances.
Now introduce Cloudflare CDN.
For some reason the cookie doesn't make it to the LB and it balances the traffic anyway. I'm assuming that there will be some kind of rule or setting within Cloudflare that will resolve this. Has anyone else came across this at all?
Cookie being used is jnAccel. It's value is a unique identifier as to which server that user belongs to. It's expiry is 2 hours.
As it would turn out the thing blocking Cookies was the HTTP/2 setting on CloudFlare.
You can find this in the network menu, It should be the first setting on the list.
The headers where scrambling and becoming non persistent with HTTTP/2.
If I find a solution to configure to load balancer to still enable HTTP/2 I will update my answer.
I want to set up a security group that will only allow my static IP to hit an EC2 server, but because I use Cloudflare and it uses some IP/reverse proxy magic, the AWS security group only sees the Cloudflare IP and not my actual static IP. I looked all over and can't find any answer to how to do this. Has anyone figured this out?
You can't do this with security group settings.
Caching proxies like Cloudflare create a separate connection to the origin server, using one of their IP addresses.
You'd need to use those addresses in your security group, and maintain the rules if Cloudflare changes the list. But, the list is public.
https://www.cloudflare.com/ips/
Next, you need to configure your web server to deny requests when the CF-Connecting-IP HTTP header doesn't contain your IP address.
https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers-
You have to do both, because if you don't restrict traffic to the Cloudflare IP address ranges with your security group, then traffic from elsewhere (not via Cloudflare) could forge the header containing the IP address.
Having done those two things, you have almost accomplished your objective, but not quite... because Cloudflare caches responses.
So, the next problem is, you also need to disable caching at Cloudflare, because once you fetch a page, it may be in the Cloudflare cache, where someone else might access it.
https://support.cloudflare.com/hc/en-us/articles/200168306-Is-there-a-tutorial-for-PageRules-
Really, for a case where you need to restrict access to a single IP (or a small set), it doesn't usually make a lot of sense to send the traffic through Cloudflare.
You could setup a lambda function that parse the IP file from Cloudflare and update your security groups dynamically. AWSlabs on github has an example lambda function that does it for CloudFront. Two problems you'll run into are that you'll have to schedule to lambda function to run since you can't subscribe to a queue like in the example and possible running out security group rules, 50 is the limit.
I need to use AWS WAF for my web application hosted on AWS to provide additional rule based security to it. I couldnt find any way to directly use WAF with ELB and WAF needs Cloudfront to add WEB ACL to block actions based on rules.
So, I added my Application ELB CNAME to cloudfront, only the domain name, WebACL with an IP block rule and HTTPS protocol was updated with cloudfront. Rest all has been left default. once both WAF and Cloudfront with ELB CNAME was added, i tried to access the CNAME ELB from one of the ip address that is in the block ip rule in WAF. I am still able to access my web application from that IP address. Also, I tried to check cloudwatch metrics for Web ACL created and I see its not even being hit.
First, is there any good way to achieve what I am doing and second, is there a specific way to add ELB CNAME on cloudfront.
Thanks and Regards,
Jay
Service update: The orignal, extended answer below was correct at the time it was written, but is now primarily applicable to Classic ELB, because -- as of 2016-12-07 -- Application Load Balancers (elbv2) can now be directly integrated with Web Application Firewall (Amazon WAF).
Starting [2016-12-07] AWS WAF (Web Application Firewall) is available on the Application Load Balancer (ALB). You can now use AWS WAF directly on Application Load Balancers (both internal and external) in a VPC, to protect your websites and web services. With this launch customers can now use AWS WAF on both Amazon CloudFront and Application Load Balancer.
https://aws.amazon.com/about-aws/whats-new/2016/12/AWS-WAF-now-available-on-Application-Load-Balancer/
It seems like you do need some clarification on how these pieces fit together.
So let's say your actual site that you want to secure is app.example.com.
It sounds as if you have a CNAME elb.example.com pointing to the assigned hostname of the ELB, which is something like example-123456789.us-west-2.elb.amazonaws.com. If you access either of these hostnames, you're connecting directly to the ELB -- regardless of what's configured in CloudFront or WAF. These machines are still accessible over the Internet.
The trick here is to route the traffic to CloudFront, where it can be firewalled by WAF, which means a couple of additional things have to happen: first, this means an additional hostname is needed, so you configure app.example.com in DNS as a CNAME (or Alias, if you're using Route 53) pointing to the dxxxexample.cloudfront.net hostname assigned to your distribution.
You can also access your sitr using the assigned CloudFront hostname, directly, for testing. Accessing this endpoint from the blocked IP address should indeed result in the request being denied, now.
So, the CloudFront endpoint is where you need to send your traffic -- not directly to the ELB.
Doesn't that leave your ELB still exposed?
Yes, it does... so the next step is to plug that hole.
If you're using a custom origin, you can use custom headers to prevent users from bypassing CloudFront and requesting content directly from your origin.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/forward-custom-headers.html
The idea here is that you will establish a secret value known only to your servers and CloudFront. CloudFront will send this in the headers along with every request, and your servers will require that value to be present or else they will play dumb and throw an error -- such as 503 Service Unavailable or 403 Forbidden or even 404 Not Found.
So, you make up a header name, like X-My-CloudFront-Secret-String and a random string, like o+mJeNieamgKKS0Uu0A1Fqk7sOqa6Mlc3 and configure this as a Custom Origin Header in CloudFront. The values shown here are arbitrary examples -- this can be anything.
Then configure your application web server to deny any request where this header and the matching value are not present -- because this is how you know the request came from your specific CloudFront distribution. Anything else (other than ELB health checks, for which you need to make an exception) is not from your CloudFront distribution, and is therefore unauthorized by definition, so your server needs to deny it with an error, but without explaining too much in the error message.
This header and its expected value remains a secret because it will not be sent back to the browser by CloudFront -- it's only sent in the forward direction, in the requests that CloudFront sends to your ELB.
Note that you should get an SSL cert for your ELB (for the elb.example.com hostname) and configure CloudFront to forward all requests to your ELB using HTTPS. The likelihood of interception of traffic between CloudFront and ELB is low, but this is a protection you should consider implenting.
You can optionally also reduce (but not eliminate) most unauthorized access by blocking all requests that don't arrive from CloudFront by only allowing the CloudFront IP address ranges in the ELB security group -- the CloudFront address ranges are documented (search the JSON for blocks designated as CLOUDFRONT, and allow only these in the ELB security group) but note that if you do this, you still need to set up the custom origin header configuration, discussed above, because if you only block at the IP level, you're still technically allowing anybody's CloudFront distribution to access your ELB. Your CloudFront distribution shares IP addresses in a pool with other CloudFront distribution, so the fact that the request arrives from CloudFront is not a sufficient guarantee that it is from your CloudFront distribution. Note also that you need to sign up for change notifications so that if new address ranges are added to CloudFront, then you'll know to add them to your security group.
I've setup a couple of EC2 instances in different regions and have configured A record sets in my hosted zone. All is working correctly, but I want to be able to check were a certain request is going. So if a request is done through webpagetest.org from Europe, is there a way I can check that Route 53 is routing it correctly to my EC2 instance in Ireland (assuming that it has the lowest latency)?
That site doesnt tell you what ip address is resolved in the request. You should try something like pingdom to see if the request is resolving to the correct ip address from different points around the world.