We have a load balancer that distributes traffic to a few servers.
In order to keep session integrity and some other bits we use a policy on the LB. Basically when a user hits the LB they receive a cookie that expires within 2 hours. This cookie tells the load balancer which server the user belongs to to stop them from switching server as they are progressing through the site.
This works fine in normal circumstances.
Now introduce Cloudflare CDN.
For some reason the cookie doesn't make it to the LB and it balances the traffic anyway. I'm assuming that there will be some kind of rule or setting within Cloudflare that will resolve this. Has anyone else came across this at all?
Cookie being used is jnAccel. It's value is a unique identifier as to which server that user belongs to. It's expiry is 2 hours.
As it would turn out the thing blocking Cookies was the HTTP/2 setting on CloudFlare.
You can find this in the network menu, It should be the first setting on the list.
The headers where scrambling and becoming non persistent with HTTTP/2.
If I find a solution to configure to load balancer to still enable HTTP/2 I will update my answer.
Related
My company is using CloudFront to serve it's services for over a year. It is supposed to redirect trafic to a Load Balancer, which distributes load to the ECS.
Last night, all of a sudden certain IP addresses, like our office ip address, call center ip address, started to receive a CloudFront HTTP 504 Gateway Timout errors.
If I switch to mobile internet over my phone - everything seems to be fine. The execution time is not long at all - one of the services is a simple fornt-end only website.
The same happened without tinkering with a settings or anything. In addition, the same happened to our Production and Development environments, which are using different AWS accounts.
The WAF is turned off, so it should not be an issue here (dev environment never had it in a first place).
Most importantly some of our integrations stopped working for this very reason, therefore it is critical.
I would appreciate any help.
There is a auto-cloudfront security group, that whitelists CloudFront edge servers ip addresses, so Load Balancer would accept only connections from them.
There is a lambda function to automatically update Security Group with new ip addresses.
On the day of issue, AWS added 12 new CloudFront edge servers. In total there was 69 of them. The root cause of issue was that Security Group can only hold up to 60 ip addresses, while there are 69 servers/ip addresses. Therefore the Security Group did not get updated and some the edge servers were not whitelisted.
This is the first time that I am using load balancer... I have spent quite a bit of time going through documentation and I am still quite confused.
I want to host my website. My website supports HTTPS only. I want to put my backend servers behind an Application Load Balancer.
I am using AWS' default VPC, I have created an ALB (myALB) and installed my SSL certificate on it. I have also created 2 EC2 instances (myBackEndServer1 & myBackEndServer2).
Questions:
Should the communication between backend servers and myALB be
through HTTP or HTTPS?
I have created an HTTPS listener on myALB, do I also need an HTTP
listener on myALB? what I want is to redirect any HTTP request to
HTTPS (I believe this should happen on myALB)?
I want to use External ID login (using Facebook). I have set up Facebook
login to work with HTTPS only. Does the communication between
Facebook and my backend servers go through myALB? I mean, I either
need HTTPS on my backend servers, or the communication with facebook
should go through myALB.
I would appreciate any general advice.
You can use both HTTP and HTTPS listeners.
Yes, you can achieve that with ALB. You can add a rule to it that says that any request that is coming to port 80 will be redirected to port 443 on a permanent basis. Check out rules for ALB.
If you make a request from your instances to Facebook - it depends on Facebook, whether your communication will be encrypted, because in such case you are a client. However if you set up some webhook, Facebook is now a client and to communicate with you, you're gonna give your load balancer's DNS name. And due to the point 2 in this list, Facebook will be forced to use TLS.
I'm not sure I fully understood your question number three, but here's something you may also find useful. ALB has some features that allows to authenticate users with Cognito. It explicitly says that your EC2 instances can be abstracted away from any authentication, also if it makes use of Facebook ID or Google Id or whatever. Never tried though.
Observations
When using an Amazon ELB Application Load Balancer and working with Sticky Sessions the load balancer inserts a cookie named AWSALB in the first request. To let the next request stick to the same target node (EC2 instance) the cookie should be included in that request. When doing so, it seems that the load balancer inserts a different cookie value in the response to the 2nd request. When including this new cookie value on the 3rd request, we get yet a new cookie value in the response. And so forth…
(This is different from how Sticky Sessions works with the Classic Load Balancer where the cookie is named AWSELB and retains its value until discarded by the client or the load balancer.)
The reason the AWSALB cookie changes value all the time seems to be (as stated by the docs):
The name of the cookie is AWSALB. The contents of these cookies are encrypted using a rotating key. You cannot decrypt or modify load balancer-generated cookies.
So even though the contents of the cookie might be the same, we cannot tell.
Question
The question is whether a request to the load balancer must always include the most recently received value of the AWSALB cookie or if it ok to send some previously received value (from the same sticky session, of course).
If this is a requirement the AWS ELB Application Load Balancer would not be able to serve a client that performs multiple parallel requests (after having received the first AWSALB cookie) but only clients that performs all requests in a sequential fashion (one at a time).
Can anybody shed some light on this?
After waiting for a reply here, I raised a support case with Amazon and got this reply:
I understand that you would like to confirm if it is required to provide the latest stickiness cookie for every request.
You are right in noting that the behaviour is different between CLB and ALB.
Due to the different functionality of Application Load Balancer to direct traffic to multiple Target Groups, each having its own stickiness, ALB encrypts the information needed to direct traffic and provides new cookie on each request. This ensures that different times for different groups are always respected correctly.
Clients can always obtain the latest cookie, as internally the information would ensure correct routing to the same target.
If you want to reuse single cookie it is also possible, ALB will respect it and correctly route the traffic as per the stickiness in the cookie. I would not recommend using the same cookie for periods longer than 60 seconds though. This is to ensure that in case of target becoming unavailable you can acquire new cookie with new stickiness information that would route you to new target.
The load balancer must always include the most recently received value of the AWSALB cookie - yes. Some text from the documentation:
The Application Load Balancer resets the expiry of the cookies it generates after every request.
Sticky sessions for your Application Load Balancer
I need to use AWS WAF for my web application hosted on AWS to provide additional rule based security to it. I couldnt find any way to directly use WAF with ELB and WAF needs Cloudfront to add WEB ACL to block actions based on rules.
So, I added my Application ELB CNAME to cloudfront, only the domain name, WebACL with an IP block rule and HTTPS protocol was updated with cloudfront. Rest all has been left default. once both WAF and Cloudfront with ELB CNAME was added, i tried to access the CNAME ELB from one of the ip address that is in the block ip rule in WAF. I am still able to access my web application from that IP address. Also, I tried to check cloudwatch metrics for Web ACL created and I see its not even being hit.
First, is there any good way to achieve what I am doing and second, is there a specific way to add ELB CNAME on cloudfront.
Thanks and Regards,
Jay
Service update: The orignal, extended answer below was correct at the time it was written, but is now primarily applicable to Classic ELB, because -- as of 2016-12-07 -- Application Load Balancers (elbv2) can now be directly integrated with Web Application Firewall (Amazon WAF).
Starting [2016-12-07] AWS WAF (Web Application Firewall) is available on the Application Load Balancer (ALB). You can now use AWS WAF directly on Application Load Balancers (both internal and external) in a VPC, to protect your websites and web services. With this launch customers can now use AWS WAF on both Amazon CloudFront and Application Load Balancer.
https://aws.amazon.com/about-aws/whats-new/2016/12/AWS-WAF-now-available-on-Application-Load-Balancer/
It seems like you do need some clarification on how these pieces fit together.
So let's say your actual site that you want to secure is app.example.com.
It sounds as if you have a CNAME elb.example.com pointing to the assigned hostname of the ELB, which is something like example-123456789.us-west-2.elb.amazonaws.com. If you access either of these hostnames, you're connecting directly to the ELB -- regardless of what's configured in CloudFront or WAF. These machines are still accessible over the Internet.
The trick here is to route the traffic to CloudFront, where it can be firewalled by WAF, which means a couple of additional things have to happen: first, this means an additional hostname is needed, so you configure app.example.com in DNS as a CNAME (or Alias, if you're using Route 53) pointing to the dxxxexample.cloudfront.net hostname assigned to your distribution.
You can also access your sitr using the assigned CloudFront hostname, directly, for testing. Accessing this endpoint from the blocked IP address should indeed result in the request being denied, now.
So, the CloudFront endpoint is where you need to send your traffic -- not directly to the ELB.
Doesn't that leave your ELB still exposed?
Yes, it does... so the next step is to plug that hole.
If you're using a custom origin, you can use custom headers to prevent users from bypassing CloudFront and requesting content directly from your origin.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/forward-custom-headers.html
The idea here is that you will establish a secret value known only to your servers and CloudFront. CloudFront will send this in the headers along with every request, and your servers will require that value to be present or else they will play dumb and throw an error -- such as 503 Service Unavailable or 403 Forbidden or even 404 Not Found.
So, you make up a header name, like X-My-CloudFront-Secret-String and a random string, like o+mJeNieamgKKS0Uu0A1Fqk7sOqa6Mlc3 and configure this as a Custom Origin Header in CloudFront. The values shown here are arbitrary examples -- this can be anything.
Then configure your application web server to deny any request where this header and the matching value are not present -- because this is how you know the request came from your specific CloudFront distribution. Anything else (other than ELB health checks, for which you need to make an exception) is not from your CloudFront distribution, and is therefore unauthorized by definition, so your server needs to deny it with an error, but without explaining too much in the error message.
This header and its expected value remains a secret because it will not be sent back to the browser by CloudFront -- it's only sent in the forward direction, in the requests that CloudFront sends to your ELB.
Note that you should get an SSL cert for your ELB (for the elb.example.com hostname) and configure CloudFront to forward all requests to your ELB using HTTPS. The likelihood of interception of traffic between CloudFront and ELB is low, but this is a protection you should consider implenting.
You can optionally also reduce (but not eliminate) most unauthorized access by blocking all requests that don't arrive from CloudFront by only allowing the CloudFront IP address ranges in the ELB security group -- the CloudFront address ranges are documented (search the JSON for blocks designated as CLOUDFRONT, and allow only these in the ELB security group) but note that if you do this, you still need to set up the custom origin header configuration, discussed above, because if you only block at the IP level, you're still technically allowing anybody's CloudFront distribution to access your ELB. Your CloudFront distribution shares IP addresses in a pool with other CloudFront distribution, so the fact that the request arrives from CloudFront is not a sufficient guarantee that it is from your CloudFront distribution. Note also that you need to sign up for change notifications so that if new address ranges are added to CloudFront, then you'll know to add them to your security group.
I've just deployed my IdentityServer3 solution out to an AWS instance. I had IdentityServer3 configured to RequireSSL. The problem seems that it's relatively common to have SSL running up to the AWS load balancer, then run non-SSL between the LB and IIS. So, I set RequireSSL to false and tried again. Now the problem seems to be that when IdentityServer tries to redirect to the sign-in page, it does so without https. When this redirect exits and re-enters through the LB, it fails because it must be https coming into the LB. So, it seems that IdentityServer is constructing the redirect to the sign-in page with consideration of the RequireSSL flag. Is there anyway around this? I'm working with my ops guy to just run SSL all the way to IIS, but i'm getting some pushback (of course).
As the docs state here:
https://identityserver.github.io/Documentation/docsv2/advanced/deployment.html
If you want to terminate SSL on the load balancer, there are two relevant settings on the options:
RequireSsl
Set this to false to allow non-SSL connections between the load balancer and IdentityServer.
PublicOrigin
Since your internal farm nodes have different names than the public reachable address, IdentityServer can’t use it for link generation. Set this property to the public name.