I recently started reading about and playing around with AWS. I have particular interest in the different high availability architectures that can be acheived using the platform. Specifically, I am looking for a reliable poor man's solution that can be implemented using the least amount of servers.
So far, I am satisfied with solutions for the main HA concerns: load balancing, redundancy, auto recovery, scalability ...
The only sticking point I have is with failover solutions.
Using an ELB might seem great, however ELB actually uses DNS balancing under the hood. See Is AWS's Elastic Load Balancer a single point of failure?. Also from a Netflix blog post: Lessons Netflix Learned from the AWS Outage
This is because the ELB is a two tier load balancing scheme. The first tier consists of basic DNS based round robin load balancing. This gets a client to an ELB endpoint in the cloud that is in one of the zones that your ELB is configured to use.
Now, I have learned DNS failover is not an ideal solution, as others have pointed out, mainly because of unpredictable DNS caching. See for example: Why is DNS failover not recommended?.
Other than ELBs, it seems to me that most AWS HA architectures rely on DNS failover using route 53.
Finally, the floating IP/Elastic IP (EIP) strategy has popped up in a very small number of articles, such as Leveraging Multiple IP Addresses for Virtual IP Address Fail-over and I'm having a hard time figuring out if this is a viable solution for production systems. Also, all examples I came across implemented this using a set of active-passive instances. It seems like a waste to have a passive for every active to achieve this.
In light of this, I would like to ask you what is a faster and more reliable way to perform failover?
More specifically, please discuss how to perform failover without using DNS for the following 2 setups:
2 active-active EC2 instances in seperate AZs. Active-active, because this is a budget setup, were we can't afford to have an instance sitting around.
1 ELB with 2 EC2 instances in region A, 1 ELB with 2 EC2 instances in region B. Again, both regions are active and serving traffic. How do you handle the failover from 1 ELB to the other?
You'll understand ELB better by playing with it, if you are the inquisitive type, as I am.
"1" ELB provisioned in 2 availability zones is billed as 1 but deployed as 2. There are 2 IP addresses assigned, one to each balancer, and 2 A records auto-created, one for each, with very short TTLs.
Each of these 2 balancers will forward traffic to the instance in its same AZ, or you can enable cross-AZ load balancing (and you should, if you only have 1 server instance in each AZ).
These IP addresses do not change often and though it stands to reason that ELBs fail like anything else, I have maybe 30 of them and have never knowingly had a dead one on my hands, presumably because the ELB infrastructure will replace a dead instance and change the DNS without your intervention.
For 2 regions, you have little choice other than using DNS at some level. Latency-based routing from Route 53 can send people to the closest site in normal operations and route all traffic to the other site in the event of an outage of an entire region (as detected by Route 53 health checks), but with this is somewhat more likely to encounter issues with DNS caching when the entire region is unavailable.
Of course, part of the active/passive dilemma in a single region using Elastic IP is easily remedied with HAProxy on both app servers. It's an http request router and load balancer like ELB, but with a broader set of features. The code is so tight that you can likely run it on your app servers with negligible CPU consumption. The instance with the EIP would then balance traffic between its local app server and the peer. Across regions, HAProxy behind ELB could forward traffic to a mate in a remote region, if the local region is up but for whatever reason the application can't serve requests from the local region. (I have used such a setup to increase availability of external services, by bouncing the request to a remote AWS region when the direct Internet path from the local region is not working.)
Related
I'm working my way through a practice exam for an AWS certification. One of the questions is as follows:
The web tier for an a pplication is running on 6 EC2 instances spread
across 2 AZs behind a classic ELB. The data tier is a MySQL database
running on an EC2 instance. What changes will increase the
availability of the application? (select TWO)
A: Turn on CloudTrail in the AWs account
B: Migrate the MySQL database to a Multi-AZ RDS MySQL database instance
C: Turn on cross-zone load balancing on the ELB
D: Launch the web tier EC2 instances in an Auto Scaling Group
E: Increase the instance size of the web tier EC2 instance
Correct answers are B and D. My question is, why is C NOT a correct answer? The instructor (an Amazon employee) glosses over C, explaining that "enabling cross-zone load balancing would have little to no effect on availability." But the way I'm looking at it, if the ELB can't send traffic to both AZ's, then we're effectively making our 6-instance system into a 3-instance system (assuming there are 3 in each AZ). And a single AZ system is never the considered a highly available architecture, since if that one AZ fails, your whole system is unavailable.
Enabling cross-zone load balancing does not impact availability because ELBs can send traffic to all configured AZs without the feature enabled. That's not what cross-zone balancing means.
An ELB configured in two availability zones always has at least two balancer nodes, one in each AZ. You can't see this, directly, but if you look under "Network Interfaces" in the EC2 console, you can find the Elastic Network Interfaces (ENIs) attached to the balancer nodes. Each node has one ENI. The service determines how many nodes a balancer has, based on load. This is managed automatically, and you are not billed based on node count.
Cross-zone load balancing controls what each node can do. "Enabled" means the balancer node in zone A can send traffic to instances in zone A or B, instead of just to instances in zone A, and the same for the balancer node in zone B.
This doesn't improve availability because if an availability zone is lost, then the balancer node in that zone is also lost, so the fact that it could have sent traffic to instances in the other zone is immaterial.
Cross-zone load balancing helps ensure that the workload is spread as evenly as possible across all instances behind the balancer, which helps if you have asymmetry -- such as 3 application instances in one AZ and 2 application instances in the other (in this case, the zone with 2 would see proportionally more traffic per instance than the zone with 3) -- or other cases where the instances are not seeing evenly-balanced workloads, which would be more likely when the number of instances behind the balancer is small or if there is wide variation in request processing time due to the complexity of certain requests compared to others.
What changes will increase the availability of the application?
Increased availability means that there are less time periods where the application is serving requests.
(B) Multi-AZ database will certainly help because if one AZ fails, it will automatically promote the secondary database server in the other AZ
(D) Auto Scaling will certainly help because failed instances will be replaced.
Cross-zone load balancing would help where there are no healthy instances available in an AZ but traffic is being handled by the ELB in that AZ. It is an unlikely scenario, especially with 3 instances in an AZ, but I could understand an argument for it. However, the other two answers are much stronger.
It's worth mentioning that official AWS Certification questions go through several levels of technical review and shouldn't leave such ambiguity in a question. Sample exam questions (be it in an AWS course or otherwise) probably haven't gone through such detailed scrutiny.
We plan to do some performance test on a web site hosted on some Amazon EC2 instances. The question is, if all the HTTP traffic come from the same IP addresses (say, in the case of many different client hosts sharing the same public IP), will the EC2 loadbalancer forward all traffic to only one of the web servers (instances)?
This used to be a problem due to DNS caching by clients indeed (see my answer to Can Elastic Load Balancers correctly distribute traffic to different size instances for more on the previous state of affairs), but has mostly been remedied apparently with the recent introduction of Elastic Load Balancing [...] Cross-Zone Load Balancing:
We are pleased to announce support for cross-zone load balancing, which changes the way that Elastic Load Balancing (ELB) routes incoming requests, making it even easier for you to deploy applications across multiple Availability Zones. [emphasis mine]
The announcements provides a bit more information already, but links to Request Routing for details:
If you enable cross-zone load balancing, you no longer have to worry that clients caching DNS information will result in requests being distributed unevenly. And now, ELB ensures that requests are distributed equally to your back-end instances regardless of the Availability Zone in which they are located. [emphasis mine]
This still doesn't outline the exact algorithm used, but the main source of uneven distribution seems to be addressed like so (I have read a post by a large AWS customer recently, who reported their metrics becoming a more or less flat curve after flipping the switch on this, but don't recall the URL right now).
This has been the case in the past, I presume it still may be an issue. Its not so much that it will forward all traffic to a single instance, but it can send all traffic to a single zone.
The ELB does some load balancing with DNS round robin. A cached DNS lookup, could result in multiple requests being routed to the same zone.
I am running couple of instances under my aws elastic load balancer. Say I have 6 large ubuntu instances running under the elb. The problem what I am facing right now is load is not evenly distributed across the availability zones. I am running 3 large instances on ap-southeast-1a and 3 in ap-southeast-1b. But elb is distributing more load on the 1b and the instances stop responding since it hits 100% CPU and elb automatically throws the instances out of it's control which causes the downtime. DNS is parked in Godaddy.com.
How do I make sure that elb distributes equally to the available regions.
Kindly help me!!!
There could be a number of reasons for this. Its without doing more digging, its hard to know which one you are experiencing.
Sticky sessions can result in instances traffic becoming unbalanced. Although this depends heavily on usage patterns and your application.
Cached DNS resolution. Part of how the ELB works is to direct traffic round robin on a DNS level. If a large number of users are all using the same DNS system provided by an ISP, they might all get sent to the same zone. Couple this with sticky sessions and you will end up with a bunch of traffic that will never switch. Using Route 53 with ALIAS records may reduce this somewhat.
If you can't get the ELB to balance your traffic better, you can set up something similar with vanish cache or other software load balancer. Not as convenient, but you will ultimately have more control.
I have runnning e-commerce based ruby on rails application on AWS stack. I am running ubuntu 10.04 ec2 instances with Elastic load balancer and I have maintained equal number of instance in the both the availability zone, 1a and 1b. But according to my observation, ELB seeming to be pushing more traffic to 1a rather then dividing it equally. Though the health of the instances running in 1b is good and also I have disabled the sticky session on the ELB. I have 2 large and 1 medium instances running on both the availability zones.
What the cause of in equal distribution of the load.
In my experience, this can happen if a disproportionate amount of traffic is coming from a single network or ip address.
ELB uses different layers of balancing. DNS load balancing will send it to a set of IP addresses in one of the two zones, and the software load balancer will distribute traffic between instances in the zone.
If you have a lot of traffic coming from the same network, its likely that a lot of users are getting the same DNS resolution on your load balancer and ending up in the same zone.
If the source traffic is coming from a single Network/IP range or IP address, ELB might load balance the traffic disproportionately to the backend. I have discussed this point as well as couple of other details to note on ELB in my blog "Dissecting ELB". I have also noticed this behavior in some popular OSS LB implementations as well, You can have balance algorithm as "source" and as well as session sticky combined . If session ID is not sent on the HTTP, then it will load balance based "source".
I'm trying to get my EC2 instances to communicate better with APIs of a 3rd party service. Latency is extremely important as voice communication is heavily involved & lag is intolerable.
I know a few of the providers use EC2, but the thing is Amazon's IP system makes it difficult to find which region the instance is in. With non elastic-ip services I could do a whois and find if it was in Australia or somewhere in Europe so I could put a server close by.
With these elastic IP's how can I find which zone they're in. I can use ping times but its a bit of a guess and I have to make all these instances in different regions to find the shortest ping time.
Amazon EC2 regularly publishes its Amazon EC2 Public IP Ranges, which clusters them by Region.
It does not cluster them by Availability Zone (AZ) (if you actually meant that literally), but this shouldn't matter much, insofar cross AZ latency should regularly be within single digit milliseconds range only.
Other than that you might also be interested in my answer to How could I determine which AWS location is best for serving customers from a particular region?, which outlines two other options for handling this based on external data/algorithms or via the Multi-Region Latency Based Routing now Available for AWS (which would likely only be useful when fully embracing Amazon Route 53 as well).
Put your server behind a Route 53 DNS and let Latency Based Routing do the rest for you - it can decide automatically for you the least latent server.
http://aws.typepad.com/aws/2012/03/latency-based-multi-region-routing-now-available-for-aws.html