I am trying to setup a VIP (virtual IP setup) - for a high availability HA setup for Redis with HA proxy, however to have a VIP for two HA instances in two different availability is proving to be difficult.
I was trying to follow this guide; https://aws.amazon.com/articles/2127188135977316 here, however this uses the same availability zone to achieve VIP floating IP, and this wouldn't work for me as my different availability zones are in different subnets.
I have tried the above mentioned example with an Elastic IP; however it is not transferring between machines as expected with the script (http://media.amazonwebservices.com/articles/vip_monitor_files/vip_monitor.sh)
Please could someone guide me how to approach this ?
The Leveraging Multiple IP Addresses for Virtual IP Address Fail-over in 6 Simple Steps article you reference is over 3 years old, so I wouldn't recommend this as a state-of-the-art method of doing failover.
The preferred method for HA is always to load balance between servers in multiple Availability Zones. Then, if one server or one AZ should fail, the other systems can take the full load of traffic (perhaps scaling-up to absorb the traffic).
For a requirement where only one server can be active at a time, switching DNS names or Elastic IP addresses would be recommended.
Option 1: Use Route 53 Health Checks to detect failure, then route the DNS name to an alternate server (may involve waiting until TTL timeout for any cached DNS resolutions)
Option 2: Use a static Elastic IP address and reassign it to an alternate server. This would involve some method to detect failure (eg the script in that article) and then an API request to associate the Elastic IP address to another server
Related
As far as high availability goes, what is the difference between using an Elastic IP or an Elastic Network Interface to mask instance failure? Is the only difference because ENIs can be used for private instances and Elastic IPs can't?
I'm trying to explain the advantages of both, so if someone can help me with this, I would appreciate it!
To achieve High Availability, you need the ability to redirect traffic in the case of instance failure. There are several options:
1. Use an Elastic Load Balancer
This is the preferred way to provide High Availability.
Run multiple Amazon EC2 instances, preferably in different Availability Zones (AZs). Users connect to the ELB (via the supplied DNS name), which redirects traffic to the EC2 instances. If an instance fails, ELB notices this via regular Health Checks, and will only direct traffic to the healthy instances.
Auto Scaling can be used to create these multiple instances across multiple Availability Zones, and it can also update the Load Balancing service when it adds/removes instances.
2. Redirect an Elastic IP address
Run multiple instances (preferably across multiple Availability Zones). Point an Elastic IP address to the instance you desire. Users connect via the Elastic IP address and are directed to the instance. If the instance fails, reassociate the Elastic IP address to a different instance, which will then start receiving the traffic immediately.
This method is not recommended because only one instance is receiving all the traffic while the other instance(s) are sitting idle. It also requires a mechanism to detect failure and reassociate the Elastic IP (which you must do yourself).
3. Reassign an Elastic Network Interface (ENI)
All EC2 instances have a primary ENI. They can optionally have additional ENIs.
It is possible to direct traffic to a secondary ENI and then move that secondary ENI to another instance. This is similar to reassigning an Elastic IP address.
This method is not recommended for the same reason as reassociating an Elastic IP address (above), but also because an ENI can only be reassigned within the same AZ. It cannot be used to direct traffic to an EC2 instance in a different AZ.
Bottom line: Use an Elastic Load Balancer. It provides true High Availability and can do it automatically.
See documentation: What Is Elastic Load Balancing?
How can I assign a static IP address to a ELB. Seems like I cannot.
Some articles online asks to create a Route 53 record but this requires changing CNAME of domain which also redirect email traffic. I just want to change A record not CNAME.
Some articles also mention that I can use a EC2 instance as a reverse proxy. But will a single proxy be able to handle a lot of traffic?
Any solution for this?
AWS' Elastic Load Balancer is actually elastic on two levels as described here:
http://shlomoswidler.com/2009/07/elastic-in-elastic-load-balancing-elb.html
The first level is the load balancer itself. In order to make sure that ELB can scale to whatever volume you have and burst to whatever volume you suddenly encounter, AWS assigns a 'static' DNS hostname (e.g. MyDomainELB-918273645.us-east-1.elb.amazonaws.com). That hostname points to multiple IP addresses. You can see that (from a command line) by running
$ host MyDomainELB-918273645.us-east-1.elb.amazonaws.com
MyDomainELB-918273645.us-east-1.elb.amazonaws.com 172.31.7.2
MyDomainELB-918273645.us-east-1.elb.amazonaws.com 172.31.11.33
The second form of elasticity within the ELB is obviously then ELB directing the query to one of your EC2 instances in the pool.
So, you can see that trying to assign a static IP address to the load balancer would be self-defeating.
Using an EC2 instance as a reverse proxy would also seem self-defeating as you would then create a bottleneck before even getting to the ELB. Might as well just create your own load balancer.
The recommended solution (which you've pointed out) is to create a CNAME that points to the ELB hostname (which won't change).
i.e. my-app.mycompany.com ->
MyDomainELB-918273645.us-east-1.elb.amazonaws.com
This would allow you to integrate your scalable application, behind the ELB within your domain.
I'm not sure I fully understand why you cannot create a CNAME in your DNS or what that has to do with directing email traffic, can you explain?
A new feature in AWS (I believe it was announced at Re:Invent 2017) allows for static IPs with Network Load Balancers (NLB). NLB can only handle layer 4 (TCP) and not HTTP specifics (layer 7).
You can assign one Elastic IP address per availability zone.
For details see the AWS blog post or the NLB documentation.
The "Classic Load Balancer" and "Application Load Balancer" do not support static IPs. If you need a feature only provided by those, you have to fall back to the CNAME solution described above.
A blog was recently published by AWS support on this topic leveraging NLB to provide static IP to Classic and Application load balancer - https://aws.amazon.com/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/
Summary of solution as described by the post
We end up with a TCP listener on a NLB that accepts traffic and forwards it to an internal ALB. The ALB terminates TLS, examines HTTP headers, and routes requests based on your configured rules to target groups with your instances, servers, or containers. The AWS Lambda function keeps everything in sync by watching the ALB for IP address changes and updating the NLB target group. In the end we’ll have a few static IP addresses that are easy for whitelisting, and we won’t lose any of the benefits of ALB. Note that we will be sending all of the traffic through two load balancers
I found setting up AWS Global Accelerator very straight forward and simple. It created 2 static IP Addresses and a static DNS pointing to my Application load balancer.
Configuring Global Accelerator
Set listeners as TCP port 80, 443
Select your load balancer endpoint (AWS Global Accelerator Configuration)
Add cname record for your dns pointing to the static dns it created
(mywebsite.com > globalacceleratorDNS.com). If any client needs to
whitelist, give them the 2 static IP it created
Pricing is $18 per month + a few pennies per GB of data transfer.
I'm pretty sure its cheaper than the NLB, Nat Gateway, Elastic IP setup.
https://docs.aws.amazon.com/global-accelerator/latest/dg/about-accelerators.html
For little traffic, it might be a solution to set up an EC2 Instance running Nginx as a forwarding proxy.
So you can use the EC2's static IP Address to forward your traffic resolving the ALB's DNS name.
However, it's a kind of a hack, but using a Global Accelerator or an NLB seems to me also like a hack :-)
Unlike the Network Load Balancer, the Application Load Balancer (ALB) does not support Elastic IPs, but that's not the worst part. If you use Route 53 together with the ALB, the DNS automatically sets the TTL to 60 seconds. This appears to be causing problems for our institutional - mainly government - customers running older Windows DNS servers. They just can't keep up with the ALB's Listener changing its public-facing IP on such a short notice. Older DNS infrastructure is either not respecting or is not capable of handling such aggressive TTL.
While I don't like it, AWS recommends to put a Network Load Balancer in front of the Application Load Balancer, per here: https://aws.amazon.com/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/
I recently started reading about and playing around with AWS. I have particular interest in the different high availability architectures that can be acheived using the platform. Specifically, I am looking for a reliable poor man's solution that can be implemented using the least amount of servers.
So far, I am satisfied with solutions for the main HA concerns: load balancing, redundancy, auto recovery, scalability ...
The only sticking point I have is with failover solutions.
Using an ELB might seem great, however ELB actually uses DNS balancing under the hood. See Is AWS's Elastic Load Balancer a single point of failure?. Also from a Netflix blog post: Lessons Netflix Learned from the AWS Outage
This is because the ELB is a two tier load balancing scheme. The first tier consists of basic DNS based round robin load balancing. This gets a client to an ELB endpoint in the cloud that is in one of the zones that your ELB is configured to use.
Now, I have learned DNS failover is not an ideal solution, as others have pointed out, mainly because of unpredictable DNS caching. See for example: Why is DNS failover not recommended?.
Other than ELBs, it seems to me that most AWS HA architectures rely on DNS failover using route 53.
Finally, the floating IP/Elastic IP (EIP) strategy has popped up in a very small number of articles, such as Leveraging Multiple IP Addresses for Virtual IP Address Fail-over and I'm having a hard time figuring out if this is a viable solution for production systems. Also, all examples I came across implemented this using a set of active-passive instances. It seems like a waste to have a passive for every active to achieve this.
In light of this, I would like to ask you what is a faster and more reliable way to perform failover?
More specifically, please discuss how to perform failover without using DNS for the following 2 setups:
2 active-active EC2 instances in seperate AZs. Active-active, because this is a budget setup, were we can't afford to have an instance sitting around.
1 ELB with 2 EC2 instances in region A, 1 ELB with 2 EC2 instances in region B. Again, both regions are active and serving traffic. How do you handle the failover from 1 ELB to the other?
You'll understand ELB better by playing with it, if you are the inquisitive type, as I am.
"1" ELB provisioned in 2 availability zones is billed as 1 but deployed as 2. There are 2 IP addresses assigned, one to each balancer, and 2 A records auto-created, one for each, with very short TTLs.
Each of these 2 balancers will forward traffic to the instance in its same AZ, or you can enable cross-AZ load balancing (and you should, if you only have 1 server instance in each AZ).
These IP addresses do not change often and though it stands to reason that ELBs fail like anything else, I have maybe 30 of them and have never knowingly had a dead one on my hands, presumably because the ELB infrastructure will replace a dead instance and change the DNS without your intervention.
For 2 regions, you have little choice other than using DNS at some level. Latency-based routing from Route 53 can send people to the closest site in normal operations and route all traffic to the other site in the event of an outage of an entire region (as detected by Route 53 health checks), but with this is somewhat more likely to encounter issues with DNS caching when the entire region is unavailable.
Of course, part of the active/passive dilemma in a single region using Elastic IP is easily remedied with HAProxy on both app servers. It's an http request router and load balancer like ELB, but with a broader set of features. The code is so tight that you can likely run it on your app servers with negligible CPU consumption. The instance with the EIP would then balance traffic between its local app server and the peer. Across regions, HAProxy behind ELB could forward traffic to a mate in a remote region, if the local region is up but for whatever reason the application can't serve requests from the local region. (I have used such a setup to increase availability of external services, by bouncing the request to a remote AWS region when the direct Internet path from the local region is not working.)
We plan to do some performance test on a web site hosted on some Amazon EC2 instances. The question is, if all the HTTP traffic come from the same IP addresses (say, in the case of many different client hosts sharing the same public IP), will the EC2 loadbalancer forward all traffic to only one of the web servers (instances)?
This used to be a problem due to DNS caching by clients indeed (see my answer to Can Elastic Load Balancers correctly distribute traffic to different size instances for more on the previous state of affairs), but has mostly been remedied apparently with the recent introduction of Elastic Load Balancing [...] Cross-Zone Load Balancing:
We are pleased to announce support for cross-zone load balancing, which changes the way that Elastic Load Balancing (ELB) routes incoming requests, making it even easier for you to deploy applications across multiple Availability Zones. [emphasis mine]
The announcements provides a bit more information already, but links to Request Routing for details:
If you enable cross-zone load balancing, you no longer have to worry that clients caching DNS information will result in requests being distributed unevenly. And now, ELB ensures that requests are distributed equally to your back-end instances regardless of the Availability Zone in which they are located. [emphasis mine]
This still doesn't outline the exact algorithm used, but the main source of uneven distribution seems to be addressed like so (I have read a post by a large AWS customer recently, who reported their metrics becoming a more or less flat curve after flipping the switch on this, but don't recall the URL right now).
This has been the case in the past, I presume it still may be an issue. Its not so much that it will forward all traffic to a single instance, but it can send all traffic to a single zone.
The ELB does some load balancing with DNS round robin. A cached DNS lookup, could result in multiple requests being routed to the same zone.
I have runnning e-commerce based ruby on rails application on AWS stack. I am running ubuntu 10.04 ec2 instances with Elastic load balancer and I have maintained equal number of instance in the both the availability zone, 1a and 1b. But according to my observation, ELB seeming to be pushing more traffic to 1a rather then dividing it equally. Though the health of the instances running in 1b is good and also I have disabled the sticky session on the ELB. I have 2 large and 1 medium instances running on both the availability zones.
What the cause of in equal distribution of the load.
In my experience, this can happen if a disproportionate amount of traffic is coming from a single network or ip address.
ELB uses different layers of balancing. DNS load balancing will send it to a set of IP addresses in one of the two zones, and the software load balancer will distribute traffic between instances in the zone.
If you have a lot of traffic coming from the same network, its likely that a lot of users are getting the same DNS resolution on your load balancer and ending up in the same zone.
If the source traffic is coming from a single Network/IP range or IP address, ELB might load balance the traffic disproportionately to the backend. I have discussed this point as well as couple of other details to note on ELB in my blog "Dissecting ELB". I have also noticed this behavior in some popular OSS LB implementations as well, You can have balance algorithm as "source" and as well as session sticky combined . If session ID is not sent on the HTTP, then it will load balance based "source".