I have an nginx AWS EC2 instance that forwards requests to an elastic load balancer (ELB) that is maintained by an auto scaling group (ASG). Now, these instances behind the elastic load balancer are born in and out of existence depending on the load on the instance that is set to 80% in the auto scaling group properties. Problem is that, this property cannot be changed and these new instances spin up with a new ip. But, nginx has cached the old IP it resolved before, for the previous version of this instance. We have to manually restart nginx everytime this happens and debugging takes time when I'm not around as well. We cannot use crontabs as we need a robust fix for this, kinda like a root cause fix.Suggestions, anyone? Thanks in advance :)
Related
I am running an app in AWS Beanstalk, I use jenkins to do automatic deploys, manage crons, ecc, jenkins connects to the EC2 behind Beanstalk using the public ip.
The problem arises when the instance scales, since the IP of the EC2 will be different, I have to manually update Jenkins every time.
One of the simplest options would be to open the port 22 in the loadbalancer, but since I am using the recommended application loadbalancer, it only allows me to open the port 80/443. I was wondering if there is a way to create a dns record in route 53, that will automatically point to the right IP every time it scales?
I would like to avoid changing load balancer, because there are at least 20 environments that will need to be reconfigured.
I tried to look but no-one seems to have this issue, so either I have the wrong architecture, or it is too easy to fix.
I will need to host a PHP Laravel application on Google Cloud Compute Engine with auto scaling and load balancing. I tried to setup and configure following:
I Created instance template, where I have added startup script to install apache2, PHP, cloning the git repository of my project, Configuring the Cloud SQL proxy, and configure all settings required to run this Laravel project.
Created Instance group, Where I have configured a rule when CPU reaches certain percent it start creating other instances for auto scale.
Created Cloud SQL instance.
Created Storage bucket, in my application all of the public contents like images will be uploaded into storage bucket and it will be served from there.
Created Load Balancer and assigned the Public IP to load balancer, configured the fronted and backed correctly for load balancer.
As per my above configuration, everything working fine, When a instance reaches a defined CPU percentage, Auto scaling start creating another instances and load balancer start routing the traffic to new instance.
The issue I'm getting, to configure and setup my environment(the startup script of instance template) takes about 20-30 minutes to configure and start ready to serve the content from the newly created instance. But when the load balancer detects if the newly created machine is UP and running it start routing the traffic to new VM instance which is not being ready to serve the content from it.
As a result, when load balancer routes the traffic to not ready machine, it obviously send me 404 error, and some other errors.
How to prevent to happen it, is there any way that the instance that created through auto scaling service send some information to load balancer after this machine is ready to serve the content and then only the load balancer route the traffic to the newly created instance?
How to prevent Google Cloud Load balancer to forward the traffic to
newly created auto scaled Instance without being ready?
Google Load Balancers use the parameter Cool Down to determine how long to wait for a new instance to come online and be 100% available. However, this means that if your instance is not available at that time, errors will be returned.
The above answers your question. However, taking 20 or 30 minutes for a new instance to come online defeats a lot of the benefits of autoscaling. You want instances to come online immediately.
Best practices mean that you should create an instance. Configure the instance with all the required software applications, etc. Then create an image of this instance. Then in your template specify this image as your baseline image. Now your instances will not have to wait for software downloads and installs, configuration, etc. All you need to do is run a script that does the final configuration, if needed, to bring an instance online. Your goal should be 30 - 180 seconds from launch to being online and running for a new instance. Rethink / redesign anything that takes longer than 180 seconds. This will also save you money.
John Hanley answer is pretty good, I'm just completing it a bit.
You should take a look at packer to create your preconfigured google images, this will help you when you need to add a new configuration or do updates.
The cooldown is a great way, but in your case you can't really be sure that your installation won't take a bit more time sometimes due to updates as you should do an apt-get update && apt-get upgrade at instance startup to be up to date it will only take more and more time...
Load balancers normally should have a health check configured and should not route traffic unless the instance is detected as healthy. In your case as you have apache2 installed I suppose you have a HC on the port 80 or 443 depending on your configuration on a /healthz path.
A way to use the health check correctly would be to create a specific vhost for the health check and you add a fake domain in the HC, let's say health.test, that would give a vhost listening for health.test and returning a 200 response on /healthz path.
This way if you don't change you conf, just activate the health vhost last so the loadbalancer don't start routing traffic before the server is really up...
Is it possible to upgrade EC2 instance without changing existing public IP address? My mobile application is live and unfortunately we didn't use elastic IP in web services. So if I upgrade current instance, it will generate new public IP and the old application users won't be able to use mobile application.
Is there any way to keep current IP as it is? Or any other way to upgrade it without loosing existing users? Please suggest.
Consider this a lesson as to why you should use a load balancer and a DNS entry, especially for anything public-facing. What were you going to do if you the instance failed? Or the availability zone went down?
Personally I would spin up a set new larger instances behind a load balancer, create a Route53 DNS entry that points to the load balancer, and then release an update to the client that points to the DNS entry. As clients update, traffic will gradually move to the load balancer. The undersized single instance's load will drop, so if it is overloaded it will eventually return to normal. Eventually you can kill the old instance when all/most clients have upgraded.
It depends on what sort of software you are running exactly
If you have an application that is sessionless then it would be simple to bring up another server on a different IP and then use route53 to switch over the traffic, with both servers running at the same time
If the application is stateful though and if it stores the sessions locally on the host then that's more of a problem
One possible approach is to bind an elastic ip to the running host, reconfigure your software to listen on all addresses ( a lot of configuration controls allow this with an address of 0.0.0.0 ) then change DNS and gradually see the traffic migrate to the elastic IP, while both addresses work
Once the new address is fully in use (depends on your TTL) it becomes much easier to switch to a new host by reassigning the EIP
We are using AutoScaling and Elastic Load Balancer from Amazon AWS. We are running three linux(ubuntu) server under a load balancer. When one of the three instance become unhealthy(Status check fails) our site become unavailable.
But other 2 instance was healthy at that time.
We figured out the actual cause of this problem. Actually apache was taking all the memory of the instance. At one point memory exhausted on the machine, so apache was not able to serve response.
Following is another question where the problem is described.
apache2 processes stuck in sending reply - W
I have an application on an Windows server EC2 with an SQL server for our database.
What I would like to do is an load balancer so the application won't fail due to overload.
I have a couple of questions that Im not certain about it.
I believe that i need to create an image of my current instance and duplicate it. my problem is that my database is based on my current instance so it would duplicate my database as well.
Do I need another instance just for my database?
If yes, then it means that I need a total of 3 instances. 2 for the application and 1 for the database.
In this case I need to change my application to connect to the new instance database instead of the current database.
After all that happens I need to add a load balancer.
I hope I made myself clear.
I would recommend using RDS (http://aws.amazon.com/rds/) for this. This way you don't have to worry about the database server and just host your application server on EC2 instances. Your AMI will then contain just the application server and thus when you scale up you will be launching additional app servers only and not database servers.
Since you are deploying a .NET application, I would also recommend taking a look at Elastic Beanstalk (http://aws.amazon.com/elasticbeanstalk/) since it will really help to make auto scaling much easier and your solution will scale up/down as well as self-heal itself.
As far the load balancer is concerned, you can either manually update your load balancer will the new instances of your application server or you can let your auto scale script do it for you. If you go for ElasticBeanstalk, then Elastic Beanstalk will take care of adding/removing instances to/from your Elastic Load Balancer for you on its own.