I've spent a few days just going over Elastic Beanstalk trying to identify the benefits of using this. I'm new to this service but also don't have much experience with EC2, so I'm really just trying to make sense of everything. My main objective is to have auto-scaling and Elastic seemed to make sense.
Last night, suddenly my existing instance terminated a new one was spun-up (automatically). Upon SSH'ing to this new instance, all my files were gone. I expected the volume to be replicated over?
I'm just not understanding what took place and why a instance was spun up and my other terminated - or more importantly, why the new instance didn't have my files.
Here is a log of what happened:
2013-12-26 23:03:23 UTC-0800 WARN Environment health has transitioned from YELLOW to RED
2013-12-26 23:01:21 UTC-0800 WARN Environment health has transitioned from GREEN to YELLOW
2013-12-26 23:01:21 UTC-0800 WARN Elastic Load Balancer awseb-e-i-AWSEBLoa-K5TNOB5OZNKU has zero healthy instances.
2013-12-26 23:00:36 UTC-0800 INFO Removed instance 'i-c75df99a' from your environment. (Reason: Instance is in 'shutting-down' state)
2013-12-26 22:55:14 UTC-0800 INFO Adding instance 'i-4d46d010' to your environment.
2013-12-26 22:54:14 UTC-0800 INFO Added EC2 instance 'i-4d46d010' to Auto Scaling Group 'awseb-e-ikszmdzite-stack-AWSEBAutoScalingGroup-TC41QI6DT3O0'.
Is this because I have 2 availability zones? I'm really confused.
Update
When I developed my Elastic Environment, I indicated that I wanted to use multiple availability zones. I then identified 2 zones to use. I indicated to use a minimum of 1 instance. I feel that this is where the problem happened -- I should have set the minimum to the same number of zones I identified. But I can't confirm except continued testing... Still looking for insight.
The storage on an EC2 instance is ephemeral, and is gone when that instance terminates. Rather than uploading your codebase to that specific instance manually, you should let Elastic Beanstalk do it for you. That way, you application's code base, including previous versions of it, are stored with Elastic Beanstalk, which is automatically deployed to new instances when they are spun up.
For example, for a PHP application, this link explains how it can be deployed using Elastic Beanstalk:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_PHP_eb.html
Related
I am using AWS Elastic Beanstalk. In there, I selected a Traffic Splitting deploy strategy, with a 100% split (so that 100% of new instances will have the new version and have their health evaluated).
Here's how (according to their documentation) that is supposed to work:
During a traffic-splitting deployment, Elastic Beanstalk creates a new set of instances in a separate temporary Auto Scaling group. Elastic Beanstalk then instructs the load balancer to direct a certain percentage of your environment's incoming traffic to the new instances. Then, for a configured amount of time, Elastic Beanstalk tracks the health of the new set of instances. If all is well, Elastic Beanstalk shifts remaining traffic to the new instances and attaches them to the environment's original Auto Scaling group, replacing the old instances. Then Elastic Beanstalk cleans up—terminates the old instances and removes the temporary Auto Scaling group.
And more specifically:
Rolling back the deployment to the previous application version is quick and doesn't impact service to client traffic. If the new instances don't pass health checks, or if you choose to abort the deployment, Elastic Beanstalk moves traffic back to the old instances and terminates the new ones.
However, it seems silly that it only looks at my internal /health health checks, and not the overall health status of the environment, from the HTTP status codes, that it already has information on.
I tried the following scenario:
Deploy a new version.
As soon as the "health evaluation period" begins, flood the server with error 500s (from an endpoint I made specifically for this purpose).
AWS then moves all my instances into "degraded" state, and "unhealthy", but then seems to ignore it, and goes on anyway.
See the following two log dump screenshots (they are oldest-first).
Is there any way that I can make AWS respect the HTTP status based health checks that it already performs, during a traffic split? Or am I bound to only rely on custom-developed health checks entirely?
Update 1: Even weirder, I tried making my own healthchecks fail always too, but it still decides to deploy the new version with the failed healthcheck!
Update 2: I noticed that the temporary auto scaling group that it creates while assessing health does only have an "EC2" type health check, and not "ELB". I think that might be the root cause. If I could only get it to use "ELB" instead.
That is interesting! I do not know if setting the health check type to "ELB" may do the job because we use CodeDeploy, which has far better rollback capabilities than AWS Elastic Beanstalk.
However, there is a well-document way in the docs [1] to apply the setting you are looking for:
[...] By default, the Auto Scaling group, created for your environment uses Amazon EC2 status checks. If an instance in your environment fails an Amazon EC2 status check, Auto Scaling takes it down and replaces it.
Amazon EC2 status checks only cover an instance's health, not the health of your application, server, or any Docker containers running on the instance. If your application crashes, but the instance that it runs on is still healthy, it may be kicked out of the load balancer, but Auto Scaling won't replace it automatically. [...]
If you want Auto Scaling to replace instances whose application has stopped responding, you can use a configuration file to configure the Auto Scaling group to use Elastic Load Balancing health checks. The following example sets the group to use the load balancer's health checks, in addition to the Amazon EC2 status check, to determine an instance's health.
Example .ebextensions/autoscaling.config
Resources:
AWSEBAutoScalingGroup:
Type: "AWS::AutoScaling::AutoScalingGroup"
Properties:
HealthCheckType: ELB
HealthCheckGracePeriod: 300
It does not mention the new traffic splitting deployment feature, though.
Thus, I cannot confirm this is the actual solution, but at least you can give it a shot.
[1] https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environmentconfig-autoscaling-healthchecktype.html
Once upon a time I thought that the Immutable Deployment option in Elastic Beanstalk was a holy panacea -- but it only works when a deployment involves no changes to the application's database schema.
We've now resorted to blue-green deployments. However, this only works if you control the DNS. If you are a SaaS solution and you allow customers to create a CNAME then B/G is often a spectacular failure as the enterprise: a) sets a very high TTL, and/or b) their internal DNS or firewalls caches the underlaying IP addresses of the ALB (which are dynamic and, of course, replaced when you swap the URL of the blue and green environments).
Traffic splitting is written as an option in the Elastic Beanstalk documentation.
But it's not actually an option in the configuration section in the console.
This wouldn't be the first time I've seen Elastic Beanstalk's docs are out of date so it could be AWS have removed that feature.
Since AWS introduced CodeStar I suspect Elastic Beanstalk is getting the cold shoulder.
I will need to host a PHP Laravel application on Google Cloud Compute Engine with auto scaling and load balancing. I tried to setup and configure following:
I Created instance template, where I have added startup script to install apache2, PHP, cloning the git repository of my project, Configuring the Cloud SQL proxy, and configure all settings required to run this Laravel project.
Created Instance group, Where I have configured a rule when CPU reaches certain percent it start creating other instances for auto scale.
Created Cloud SQL instance.
Created Storage bucket, in my application all of the public contents like images will be uploaded into storage bucket and it will be served from there.
Created Load Balancer and assigned the Public IP to load balancer, configured the fronted and backed correctly for load balancer.
As per my above configuration, everything working fine, When a instance reaches a defined CPU percentage, Auto scaling start creating another instances and load balancer start routing the traffic to new instance.
The issue I'm getting, to configure and setup my environment(the startup script of instance template) takes about 20-30 minutes to configure and start ready to serve the content from the newly created instance. But when the load balancer detects if the newly created machine is UP and running it start routing the traffic to new VM instance which is not being ready to serve the content from it.
As a result, when load balancer routes the traffic to not ready machine, it obviously send me 404 error, and some other errors.
How to prevent to happen it, is there any way that the instance that created through auto scaling service send some information to load balancer after this machine is ready to serve the content and then only the load balancer route the traffic to the newly created instance?
How to prevent Google Cloud Load balancer to forward the traffic to
newly created auto scaled Instance without being ready?
Google Load Balancers use the parameter Cool Down to determine how long to wait for a new instance to come online and be 100% available. However, this means that if your instance is not available at that time, errors will be returned.
The above answers your question. However, taking 20 or 30 minutes for a new instance to come online defeats a lot of the benefits of autoscaling. You want instances to come online immediately.
Best practices mean that you should create an instance. Configure the instance with all the required software applications, etc. Then create an image of this instance. Then in your template specify this image as your baseline image. Now your instances will not have to wait for software downloads and installs, configuration, etc. All you need to do is run a script that does the final configuration, if needed, to bring an instance online. Your goal should be 30 - 180 seconds from launch to being online and running for a new instance. Rethink / redesign anything that takes longer than 180 seconds. This will also save you money.
John Hanley answer is pretty good, I'm just completing it a bit.
You should take a look at packer to create your preconfigured google images, this will help you when you need to add a new configuration or do updates.
The cooldown is a great way, but in your case you can't really be sure that your installation won't take a bit more time sometimes due to updates as you should do an apt-get update && apt-get upgrade at instance startup to be up to date it will only take more and more time...
Load balancers normally should have a health check configured and should not route traffic unless the instance is detected as healthy. In your case as you have apache2 installed I suppose you have a HC on the port 80 or 443 depending on your configuration on a /healthz path.
A way to use the health check correctly would be to create a specific vhost for the health check and you add a fake domain in the HC, let's say health.test, that would give a vhost listening for health.test and returning a 200 response on /healthz path.
This way if you don't change you conf, just activate the health vhost last so the loadbalancer don't start routing traffic before the server is really up...
EB dashboard:
Removed instance [i-0c6e4cba4392d1ace] from your environment.
And if I'm on the EC2 instance, I get these messages on console:
Broadcast message from root#ip-172-31-20-119
(unknown) at 21:20 ...
The system is going down for power off NOW!
Connection to 54.186.171.133 closed by remote host.
Connection to 54.186.171.133 closed.
Any pointers on why this is happening and how can I debug this? Are there any logs which can I look after the instance has terminated?
It is likely that the Auto Scaling group associated with your Elastic Beanstalk application decided to scale-in the number of instances.
You can go to Auto Scaling in the EC2 console, find the Auto Scaling group and look at the History tab to determine why it happened (eg due to low CPU load).
It might also be because the instance failed a Health Check, so Auto Scaling removed it.
I have previously seen it done by having one EC2 instance running HAProxy, configured via a json file/lambda function, that in turn controlled the traffic with sticky sessions, into two separate elasticbeanstalk applications. So we have two layers of load balancing.
However, this has a few issues, one being: Testing several releases becomes expensive, requires more and more EB applications.
By canary release, I mean, being able to release to only a percentage of traffic, to figure out any errors that escaped the devs, the review process, and the QA process, without affecting all traffic.
What would be the best way to handle such a setup with AWS resources and not break the bank? :)
I found this Medium article that explain the usage of passive autoscaling group where you deploy the canary version into it and monitor for statistics. Once you are satisfied with the result, you can change the desired count for the canary autoscaling group to 0, and perform rolling upgrade to the active autoscaling group.
Here is the link to the article: https://engineering.klarna.com/simple-canary-releases-in-aws-how-and-why-bf051a47fb3f
The way you would achieve canary testing with elastic beanstalk is by
Create a 2nd beanstalk environment to which you deploy the canary release
Use a Route53 Weighted routing policy to send a percentage of the DNS requests to your canary environment.
If you're happy with the performance of the canary you can then route 100% of the traffic to the canary env, etc.
Something to keep in mind with DNS routing is, that the weighted routing is not an exact science since clients cache DNS based on the TTL you set in Route53. In the extreme scenario where you would have e.g. only one single client calling your beanstalk environment (such as a a single web server) and the TTL is set to 5 minutes, it could happen that the switching between environments only happens every 5 minutes.
Therefore for weighted routing it is recommended to use a fairly low TTL value. Additionally having many clients (e.g. mobile phones) works better in conjunction with DNS routing.
Alternatively it might be possible to create a separate LB in front of the two beanstalk environments that balances requests between the beanstalk environments. However I'm not 100% sure if a LB can sit in front other (beanstalk) LBs. I suspect the answer is not but I haven tried yet.
Modifying the autoscaling group in elastic beanstalk is not possible, since the LB is managed by beanstalk and beanstalk can decide to revert the changes you did manually on the LB. Additionally beanstalk does not allow you to deploy to a subset of instances while keeping the older version on another subset.
Hope this helps.
Traffic splitting is supported natively by Elastic Beanstalk.
Be sure to select a "high availability" config preset when creating your application environment (by clicking on "configure more options"), as this will configure a load balancer for your env:
Then edit the "Rolling updates and deployments" section of your environment and choose "Traffic splitting" as your deployment strategy.
I have a test environment setup (along with a staging and production). The staging environment has spun down automatically in to red (health).
I would like to spin down the test server as well, is this possible without destroying the actual instance? I will be using it again in a few months, but until then I don't need it consuming resources.
Server == Instance.
Termination ==> killing a server, no chance to get it back.
Stop => Stopping a server, it can be restarted. No EC2 costs while it's stopped. ( EBS costs may apply.)
Having a single node Beanstalk environment means having 1 EC2 instance. I never tried to stop it, but you should be able to do it. ( Just check it a few minutes ago, if AWS spins up a new instance or not. In a single node setup it shouldn't.)