I have an ubuntu server (ami-714ba518) which has a webserver on it (Apache and PHP). It is connected to a MySQL server on RDS. I want to setup auto scaling when load goes above 60-70% , but I'm having trouble getting my head around how it works.
My concern is that, when a change is made to a file in server, how does that change reflect on the other instances that auto scaling has started?
Regards!
Scaling http servers assumes that they are stateless, meaning that the state of them is dedicated to persistence layer - database, and not to servers themselves. If one of your instances writes something on its disk, this data won't be available to autoscaled new instances. Changes like this do not replicate by autoscaling feature. You can achieve that with shared file system via EFS, though, but it won't be replication. It'll be a shared resource.
P.S. Automatic EFS volumes attachment described here: https://docs.aws.amazon.com/efs/latest/ug/mount-fs-auto-mount-onreboot.html
The Autoscaling doesn't replicate the changes from your host server. If you already done the changes in your site, then may be you can create Custom AMI from your existing EC2 instance, and then use the same for launching new instances using autoscaling.
You need to use some common back-end service which will serve your changes to all of your web servers (existing or newly launched).
Take a look at s3 or efs.
Related
I have Amazon Aurora for MySQL t3.db.medium instances. I would like to scale down to t3.db.small.
If I modify the instance settings in AWS console, will my DB data be preserved? So can I scale down without service interruption? I think I should be able to do this, but I just wanna make sure. There is prod instance involved.
I have the same question about Elastic Cache (AWS redis). Can I scale that down without service interruption?
According to Docs, there is table(DB instance class) which tells which settings can be changed, you can change your instance class for your aurora, as a note An outage occurs during this change.
For redis
according to docs, you can scale down node type of your redis cluster (version 3.2 or newer). During scale down ElastiCache dynamically resizes your cluster while remaining online and serving requests.
In both the cases your data will be preserved.
I'm trying AWS auto-scaling for the first time, as far as I understand it creates instances if for example my CPU Utilization reaches critical level, that I define.
So I am curious, after I lunch my instance I spend a fair amount of time configuring it and copying the data, if AWS auto-scales my instance how will it configure the new instances and move the data to it?
You can't store any data that you want to keep on an instance that is part of an autoscaling group (well you can, but you will lose it).
There are (at least) two ways to answer your question:
Create a 'golden image', in other words spin-up an instance, configure it, install the software etc and then save it as an AMI (amazon machine image). Then tell the autoscaling group to use that AMI each time an instance starts - it will be pre-configured when it starts.
Put a script on the instance that tells the instance how to configure itself when it starts up (in the user data). SO basically each time an instance scales up, it runs the script and does all the steps it needs to to configure itself.
As for you data, best practice would be to store any data you want to keep in a database or object store that is not on the instance - so something like RDS, DynamoDB or even S3 objects.
You could also use AWS EFS, store there your data/scripts that the EC2 Instances will be sharing, and automatically mount it every time a new EC2 Instance is created via /etc/fstab.
Once you have configured the EFS to be mounted on the EC2 Instance (/etc/fstab), you should create a new AMI, and use this new AMI to create a new Launch Configuration and AutoScaling Group, so that the new Instances automatically mount your EFS and are able to consume that shared data.
https://aws.amazon.com/efs/faq/
Q. What use cases is Amazon EFS intended for?
Amazon EFS is designed to provide performance for a broad spectrum of
workloads and applications, including Big Data and analytics, media
processing workflows, content management, web serving, and home
directories.
Q. When should I use Amazon EFS vs. Amazon Simple Storage Service (S3)
vs. Amazon Elastic Block Store (EBS)?
Amazon Web Services (AWS) offers cloud storage services to support a
wide range of storage workloads.
Amazon EFS is a file storage service for use with Amazon EC2. Amazon
EFS provides a file system interface, file system access semantics
(such as strong consistency and file locking), and
concurrently-accessible storage for up to thousands of Amazon EC2
instances. Amazon EBS is a block level storage service for use with
Amazon EC2. Amazon EBS can deliver performance for workloads that
require the lowest-latency access to data from a single EC2 instance.
Amazon S3 is an object storage service. Amazon S3 makes data available
through an Internet API that can be accessed anywhere.
https://docs.aws.amazon.com/efs/latest/ug/mount-fs-auto-mount-onreboot.html
You can use the file fstab to automatically mount your Amazon EFS file
system whenever the Amazon EC2 instance it is mounted on reboots.
There are two ways to set up automatic mounting. You can update the
/etc/fstab file in your EC2 instance after you connect to the instance
for the first time, or you can configure automatic mounting of your
EFS file system when you create your EC2 instance.
I recommend using a shared data container if it is data that is updated and the updated data is needed by all instances that might be spinning up.
If it is database data or you could store the needed data in a database I would consider using an RDS.
If it is static data only used to configure the instances like dumps or configuration files which are not updated by running instances then I would recommend pulling them from CloudFlare or S3 of iT is not possible to pull them from a repository.
Good luck
Last day I wanted, according to AWS recommendations, put my ec2 instance inside of an autoscaling group. I created my ec2 instance by using the standard linux AMI instance and then I installed a full LAMP server.
The next morning I tried accessing my apache and guess what? My LAMP wasn't there anymore! Everything was wiped away.
I guess this is because, for some reason, the autoscaling group deleted my instance and recreated it vanilla.
Now I still want to autoscale my instance but, of course, I want to keep my LAMP and the stored data.
So here's my questions:
How to create a customized image starting from my actual instance?
Would it be correct to create the mysql DB using AWS RDS so to not keep it linked to my instance?Is it more or less expensive than dedicating a EBS storage?
I also want to keep my /var/www/html data somewhere shared between instances: while it is true that, on production, I won't update those files often it is also true that I don't want to lose them just because the autoscaling resets my instance state. I also don't want to re-create an image each time I update said files... What's the best way?Maybe an s3 bucket? Or, still, an EBS storage shared between instances?
I would assume that the reason that your "LAMP [server] wasn't there anymore" was because the web server failed health checks and was terminated and replaced by AutoScaling.
Elastic Beanstalk would be a good way to manage some of the complexity here. If that's not an option then you should read up on AutoScaling, ALB, and health checks.
In response to your specific questions:
you can create an Amazon Machine Image (AMI) from an instance. When you, or AutoScaling, launch a new instance from that AMI, you can get the instance up to date by running a script in userdata
move the DB from the web/app server to RDS, or to a DB server that you manage yourself
maintain the html/js/css etc. in S3 and sync them to your web server periodically (there are other options, but that's simple)
I am working on a project and am at a point where the POC is done and now want to move towards a real product. I am trying to understand the Amazon cloud offerings just to see if I need to be aware of them at development time. I have a bunch of questions that I cannot get answered from the Amazon site. Its probably because I am new to the whole web services thing and have never hosted a site before. I am hoping someone out here will explain this to me like I am a C programmer :)
I see amazon has a bunch of offerings -
EC2
Elastic Block Store
Simple DB
AuotScaling
Elastic Load Balancing
I understand EC2 is virtual server instances that I can use and these could come pre-loaded with what I want (say Apache + python). I have the following questions -
If I want a custom instance of something (like say a custom apache module I wrote for my project). Can I create a server instance using the exact modules and make it the default the next time I create a new instance or in Autoscaling?
Do I get an IP Address to access this? Can I set my own hostname to it? I mean do I get a DNS record? Or is it what Elastic IP is?
How do I access it from the outside? SSH? Remote Desktop? Or is it entirely up to how I configure the instance?
What do they mean by Inter-Region or Intra-Region data transfer? What is data transfer to begin with? Is it just people using my instance? So if I go live with it that will be the cost I have to pay for people using it?
What is the difference between AutoScaling and Elastic Load Balancing?
What is Elastic Block Store? Is it storage? If so do I have to worry about backups or do they take care of it?
About the Simple DB -
It looks like the interface to use this is different to my regular SQL calls. Am I correct?
If so the whole development needs to be tailored specifically for Amazon. Which kind of sucks. Is there a better alternative?
Do I get data backups or do I have to worry about it myself?
Will I be able to connect to the DB using regular tools to inspect the DB (during or afte development). Or do I get other tools made by Amazon for it?
What about security? The DB is obviously somewhere in the cloud farm away from the EC2 instance. My DB password is going over the wire and so is all my data totally unencrypted. Don't I have to worry about that? The question comes up only because I don't own any of the hardware.
I really hope some one points me in the right direction here.
Thanks for taking the time to read.
P
I just went through the question and here I tried to answer few of them,
1) AWS EC2 instances doesnt publish pre-configured instances, in fact its configured by the developers and made it publicly available to the users so that they can use it. One can any one of those instances or you can just opt for what ever OS you want which is raw and provision it accordingly and create a snap shot of it so that you can use it for autos caling.The snap shot becomes the base AMI in your case.
2) Every instance you boot will have a public DNS attach to it, you can use the public DNS to connect to that instance using ssh if your are a linux user or using putty if you are a windows users. Apart from that, you can also attach a elastic IP which comes with a cost will is like peanuts and attach it to the instance and access your instance through the elastic IP and you can either map the public DNS or elastic ip to map to a website by adding a A record or Cname respectively.
3)AWS owns databases in the different parts of the world. For example you deploy your application depending upon your customer base, if you target customers are based out of India, the nearest region available is Singapore which is called as ap-southeast-1 by AWS. Each region will have multiple availability zones, example ap-southeast-1a and ap-southeast-1b, which are two different databases and geographically part. Intre region means from ap-southeast-1a to ap-southeast-1b. Inter Region means, from ap-southeast-1 to us-east-1 which is Northern Virginia Data centre. AWS charges from in coming and out going bandwidth, trust me its nothing.
They chargge 1/8th of a cent per GB. Its a thing to even think about it.
4)Elastic Load balancer is cluster which divides the load equally to all your regions across availability zones (if you are running in multi AZ) ELB sits on top the AWS EC2 instances and monitors the instance health periodically and enables auto scaling
5) To help you understand what is autoscaling please go through this document http://aws.amazon.com/autoscaling/
6)Elastic Block store or EBS are like hard disk which is a persistent data storage which can be attached to your instance.Regarding back up yes dependents upon your use case. I do backups of EBS periodically.
7)Simple Db now renamed as dynamo DB is nosql DB, I hope you understand what is nosql db, its a non RDMS db systems. Please read some documentation to understand what is nosql db is.
8)If you have mysql or oracle db you can opt for RDS, please read the documents.
9)I personally feel you are newbie to the entire cloud eco system, you need to understand what exactly cloud does first.
10)You dont have to make large number of changes to development as such, just make sure it works fine in your local box, it can be deployed to cloud with out much ado.
11) You dont have to use any extra tool for that, change the database end point to RDS(if your use it) or else install mysql in your ec2 instance and connect to the local db which resides in the ec2 instance and connect to it,which is as simple as your development mode.
12)You dont have to worry about any security issues aws, it is secured. Dont follow the myths, I am have been using aws since 3 years running I dont even know remember how many applications, like(e-commerce,m-commerce,social media apps) I never faced any kind of security issues and also aws allows to set your security how ever you want.
Go ahead, happy coding. Contact me if you have any problem.
The answer above is a good summary on AWS. Just wanted to add
AWS offers full data center, so it depends what you are trying to achieve. For starters you will need,
EC2 - This is your server, it comes with instance storage, which will be lost on restart
EBS - Your mounted storage, the data is persisted across reboots
S3 - Provides storage (RESTful API's on top, the cost is usage based rather than "provisioned" as in EBS)
Databases - can start with Amazon RDS, which provides managed database services, you can chose between various available databases. You can also install your own database using EC2 + EBS, you will have to take care of managing the database yourself.
Elastic IP: Public facing IP address, you can point your DNS server to this.
One great tool to calculate the pricing,
http://calculator.s3.amazonaws.com/calc5.html
Some other services to take in account are:
VPC (Virtual Private Cloud). This is your own private network. You can define subnets, route tables and internet gateways there. I would strongly recommend to use VPC for any serious deployment of more than one instance.
Glacier - this will replace your tape library to storing backups.
Cloud Formation - great tool for deployment and automation of instances.
I'm just getting started with AWS EC2 and not entirely sure I understand it.
From what I've read, an instance is basically a virtual server, and you should be able to somehow "duplicate" that virtual server from the AWS console somehow. Then use Load Balancer or Elastic IP to route requests to one or the other.
The problem comes in when I try to "duplicate" my instance. I tried a million things, but the only thing that got me close was creating an AMI of my current instance then launching an instance from that, but when I did that, the new instance was basically the default server config. None of my files were there.
What am I doing wrong?
You don't really "duplicate" the instance. You more copy it as a "blueprint". Then when you boot an instance later, you can base that instance off of your snapshot or "blueprint".
The ELB can be configured to point at any instance you want, so when you boot a new server off this snapshot/"blueprint" it can be automatically added to the ELB.
Now that is cleared up, to answer the question:
I would make sure to use EBS backed instances. You can find them all over. But not S3 backed. If they EBS backed then the exact volume with all your configs will be there.
I would make sure your instance is configured how you like it and has proper scripts installed for when it boots up. You will want your services started, config files pulled down from repositories, etc. The config files should be there, but I would not rely on that. If you have them in a repository and then make a startup script to pull them down and copy them where you want, you will be in much better shape.
With the instance running and selected, click on the instance actions drop down and click "Create AMI"
The instance will REBOOT. So be careful.
Launch a new instance. And pick the AMI/Snapshot that #3 created.
Done. Check this https://stackoverflow.com/a/8919031/667608 that could help with the above.
Oh, one other thing, if you have any EBS Volumes attached, they will also be copied, but you will need to mount them once the server boots.
Under instances, click on the image you want to duplicate and then go to instance action(its near the top) and create ami.
This creates a snapshot of your image as it is right now. Then when you need to add more power, you can simply launch that ami and have the load balancer distrubute the traffic between those ami's.
On a side note, unless really required, I would not suggest you store data on the ami if its changing and you plan to use it on another launched ami. You'll pretty much have to keep taking ami snapshots to update it with the new data, so instead figure a way to maintain state somewhere else(not sure about your data but you can consider a database, s3, or another server that these servers can mount to get the same data).
Hope that helps!
Create a webserver AMI using EBS backed instance. This will serve as your template for running multiple web-server instance later.
For the app codes, depending on your strategy and amount of files to transfer, you can pull them from S3 or git or maybe using a centralized filesystem such as NFS.
Configure the ELB, add one or more web server instances to it. CNAME your ELB's public dns to your www.domain.com.