Last day I wanted, according to AWS recommendations, put my ec2 instance inside of an autoscaling group. I created my ec2 instance by using the standard linux AMI instance and then I installed a full LAMP server.
The next morning I tried accessing my apache and guess what? My LAMP wasn't there anymore! Everything was wiped away.
I guess this is because, for some reason, the autoscaling group deleted my instance and recreated it vanilla.
Now I still want to autoscale my instance but, of course, I want to keep my LAMP and the stored data.
So here's my questions:
How to create a customized image starting from my actual instance?
Would it be correct to create the mysql DB using AWS RDS so to not keep it linked to my instance?Is it more or less expensive than dedicating a EBS storage?
I also want to keep my /var/www/html data somewhere shared between instances: while it is true that, on production, I won't update those files often it is also true that I don't want to lose them just because the autoscaling resets my instance state. I also don't want to re-create an image each time I update said files... What's the best way?Maybe an s3 bucket? Or, still, an EBS storage shared between instances?
I would assume that the reason that your "LAMP [server] wasn't there anymore" was because the web server failed health checks and was terminated and replaced by AutoScaling.
Elastic Beanstalk would be a good way to manage some of the complexity here. If that's not an option then you should read up on AutoScaling, ALB, and health checks.
In response to your specific questions:
you can create an Amazon Machine Image (AMI) from an instance. When you, or AutoScaling, launch a new instance from that AMI, you can get the instance up to date by running a script in userdata
move the DB from the web/app server to RDS, or to a DB server that you manage yourself
maintain the html/js/css etc. in S3 and sync them to your web server periodically (there are other options, but that's simple)
Related
I have an ubuntu server (ami-714ba518) which has a webserver on it (Apache and PHP). It is connected to a MySQL server on RDS. I want to setup auto scaling when load goes above 60-70% , but I'm having trouble getting my head around how it works.
My concern is that, when a change is made to a file in server, how does that change reflect on the other instances that auto scaling has started?
Regards!
Scaling http servers assumes that they are stateless, meaning that the state of them is dedicated to persistence layer - database, and not to servers themselves. If one of your instances writes something on its disk, this data won't be available to autoscaled new instances. Changes like this do not replicate by autoscaling feature. You can achieve that with shared file system via EFS, though, but it won't be replication. It'll be a shared resource.
P.S. Automatic EFS volumes attachment described here: https://docs.aws.amazon.com/efs/latest/ug/mount-fs-auto-mount-onreboot.html
The Autoscaling doesn't replicate the changes from your host server. If you already done the changes in your site, then may be you can create Custom AMI from your existing EC2 instance, and then use the same for launching new instances using autoscaling.
You need to use some common back-end service which will serve your changes to all of your web servers (existing or newly launched).
Take a look at s3 or efs.
I'm trying AWS auto-scaling for the first time, as far as I understand it creates instances if for example my CPU Utilization reaches critical level, that I define.
So I am curious, after I lunch my instance I spend a fair amount of time configuring it and copying the data, if AWS auto-scales my instance how will it configure the new instances and move the data to it?
You can't store any data that you want to keep on an instance that is part of an autoscaling group (well you can, but you will lose it).
There are (at least) two ways to answer your question:
Create a 'golden image', in other words spin-up an instance, configure it, install the software etc and then save it as an AMI (amazon machine image). Then tell the autoscaling group to use that AMI each time an instance starts - it will be pre-configured when it starts.
Put a script on the instance that tells the instance how to configure itself when it starts up (in the user data). SO basically each time an instance scales up, it runs the script and does all the steps it needs to to configure itself.
As for you data, best practice would be to store any data you want to keep in a database or object store that is not on the instance - so something like RDS, DynamoDB or even S3 objects.
You could also use AWS EFS, store there your data/scripts that the EC2 Instances will be sharing, and automatically mount it every time a new EC2 Instance is created via /etc/fstab.
Once you have configured the EFS to be mounted on the EC2 Instance (/etc/fstab), you should create a new AMI, and use this new AMI to create a new Launch Configuration and AutoScaling Group, so that the new Instances automatically mount your EFS and are able to consume that shared data.
https://aws.amazon.com/efs/faq/
Q. What use cases is Amazon EFS intended for?
Amazon EFS is designed to provide performance for a broad spectrum of
workloads and applications, including Big Data and analytics, media
processing workflows, content management, web serving, and home
directories.
Q. When should I use Amazon EFS vs. Amazon Simple Storage Service (S3)
vs. Amazon Elastic Block Store (EBS)?
Amazon Web Services (AWS) offers cloud storage services to support a
wide range of storage workloads.
Amazon EFS is a file storage service for use with Amazon EC2. Amazon
EFS provides a file system interface, file system access semantics
(such as strong consistency and file locking), and
concurrently-accessible storage for up to thousands of Amazon EC2
instances. Amazon EBS is a block level storage service for use with
Amazon EC2. Amazon EBS can deliver performance for workloads that
require the lowest-latency access to data from a single EC2 instance.
Amazon S3 is an object storage service. Amazon S3 makes data available
through an Internet API that can be accessed anywhere.
https://docs.aws.amazon.com/efs/latest/ug/mount-fs-auto-mount-onreboot.html
You can use the file fstab to automatically mount your Amazon EFS file
system whenever the Amazon EC2 instance it is mounted on reboots.
There are two ways to set up automatic mounting. You can update the
/etc/fstab file in your EC2 instance after you connect to the instance
for the first time, or you can configure automatic mounting of your
EFS file system when you create your EC2 instance.
I recommend using a shared data container if it is data that is updated and the updated data is needed by all instances that might be spinning up.
If it is database data or you could store the needed data in a database I would consider using an RDS.
If it is static data only used to configure the instances like dumps or configuration files which are not updated by running instances then I would recommend pulling them from CloudFlare or S3 of iT is not possible to pull them from a repository.
Good luck
We have created an instance (m4.large) where we installed Windows Server 2012 r2. It is currently used as a terminal server for SAP business one software with license Remote Desktop Service per device (CAL). We made a mistake in choosing the instance type for it is recommended to use m4.xlarge for terminal servers. We would like to upgrade our instance type to resolve memory requirement issue.
We are new to AWS and would like to seek advise on what would be our best approach. Is it safe to directly upgrade the instance? We have no concerns of downtime for now.
We are also considering in making a snapshot first then launch it on an new and upgraded instance. We would like to ask if RDS licenses will also be migrated?
Thanks you for your kind replies.
You can change instance type anytime, but the instance must be stopped. If you are not concerned with some minutes of downtime just stop the instance, change the instance type and start the instance. Taking a snapshot as a backup before and frequently is a good precaution.
This change will have no impact on RDS, as that is a separate service.
One way would be simply to stop instance and upgrade it.
Warning
When you stop an instance, the data on any instance store volumes is
erased. Therefore, if you have any data on instance store volumes that
you want to keep, be sure to back it up to persistent storage.
Any Amazon EBS volumes remain attached to the instance, and their data
persists.
Any data stored in the RAM of the host computer or the instance store
volumes of the host computer is gone.
Another Way if you want minimal Downtime is you can create an AMI of it and Spin up new instance with Instance type and size what ever you desire and assign the domain or Elastic IP of current instance to new one and terminate old instance.
Also Regarding The RDS Licensing Please Mention what you mean by upgrading , but there will an outage when you apply for instance upgrade or License is Modified.
Also Upgrading EC2 Instance has no affect on Existing RDS as both are different services.
For Reference on Ms SQL DB , Please read these AWS Documentation:
Modifying a DB Instance Running the Microsoft SQL Server Database Engine
Microsoft SQL Server on Amazon RDS
As m4.large is EBS backed instance. so the loss of data is not a concern in the above question. Just select->right click-> stop the instance and change the instance type to m4.xlarge by right click->instance settings->instance type.
Also for your second question RDS is a separate service, so it won't get affected.
I have a php/mysql website that I want to deploy on AWS. Ultimately, I'm going to want auto-scaling (but don't need it right away).
I'm looking at an EBS based AMI. I see that by default the "Root Device Volume" is deleted when an instance "terminates". I realize I can also attach other EBS devices/drives to an instance (that will persist after termination) but I'm going to save most user content in S3 so I dont think that's necessary. I'm not sure how often I'll start/stop vs when i would ever want to terminate. So that's a bit confusing.
I'm mostly confused with where changes to the system get saved. Say I run a YUM install or update. Does that get saved in the "root device volume"? If i stop/start the instance, the changes should be there? What about if I setup cron jobs?
How about if I upload files? I understand to an extent that it depends where I put the files and if I attached a second EBS. Say I just put them in the root folder "/" (unadvised, but for simplicity sake). I guess that they are technically saved in the "root device volume"? If I start/stop the instance, they should still be there?
However, if I terminate an instance, then those changes/uploads are lost. But if I set the "root device volume" to not delete on termination, then I can launch a new instance with the changes there?
In terms of auto-scaling. Someone said to leave the "root device volume" to default delete so that when new instances are spun up/shut down, they don't leave behind zombie EBS volumes that are no longer needed (and would require manual clean-up)?
Would something like this work: ?
Setup S3 bucket (for shared image uploads)
Setup Amazon RDS / mysql
Setup DynamoDB (for sharing php sessions)
Launch EBS-backed AMI (leave as default to delete "root device volume" on
termination). Make system updates using yum/etc. Upload via sftp
PHP/HTML/JS/CSS files (ex: /var/www/html). Validate site can save
images to S3, share sessions via DynamoDB, access mysql via RDS.
Make/clone your own AMI image from your currently running/configured
one. Save it with a name that indicates site version/date/etc.
Setup auto-scaling to launch the image created in #5
I'm mostly concerned with how to save my configuration so that 1) changes are saved in-case i ever need to terminate an instance (before using autoscale) and 2) that auto-scaling will have access to the changes when I'm ready for it. I also don't want something like the same cron-job running on all auto-scaling instances.
I guess I'm confused with "does creating my own AMI image in #4" basically replace the "saving EBS root device volume" on termination? I can't wrap my head around the image part of things vs the storage part of things.
I get even more confused when I read about people talking about if you use "Amazon Linux" then the way they deploy updates every 6 months makes it difficult to use because you are forced to use new versions of software. How does that affect my custom AMI (with my uploaded code)? Can I just keep running yum updates on my custom AMI (for security patches) and ignore any changes to amazon's standard AMIs? When does the yum approach put me at risk for being out-of-date?
I know there are a host of things I'm not covering (dns/static IPs/scaling metrics/etc). That instead of uploading files then creating an AMI image, some people have their machine set to pull files from git on startup (i dont mind my more manual approach for now). Or that i could technically put the php/html/css/js on S3 too.
Sorry for all of the random questions. I know my question might not even be totally clear, but I'm just looking for confirmation/advice in-general. There are so many concepts to tie-together.
Thanks and sorry for the long post!
Yes, if you install packages, upload files, set up cron jobs, etc. and then stop the EBS-based instance, everything will still be there when you restart it.
Consequently, if you create an AMI from that instance and then use it for your autoscaling group, all the instances of the autoscaling group will run the cron jobs.
Your steps look good. As you are creating an AMI, your changes will be saved in that AMI. If the instance is terminated, it can be recreated via the AMI. The modifications made on that instance since the AMI creation will not be saved though. You need to create an AMI or take a snapshot of the EBS volume if you want a backup.
If you make a change and want to apply it to all the instances in the autoscaling group, you need to create a new AMI and apply it to your autoscaling group.
Concerning the cron jobs, I guess you have 2 options:
Have 1 instance that is not part of the autoscaling group running them (and disable the cron jobs before creating the AMI for the autoscaling group)
Do something smart so only one instance of the autoscaling group runs them. Here is the first page I hit on Google: https://gist.github.com/kixorz/5209217 (not tested)
Yes, creating your own AMI image basically replaces the "saving EBS root device volume" on termination.
An EBS boot AMI is an EBS snapshot of the EBS root volume plus some
metadata like the architecture, kernel, AMI name, description, block
device mappings, and more.
(From: AWS Difference between a snapshot and AMI)
Yes, you can automatically run yum security updates. To be completely identical to the latest Amazon Linux AMI, you should run all yum updates (not only security). But I wouldn't run those automatically.
Let me know if I forgot to answer some of your questions or if some points are still unclear.
I'm just getting started with AWS EC2 and not entirely sure I understand it.
From what I've read, an instance is basically a virtual server, and you should be able to somehow "duplicate" that virtual server from the AWS console somehow. Then use Load Balancer or Elastic IP to route requests to one or the other.
The problem comes in when I try to "duplicate" my instance. I tried a million things, but the only thing that got me close was creating an AMI of my current instance then launching an instance from that, but when I did that, the new instance was basically the default server config. None of my files were there.
What am I doing wrong?
You don't really "duplicate" the instance. You more copy it as a "blueprint". Then when you boot an instance later, you can base that instance off of your snapshot or "blueprint".
The ELB can be configured to point at any instance you want, so when you boot a new server off this snapshot/"blueprint" it can be automatically added to the ELB.
Now that is cleared up, to answer the question:
I would make sure to use EBS backed instances. You can find them all over. But not S3 backed. If they EBS backed then the exact volume with all your configs will be there.
I would make sure your instance is configured how you like it and has proper scripts installed for when it boots up. You will want your services started, config files pulled down from repositories, etc. The config files should be there, but I would not rely on that. If you have them in a repository and then make a startup script to pull them down and copy them where you want, you will be in much better shape.
With the instance running and selected, click on the instance actions drop down and click "Create AMI"
The instance will REBOOT. So be careful.
Launch a new instance. And pick the AMI/Snapshot that #3 created.
Done. Check this https://stackoverflow.com/a/8919031/667608 that could help with the above.
Oh, one other thing, if you have any EBS Volumes attached, they will also be copied, but you will need to mount them once the server boots.
Under instances, click on the image you want to duplicate and then go to instance action(its near the top) and create ami.
This creates a snapshot of your image as it is right now. Then when you need to add more power, you can simply launch that ami and have the load balancer distrubute the traffic between those ami's.
On a side note, unless really required, I would not suggest you store data on the ami if its changing and you plan to use it on another launched ami. You'll pretty much have to keep taking ami snapshots to update it with the new data, so instead figure a way to maintain state somewhere else(not sure about your data but you can consider a database, s3, or another server that these servers can mount to get the same data).
Hope that helps!
Create a webserver AMI using EBS backed instance. This will serve as your template for running multiple web-server instance later.
For the app codes, depending on your strategy and amount of files to transfer, you can pull them from S3 or git or maybe using a centralized filesystem such as NFS.
Configure the ELB, add one or more web server instances to it. CNAME your ELB's public dns to your www.domain.com.