I'm looking into removing EBS from my EC2 instances and instead using instance storage (primarily to remove the cost of EBS - my applications can live with ephemeral storage just fine).
Trying to attach instance storage when creating an instance via the management console results in a Javscript error though, and searching for advice online I haven't been able to find anyone online who seems to be doing this - which is leading me to wonder if it's even possible.
Does anyone have experience with this and could lend some advice or pointers? Thanks.
Make sure you select an AMI that has Instance Store as its Root Device, by default you would get and EBS as your root volume.
You can set such filter in the AMI selection screen of the launch instance tool in the EC2 Management Console, here: http://grab.by/x3Lm
Related
I wanted to know is it possible to automate the creation of AMI in AWS without downtime and data loss, if possible how can we achieve it.
I have use system manager-> maintenance window in which i have set the reboot to true for data integrity, but i need a way so that the data is not lost.
Any help will be appreciated.
Thank-you.
Answering it as per comments discussion, question is somehow still vague to me
You have EBS right now. I'm not sure if your Instances are in Same AZ or not. If they are in same AZ then you can use EBS multi attach feature (available for IO volumes only) to share same storage with all of them.
Regarding backup you can choose EBS snapshots
Ideally my suggestion to you would be create a launch template, use EFS that can be mounted to multiple instances in same region, if you want it across regions then create mount targets. EFS is natively integrated with AWS backup.
Whenever any failover happens or your EC2 crashes for any reason and it goes less than your target capacity, auto scaling would automatically provision a new instance using launch template which would be using same EFS
but i need a way so that the data is not lost.
if you want to achieve this, then According to Docs, you need to ensure that application or os is not writing to ebs, which can be managed by either a script or a custom logic.
You can take a snapshot of an attached volume that is in use. However, snapshots only capture data that has been written to your Amazon EBS volume at the time the snapshot command is issued. This might exclude any data that has been cached by any applications or the operating system. If you can pause any file writes to the volume long enough to take a snapshot, your snapshot should be complete. However, if you can't pause all file writes to the volume, you should unmount the volume from within the instance, issue the snapshot command, and then remount the volume to ensure a consistent and complete snapshot.
if you achieved the above then you can automate the creation, retention, and deletion of EBS snapshots and EBS-backed AMIs it using Data Lifecycle Manager
I haven't tried this but I think exporting VM to S3 and then automating the entire pipeline with Ec2 image builder should do the trick, you can customise your further images with build components
Refers importing and exporting vm's
Unfortunately there is not of box solution other than compromising data integrity but you can try above mentioned which can ensure data integrity and automation
I have one r5.xlarge windows ec2 instance which is attached to 6TB EBS volume and the backup for EBS is taken every week.
Now I want a better solution with zero downtime.
if ec2 instance fails I want new ec2 instance to be to created and it should be attached to EBS volume automatically.
if EBS volume crashes I want backup snapshot to be available and gets attached to ec2 instance with zero downtime and volume content available immediately.
Is there a way to implement solution for this and how?
If you want to improve the availability of the large drive, I would recommend FSx. It's not so trivial - you need to set up Active Directory, and your EC2 needs to join AD - but once you have all these components (and there is additional cost as well) - FSx provides significantly higher availability than EBS assuming you're using a multi-AZ setup.
More information here
Note that load balancer tags are irrelevant to your question - although to address your first situation you probably need load balancer balancing two EC2 instances
You should try and use Elastic BeanStalk, this will handle the autoscaling on its own. You just have to configure the settings while creating it.
This should be the nearest solution that you need, as explained by Marcin above there is no zero downtime for 2 single point of failures. So Elastic BeanStalk seems to be the best solution for you.
Updated:
So in this case what you can do is add a load balance and add auto scaling, so that the first point of your questions is taken care and if the ec2 instance goes down through autoscaling a new instance will be create and in the shell script you can add commands to map to the EBS.
And for EBS, there isn't much you can do, you can take regular backups and if the EBS fails then replace it with the latest backup or snapshot, also check the below link:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-restoring-volume.html
I am working with a p2.xlarge instance on AWS, and it has 120 GiB storage. I am in need of more storage as it is running out of memory. I am looking into options of changing the storage, but I can't find a way to alter it. Does anyone know if this is possible at all? Or alternatively, should I launch a new instance and move all my work to the new instance? Tips on how to do this most efficiently are much appreciated.
Based on the comments.
You can extend EBS volume as shown in the AWS documentation:
Extending a Linux file system after resizing a volume and
How do I extend my Linux file system after increasing my EBS volume on my EC2 instance?
I have a php/mysql website that I want to deploy on AWS. Ultimately, I'm going to want auto-scaling (but don't need it right away).
I'm looking at an EBS based AMI. I see that by default the "Root Device Volume" is deleted when an instance "terminates". I realize I can also attach other EBS devices/drives to an instance (that will persist after termination) but I'm going to save most user content in S3 so I dont think that's necessary. I'm not sure how often I'll start/stop vs when i would ever want to terminate. So that's a bit confusing.
I'm mostly confused with where changes to the system get saved. Say I run a YUM install or update. Does that get saved in the "root device volume"? If i stop/start the instance, the changes should be there? What about if I setup cron jobs?
How about if I upload files? I understand to an extent that it depends where I put the files and if I attached a second EBS. Say I just put them in the root folder "/" (unadvised, but for simplicity sake). I guess that they are technically saved in the "root device volume"? If I start/stop the instance, they should still be there?
However, if I terminate an instance, then those changes/uploads are lost. But if I set the "root device volume" to not delete on termination, then I can launch a new instance with the changes there?
In terms of auto-scaling. Someone said to leave the "root device volume" to default delete so that when new instances are spun up/shut down, they don't leave behind zombie EBS volumes that are no longer needed (and would require manual clean-up)?
Would something like this work: ?
Setup S3 bucket (for shared image uploads)
Setup Amazon RDS / mysql
Setup DynamoDB (for sharing php sessions)
Launch EBS-backed AMI (leave as default to delete "root device volume" on
termination). Make system updates using yum/etc. Upload via sftp
PHP/HTML/JS/CSS files (ex: /var/www/html). Validate site can save
images to S3, share sessions via DynamoDB, access mysql via RDS.
Make/clone your own AMI image from your currently running/configured
one. Save it with a name that indicates site version/date/etc.
Setup auto-scaling to launch the image created in #5
I'm mostly concerned with how to save my configuration so that 1) changes are saved in-case i ever need to terminate an instance (before using autoscale) and 2) that auto-scaling will have access to the changes when I'm ready for it. I also don't want something like the same cron-job running on all auto-scaling instances.
I guess I'm confused with "does creating my own AMI image in #4" basically replace the "saving EBS root device volume" on termination? I can't wrap my head around the image part of things vs the storage part of things.
I get even more confused when I read about people talking about if you use "Amazon Linux" then the way they deploy updates every 6 months makes it difficult to use because you are forced to use new versions of software. How does that affect my custom AMI (with my uploaded code)? Can I just keep running yum updates on my custom AMI (for security patches) and ignore any changes to amazon's standard AMIs? When does the yum approach put me at risk for being out-of-date?
I know there are a host of things I'm not covering (dns/static IPs/scaling metrics/etc). That instead of uploading files then creating an AMI image, some people have their machine set to pull files from git on startup (i dont mind my more manual approach for now). Or that i could technically put the php/html/css/js on S3 too.
Sorry for all of the random questions. I know my question might not even be totally clear, but I'm just looking for confirmation/advice in-general. There are so many concepts to tie-together.
Thanks and sorry for the long post!
Yes, if you install packages, upload files, set up cron jobs, etc. and then stop the EBS-based instance, everything will still be there when you restart it.
Consequently, if you create an AMI from that instance and then use it for your autoscaling group, all the instances of the autoscaling group will run the cron jobs.
Your steps look good. As you are creating an AMI, your changes will be saved in that AMI. If the instance is terminated, it can be recreated via the AMI. The modifications made on that instance since the AMI creation will not be saved though. You need to create an AMI or take a snapshot of the EBS volume if you want a backup.
If you make a change and want to apply it to all the instances in the autoscaling group, you need to create a new AMI and apply it to your autoscaling group.
Concerning the cron jobs, I guess you have 2 options:
Have 1 instance that is not part of the autoscaling group running them (and disable the cron jobs before creating the AMI for the autoscaling group)
Do something smart so only one instance of the autoscaling group runs them. Here is the first page I hit on Google: https://gist.github.com/kixorz/5209217 (not tested)
Yes, creating your own AMI image basically replaces the "saving EBS root device volume" on termination.
An EBS boot AMI is an EBS snapshot of the EBS root volume plus some
metadata like the architecture, kernel, AMI name, description, block
device mappings, and more.
(From: AWS Difference between a snapshot and AMI)
Yes, you can automatically run yum security updates. To be completely identical to the latest Amazon Linux AMI, you should run all yum updates (not only security). But I wouldn't run those automatically.
Let me know if I forgot to answer some of your questions or if some points are still unclear.
I wanted to take a pre-existing t2.medium EC2 instance and essentially clone it. Knowing no exact feature exists within AWS I scoured their documentation, asked people I know and came up with:
Create snapshot of said EC2 instance
Create AMI of snapshot
Build EC2 instance from AMI
When I went to build the new EC2 I was only given the option for a t1.micro OR m3.medium and above. I tried both (in and out of same region as original) and kept getting "insufficient data" under Status Checks.
Any ideas on what is going on here?
It sounds like you selected "paravirtual" as the virtualization type when you created the AMI, when you should have selected "hvm".
If you are doing this from the console you can create an AMI directly from the pre-existing EC2 instance, skipping the manual snapshot step, which should automatically use the correct settings for the AMI.