So I am trying to upload my entire windows volume to S3 glacier and I am following this guide: https://docs.aws.amazon.com/cli/latest/userguide/cli-services-glacier.html but this works in chunks. What if my volume is of 500GB? I want to automate this so that I don't have to upload in chunks. I see that tools like FastGlacier have this capability but it is not what I am looking for, I am looking for a CLI-based solution.
For reference, the subject volume is live on an EC2 instance and target is a vault in S3 glacier.
Any help would be highly appreciated. Thanks!
Rather than backing up your volume to Glacier, I'd recommend using an Amazon EBS Snapshot, or create an AMI. This works very nicely and is fast to restore.
Also, rather than using Glacier, I would recommend you store the data in Amazon S3 and choose a Glacier or Glacier Deep Archive storage class. This is much easier to manage.
Related
I am looking to build a lambda function as part of a forensics workflow that will copy a particular EBS snapshot to a manually created S3 bucket in order to store for short/long term forensics requirements. Looking for any pointers!
The copy_snapshot option is not helpful here. It copies an EBS snapshot to an AWS-controlled S3 bucket (in a different region). It's not to an S3 bucket under your control and you have no direct access to it.
If you genuinely want to export an EBS snapshot to your own S3 bucket, or even to some storage device external to AWS, then you need to do it manually.
One way is as follows (some details are thanks to this serverfault answer):
launch an EC2 instance
create an EBS volume from your EBS snapshot
attach, but do not mount, the EBS volume to your instance
export the data (to S3 or elsewhere) using a tool such as dd
There may be tools available that actually implement this series of steps for you, though I was not able to locate any with a quick search.
I want examples on how to backup an EC2 snapshots to S3 bucket, and import it back afterwards.
I found the AWS CLI can export the snapshots to S3, and was explained here
Copying aws snapshot to S3 bucket
I also found the import command from AWS CLI reference, but I failed to execute that command, as I don't follow understand the option
https://docs.aws.amazon.com/cli/latest/reference/ec2/import-snapshot.html
can someone explain how to use this command? especially on how to specific which file on the S3 bucket to import from?
EC2 snapshots are by default stored on S3 standard storage. However, you cannot copy the snapshot to a specific S3 bucket using the AWS CLI.
There may be some third party tool out there somewhere that can do it, but I do not see any reason why you would need to download a snapshot to your s3 bucket? It's like paying for the snapshot twice!!!
Could you mention why you have this requirement? An easier alternate to your problem might exist.
Note:
The two links that you shared in your question, do not copy a snapshot to S3.
The first link shows how to copy a snapshot from one region to another, while the second link is to export a disk image into an EBS snapshot and only the following disk formats are supported for this import:
Virtual Hard Disk (VHD/VHDX)
ESX Virtual Machine Disk (VMDK)
Raw
If I am reading your question correctly, you are having trouble with choosing the bucket from which to restore your backup. You might find this easier using the EC2 console.
In the console - Navigation bar - select Snapshots
Select the snapshot you want to copy from the list
Choose Copy from Action list, complete the dialog box and click Copy
When the confirmation dialog box comes up if you click Snapshots then you can monitor the progress.
Here's some additional information on AWS backups that might help you.
Amazon suggests using a backup EFS and data pipeline for EFS backups but I am thinking of a simpler method to do the backup on a s3 bucket. What are your thoughts/suggestions, any scripts?
I would recommend to use whatever Amazon does. However, for smaller projects where cost effectiveness overweights reliability, custom approaches could be utilized.
Checkout the s3 sync AWS CLI command. I believe that is what you are looking for:
http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
I am having an Ubuntu ec2 instance at AWS. I took AMI for the instance.
I want to store the AMI to S3 bucket. Is there any way? Also is there anyway to export AMI from S3 bucket?
Update: This feature is now available
From Store and restore an AMI using S3 - Amazon Elastic Compute Cloud:
You can store an Amazon Machine Image (AMI) in an Amazon S3 bucket, copy the AMI to another S3 bucket, and then restore it from the S3 bucket. By storing and restoring an AMI using S3 buckets, you can copy AMIs from one AWS partition to another, for example, from the main commercial partition to the AWS GovCloud (US) partition. You can also make archival copies of AMIs by storing them in an S3 bucket.
--- Old Answer ---
It is not possible to export an AMI.
An Amazon Machine Image (AMI) is a copy of an Elastic Block Store (EBS) volume. The AMI is stored in Amazon S3, but it is not accessible via the S3 service. Think of it as being stored in AWS's own S3 bucket, rather than yours.
If you wish to export a disk image, use a standard disk utility to copy the disk to ISO format, which can then be copied and mounted on other VMs.
Thank you John.
Hi Guys, I had chat with AWS support also. For your reference ,
10:15:45 AM Myself: Well i have some doubts. I will ask and just clear in me on that.
10:15:49 AM AWS support: Sure
10:15:59 AM AWS support: I'll be happy to do so
10:16:25 AM Myself: Is there any option to sore AMI in S3 bucket.
10:16:46 AM AWS support: No, this is not possible
10:17:05 AM AWS support: AMI data is a simple configuration file
10:17:11 AM AWS support: This is backed by S3
10:17:18 AM AWS support: But not stored in an S3 bucket
10:17:27 AM AWS support: The exact same is true for Snapshots
10:17:45 AM AWS support: It is stored and backed by S3- but not something that can be placed in one of your buckets
10:17:59 AM Myself: is it possible to view that in s3?
10:18:51 AM AWS support: No, this is not something that is visible in S3, I am sorry to say
10:19:57 AM Myself: OK. I need to download the AMI . what can i do?
10:20:19 AM AWS support: AMI data is not something that is downloadable
10:20:35 AM AWS support: Are you seeking to Download your whole instance?
10:20:46 AM AWS support: Or download a complete volume?
10:21:07 AM AWS support: If you originally imported your instance from a VM, you can Export the VM 10:21:29 AM AWS support: But its an EC2 instance that was created in EC2, you can not- I am really sorry to say
10:22:02 AM Myself: Okay fine.
I ran into the same problem and to my delight, AWS has since innovated something about this.
You can store and restore your AMI in S3 now.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ami-store-restore.html#store-ami
--
UPDATE:
the version of AWS CLI is matter
I am launching Mobile application with backend as PHP hosted on 4 instances of AWS Elastic beanstalk. For media storage (images and videos) I am not sure if S3 is a better option or having an EC2 instance with a share directory will be fine.
My consideration will be based on performance and throughput. For S3 i never came across any documentation or reference which can give me the throughput between EC2 and S3.
As per your use case S3 is the best option as per the images durability goes. And the data transfer speeds between an EC2 instance and S3 is super fast so you don't have to worry about that.
And if you come across issue where there is latency in data transferred between the EC2 instance and S3 due to the Instance and S3 bucket regions being different AWS just introduced S3 Accelerated transfer http://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html
So using S3 for image file storage is most durable and reliable option for your use case.