How do I move an S3 "Deep Glacier Archive" object? - amazon-web-services

I have a number of "Deep Glacier Archive" class objects in the root level of my Amazon S3 bucket.
As the number of objects grows, I've added some top-level folders to the same bucket that I'd like to move the other objects into for organizational reasons. While I can add new objects to these folders, I've noticed that the "Move" action option is grayed out while when I have existing objects selected.
Is there a way that I can move these glacier objects into the other folders in the same bucket? (I'm using the Amazon AWS S3 web console interface.)

Objects cannot be 'moved' in Amazon S3. Doing so actually involves performing a copy and then delete.
The S3 management console is unable to move/copy an object with a Glacier storage class because the data is not immediately available. Instead, you should:
Restore the object (Charges might apply)
Once restored, perform the move/copy

You have to first restore the objects and wait around 48h until the process completes (you can do that directly from the management console). Once it is done you should see the download button enabled in the console and a countdown of the days you set them to be available.
Then you can move them using the AWS CLI with:
aws s3 mv "s3://SOURCE" "s3://DEST" --storage-class DEEP_ARCHIVE --force-glacier-transfer
I don't think is possible to move them from the management console directly, after the restoration.

Related

Approach to move file from s3 to s3 glacier

I need to create a python flask application that moves a file from s3 storage to s3 glacier. I cannot use the lifetime policy to do this as I need to use glacier vault lock which isn't possible with the lifetime policy method since I won't be able to use any glacier features on those files. The files will be multiple GBs in size so I need to download these files and then upload them on glacier. I was thinking of adding a script on ec2 that will be triggered by flask and will start downloading and uploading files to glacier.
This is the only solution I have come up with and it doesn't seem very efficient but I'm not sure. I am pretty new to AWS so any tips or thoughts will be appreciated.
Not posting any code as I don't really have a problem with the coding, just the approach I should take.
It appears that your requirement is to use Glacier Vault Lock on some objects to guarantee that they cannot be deleted within a certain timeframe.
Fortunately, similar capabilities have recently been added to Amazon S3, called Amazon S3 Object Lock. This works at the object or bucket level.
Therefore, you could simply use Object Lock instead of moving the objects to Glacier.
If the objects will be infrequently accessed, you might also want to change the Storage Class to something cheaper before locking it.
See: Introduction to Amazon S3 Object Lock - Amazon Simple Storage Service

Unable to upload files to Amazon Glacier

I have an AWS EC2 instance yielding some data, which in turn is meant to be moved to AWS Glacier. According to Is it possible to move EC2 volumes to Amazon Glacier without having to download and upload it? - Stack Overflow there are only two ways to put data in Glacier:
Upload data directly as described in Using Amazon Glacier with the AWS Command Line Interface - AWS Command Line Interface
Copy/move the data to S3 and create a lifecycle rule.
Unfortunately neither approaches worked, meaning that I access my vault and nothing is there, even after a week. Furthermore once I complete the example provided, the "aws glacier describe-vault" command
outputs:
{
"SizeInBytes": 0,
"NumberOfArchives": 0,
"CreationDate": "2018-08-14T12:59:31.456Z",
...
}
What am I missing?
For Option #2, where you created a lifecycle rule to move objects to Glacier, you will not see the objects in Glacier itself.
When Amazon S3 lifecycles objects to Glacier, the objects are kept in a Glacier vault that is managed by Amazon S3 and is not visible to you. Instead, the objects in S3 will show a Storage Class of Glacier, which means that the object metadata is kept in S3 (name, size, etc) but the actual contents of the object has been moved to Glacier.
As long as you can see the Storage Class of Glacier, your objects have been successfully moved to Glacier.

aws s3 glacier storage and transfer

I am trying to see if there is a way to transfer s3 objects in glacier in one bucket to another bucket but keep the storage type the same? I can restore the glacier object and transfer it, but in the new bucket, the file is saved in standard storage. I would like it to know if there is a way that the file is directly stored in glacier outside of enforcing life cycle policies on the bucket.
There isn't.
Objects can only be copied to another bucket once restored, and objects can only be transitioned into the Glacier storage class by lifecycle policies, not by creating them with this storage class ... which essentially rules out the possibility of the desired outcome for two different reasons.
S3 does not have either a "move" or a "rename" feature -- both of these can only be emulated by copy-and-delete.

Lifecycle policy on S3 bucket

I have an S3 bucket on which I've configured a Lifecycle policy which says to archive all objects in the bucket after 1 day(s) (since I want to keep the files in there temporarily but if there are no issues then it is fine to archive them and not have to pay for the S3 storage)
However I have noticed there are some files in that bucket that were created in February ..
So .. am I right in thinking that if you select 'Archive' as the lifecycle option, that means "copy-to-glacier-and-then-delete-from-S3"? In which case this issue of the files left from February would be a fault - since they haven't been?
Only I saw there is another option - 'Archive and then Delete' - but I assume that means "copy-to-glacier-and-then-delete-from-glacier" - which I don't want.
Has anyone else had issues with S3 -> Glacier?
What you describe sounds normal. Check the storage class of the objects.
The correct way to understand the S3/Glacier integration is the S3 is the "customer" of Glacier -- not you -- and Glacier is a back-end storage provider for S3. Your relationship is still with S3 (if you go into Glacier in the console, your stuff isn't visible there, if S3 put it in Glacier).
When S3 archives an object to Glacier, the object is still logically "in" the bucket and is still an S3 object, and visible in the S3 console, but can't be downloaded from S3 because S3 has migrated it to a different backing store.
The difference you should see in the console is that objects will have A "storage class" of Glacier instead of the usual Standard or Reduced Redundancy. They don't disappear from there.
To access the object later, you ask S3 to initiate a restore from Glacier, which S3 does... but the object is still in Glacier at that point, with S3 holding a temporary copy, which it will again purge after some number of days.
Note that your attempt at saving may be a little bit off target if you do not intend to keep these files for 3 months, because any time you delete an object from Glacier, you are billed for the remainder of the three months, if that object has been in Glacier for a shorter time than that.

How do I restore an entire S3 bucket from Glacier permanently?

A while ago when the price difference between standard storage and glacier was closer to 10:1 than 3:1, I moved a couple of bucket completely to Glacier using a life-cycle policy. Admittedly, I hadn't investigated how to reverse that process permanently.
I know the documentation states that I would have to "use the copy operation to overwrite the object as a Standard or RRS object", but I guess I'm unclear what that looks like. Do I just copy and paste within that bucket?
Restoration can only be done at the Object level, not Bucket or Folder. When an object is restored, it is a available in Amazon S3 for the period requested (eg 2 days), then reverts to only being in Glacier. You'll need to copy the objects out of the bucket during that time to keep a copy of them in S3.
The easiest method would be to use a utility to restore multiple Objects for you, eg:
How to restore whole directories from Glacier?
Also, see Stackoverflow: How to restore folders (or entire buckets) to Amazon S3 from Glacier?