I have 1.5TB of data in bucket with Standard storage class. I want to move all objects to Glacier Deep Archive storage class.
S3 makes a copy (version) of an object in Standard storage class If I move it to Glacier Deep Archive via GUI (select objects->actions->edit storage class).
Do I get charged for both objects (versions?) or just for the Glacier one? Both versions are the same size.
I could use Lifecycle rules but not all files were created >180days ago.
Yes, you are getting charged for the whole volume including the versions.
But instead of manual update of the storage class, maybe a lifecycle policy is better option, where the objects you can archive the the objects and specify to delete the versionins after some time. No need to wait 180 days
https://aws.amazon.com/blogs/aws/amazon-s3-lifecycle-management-update/
I could use Lifecycle rules but not all files were created >180days ago.
You can transition them at any time. They don't have to be 180 days old. The 180 days limit is for getting them back from Deep Archive. If you try to restore your files before they have been in DA for 180, you will be charged penalties for that.
You'll get charged for both objects (versions). From S3 FAQs:
Normal Amazon S3 rates apply for every version of an object stored or requested
Related
I have some files in my AWS S3 bucket which i would like to put in Glacier Deep Archive from Standard Storage. After selecting the files and changing the storage class, it gives the following message.
Since the message says that it will make a copy of the files, my question is that will I be charged extra for moving my existing files to another storage class?
Thanks.
"This action creates a copy of the object with updated settings and a new last-modified date. You can change the storage class without making a new copy of the object using a lifecycle rule.
Objects copied with customer-provided encryption keys (SSE-C) will fail to be copied using the S3 console. To copy objects encrypted with SSE-C, use the AWS CLI, AWS SDK, or the Amazon S3 REST API."
Yes, changing the storage class incurs costs, regardless of whether it's done manually or via a lifecycle rule.
If you do it via the console, it will create a deep archive copy but will retain the existing one as a previous version (if you have versioning enabled), so you'll start being charged for storage both (until you delete the original version).
If you do it via a lifecycle rule, it will transition (not copy) the files, so you'll only pay for storage for the new storage class.
In both cases, you'll have to pay for LIST ($0.005 per 1000 objects in STANDARD class) and COPY/PUT ($0.05 per 1000 objects going to DEEP_ARCHIVE class) actions.
Since data is being moved within the same bucket (and therefore within the same region), there will be no data transfer fees.
The only exception to this pricing is the "intelligent tiering" class, which automatically shifts objects between storage classes based on frequency of access and does not charge for shifting classes.
No additional tiering fees apply when objects are moved between access tiers within the S3 Intelligent-Tiering storage class.
We have a good amount of files that were on the wrong folder in S3 but has since transitioned to Glacier storage class. First thing is we want to restore them so we can move them to the right folders. Once moved we transition it back to Glacier storage class. The main question is, do those files get duplicated on the Glacier side? Since restoring them (to Standard) doesn't mean that they're being deleted or moved on the Glacier side. How do we verify that they are not duplicated after moving to a different folder in the Standard class?
Amazon S3 objects are immutable. You cannot "move" objects in S3 or Glacier.
The process would be:
Restore the objects from Glacier storage class
Rename/move them: This actually doesn't happen. Rather, the objects are copied to the new Key and then the original object is deleted. Using "Rename" in the console does this for you, as does the AWS CLI aws mv command.
Create a lifecycle rule to transition them to Glacier storage class
From CopyObject - Amazon Simple Storage Service:
If the source object's storage class is GLACIER, you must restore a copy of this object before you can use it as a source object for the copy operation.
The transition of objects to the S3 Glacier Deep Archive storage class can go only one way.
As per, https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html:
You cannot use a lifecycle configuration rule to convert the storage class of an object from S3 Glacier Deep Archive to any other storage class. If you want to change the storage class of an archived object to another storage class, you must use the restore operation to make a temporary copy of the object first. Then use the copy operation to overwrite the object specifying STANDARD, INTELLIGENT_TIERING, STANDARD_IA, ONEZONE_IA, S3 Glacier, or REDUCED_REDUNDANCY as the storage class.
To delete the data permananently from Glacier, Refer https://docs.aws.amazon.com/amazonglacier/latest/dev/deleting-an-archive.html
Renaming folder involves cost. See https://stackoverflow.com/a/33006139/945214
In my case, the best solution was to delete everything and re-upload.
This is unfortunately only an option if you still have another copy.
I have created a s3 life cycle policy which will Expire the current version of the object in 540 days.
I am a bit confused here, whether it deletes the objects from s3 or glacier,
if not I want to delete the objects from a bucket in 540 days and the glacier in some 4 years! how will I set it up?
Expiring an object means "delete it", regardless of its storage class.
So, if it has moved to a Glacier storage class, it will still be deleted.
When you store data in Glacier via S3, then the object is managed by Amazon S3 (but stored in Glacier). Thus, the lifecycle rules apply.
If, however, you store data directly in Amazon Glacier (without going via Amazon S3), then the data would not be impacted by the lifecycle rules, nor would it be visible in Amazon S3.
Bottom line: Set your rules for deletion based upon the importance of the data, not its current storage class.
In the same bucket I've folders named Backup1 and Backup2.
Files in Backup1 folder transition from S3 Standard storage to Glacier after 5 days.
Now from the S3 console, if I want to copy some of these files from Backup1 folder to Backup2:
Will I incur any charges if these files are less than 90 days old?
Will the copy be done from Glacier to Glacier or will it be from Glacier to S3 Standard?
Since this involves file copy from Glacier, will it be time consuming?
First, it's worth mentioning that there have been some recent updates to Amazon S3:
It is now possible to upload directly to the Glacier storage class, rather than having to specify a Lifecycle rule. This is good for files you want to archive immediately (not necessarily your use-case).
There is a new S3 Intelligent-Tiering Storage Class that will automatically move objects in/out of Glacier based on usage patterns.
Will I incur any charges if these files are less than 90 days old?
When you copy objects to Backup2, you will be storing another copy of the data. Therefore, you will be charged for this additional storage at whatever storage class it is using. When you restore objects from the Glacier storage class, they are temporarily stored as Reduced Redundancy Storage. You'll have to check what storage class is used after you copy the files to Backup2.
Will the copy be done from Glacier to Glacier or will it be from Glacier to S3 Standard?
To copy objects from Backup1, you will need to first Restore the object(s) from Glacier. Once they have been restored, they will be available in S3.
Since this involves file copy from Glacier, will it be time consuming?
The file copy is fast, but you will first need to restore from Glacier. You can choose how long this should take, depending on whether you wish to pay for an expedited retrieval (1-5 minutes) or standard (3-5 hours) or bulk (5-12 hours).
A while ago when the price difference between standard storage and glacier was closer to 10:1 than 3:1, I moved a couple of bucket completely to Glacier using a life-cycle policy. Admittedly, I hadn't investigated how to reverse that process permanently.
I know the documentation states that I would have to "use the copy operation to overwrite the object as a Standard or RRS object", but I guess I'm unclear what that looks like. Do I just copy and paste within that bucket?
Restoration can only be done at the Object level, not Bucket or Folder. When an object is restored, it is a available in Amazon S3 for the period requested (eg 2 days), then reverts to only being in Glacier. You'll need to copy the objects out of the bucket during that time to keep a copy of them in S3.
The easiest method would be to use a utility to restore multiple Objects for you, eg:
How to restore whole directories from Glacier?
Also, see Stackoverflow: How to restore folders (or entire buckets) to Amazon S3 from Glacier?