I am trying to see if there is a way to transfer s3 objects in glacier in one bucket to another bucket but keep the storage type the same? I can restore the glacier object and transfer it, but in the new bucket, the file is saved in standard storage. I would like it to know if there is a way that the file is directly stored in glacier outside of enforcing life cycle policies on the bucket.
There isn't.
Objects can only be copied to another bucket once restored, and objects can only be transitioned into the Glacier storage class by lifecycle policies, not by creating them with this storage class ... which essentially rules out the possibility of the desired outcome for two different reasons.
S3 does not have either a "move" or a "rename" feature -- both of these can only be emulated by copy-and-delete.
Related
I am trying to understand aws s3 object transitioning across various storage classes, and I am using https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html to go through details, however I find this confusing. Below is what I am trying to understand
as per this document and nice image provided on this page says that -
"Amazon S3 supports a waterfall model for transitioning between storage classes, as shown in the following diagram.".
For example, object from:
S3 Standard can be transitioned to any of --> Standard IA, Intelligent Tiering, One Zone IA, Glacier Instant, Glacier Flexible , Glacier Deep.
Which means reverse explicit transitioning is not possible
However, on same page it then says
"If you want to change the storage class of an object that is stored in S3 Glacier Flexible Retrieval to a storage class other than S3 Glacier Deep Archive, you must use the restore operation to make a temporary copy of the object first. Then use the copy operation to overwrite the object specifying S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA, or Reduced Redundancy as the storage class."
Here confusing part is that it shows option of overwrite the storage class (off course after restoring copy), to S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA
Confusion is that - how does the aws treat this "restored" object as - a new object ? or existing object?
if it treats this as a "new" object - how can it transition directly to S3 Intelligent-Tiering, S3 Standard-IA, S3 One Zone-IA ? this contradicts to waterfall rule that they have mentioned
Thank you.
My expectation from this was that if object should be transitioned to say One Zone IA , the restored object should be first only treated as S3 Standard object - wait for 30 days and then should allow transition to S3 One Zone IA.
This question might look like an easy/idiot/beginner question but I'm really confused between both of them.
Why do I need to use Amazon S3 Glacier if I can use the normal S3 Bucket and just change the storage class of the objects inside to Glacier manually or by using Lifecycle rule?
Thanks in advance,
In the old days, Amazon Glacier was only available as a separate product. Frankly, the Glacier service is a pain to use.
Every request has to be submitted as a Job, which takes a long time to return. Even obtaining a list of archives is slow, let alone restoring a file from the archive.
The best way to use the Amazon Glacier service is with a third-party tool (eg Cloudberry Backup) that knows how to interface with Glacier, isolating you from having to use it directly.
Then, in 2012, the Amazon S3 team introduced a new Glacier Storage Class where S3 would move the data to Glacier, but still present the objects as being "in S3". (Well, the objects appear in S3 and their metadata is accessible, but the contents of the objects is stored in Glacier.) Then, in 2019, an even lower-cost Glacier Deep Archive storage class offered even lower prices than available through Amazon Glacier itself.
Therefore, it is now both easier and lower cost to use Glacier via Amazon S3 storage classes.
Amazon Glacier still remains available for use, and has been renamed Amazon S3 Glacier to further confuse things. There might be some use-cases where it is preferable to use (eg acting like traditional tape backups for AWS Storage Gateway Tape Gateways), but Glacier Deep Archive in S3 would be the lowest-cost option.
These days most people will just use S3 glacier storage class, because S3 api is much more convenient to work with then Glacier api.
However, Amazon S3 Glacier offers some extra functionality, not available in regular S3. Most notably this would be Vault Lock Policies which allow for fine-grain control of locking your vaults with archives for regulatory purposes.
S3 offers Object Lock which performs similar function, but it is not as versatile as vault lock policies. For example, the s3 object locks can be only enabled on bucket creation, and legal holds apply only to individual versions of objects. In contrast, vault lock policies, as the names suggest, are policy documents written in json, which don't have such limitations.
I have created a s3 life cycle policy which will Expire the current version of the object in 540 days.
I am a bit confused here, whether it deletes the objects from s3 or glacier,
if not I want to delete the objects from a bucket in 540 days and the glacier in some 4 years! how will I set it up?
Expiring an object means "delete it", regardless of its storage class.
So, if it has moved to a Glacier storage class, it will still be deleted.
When you store data in Glacier via S3, then the object is managed by Amazon S3 (but stored in Glacier). Thus, the lifecycle rules apply.
If, however, you store data directly in Amazon Glacier (without going via Amazon S3), then the data would not be impacted by the lifecycle rules, nor would it be visible in Amazon S3.
Bottom line: Set your rules for deletion based upon the importance of the data, not its current storage class.
I have an AWS EC2 instance yielding some data, which in turn is meant to be moved to AWS Glacier. According to Is it possible to move EC2 volumes to Amazon Glacier without having to download and upload it? - Stack Overflow there are only two ways to put data in Glacier:
Upload data directly as described in Using Amazon Glacier with the AWS Command Line Interface - AWS Command Line Interface
Copy/move the data to S3 and create a lifecycle rule.
Unfortunately neither approaches worked, meaning that I access my vault and nothing is there, even after a week. Furthermore once I complete the example provided, the "aws glacier describe-vault" command
outputs:
{
"SizeInBytes": 0,
"NumberOfArchives": 0,
"CreationDate": "2018-08-14T12:59:31.456Z",
...
}
What am I missing?
For Option #2, where you created a lifecycle rule to move objects to Glacier, you will not see the objects in Glacier itself.
When Amazon S3 lifecycles objects to Glacier, the objects are kept in a Glacier vault that is managed by Amazon S3 and is not visible to you. Instead, the objects in S3 will show a Storage Class of Glacier, which means that the object metadata is kept in S3 (name, size, etc) but the actual contents of the object has been moved to Glacier.
As long as you can see the Storage Class of Glacier, your objects have been successfully moved to Glacier.
Our S3 bucket contains a mix of objects in Standard and Glacier storage class(due to the lifecycle rules we setup). I want to understand how to delete the Glacier storage class objects. I looked at the Glacier console and we don't have a Vault/archive to delete. So, my guess is that S3 is managing the Glacier vault for us. I also looked at the lifecycle rule blog post(https://aws.amazon.com/blogs/aws/archive-s3-to-glacier/) to validate this.
TL;DR
So, is deleting an object (in Glacier storage class) from S3 (using the aws CLI or lifecycle rules) sufficient to delete it from Glacier as well?
So, is deleting an object (in Glacier storage class) from S3 (using the aws CLI or lifecycle rules) sufficient to delete it from Glacier as well?
Yes, it is. Deleting the object from S3 removes it from Glacier.
The GLACIER storage class uses the very low-cost Amazon Glacier storage service, but you still manage objects in this storage class through Amazon S3.
https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html
When S3 stores objects in the GLACIER storage class, S3 doesn't put that data into your Glacier in your AWS account. S3 seems to have its own, separate interface to Glacier. (Similarly, EBS snapshots are stored "in S3," but not in "your" S3.)