how can I prevent the accidental deletion of a bucket? I've enabled versioning to prevent the accidental deletion of objects, but if someone just deletes the whole bucket, my data is down the drain, right? I've read upon the retention policy and locking it, but then I would need to disable object versioning. I could implement the following:
create a different project where the bucket will live and have no storage admin there (just the owner of the project)
mirror the bucket to another bucket with retention policy set
Any other options that I could implement?
Thanks
Christian
Add a Retention Policy to the bucket and then lock the retention policy. This prevents a bucket from being deleted until all objects have exceeded the retention period.
Retention policies and retention policy locks
You can add a retention policy to a bucket to specify a retention
period.
If a bucket does not have a retention policy, you can delete or replace objects in the bucket at any time.
If a bucket has a retention policy, objects in the bucket can only be deleted or replaced once their age is greater than the retention
period.
A retention policy retroactively applies to existing objects in the bucket as well as new objects added to the bucket.
You can lock a retention policy to permanently set it on the bucket.
Once you lock a retention policy, you cannot remove it or reduce the
retention period it has.
You cannot delete a bucket with a locked retention policy unless every object in the bucket has met the retention period.
You can increase the retention period of a locked retention policy.
Locking a retention policy can help your data comply with record retention regulations.
The solution to my problem was to implement a deny policy and deny the storage.buckets.delete permission for the whole project. See also https://cloud.google.com/iam/docs/deny-overview.
Related
Reading about Bucket Locks in Cloud Storage made me think of something very evil and bad that one could do:
Create a Cloud Storage Bucket.
Set a retention policy of 100 years on the bucket.
Lock the retention policy to the bucket.
Upload many petabytes of objects to the bucket.
The project is now stuck with a bucket that cannot be deleted for 100 years and the project can never be deleted either due to a "lien". And theoretically, someone is stuck paying the bill to store the petabytes. For 100 years.
Is there any way, preferably programmatically or through configuration, to prevent users from locking a retention policy on a bucket but still permitting them to create and manage other aspects of Cloud Storage buckets that can't be bucket locked?
The more blunt permission system doesn't seem like it's fine grained enough to permit or deny locking:
https://cloud.google.com/storage/docs/access-control/iam-json
I'm thinking there's some way to use IAM Conditions to accomplish what I want, but I'm not sure how.
Update: I'm looking for a solution that does not force a retention policy to be set. John Hanley's organization policy contraint solution is interesting, but it forces a retention policy to be set with at least a 1 second retention across all applicable projects and it also disables the option to have bucket versioning enabled in the bucket.
A forced retention of 1 second can cause certain issues with applications that write and delete objects at the same key multiple times a second.
FWIW, AWS identifies these kinds of radioactive waste creation actions and lets policies be set on them accordingly.
Method 1:
Select or create a custom role for bucket users that does not have the permission resourcemanager.projects.updateLiens. That permission is required to create a Retention Policy.
Method 2:
This method has side effects such as not supporting object versioning but can prevent a long bucket lock such as 100 years.
You can set an Organization Policy Constraint to limit the maximum duration of a Retention Policy.
Name:
constraints/storage.retentionPolicySeconds
Description:
Retention policy duration in seconds
I have a bucket with an object lock policy, for security reasons.
The problem is, I don't want the bucket to accidentally grow to hundreds of gigabytes for which I will have to pay for a long long time.
Is there a way using an alarm or any suitable amazon service to revoke access to a user if a bucket exceeds a certain size (or any other solution to this problem)?
I want to delete particular Folder/Objects after prescribed time, can we set a Retention Policy or Object lifecycles?
Thanks!
You can set lifecycle on a bucket. After a period of time, you can chose to delete the objects that match the rules.
It's not possible at object or folder level (because folder doesn't exist!)
I have created a lifecycle policy for one of my buckets as below:
Name and scope
Name MoveToGlacierAndDeleteAfterSixMonths
Scope Whole bucket
Transitions
For previous versions of objects Transition to Amazon Glacier after 1 days
Expiration Permanently delete after 360 days
Clean up incomplete multipart uploads after 7 days
I would like to get answer for the following questions:
When would the data be deleted from s3 as per this policy ?
Do i have to do anything on the glacier end inorder to move my s3 bucket to glacier ?
My s3 bucket is 6 years old and all the versions of the bucket are even older. But i am not able to see any data in the glacier console though my transition policy is set to move to glacier after 1 day from the creation of the data. Please explain this behavior.
Does this policy affect only new files which will be added to the bucket post lifepolicy creation or does this affect all the files in s3 bucket ?
Please answer these questions.
When would the data be deleted from s3 as per this policy ?
Never, for current versions. A lifecycle policy to transition objects to Glacier doesn't delete the data from S3 -- it migrates it out of S3 primary storage and over into Glacier storage -- but it technically remains an S3 object.
Think of it as S3 having its own Glacier account and storing data in that separate account on your behalf. You will not see these objects in the Glacier console -- they will remain in the S3 console, but if you examine an object that has transitioned, is storage class will change from whatever it was, e.g. STANDARD and will instead say GLACIER.
Do i have to do anything on the glacier end inorder to move my s3 bucket to glacier ?
No, you don't. As mentioned above, it isn't "your" Glacier account that will store the objects. On your AWS bill, the charges will appear under S3, but labeled as Glacier, and the price will be the same as the published pricing for Glacier.
My s3 bucket is 6 years old and all the versions of the bucket are even older. But i am not able to see any data in the glacier console though my transition policy is set to move to glacier after 1 day from the creation of the data. Please explain this behavior.
Two parts: first, check the object storage class displayed in the console or with aws s3api list-objects --output=text. See if you don't see some GLACIER-class objects. Second, it's a background process. It won't happen immediately but you should see things changing within 24 to 48 hours of creating the policy. If you have logging enabled on your bucket, I believe the transition events will also be logged.
Does this policy affect only new files which will be added to the bucket post lifepolicy creation or does this affect all the files in s3 bucket ?
This affects all objects in the bucket.
I have an S3 bucket on which I activated a Lifecycle Policy that transitions objects to Glacier class storage after 14 days.
I desire to disable this policy.
Is it safe to assume that disabling/deleting the Lifecycle Policy will not alter/affect the already transitioned objects in any way?
This links specifies what happens to transition scheduling, in the event of Policy deletion.
https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-set-lifecycle-configuration-intro.html
Although it would seem evident, I didn't find it to imply about already transitioned objects remaining untouched.
My concern with this is critical due to AWS:Glacier having extreme costs for retrieval over 5% of total Glacier usage, but also for deleting files prior to 90 days.
The lifecycle policy works on the bucket itself. When you delete it/disable it, it stops performing the defined actions.
It will NOT perform actions to reverse anything. This includes glacier, deleted files, moved files, etc.