I want to change default retention time in Cloudwatch log groups or create an organizational log retention policy. I don't want to set "never expire" as default. Is it possible?
Related
Reading about Bucket Locks in Cloud Storage made me think of something very evil and bad that one could do:
Create a Cloud Storage Bucket.
Set a retention policy of 100 years on the bucket.
Lock the retention policy to the bucket.
Upload many petabytes of objects to the bucket.
The project is now stuck with a bucket that cannot be deleted for 100 years and the project can never be deleted either due to a "lien". And theoretically, someone is stuck paying the bill to store the petabytes. For 100 years.
Is there any way, preferably programmatically or through configuration, to prevent users from locking a retention policy on a bucket but still permitting them to create and manage other aspects of Cloud Storage buckets that can't be bucket locked?
The more blunt permission system doesn't seem like it's fine grained enough to permit or deny locking:
https://cloud.google.com/storage/docs/access-control/iam-json
I'm thinking there's some way to use IAM Conditions to accomplish what I want, but I'm not sure how.
Update: I'm looking for a solution that does not force a retention policy to be set. John Hanley's organization policy contraint solution is interesting, but it forces a retention policy to be set with at least a 1 second retention across all applicable projects and it also disables the option to have bucket versioning enabled in the bucket.
A forced retention of 1 second can cause certain issues with applications that write and delete objects at the same key multiple times a second.
FWIW, AWS identifies these kinds of radioactive waste creation actions and lets policies be set on them accordingly.
Method 1:
Select or create a custom role for bucket users that does not have the permission resourcemanager.projects.updateLiens. That permission is required to create a Retention Policy.
Method 2:
This method has side effects such as not supporting object versioning but can prevent a long bucket lock such as 100 years.
You can set an Organization Policy Constraint to limit the maximum duration of a Retention Policy.
Name:
constraints/storage.retentionPolicySeconds
Description:
Retention policy duration in seconds
how can I prevent the accidental deletion of a bucket? I've enabled versioning to prevent the accidental deletion of objects, but if someone just deletes the whole bucket, my data is down the drain, right? I've read upon the retention policy and locking it, but then I would need to disable object versioning. I could implement the following:
create a different project where the bucket will live and have no storage admin there (just the owner of the project)
mirror the bucket to another bucket with retention policy set
Any other options that I could implement?
Thanks
Christian
Add a Retention Policy to the bucket and then lock the retention policy. This prevents a bucket from being deleted until all objects have exceeded the retention period.
Retention policies and retention policy locks
You can add a retention policy to a bucket to specify a retention
period.
If a bucket does not have a retention policy, you can delete or replace objects in the bucket at any time.
If a bucket has a retention policy, objects in the bucket can only be deleted or replaced once their age is greater than the retention
period.
A retention policy retroactively applies to existing objects in the bucket as well as new objects added to the bucket.
You can lock a retention policy to permanently set it on the bucket.
Once you lock a retention policy, you cannot remove it or reduce the
retention period it has.
You cannot delete a bucket with a locked retention policy unless every object in the bucket has met the retention period.
You can increase the retention period of a locked retention policy.
Locking a retention policy can help your data comply with record retention regulations.
The solution to my problem was to implement a deny policy and deny the storage.buckets.delete permission for the whole project. See also https://cloud.google.com/iam/docs/deny-overview.
I have my app writing logs to /var/log/my_app.log. I have the logrotator set up daily to rotate the log, so presumably when the log rotate condition is met it will copy over my_app.log to my_app<date>.log. I also have the Cloudwatch agent on the same ec2 instance sending files over to Cloudwatch logs. There they will stay indefinitely I assume (or until a set time set in the aws console). Is it correct to assume that Cloudwatch will always have the first log created and logged regardless of how I rotate the actual log files on the ec2 instance? That is to say, no matter what happens with the rotated logs, I will always have ALL the logs that have been created because they've been sent to cloudwatch?
Any logs that is sent to CloudWatch will not be deleted because of the log rotation. Check out the FAQ section in the following link that has some important questions answered including the log rotation naming schemes and the scenarios in which log events can be truncated or skipped.
(Search for CloudWatch Logs Agent FAQs in the following link)
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html
Your assumption is correct on the log retention. CloudWatch logs are stored indefinitely by default.
Here is the quote from Amazon documentation
Log Retention – By default, logs are kept indefinitely and never expire. You can adjust the retention policy for each log group, keeping the indefinite retention, or choosing a retention period between 10 years and one day.
Here is documentation about creating Cloud Watch LogGroup via Cloud Formation. They said:
RetentionInDays
The number of days log events are kept in CloudWatch
Logs. When a log event expires, CloudWatch Logs automatically deletes
it. For valid values, see PutRetentionPolicy in the Amazon CloudWatch
Logs API Reference.
Required: No
So if I create LogGroup without RetentionInDays parameter will Cloud Watch keep those logs forever? Or what RetentionInDays value they use by default?
By default, log data is stored in CloudWatch Logs indefinitely. However, you can configure how long to store log data in a log group. Any data older than the current retention setting is automatically deleted. You can change the log retention for each log group at any time.
Source :- http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SettingLogRetention.html
Is there any expiry date of log files, generated by EC2 instance in CloudWatch logs using CloudWatch Logs Agent ?
By default, log data is stored indefinitely. However, you can
configure how long you want to store log data in a log group. Any data
older than the current retention setting is automatically deleted. You
can change the log retention for each log group at any time.
For more information:
Changing Log Retention