Is there any expiry date of log files, generated by EC2 instance in CloudWatch logs using CloudWatch Logs Agent ?
By default, log data is stored indefinitely. However, you can
configure how long you want to store log data in a log group. Any data
older than the current retention setting is automatically deleted. You
can change the log retention for each log group at any time.
For more information:
Changing Log Retention
Related
I want to change default retention time in Cloudwatch log groups or create an organizational log retention policy. I don't want to set "never expire" as default. Is it possible?
We use AWS Cloudwatch Logs, and are retrieving a set of logs using AWS SDKs cloudwatch client's filterLogEvents function, and are pushing the logs to S3. For log groups with about 5000 events, we could see that the retrieved logs were out of order. For example, we would receive the logs till 11:35 UTC and the next log would be 14:42 UTC. We checked in cloudwatch and found that there were more logs between 11:35 UTC - 14:42 UTC. The logs during that time were present after a bunch of other logs.
I want to know why there is an inconsistency in the logs. Why are some set of logs obtained in the correct order, while some logs are placed somewhere else. All the logs seem to be present when we try to retrieve them, but the order for some of the logs is messed up.
I have my app writing logs to /var/log/my_app.log. I have the logrotator set up daily to rotate the log, so presumably when the log rotate condition is met it will copy over my_app.log to my_app<date>.log. I also have the Cloudwatch agent on the same ec2 instance sending files over to Cloudwatch logs. There they will stay indefinitely I assume (or until a set time set in the aws console). Is it correct to assume that Cloudwatch will always have the first log created and logged regardless of how I rotate the actual log files on the ec2 instance? That is to say, no matter what happens with the rotated logs, I will always have ALL the logs that have been created because they've been sent to cloudwatch?
Any logs that is sent to CloudWatch will not be deleted because of the log rotation. Check out the FAQ section in the following link that has some important questions answered including the log rotation naming schemes and the scenarios in which log events can be truncated or skipped.
(Search for CloudWatch Logs Agent FAQs in the following link)
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html
Your assumption is correct on the log retention. CloudWatch logs are stored indefinitely by default.
Here is the quote from Amazon documentation
Log Retention – By default, logs are kept indefinitely and never expire. You can adjust the retention policy for each log group, keeping the indefinite retention, or choosing a retention period between 10 years and one day.
Here is documentation about creating Cloud Watch LogGroup via Cloud Formation. They said:
RetentionInDays
The number of days log events are kept in CloudWatch
Logs. When a log event expires, CloudWatch Logs automatically deletes
it. For valid values, see PutRetentionPolicy in the Amazon CloudWatch
Logs API Reference.
Required: No
So if I create LogGroup without RetentionInDays parameter will Cloud Watch keep those logs forever? Or what RetentionInDays value they use by default?
By default, log data is stored in CloudWatch Logs indefinitely. However, you can configure how long to store log data in a log group. Any data older than the current retention setting is automatically deleted. You can change the log retention for each log group at any time.
Source :- http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SettingLogRetention.html
I'm trying to understand how long the details associated with an AWS Batch job are retained. For example, the Kinesis limits page describes how each stream defaults to a 24 hour retention period that is extendable up to 7 days.
The AWS Batch limits page does not include any details about either the maximum time or count allowed for jobs. It does say that one million is the limit for SUBMITTED jobs, but its unclear if that is exclusively for SUBMITTED or includes other states as well.
Does anybody know the details of batch job retention?
Job metadata for SUCCEEDED and FAILED jobs are retained for 24 hours. Metadata for Jobs in SUBMITTED, PENDING, RUNNABLE, STARTING, and RUNNING remain in the queue until the job completes. Your AWS Batch Jobs also log STDERR/STDOUT to CloudWatch Logs where you control the retention policy.
From AWS Batch official doc - https://docs.aws.amazon.com/batch/latest/userguide/batch_user.pdf
Under Jobs -> Job States (Page 23)
FAILED
The job has failed all available attempts. The job state for FAILED jobs is persisted in AWS Batch for at least 24 hours.
Note
Logs for FAILED jobs are available in CloudWatch Logs; the log group is /aws/batch/job, and the log stream name format is first200CharsOfJobDefinitionName/default/ecs_task_id (this format may change in the future). After a job reaches the RUNNING status, you can programmatically retrieve its log stream with the DescribeJobs API operation. For more information, see View Log Data Sent to CloudWatch Logs in the Amazon CloudWatch Logs User Guide. By default, these logs are set to never expire, but you can modify the retention period. For more information, see Change Log Data Retention in CloudWatch Logs in the Amazon CloudWatch Logs User Guide.