I have some AWS EC2 instances logging to CloudWatch Log Groups. When I open a Log Group, I can see my log streams listed:
However, when I click on one of the streams, I get an error:
Log stream not found The log stream version_0.2.41 date_2018-06-25
instance_i-02962ec6fbfd20c4c could not be found. Check if it was
correctly created and retry.
All this used to work before (it broke down somewhere in the last two weeks) and nothing has been modified on our part.
Any idea what's wrong here?
Related
We are making use of AWS Lambda and have configured cloudwatch for logging. There is a cron job running every 5 minutes which is triggering the lambda function. The logs that are generated for the hit are getting created in different log streams. For reference, please check the image attached here:
So, let's say there is an API hit at 11:45, then for checking the logs I have to go through the log streams having last event time 2022-05-05 11:43:10 (UTC+05:30) , 2022-05-05 11:43:00 (UTC+05:30), 2022-05-05 11:38:11 (UTC+05:30) and 2022-05-05 11:38:02 (UTC+05:30) and so on. The reason is, for a single hit logs are getting created in different log streams. Some of the logs are in first log stream, some are in second, a few are in third one. Previously, all the logs were created in single log stream corresponding to a single hit. Is there anything that can be done to avoid this? as this makes debugging a time taking process.
This is how Lambda works: each Lambda execution environment gets its own log stream. If you need to look at logs across log streams, then the best "built-in" solution is CloudWatch Logs Insights, which works at the log-group level.
Update: this document describes the Lambda execution environment, and the conditions that cause creation/destruction of an environment.
Thinking that I wanted to clear out old logs, I made the mistake of deleting my Lambda's "Log Stream" on CloudWatch.
The result, as I should have expected if I was awake, is that now CloudWatch isn't getting the Lambda's console logs at all. Oops.
The log group still exists.
I can see how to create a new log stream.
What I haven't been able to find on the web is clear instructions to get the existing Lambda to output to this new stream... ie, to repair what I did.
Can someone provide instructions or a pointer to them, please? I'm sure I'm not the only one who's made this mistake, so I think it's an answer worth having on tap.
UPDATE: Decided to try recovering by creating an entirely new Lambda, running the same code and configured the same way, expecting that it would Just Work; my understanding was that a new Lambda binds to a CloudWatch group automagically.
Then I ran my test, clicked the twist-arrow to see the end of the output, and hit "Click here to view the corresponding CloudWatch log group.". It opened Cloudwatch looking at the expected log group name -- with a big red warning that this group did not exist. Clicking "(Logs)" at the top of the test output gave the same behavior.
I tried creating the group manually, but now I'm back where I was -- lambda runs, I get local log output, but the logs are not reaching CloudWatch.
So it looks like there's something deeper wrong. CloudWatch is still getting logs from the critical lambda (the one driving my newly-released Alexa skill), and the less-critical one (scheduled update for the skill's database) is running OK so I don't absolutely need its logs right now -- but I need to figure this out so I can read them if that background task ever breaks.
Since this is now looking like real Unexpected Behavior rather than user error, I'll take it to the AWS forums and post here if they come up with an answer. On that system, the question is now at https://repost.aws/questions/QUDzF2c_m0TPCwl3Ufa527Wg/lambda-logging-to-cloud-watch-seems-to-be-broken
Programmer's mantra: "If it was easy, they wouldn't need us..."
After a Lambda function is executed, you can go to the Monitoring tab and click View logs in CloudWatch -- it will take you to the location where the logs should be present.
If you know that the function has executed but no logs are appearing, then confirm that your Lambda function has the AWSLambdaBasicExecutionRole assigned to the IAM Role being used by the Lambda function. This grants permission for the Lambda function to write to CloudWatch Logs.
See: AWS Lambda execution role - AWS Lambda
I have my app writing logs to /var/log/my_app.log. I have the logrotator set up daily to rotate the log, so presumably when the log rotate condition is met it will copy over my_app.log to my_app<date>.log. I also have the Cloudwatch agent on the same ec2 instance sending files over to Cloudwatch logs. There they will stay indefinitely I assume (or until a set time set in the aws console). Is it correct to assume that Cloudwatch will always have the first log created and logged regardless of how I rotate the actual log files on the ec2 instance? That is to say, no matter what happens with the rotated logs, I will always have ALL the logs that have been created because they've been sent to cloudwatch?
Any logs that is sent to CloudWatch will not be deleted because of the log rotation. Check out the FAQ section in the following link that has some important questions answered including the log rotation naming schemes and the scenarios in which log events can be truncated or skipped.
(Search for CloudWatch Logs Agent FAQs in the following link)
https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AgentReference.html
Your assumption is correct on the log retention. CloudWatch logs are stored indefinitely by default.
Here is the quote from Amazon documentation
Log Retention – By default, logs are kept indefinitely and never expire. You can adjust the retention policy for each log group, keeping the indefinite retention, or choosing a retention period between 10 years and one day.
I created AWS Kinesis stream.
several different EKS log groups destination point to the same Kinesis stream.
i didn't find any documentation that make sure that each record i get is related to ONLY one log group and it doesn't include logs from different log groups.
I would like to make sure that its the way its works.
Anyone know?
I have recently started learning about AWS cloud watch and I want to understand the concept of creating Logs so I went through a lot of links like
https://aws.amazon.com/answers/logging/centralized-logging/
I could understand that we can create log groups but and logs are basically to track activity. Is there anything more to it. When do the logs get created.
Any help would be highly appreciated!
You can get more details about Log Groups and CloudWatch Logs Concepts here
Following is the extract from that page
Log Events
A log event is a record of some activity recorded by the application or resource being monitored. The log event record that
CloudWatch Logs understands contains two properties: the timestamp of
when the event occurred, and the raw event message. Event messages
must be UTF-8 encoded.
Log Streams
A log stream is a sequence of log events that share the same source. More specifically, a log stream is generally intended to
represent the sequence of events coming from the application instance
or resource being monitored. For example, a log stream may be
associated with an Apache access log on a specific host. When you no
longer need a log stream, you can delete it using the aws logs
delete-log-stream command. In addition, AWS may delete empty log
streams that are over 2 months old.
Log Groups
Log groups define groups of log streams that share the same retention, monitoring, and access control settings. Each log stream
has to belong to one log group. For example, if you have a separate
log stream for the Apache access logs from each host, you could group
those log streams into a single log group called
MyWebsite.com/Apache/access_log.
And to answer your question "When do the logs get created.", basically that is completely dependent on your application. However, whenever they are created they get streamed to cloudwatch streams (if you have installed the cloudwatch agent and are streaming that particular log)
The advantage of using cloudwatch is that you can retain logs even after your EC2 instance is terminated and you dont need to SSH into the resource to check the logs, you can simply get that from AWS Console