In AWS Gateway API, if we enable logging using cloud watch settings as shown below, I see quite a few LogStreams in cloud watch.
What are these streams? Cloudwatch document mentioned below
Represents a log stream, which is a sequence of log events from a single emitter of logs.
I expect only 1 stream for my API in this case. But I see multiple entries.
Why do I have so many streams?
CloudWatch generates multiple streams due to concurrent access. Only one active connection to API gateway can write to one stream at a time. So, it will generate a new stream for the blocked sessions. Then, when a new session comes up it will choose the newest log stream to write to, which in turn will cause any other concurrent new sessions to generate new streams. Unfortunately, in my experience, it appears that CloudWatch Logs does not have a buffering system in place, so it's a 1-to-1 realtime relationship between sessions and streams.
Related
We are making use of AWS Lambda and have configured cloudwatch for logging. There is a cron job running every 5 minutes which is triggering the lambda function. The logs that are generated for the hit are getting created in different log streams. For reference, please check the image attached here:
So, let's say there is an API hit at 11:45, then for checking the logs I have to go through the log streams having last event time 2022-05-05 11:43:10 (UTC+05:30) , 2022-05-05 11:43:00 (UTC+05:30), 2022-05-05 11:38:11 (UTC+05:30) and 2022-05-05 11:38:02 (UTC+05:30) and so on. The reason is, for a single hit logs are getting created in different log streams. Some of the logs are in first log stream, some are in second, a few are in third one. Previously, all the logs were created in single log stream corresponding to a single hit. Is there anything that can be done to avoid this? as this makes debugging a time taking process.
This is how Lambda works: each Lambda execution environment gets its own log stream. If you need to look at logs across log streams, then the best "built-in" solution is CloudWatch Logs Insights, which works at the log-group level.
Update: this document describes the Lambda execution environment, and the conditions that cause creation/destruction of an environment.
We have a service that outputs application logs to cloudwatch. We structure the logs into json format, and output them through stdout, which is forwarded by fluentbit to cloudwatch. We then have a stream set up to forward the logs from cloudwatch to s3, followed by glue crawlers, Athena, and quick sight for dashboards.
We go all this working, and I just saw today that there is a 256kb limit in cloudwatch which we went over for some of our application logs. How else can we get our logs out of our service to s3 (or maybe a different data store?) for analysis? Is cloudwatch not the right approach for this? Other option I thought of us to break up the application logs into multiple events, but then we need to plumb through a joinable ID, as well as write etl logic that does more complex joins. Was hoping to avoid it unless it’s considered a better practice than what we are doing.
Thanks!
I need to monitor the CloudFront real time logs in cloud watch. Is there any way to stream CloudFront real time logs to the cloudwatch?
I know how to stream CloudFront standard(access)logs to cloudwatch(But it won't stream live logs and there will be a huge delay between live log time and log streaming time. so I won't prefer it) and how to stream real time logs to AWS OpenSearch through kinesis firehouse.
But in our project, we stream all service logs to the cloudwatch. So it would be better if any way to stream CloudFront logs to the cloudwatch.
Is there any possibility to do as my request?
Barring any unannounced product that will stream CloudFront to CloudWatch (talk to your AWS account manager), you will need to write a Lambda to do this.
Configure CloudFront to write to Kinesis, and then attach a Lambda function to that Kinesis stream.
Where things will get tricky is that you may have multiple Lambdas running concurrently, reading from different shards in the Kinesis stream. If you attempt to write to the same log stream, you will need to retrieve the latest sequence token before calling PutLogEvents, and be prepared for (1) collisions, and (2) running up against quotas for writing to log streams.
Instead, I recommend that the first call to a handler Lambda create a new log stream, and subsequent calls to that Lambda write to that stream using the sequence token from the previous request. This may result in a lot of log streams, which may be challenging to read in real-time (although CloudWatch Logs Insights helps there).
I have recently started learning about AWS cloud watch and I want to understand the concept of creating Logs so I went through a lot of links like
https://aws.amazon.com/answers/logging/centralized-logging/
I could understand that we can create log groups but and logs are basically to track activity. Is there anything more to it. When do the logs get created.
Any help would be highly appreciated!
You can get more details about Log Groups and CloudWatch Logs Concepts here
Following is the extract from that page
Log Events
A log event is a record of some activity recorded by the application or resource being monitored. The log event record that
CloudWatch Logs understands contains two properties: the timestamp of
when the event occurred, and the raw event message. Event messages
must be UTF-8 encoded.
Log Streams
A log stream is a sequence of log events that share the same source. More specifically, a log stream is generally intended to
represent the sequence of events coming from the application instance
or resource being monitored. For example, a log stream may be
associated with an Apache access log on a specific host. When you no
longer need a log stream, you can delete it using the aws logs
delete-log-stream command. In addition, AWS may delete empty log
streams that are over 2 months old.
Log Groups
Log groups define groups of log streams that share the same retention, monitoring, and access control settings. Each log stream
has to belong to one log group. For example, if you have a separate
log stream for the Apache access logs from each host, you could group
those log streams into a single log group called
MyWebsite.com/Apache/access_log.
And to answer your question "When do the logs get created.", basically that is completely dependent on your application. However, whenever they are created they get streamed to cloudwatch streams (if you have installed the cloudwatch agent and are streaming that particular log)
The advantage of using cloudwatch is that you can retain logs even after your EC2 instance is terminated and you dont need to SSH into the resource to check the logs, you can simply get that from AWS Console
We are hosting our services in AWS beanstalk managed instances. That is forcing us to move away from files based logging to use database based logging.
Is DynamoDB a good choice for replacing file based logging. If so, what should be the primary key. I thought of using timestamp but multiple messages may be logged by the same service within the same timeStamp so that might not be reliable.
Any advice would be appreciated.
Don't use DynamoDB to store logs. You'll be paying for throughput and space needlessly.
Amazon CloudWatch has built-in logging capabilities.
http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/WhatIsCloudWatchLogs.html
Another alternative is a dedicated logging service such as Loggly which is cloud-based and can receive logs in many common formats, plus they have an API to send custom logs. In the web-based console, you can search and filter through the logs.
As an alternative, why don't you use cloudwatch? I ended up writing a whole app to consolidate logs across ec2 instances in a beanstalk app, then last year AWS opened up cloudwatch as a service, so I junked my stuff. You tell cloudwatch where your logs are on the instance, give it a log group and stream name, and all your logs are consolidated in one spot, in cloudwatch. You can also run alarms off them using the standard AWS setup. It's pretty slick, and easy - don't have to write a front end to do lookups, it's already there.
Don't know what you're using for logging - we are a node.js shop, used winston for logging, and there is a nice NPM module that works with Winston to log automatically, called winston-cloudwatch.