I need to monitor the CloudFront real time logs in cloud watch. Is there any way to stream CloudFront real time logs to the cloudwatch?
I know how to stream CloudFront standard(access)logs to cloudwatch(But it won't stream live logs and there will be a huge delay between live log time and log streaming time. so I won't prefer it) and how to stream real time logs to AWS OpenSearch through kinesis firehouse.
But in our project, we stream all service logs to the cloudwatch. So it would be better if any way to stream CloudFront logs to the cloudwatch.
Is there any possibility to do as my request?
Barring any unannounced product that will stream CloudFront to CloudWatch (talk to your AWS account manager), you will need to write a Lambda to do this.
Configure CloudFront to write to Kinesis, and then attach a Lambda function to that Kinesis stream.
Where things will get tricky is that you may have multiple Lambdas running concurrently, reading from different shards in the Kinesis stream. If you attempt to write to the same log stream, you will need to retrieve the latest sequence token before calling PutLogEvents, and be prepared for (1) collisions, and (2) running up against quotas for writing to log streams.
Instead, I recommend that the first call to a handler Lambda create a new log stream, and subsequent calls to that Lambda write to that stream using the sequence token from the previous request. This may result in a lot of log streams, which may be challenging to read in real-time (although CloudWatch Logs Insights helps there).
Related
I have been Stocked on how to send Lambda logs(Prints) directly to Amazon Kinesis Data Stream. I have Found the way to send Logs from Cloud watch but I would like to send every single prints to kinesis data streams. I have a doubt if I send data from cloud watch does it stream real time prints records to kinesis or not? On this case I would like to use lambda as producer and through the kinesis data S3 as a consumer .
below I have attached a flow work of my conditions.
You can also check the lambda extensions, which helps into direct ingestion of the logs to custom destinations. Its helpful incase you want to avoid cloudwatch costs
https://aws.amazon.com/blogs/compute/using-aws-lambda-extensions-to-send-logs-to-custom-destinations/
You have to create CouldWatch Subscription filter for the Lambda's log stream you want to save to S3. So you would do:
CW Logs subscription ---> Firehose ---> S3
Is there a way to buffer X log messages from a CloudWatch log group and only then stream it to a lambda function? I'll elaborate:
I have an app that I registered it's CloudWatch logs to stream to a lambda function which formats the logs and pushes them to Elastic Search.
So the flow is the following:
(app logs) -> (CloudWatch) -->(Lambda)-->(Elastic Search)
My problem is that my lambda function is invoked very often (most of the time single log message) and bombards ES with write requests, I would like to write the logs in bulks, i.e wait until 30 new logs and then invoke the lambda for the 30 logs bulk.
The only way I found to achieve this is to use Kinesis and Firehose but those services cost extra and I want to avoid this.
Are there any other alternatives to achieve this without using something like LogStash?
I am assuming this is a very common usage so there must be some easy way to solve this.
Thanks,
I would investigate Functionbeat whose main goal is to stream Cloudwatch logs (among others) to ES. Extremely easy to deploy and operate, no fiddling with Lambda code, etc. A MUST if you're evolving in the AWS environment yet still want to leverage ES as a log engine
I was wondering what you ended up doing in the situation. I believe if you use functionbeats you can not use aws ES you have to create it manually.
Is it possible to capture lambda #edge requests, after transformation, to a service like kinesis.
I was thinking I could maybe write to cloudwatch logs as json, and then somehow automatically write every entry to kinesis somehow, maybe through lambda.
Is this possible? This would be used for traffic logs and identity matching.
You can write to kinesis directly from a lambda#edge function if you like. Though this would happen synchronously with the execution of the function and, thus, would delay the processing of a request/response by CloudFront.
The use case you are describing would fit nicely in some sort of a tear-down event that can be triggered asynchronously after the request has been fully processed by CloudFront. CloudFront currently does not support such kind of a trigger.
Another option available today is to configure CloudFront access logs delivery to your s3 bucket and parse them as they are delivered. The access logs are delivered to your bucket with up to 24h delay though.
In AWS Gateway API, if we enable logging using cloud watch settings as shown below, I see quite a few LogStreams in cloud watch.
What are these streams? Cloudwatch document mentioned below
Represents a log stream, which is a sequence of log events from a single emitter of logs.
I expect only 1 stream for my API in this case. But I see multiple entries.
Why do I have so many streams?
CloudWatch generates multiple streams due to concurrent access. Only one active connection to API gateway can write to one stream at a time. So, it will generate a new stream for the blocked sessions. Then, when a new session comes up it will choose the newest log stream to write to, which in turn will cause any other concurrent new sessions to generate new streams. Unfortunately, in my experience, it appears that CloudWatch Logs does not have a buffering system in place, so it's a 1-to-1 realtime relationship between sessions and streams.
For example I have lambda functions that consume messages from a KinesisStream. How do stop and resume the function so that I don't incur charges and I don't loose data in the stream.
I know that if the events keep failing, Kinesis will keep retrying and the cost can be very high.
I cannot delete the function because there is lots of automation around it through CloudFormation. Is there a way to stop and restart the function?
SOLUTION: http://alestic.com/2015/11/aws-lambda-kinesis-pause-resume
NOTE: Event sources for rules, log streaming, cannot be disable using the event source. You will not event get it in the list when calling the API using the SDK. For those you have to disable the Event Rule, or the Log Subscription.
The updated Lambda console on AWS supports this in the UI now. Click on the Kinesis stream feeding your lambda function, toggle the "Enabled/Disabled" toggle at the bottom, and Save. This will essentially pause/resume your function.Screenshot - Toggling Kinesis input into Lambda
Let's talk about Kinesis for a moment. When you pull records off the stream, Kinesis will not 'delete' those records until you 'checkpoint' the stream. You can read the same records over and over until you confirm with Kinesis that you don't need them anymore.
AWS Lambda does not checkpoint the stream until the function completes its execution without an error. (context.success())
If you deploy a Lambda function and it is broken in some way (exits with an exception/error), the Lambda function will not checkpoint the stream, and your records will stay in the stream for as long until retention period expires (24 hours, by default). The 'un-checkpointed' records can then be read in a subsequent Lambda execution.
During deployment, the same thing applies. Any currently executing Lambdas that are interrupted will not checkpoint the stream, and any currently executing Lambdas that complete successfully will checkpoint as you expect.