I have a question about streaming CloudFront logs to CloudWatch. I know that it is possible to save CloudFront logs to S3 bucket and later, I can use lambda function which can be triggered by S3 to send logs to CloudWatch. There is also a possibility to look at streamed logs using Amazon Kinesis Data Streams. However I want to know that if some CloudFront configurations exist to send the logs to CloudWatch. Helpful answers are appreciated. Thanks in advance!
Related
How I can send or export logs from my AWS Lambda function to S3 without using by CloudWatch?
Is there another option?
Yes, you can stop your cloudwatch log ingesting from function and send the log to another destination.
Please check- Disabling logging to CloudWatch Logs
This AWS Sample shows you have to use a Lambda extension to send logs to S3. But there is currently no way to stop logs from going to CloudWatch.
I use a lot StackDriver sinks to BigQuery. It helps on keeping the Data and have them in a convenient queryable form.
I am searching for the equivalent on AWS using CloudWatch but it seems only S3 is integrated. Any workarounds or should I code it?
This can be done with a workaround. It needs only one code component reading from S3 and posting to Stackdriver, then GCP sinks can pickup.
Create a CloudWatch Sink to S3 bucket.
The S3 bucket will be integrated to send SNS messages on object Creation.
A process should listen to those SNS notifications and copy the contents of the bucket into StackDriver.
Since the guarantees of SNS and Cloudwatch to S3 are there, a Lambda function on the SNS notification messages can make the solution more seamless.
I created a custom app that automatically uploads logs to s3.
Is there a way to push those logs to cloudwatch from s3 for analysis and alerting?
I'm aware that I can use a cloudwatch agent to push directly to cloudwatch from the app but there are complications involved in that option.
Thank you!
You could probably use Cloudwatch Events to listen to S3 changes. Not sure about if you can get the data from the S3 file, or just a trigger saying that a new log has been added.
You could also use S3 event notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) connected either to a lambda or SQS, and from there create the logs to Cloudwatch. (similar to what was suggested by #marcin)
A better solution, but one that is a bit beyond the scope of the question, would be to send your logs through Kinesis Firehose and from there add the Cloudwatch and S3 logs.
I'm not aware of any out-of-the-box mechanism for that provided by AWS. But I think it could be relatively easy to develop.
Namely, you can create S3 notification for a PUT of a new log file from your app to S3. The event would trigger a lambda function. The function would get the file and using AWS SDK, e.g. boto3's put_log_events, it would send the log events to CloudWatch logs.
Is there a way where we can check how many S3 List requests are happening on a bucket
The general solution on AWS to finding out what is going on at the API level is to use CloudTrail.
See Logging Amazon S3 API Calls by Using AWS CloudTrail.
This can be done by enabling the request metrics on S3 bucket by using CloudWatch service. The metrics are available at 1-minute intervals after some latency to process. These CloudWatch metrics are billed at the same rate as the Amazon CloudWatch Custom Metrics.
You can refer this Document to Configure Request Metrics for an S3 Bucket.
Looking to create a rule to notify on if someone happen to change permissions on an s3 bucket to everyone essentially. Hoping someone has some experience with a similar request and would love to hear what you did.
Configure CloudTrail to send logs to CloudWatch and setup an alarm to get notified when particular API request happened.
Read here how to enable logging to CloudWatch: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/send-cloudtrail-events-to-cloudwatch-logs.html
Read here how to filter particular API requests: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudwatch-alarms-for-cloudtrail.html#cloudwatch-alarms-for-cloudtrail-s3-bucket-activity