I've setup some aws cloudwatch agents to send logs to a cloudwatch log group and what I want to do now is get those into an elasticsearch cluster (on ec2). Ideally, what I want to do is write a lambda function that parses it and then sends it to elasticsearch, but also have the lambda function be triggered anytime a message is written to the log group. Is it possible to trigger lambda from a cloudwatch log group?
Subscribing AWS Lambda functions to CloudWatch Logs is documented here. Also, streaming CloudWatch Logs to the AWS ElasticSearch service is documented here.
Here you can find the documentation about adding a subscription to CloudWatch logs for Lambda.
For streaming to AWS ElasticSearch see this (as mentioned before).
Related
How I can send or export logs from my AWS Lambda function to S3 without using by CloudWatch?
Is there another option?
Yes, you can stop your cloudwatch log ingesting from function and send the log to another destination.
Please check- Disabling logging to CloudWatch Logs
This AWS Sample shows you have to use a Lambda extension to send logs to S3. But there is currently no way to stop logs from going to CloudWatch.
There is an option when creating a CloudTrail trail to enable CloudWatch Logs. The description states "You can enable Sns notifications in CloudWatch Logs for specific API actions. Standard CloudWatch and CloudWatch Logs charges apply." This description makes it seem like you would enable CloudWatch Logs to use Sns specifically.
If CloudTrail can interface with Sns on its own just fine, what would be the purpose of going through CloudWatch Logs to get there? Is it just organizational?
It is referring to the fact the you can filter specific API calls in CloudTrail trails by creating metric filters on the log entries.
CloudTrail has integration with SNS by default, but it will create a notification for every event in the trail. You need CloudWatch if you want something specific.
I have to monitor CloudWatch logs for particular lambda. When the specific error message such as HTTP-50X is logged to CloudWatch, then we need to send an email notification to alert that the something is wrong.
We need help to create alert email in AWS to monitor manually the logs to see if there is any such error logged.
This is exactly what metric filter is for. Create a metric filter that monitors your log group for specific pattern. Then create a CloudWatch alarm based on a new metric and configure it with an SNS action. Then simply subscribe your email to that SNS topic.
All of this can be easily done via CloudFormation(Metric filter, Alarm, SNS). You can also use CDK.
You can process/monitor logs in real-time using lambda functions. Checkout Using AWS Lambda with Amazon CloudWatch Logs and Real-time Processing of Log Data with Subscriptions for details.
The answer to this post states that Amazon Simple Notification Service (SNS) topic can be triggered by Amazon CloudWatch Events schedule without utilizing an AWS Lambda function:
Scheduled rest requests from aws
Is there an ability to automatically publish to SQS when a scheduled Amazon CloudWatch Event occurs without using lambda? Can I set this up by means of Cloud Formation template?
This functionality is supported.
See: https://aws.amazon.com/ru/about-aws/whats-new/2016/03/cloudwatch-events-now-supports-amazon-sqs-queue-targets/
The example of Cloud Formation template:
https://stackoverflow.com/a/52029876/4251336
In AWS, I am trying to detect Critical alerts in Guardduty and configure cloudwatch which will trigger mails when there is a critical alert in guardduty.
I found a link http://docs.aws.amazon.com/guardduty/latest/ug/guardduty_findings_cloudwatch.html which uses Lambda to create fuctions which will do that but I was looking for more simpler ways without AWS lamda.
Is there is a way to directly configure a cloudwatch alarm which will keep an eye on Guardduty and can use SNS notifications to send mails to desired team.
I implemented the CloudWatch event as described in the link you posted. I get alerts via SNS. No Lambda required (although you COULD trigger a Lambda function instead of SNS if you wanted).