I am looking for the best way to detect errors in cloudwatch logs which are logged by lambda functions, the log output is structured.
I was considering using a metric filter to trigger a lambda but I think eventbridge is now the preferred way to do this sort of thing but from the documentation I cannot work out what is the right way to approach it.
I would like to trigger the same eventbridge rule for any error in any log group if this is possible as all the logs have the same format.
Is it possible to do this purely from cloudwatch log entries so I do not need to add additional code to my functions to call event bridge using the AWS api's?
Instead I would like to trigger the rule whenever a matching json object gets inserted into cloudwatch logs.
I was not even able to find the event structure for cloudwatch log updates.
Amazon EventBridge is a serverless event bus for building event-driven applications. It is best suited for application to application integration with event filtering. Your use case seems to be of pure monitoring ( or notification)
For your use case (monitoring) using the metric filter will be the simple and elegant option.
For implementation (nodejs) refer :CloudWatch log multiple custom metric filters to trigger lambda function
Related
How to pull data from AWS Security hub automatically using a scheduler ?
I am new to AWS on doing some analysis I found below :
In Security Hub data is in Json format , we don't have option to do Export to csv/excel ?
All Security hub findings/insights are automatically sent to eventbridge ? Is it true ? If yes where i can check the same in eventbridge ?
Are there any other options in order to pull data from security hub , every 12 hours automatically. I want to take the data from security hub and pass it to the ETL Process in order to apply some logic on this data ?
Is Eventbridge the only and best approach for this ?
On:
It is a JSON based but it's their own format named AWS Security Finding Format (ASFF)
It is true (for all resources that SecurityHub supports and is able to see). It should be noted that Each Security Hub Findings - Imported event contains a single finding.
In order to see those events you'll need to create an EventBridge rule based on the format for each type of event.
how to create rule for automatically sent events (Security Hub Findings - Imported)
In addition you can create a custom action in SecurityHub and then have an EventBridge event filter for it too.
Once you have that set up, the event could trigger an automatic action like:
Invoking an AWS Lambda function
Invoking the Amazon EC2 run command
Relaying the event to Amazon Kinesis Data Streams
Activating an AWS Step Functions state machine
Notifying an Amazon SNS topic or an Amazon SQS queue
Sending a finding to a third-party ticketing, chat, SIEM, or incident response and management tool.
In general, EventBridge is the way forward, but rather than using a scheduled based approach you'll need to resort to an event-based one.
In order to intercept all findings, instead of rule being triggered by just specific one, you'll need to adjust the filter and essentially create a catch-all rule for SecurityHub which will then trigger your ETL job.
EDIT (as requested in comment):
The filter in the rule would look like this:
{
"source": [
"aws.securityhub"
]
}
with regard to the ETL, it really depends on your use case, having Kinesis Data Firehose dumping it to S3 and then using Athena as you suggest on your own would work. Another common approach is to send the data to ElasticSearch (or now OpenSearch).
This blog post described them both, you can adjust it based on your needs.
EDIT 2:
Based on the discussion in the comments section if you really want to use a cron based approach you'll need to use the SDK based on your preferred language and create something around the GetFindings API that will poll for data from SecurityHub.
You can use this function in Python, which extracts data from SecurityHub to Azure Sentinel as an example
Requirement
I'd like to push DLQ messages to Lambda function periodically like cron.
Situation
I created a batch processing system using SQS and Lambda.
So, I also prepare DLQ(Dead Letter Queue) to store the failed messages.
AWS proposed re-run failed messages in the DLQ with Lambda trigger.
https://docs.aws.amazon.com/lambda/latest/dg/invocation-retries.html
I'd like to re-run periodically and automatically like cron, but AWS doesn't show how to do that except for manually.
In case I re-run manually, I JUST map the lambda and the DLQ. It's all done.
After that, I can re-run messaging dynamically with Enable and Disable button.
Otherwise, it's more complicated because no API switching Enable and Disable of Lambda trigger.
boto3 API: create_event_source_mapping() and delete_event_source_mapping() are seemed to better way.
But delete_event_source_mapping() requires UUID of the event mapping. I think it indicates that I need any datastore like ElastiCache or else.
However, I don't wanna prepare other resources in this situation if possible.
My Solution
Cloud Watch Event call lambda.
lambda activate(Enable) event source mapping using create_event_source_mapping().
lambda deactivate(Disable) event source mapping using delete_event_source_mapping()
It looks good to me at first, but in 3rd process lambda want to know UUID from the 1st event. I think this case needs datastore including UUID.
Do you have any solutions without datastore?
Thanks.
Is there a way to buffer X log messages from a CloudWatch log group and only then stream it to a lambda function? I'll elaborate:
I have an app that I registered it's CloudWatch logs to stream to a lambda function which formats the logs and pushes them to Elastic Search.
So the flow is the following:
(app logs) -> (CloudWatch) -->(Lambda)-->(Elastic Search)
My problem is that my lambda function is invoked very often (most of the time single log message) and bombards ES with write requests, I would like to write the logs in bulks, i.e wait until 30 new logs and then invoke the lambda for the 30 logs bulk.
The only way I found to achieve this is to use Kinesis and Firehose but those services cost extra and I want to avoid this.
Are there any other alternatives to achieve this without using something like LogStash?
I am assuming this is a very common usage so there must be some easy way to solve this.
Thanks,
I would investigate Functionbeat whose main goal is to stream Cloudwatch logs (among others) to ES. Extremely easy to deploy and operate, no fiddling with Lambda code, etc. A MUST if you're evolving in the AWS environment yet still want to leverage ES as a log engine
I was wondering what you ended up doing in the situation. I believe if you use functionbeats you can not use aws ES you have to create it manually.
I have written some cronjobs in my django app and I want to schedule these jobs using AWS Lambda service. Can someone please recommend a good approach to get this done?
I will answer this based on the question's topic rather than the body, since I am not sure what the OP means with "I want to schedule these jobs using AWS Lambda".
If all you want is trigger your Lambda function based in a cronjob, you can use CloudWatch Events to achieve this. You can specify regular cron expressions or some built-in expressions that AWS makes available, like rate(1 min) will run your function every minute. You can see how to trigger a Lambda function via CloudWatch Events on the docs. See cron/rate to see all the available options.
CloudWatch Events is only one of the many options to trigger you Lambda function. Your function can react to a whole bunch of AWS Events, including S3, SQS, SNS, API Gateway, etc. You can see the full list of events here. Just pick one that fits your needs and you are good to go.
EDIT AFTER OP'S UPDATE:
Yes, what you're looking for is CloudWatch Events. Once you have the Lambda to poll your database in place, you can just create a rule in CloudWatchEvents and have your Lambda be triggered by it. Please see the following images for guidance.
Go to CloudWatch, click on Events and choose Schedule as the Event Source
(make sure to setup your own Cron expression or select the pre-defined rate values)
On the right-hand side, choose your Lambda function accordingly.
Click on "Configure Details" when you are done, give it a name, leave the "Enabled" box checked and finally click on Create.
Go back to your Lambda function and you should see it's now triggered by CloudWatch Events (column on the left-hand side)
Your lambda is now configured properly and will execute once a day.
With CloudWatch you can monitor applications running on AWS. Is it also possible to monitor an external service?
For example, I have a REST API and I want to get notified once that API is not accessible anymore. Does AWS offer you a monitoring tool for that purpose?
Not Cloudwatch just by itself, but you can use a combination of Cloudwatch and Lambdas to do what you're asking. You can use cloudwatch events to run lambdas on a schedule, something like once every 5 mins.
CloudwatchEvents -> HealthCheck Lambda -> Cloudwatch Custom Metrics
Your lambda can then ping the API you're monitoring the health of, and either send its status to cloudwatch as a custom metric; or potentially if your lambda throws an error when the API fails, the lambda error metric which is already in cloudwatch becomes your API failure metric
Once the metric exists in cloudwatch, either as a custom metric or the lambda metric by proxy, you're able to do usual cloudwatch things like alarms and notifications.
Now there is a simple way to monitor external resources - CloudWatch Synthetics. Just create a canary to regularly monitor a website, API or even validate a multi-step UI flow.
Read more in the docs: CloudWatch > Using Synthetic Monitoring
Amazon CloudWatch supports custom metrics generated by your applications and services that you do not run on AWS. In this way, CloudWatch can be an integrated storage and aggregation point, allowing you to monitor all of the metrics that you collect, and track on a single platform.
There might be more than one way to reach your goal by using the AWS CLI, an API/SDK, or the CloudWatch collectd plugin etc. I'd recommend you take a look at these links for more details: link-1, link-2, link-3, link-4