I have a AWS Lambda that is running every minute. It will either succeed or pass. I have a AWS Alarm that is monitoring this Lambda and going into an ALERT or SUCCESS state based on the Lambda execution. When the Alarm state changes a SNS message is fired off an another Lambda is triggered. This Lambda uses a webhook and sends out a message.
Is there a way of getting the error message from the 1st Lambda to be viewed by the 2nd Lambda (and ultimatley passed to the webhook)? I can see the error message on the CloudWatch logs.
Any ideas on this would be great.
The reason I have a Alarm inbetween the 1st Lambda and the SNS is I only want a message when the state changes not every time the 1st Lambda runs.
Related
Purpose:
My goal is to add items into dynamo using a lambda function, which then will set off another lambda trigger that will send a SNS notification.
The first lambda function successfully adds in items into Dynamo. However the second lambda trigger is not sending SNS notifications.
However, I did a JSON test event for the second lambda trigger and it was then successfully able to send a sns notification. So, theres is something wrong with dynamo and the second lambda function.
In DynamoDB, under 'DynamoDB stream details' where I enabled the lambda tigger I get an error 'PROBLEM: Function call failed'
In our team's infrastructure we have a Databricks job which sends data to an SQS queue which triggers a Lambda function. The Databricks job runs one in every 30 minutes. A week ago the Databricks job was failing continuously so it was not sending data, therefore the Lambda function was not triggered. Is there any way to set up an alert so that I get notified if the lambda function is not triggered for a period of 2 hours?
When I searched for a solution I was only able to see to get an alert if and when a Lambda fails or if a specific log type is found in its cloudwatch logs etc, but couldn't see any solution for the above scenario.
You can create a Cloudwatch alarm for the Invocation metrics for that lambda; you can configure the alarm so that if there are no invocations over a timespan of two hours, it goes into an ALARM state.
If you wish to be notified, you can also configure the Cloudwatch alarm to send a message to an SNS topic, which can then be configured to trigger SES so that it sends you an email (for example).
Currently I'm using SQS - Lambda integration
The concurrency for Lambda is available. SQS batch is set to 1 record, 0 delay.
Visibility timeout for SQS is 15 Minutes, Lambda max exec time is 15 Minutes
I would notice that sometimes SQS Messages are stuck in-flight without being processed by any Lambda at all ( They fall into the dead letter queue after 15 minutes, CloudWatch show no Lambda being invoked with the message )
Has anyone faced the same issue?
I run Lambda inside VPC, if that matters
The Lambda backend polls SQS on your behalf and invokes a Lambda function if a message is returned. If the invocation succeeds the message will be deleted if however the function fails the message will be returned to the queue (or DLQ depending on your redrive policy) after the visibility timeout has expired. Check this blog post.
Check if you can see any error metrics for the function in Cloudwatch. Your Lambda function might be failing before it gets a chance to run any code. When this happens there's an error metric but no invocation metric/logs and it's most likely due to an incorrect permission.
I want to design AWS lambda in manner where if its failed then lambda should attempt to retry for given no of time & if after those many attempts it still fail then alert user.
I tied to configure AWS CloudWatch alarm for failure but looks like user is alert on first failure of lambda.
how about using SQS for the DLQ(Dead Letter Queue)?
You can make fault tolerant architecture using SQS and Lambda together.
Briefly saying, you can make two functions at lambda.
Function1: the first one which is triggered by initial trigger and do it's job. If it fails, it will go to SQS.
Function2 : this is triggered by SQS polling, which means that this function runs when there is a message in SQS queue. It reads SQS message so the event handler should be little bit different.
We have aws alarms set up to email on alarm but we would like to continue to get the alarm notification even if the state is in Alarm without a state change. How could I achieve this (would be happy to use a lambda but no idea how to do it)
Amazon CloudWatch alarm notifications are only sent when the state of the alarm changes. It is not possible to configure CloudWatch to continually send notifications while in the ALARM state.
You would need to write your own code to send such notifications. This could be accomplished via a cron job, scheduled AWS Lambda function or your own application.
Try with a script using Cloudwatch API for example with Boto3 + Python or a Lambda running every X minutes. I have a python script to get values from cloudwatch you can adapt it. http://www.dbigcloud.com/cloud-computing/230-integrando-metricas-de-aws-cloudwatch-en-zabbix.html
One alternative is, to create a Lambda function to send email and host that function using CloudWatch Rule with Scheduled option and target as Lambda function that you have created. In Schedule option, you can set the frequency of time that you expect to receive email. In defined frequency, the Rule will trigger Lambda Function to send email.