I have 2 serverless projects (P1,P2), where in P1 there is EventBridge that routes events to P1 SQS that has P2 Lambda subscriber.
In other words:
P1 has EventBridge, SQS Queue and SQS DLQ.
P2 has Lambda that is subscribed to P1 SQS Queue.
My Question: what is a best practice to define all these resources:
EventBridge + SQS/SQS DLQ in P1, Lambda - P2.
EventBridge - P1, SQS/SQS DLQ, Lambda - P2.
In case 1 everything works fine, but if I remove P1 stack and redeploy again, Lambda in P2 does not pull messages from P1 SQS even though after redeployment SQS arn remains the same in P1. I need to remove and deploy P2 stack again.
In case 2 I deploy P2 first, provide SQS arn and url to P1 (as a string values) and deploy P1. But in this case EventBridge does not route messages to P2 SQS (this is the 2nd question - why it does not work?). I solved it by exporting SQS arn/url as CloudFormation outputs and importing into P1 stack making P1 stack dependent on P2 stack but I don't want to create dependency between P1 and P2.
Related
I created a lambda function in us-east-1 and sns topic to send notifications to a slack channel.
Now I also want to use logs from a service in us-west-2 to trigger the notifications but I can't because they are in different regions.
Whats the best way to handle this? I could just copy the Lambda function/sns topic into us-west-2 but that seems redundant....
Thanks
I decided to go with separate lambda functions in each region.
Since Network Manager is only available in US West 2 and the messages being processed will be specific to that region.
What is the difference between setting a Dead Letter Queue on an SNS Topic or on a Lambda function?
I was wondering, because if you set the DLQ on the SNS subscription, then that subscription message will fail over to DLQ when the Lambda (the subscriber) fails, correct? So in that scenario would setting the DLQ in these two places have the same effect?
I have set a DLQ on a SNS Topic Subscription, and it didn't "automagically" appear as the DLQ on the Lambda screen settings, so I assume there may be some difference?
SNS dead letter queue ref: https://docs.aws.amazon.com/sns/latest/dg/sns-dead-letter-queues.html
In general, message delivery fails when Amazon SNS can't access a subscribed endpoint due to a client-side or server-side error.
Lambda dead letter queue ref: https://aws.amazon.com/about-aws/whats-new/2016/12/aws-lambda-supports-dead-letter-queues/
AWS Lambda will write the event object invoking the Lambda function to this [DLQ] endpoint after the standard retry policy (2 additional retries on failure) is exhausted.
Lambda:
SNS subscription:
The SNS DLQ is specific for SNS, the benefit here is that it also accounts for client errors such as the Lambda service being down. This will ensure if the Lambda service is down the messages can be replayed to it later, whereas if the DLQ is attached to the Lambda this would only account for a replay if the service is running.
However as I mentioned the SNS DLQ is only for notifications that are from SNS, whereas Lambda can support the DLQ from any incoming events. This means if you have multiple SNS topics, or an SNS topic and some SQS queues you only need to apply it to the Lambda itself.
Both services use SQS for their DLQs so the ingestion/retrieval from both would be identical. If you have a DLQ on both services then yes you might end up with 2 copies of the event/notification, however it is unlikely that you will get both as in theory would the Lambda endpoint acknowledges it SNS would treat as sent, this would then be Lambdas responsibility to add to the DLQ if it fails.
SNS DLQ and Lambda DLQ protect your workload from different failure modes. The former is for failed message delivery from a topic, whereas the latter is for failed function execution. You should use both simultaneously. Here more info: https://aws.amazon.com/blogs/compute/designing-durable-serverless-apps-with-dlqs-for-amazon-sns-amazon-sqs-aws-lambda/
We are trying to work on a Live - DR model for SQS queue.
We have two different account in AWS:
eu-west (Account no 1234)
us-east (Account no 4567)
Our application resides in both account (ACTIVE - PASSIVE).
In a normal scenario, EU-WEST is active and US-EAST inactive.
When we failover for DR, US-EAST will be active and EU-WEST inactive.
We want to have two SQS queue in each account (eu-west, us-east).
When EU-WEST is active, we want only SQS queue in EU-WEST working and processing events.
When we switch to DR we want to make EU-WEST SQS inactive and make SQS in US-EAST active.
There is a Lambda trigger on each SQS.
The problem we might face here is: Both SQS queues will process events since it subscribes to the same SNS topic. And since it is connected to a Lambda function, both will process events.
I don't want this to happen. I want only one pair of SQS and Lambda functions working at a time - either EU-WEST or US-EAST. I know this can be achieved by removing the Lambda trigger on the inactive region.
Just looking for a better approach.
I got a solution here.
We need to check the Route53 to find the current active region for the application.
In DR region, when we receive a message on SQS , it will trigger the lambda. The lambda checks for active region based on route53 or ALB dns . If it finds the region is not active/live , it will skip processing of the message and hence the SQS queue will clear up on DR.
So live region Lambda will be actively processing SQS messages whereas the DR one will skip all the processing.
This idea should work for the scenario I mentioned above.
I have my SQS URL which is configured in one VPC and want to trigger Lambda function from another VPC based on any new message that comes in SQS. How can i achieve it for real time analysis.
Amazon SQS is not associated with Amazon VPC. It exists "outside" of VPCs.
Therefore, you can simply configure the Lambda function to use the SQS queue as a trigger.
I have a lambda function with 2 aliases.
Dev - Points to the $LATEST version
Test - Points to a specific version.
I have setup my SQS queue to trigger both aliases when it receives a message in the queue. However, it only seems to trigger one of them (The one that got registered the latest) and not both. Has anyone else come across this issue?
arn:aws:lambda:us-east-1:XXXXXXXXXXXXXX:function:Amoel:Dev
arn:aws:lambda:us-east-1:XXXXXXXXXXXXXX:function:Amoel:Test
A message on the queue would be consumed only once, by whichever Lambda function grabs it.
If you wish to send the same message to multiple AWS Lambda functions, combine it with Amazon SNS:
SQS queue -> SNS topic -> 2 x Lambda function subscriptions
Be sure to activate Raw Message Delivery on the Amazon SNS topic to preserve the original format of the message from Amazon SQS.