What is the best way to migrate invocation of Lambda function from Cloudwatch event to Event Source Mapping(Triggers) with minimum risk and downtime?
Our application uses cloudwatch event rule to invoke Lambda at fixed rate, which then polls SQS queue to get messages, we want to automatically invoke the lambda using SQS trigger(ESM).
Brainstormed a couple approaches:
Option 1. Hookup both, trigger and cloudwatch event to existing Lambda, then slowly deprecate event rule.
Option 2. Using duplicate lambda with event source as SQS queue and divide the traffic, deprecate original lambda later.
Option 3. Add additional SQS(subscribed to same SNS) and Lambda pair which uses the trigger, deprecate the original later.
I am sure many teams might have done similar migration, any insights would be appreciated.
Your option 2 in the simpler way to switch.
Since you have CW event schedule, you lambda can execute maximum once per minute as the minimum precision for schedules is 1 minute.
Create new lambda
Set concurrency for old lambda to zero
Enable SQS trigger for new lambda
Delete event schedule trigger in old lambda
Related
I am wondering about how to configure AWS Lambda, SNS, and SQS for processing background tasks.
There are three ways I thought.
Option 1. A function called consumer can execute workers by receiving tasks from the queue.
Option 2. Send all tasks to SNS. One worker and one SQS receive and work from SNS.
Option 3. Directly forward the task to one SQS and one lambda from the APP.
The biggest concern is whether to directly invoke Lambda in the app or use task consumer using SQS or SNS.
My idea is from Triggering multiple lambda functions from one SQS trigger
It depends on your current and future requirements:
Options 1: Choosing consumer lambda will allow you to add validations and manipulation in the event.
But your consumer lambda will be running until your worker lambdas are running.
Option 2: SNS gives you flexibility to add new events in future and new subscribers as well and your App will have to deal with only SNS.
Option 3: If you are sure in future there will be no such other lambdas. In this case your app need to have the configuration which type of event will go to which SQS.
You can choose any option based on your requirement but I will suggest you to choose option 2 as your app will be required to push notification to SNS only(Single integration). In SNS you can add filters for different types of event.
From SNS you can directly trigger lambda as well.
If you do not need output of lambda functions in your App you should use SNS/SQS for async processing.
The typical pattern is:
Push jobs/tasks to an Amazon SQS queue
Configure an AWS Lambda function to subscribe to the SQS queue
Lambda will automatically execute the Lambda function for each message in the SQS queue
I guess this matches your Option 1, but with the AWS Lambda service acting as the "consumer" that triggers the individual Lambda functions.
If there are different types of inputs (eg three different tasks) that each require a different Lambda function, then create 3 separate queues each linked to its own Lambda function (your Option 3).
Inserting Amazon SNS in-between (shown in your Option 2) makes it easier to 'fork' information, such as adding another subscriber to each message in case they need to be processed in parallel. Otherwise, it is not necessary.
I have a Lambda function that takes a list of tasks to be run at the time specified. This time can vary.
I am using SNS to trigger another Lambda function that in turn runs the tasks.
These tasks need to be run at specified time. Is it possible to publish a message to SNS using Lambda at the specified time?
Or send the message to SNS, but SNS in turn triggers Lambda at the specified time?
Any option would do.
P.S. I know there is an option of using Cloud Watch events, but I do not want to use any more services.
It appears that your requirement is to trigger an AWS Lambda function at specific times.
This can be done by using Amazon CloudWatch Events, which can take a cron-like expression to run a Lambda function at desired time intervals or specific times. This functionality was originally in the Lambda console, but was moved to CloudWatch Events when more services added scheduling capabilities.
However, CloudWatch Events cannot trigger an Amazon SNS message. If you need SNS to trigger Lambda, then you'll need CloudWatch Events to trigger a Lambda function that sends a message to SNS (which then triggers Lambda functions). Obviously, it would be cleaner to avoid SNS altogether unless you specifically need to fan-out the message to multiple subscriptions/Lambda functions.
i have an aws lambda function to do some statistics on over 1k of stock tickers after market close. i have an option like below.
setup a cron job in ec2 instance and trigger a cron job to submit 1k http request asyn (e.g. http://xxxxx.lambdafunction.xxxx?ticker= to trigger the aws lambda function (or submit 1k request to SNS and let lambda to pickup.
i think it should run fine, but much appreciate if there is any serverless/PaaS approach to trigger task
On top of my head, Here are a couple of ways to achieve what you need:
Option 1: [Cost-Effective]
Post all the ticks to AWS FIFO SQS queue.
Define triggers on this queue to invoke lambda function.
Result: Since you are posting all the events in FIFO queue that maintains the order, all the events will be polled sequentially. More-over SQS to lambda trigger will help you scale automatically based on the number of message in the queue.
Option 2: [Costly and can easily scale for real-time processing]
Same as above, but instead of posting to FIFO queue, post to Kinesis Stream.
Enable Kinesis stream to trigger lambda function.
Result: Kinesis will ensure the order of event arriving in the stream and lambda function invocation will be invoked based on the number of shards in the stream. This implementation scales significantly. If you have any future use-case for real-time processing of tickers, this could be a great solution.
Option 3: [Cost Effective, alternate to Option:1]
Collect all ticker events(1k or whatever) and put it into a file.
Upload this file to AWS S3 bucket.
Enable S3 event notification to trigger proxy lambda function.
This proxy lambda function reads the s3 file and based on the total number of events in the file, it will spawn n parallel actor lambda function.
Actor lambda function will process each event.
Result: Easy to implement, cost-effective and provides easy scaling based on your custom algorithm to distribute the load in the proxy lambda function.
Option 4: [All-serverless]
Write a lambda function that gets the list of tickers from some web-server.
Define an AWS cloud watch rule for generating events based on cron/frequency.
Add a trigger to this cloudwatch rule to invoke proxy lambda function.
Proxy lambda function will use any combination of above options[1, 2 or 3] to trigger the actor lambda function for processing the records.
Result: Everything can be configured via AWS console and easy to use. Alternatively, you can also write your AWS cloud formation template to generate all the required resources in a single go.
Having said that, now I will leave this up to you to choose the right solution based on your business/cost requirements.
You can use lambda fanout option.
You can follow these steps to process 1k or more using serverless aproach.
1.Store all the stock tickers in a S3 file.
2.Create a master lambda which will read the s3 file and split the stocks in groups of 10.
3. Create a child lambda which will make the async call to external http service and fetch the details.
4. In the master lambda Loop through these groups and invoke 100 child lambdas passing in each group and return the results to the
Master lambda
5. Collect all the information returned from the child lambdas and continue with your processing here.
Now you can trigger this master lambda at the end of markets everyday using CloudWatch time based rule scheduler.
This is a complete serverless approach.
I have an AWS Lambda function to read from an SQS queue. The lambda logic is basically to read off one message from SQS and then it processes and deletes the message. Code to read the message being something like.
ReceiveMessageRequest messageRequest =
new ReceiveMessageRequest(queueUrl).withWaitTimeSeconds(5).withMaxNumberOfMessages(1);
Now my question is what is the best way to trigger this lambda and how does this lambda scale for instance, if there are let's say 1000 messages in the queue so will there be a 1000 lambdas running together, since in my case one lambda can read only one message off the queue.
Any pointers on best practices around this kind of design.
Right now you best option is probably to setup an AWS Cloudwatch event rule that calls the lambda function on the interval that you need.
Here is a sample app from AWS to do just that:
https://github.com/awslabs/aws-serverless-sqs-event-source
I do believe that AWS will eventually support SQS as a event type for AWS lambda, which should make this even easier, but for now you best choice is probably a version of the code I linked above.
We can now use SQS messages to trigger AWS Lambda Functions. Moreover, no longer required to run a message polling service or create an SQS to SNS mapping.
Further details:
https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
AWS added native support in June 2018: https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/
There are probably a few ways to do this, but I found this guide to be fairly helpful when I tried to implement the same sort of functionality you are describing in Node.js. One downside to this strategy is that you can only poll the queue every 60s.
The basic workflow would look something like this:
Set up a CloudWatch Alarm that gets triggered when the queue has a certain number of messages.
The Cloudwatch alarm then posts to SNS
The SNS message triggers a Lambda scale() function
The scale() function updates a configuration record in a DynamoDB table that sets the number of worker processes needed
You then have a main CloudWatch Schedule that invokes a worker() function every 60s
The worker() function reads configuration from DynamoDB to determine how many concurrent processes are needed, based on the queue size.
Worker() then invokes the appropriate number of process() functions
Process() function consumes messages from SQS, performs your main application logic, and then removes the item from the queue.
You can find an example of what the scaling functions would look like in Node.js here
I have used this solution in a production environment for almost a year without any issues, even with thousands of messages in the queue. If you cut out the scaling portion it is only going to do one message a time.
I'm using an AWS Lambda function that is triggered from an SNS event trigger to consume from an SQS queue. When the Lambda function executes, it pulls 10 messages from the queue, processes them, pulls another 10, and so on and so forth - up to a certain time limit that's coded into the Lambda function (less than the max of 5 minutes, obviously).
It's my understanding that a Lambda function triggered by an SNS event is one-to-one, is that correct? In other words, one SNS event won't trigger multiple Lambda functions (up to the maximum concurrent execution limit). There's no scaling based on load.
Are there any other potential solutions, leveraging Lambda, that would let me consume from SQS as frequently/fast as possible? I had considered trying to auto-scale my Lambda functions by leveraging CloudWatch alarms (and SNS event triggers) based on SQS queue size, but it seems like those alarms can fire, at most, every 5 minutes. I've also considered developing a master Lambda function that can automatically execute (many) slave Lambdas based on querying the queue size.
I understand that the more optimal design may be to leverage Kinesis instead of SNS. I may consider incorporating Kinesis in the future, but let's just pretend that Kinesis is not an option at this time.
There is no best way to do this. One approach (which you've kind of already mentioned) is to use CloudWatch and schedule a Lambda function to run every minute (that's the minimum schedule time for Lambda). This Lambda function will then look for new SQS messages and invoke other Lambda functions to handle new message(s). Here is a very good article for that use case: https://cloudonaut.io/integrate-sqs-and-lambda-serverless-architecture-for-asynchronous-workloads/
Personally, I do not recommend triggering your Lambda by SNS for this use case, because SNS doesn't give a full guarantee for delivery and recommend sending the SNS notifications to SQS - which does not solve your problem. From the FAQ's:
[...] If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).
Source: https://aws.amazon.com/sns/faqs/
For this kind of processing, instead of SQS if you push messages to Kinesis Stream you should be able to flexibly process(In batches of needed size) the messages.
Note: If you use SQS, after triggering a Lambda function through SNS (or using a Scheduled Lambda), it can invoke inner Lambda functions to check the queue where multiple concurrent inner Lambdas are spawned. However the problem is that its not practical to process SQS items in batches.