Our system differentiates between what we're calling 'single events' and 'bulk events'. Single events are the result of user interactions and arrive just a few at a time, now and then. Bulk events are the result of actions by an admin and can contain hundreds of events that arrive at essentially the same time.
All the events arrive at a lambda that detects the bulk events and sends them into an SNS FIFO topic. Next is an SQS FIFO queue that's subscribed to the SNS, followed by a second lambda that's triggered by the SQS. The second lambda sends the bulk events into the system. We need the bulk events to enter the system slowly so that we don't exceed the API rate limit of a third-party downstream system.
We're experimenting with batch size on the lambda SQS trigger and setting the reserved concurrency on the second lambda to a low value, but it appears we need to slow things down quite a bit more. We're considering adding a wait timer to the first lambda so that it would insert a 5-10 second delay before each individual event being sent into SNS. That should more dramatically spread the events out in time, and it seems like it might work, but isn't a very satisfying approach.
Are there other options for slowing down the throughput? Any ideas would be appreciated. Thanks.
Related
My requirement is like this.
Read from a SQS every 2 hours, take all the messages available and then process it.
Processing includes creating a file with details from SQS messages and sending it to an sftp server.
I implemented a AWS Lambda to achieve point 1. I have a Lambda which has an sqs trigger. I have set batch size as 50 and then batch window as 2 hours. My assumption was that Lambda will get triggered every 2 hours and 50 messages will be delivered to the lambda function in one go and I will create a file for every 50 records.
But I observed that my lambda function is triggered with varied number of messages(sometimes 50 sometimes 20, sometimes 5 etc) even though I have configured batch size as 50.
After reading some documentation I got to know(I am not sure) that there are 5 long polling connections which lambda spawns to read from SQS and this is causing this behaviour of lambda function being triggered with varied number of messages.
My question is
Is my assumption on 5 parallel connections being established correct? If yes, is there a way I can control it? I want this to happen in a single thread / connection
If 1 is not possible, what other alternative do I have here. I do not want to have one file created for every few records. I want one file to be generated every two hours with all the messages in sqs.
A "SQS Trigger" for Lambda is implemented with the so-called Event Source Mapping integration, which polls, batches and deletes messages from the queue on your behalf. It's designed for continuous polling, although you can disable it. You can set a maximum batch size of up to 10,000 records a function receives (BatchSize) and a maximum of 300s long polling time (MaximumBatchingWindowInSeconds). That doesn't meet your once-every-two-hours requirement.
Two alternatives:
Remove the Event Source Mapping. Instead, trigger the Lambda every two hours on a schedule with an EventBridge rule. Your Lambda is responsible for the SQS ReceiveMessage and DeleteMessageBatch operations. This approach ensures your Lambda will be invoked only once per cron event.
Keep the Event Source Mapping. Process messages as they arrive, accumulating the partial results in S3. Once every two hours, run a second, EventBridge-triggered Lambda, which bundles the partial results from S3 and sends them to the SFTP server. You don't control the number of Lambda invocations.
Note on scaling:
<Edit (mid-Jan 2023): AWS Lambda now supports SQS Maximum Concurrency>
AWS Lambda now supports setting Maximum Concurrency to the Amazon SQS event source, a more direct and less fiddly way to control concurrency than with reserved concurrency. The Maximum Concurrency setting limits the number of concurrent instances of the function that an Amazon SQS event source can invoke. The valid range is 2-1000 concurrent instances.
The create and update Event Source Mapping APIs now have a ScalingConfig option for SQS:
aws lambda update-event-source-mapping \
--uuid "a1b2c3d4-5678-90ab-cdef-11111EXAMPLE" \
--scaling-config '{"MaximumConcurrency":2}' # valid range is 2-1000
</Edit>
With the SQS Event Source Mapping integration you can tweak the batch settings, but ultimately the Lambda service is in charge of Lambda scaling. As the AWS Blog Understanding how AWS Lambda scales with Amazon SQS standard queues says:
Lambda consumes messages in batches, starting at five concurrent batches with five functions at a time. If there are more messages in the queue, Lambda adds up to 60 functions per minute, up to 1,000 functions, to consume those messages.
You could theoretically restrict the number of concurrent Lambda executions with reserved concurrency, but you would risk dropped messages due to throttling errors.
You could try to set the ReservedConcurrency of the function to 1. That may help. See the docs for reference.
A simple solution would be to create a CloudWatch Event Trigger (similar to a Cronjob) that triggers your Lambda function every two hours. In the Lambda function, you call ReceiveMessage on the Queue until you get all messages, process them and afterward delete them from the Queue. The drawback is that there may be too many messages to process within 15 minutes so that's something you'd have to manage.
We noticed that when setting up an AWS lambda to trigger from SQS that a lot of times the trigger happens minutes and sometimes up to an hour delay to trigger. I know AWS lambda does polling internally and when the queue is empty it probably does some exponential backoff.
However, we have a scheduler that runs every 30 min and pushes data into the queue. However, lambda is triggered much much later for a % of messages. Our business requirement that it triggers within a min.
Is there a way to force lambda to check the queue consistently? An alternative was to uses step functions but this is not possible due to another answer in this thread --> How do you run functions in parallel?
I was also thinking about pushing data into s3 and have lambda trigger from s3 asynchronously vs being polled but s3 does not have a batch api when we want to push a lot of records so that's out.
It turned out to be the wrong use of async/await when using AWS SDK. They only support .promise(). That was the reason that not all messages ended up in sqs.
Hope it helps others. AWS is working on a new sdk that will support async/await. Here is the link for their
https://github.com/aws/aws-sdk-js-v3/issues/153#issuecomment-457769969
I would check to make sure the SQS is either long pulling or short pulling.
"In almost all cases, Amazon SQS long polling is preferable to short polling. Long-polling requests let your queue consumers receive messages as soon as they arrive in your queue while reducing the number of empty ReceiveMessageResponse instances returned.
Amazon SQS long polling results in higher performance at reduced cost in the majority of use cases. However, if your application expects an immediate response from a ReceiveMessage call, you might not be able to take advantage of long polling without some modifications to your application.
For example, if your application uses a single thread to poll multiple queues, switching from short polling to long polling will probably not work, because the single thread will wait for the long-poll timeout on any empty queues, delaying the processing of any queues that might contain messages.
In such an application, it is a good practice to use a single thread to process only one queue, allowing the application to take advantage of the benefits that Amazon SQS long polling provides."
Also, if you're looking to fire off your lambda function, could you set up an SNS notification system to your SQS? Something along the lines of SQS SNS Lambda. This should get you sub minute and you're not constantly pulling the queue for messages. You'll just do it on a SNS.
https://aws.amazon.com/sqs/faqs/
I have an AWS Lambda function which processes events from S3. I'd like to aggregate them before processing and let lambda process the batch.
This is depicted below:
Ideally, I'd like to be able to specify a batch size, and a timeout (say a single even, and then nothing for 5 sec, I'd like to send an 1-event batch).
Is there an idiomatic way to do it using Lambda or other AWS services?
There are a few things you can do:
1. Make upstream do the aggregation:
Make publishing the publisher's responsibility, and get the publisher to give you one event per group of objects to process. This works well if the publisher is already working in batches.
2. Insert your own aggregation step:
Trigger on each event.
Store the event somewhere.
If enough events have been stored, empty the store and pass all the contents to the processing step.
This works well if your processing step is much more expensive per event than just handling the event. Often, this can take the form of {aggregating lambda} -> {processing batch job}, since Lambda isn't great for very expensive processing.
3. Do aggregation on a time basis:
Send your events to an SQS queue.
Trigger on a timer (e.g. Cloudwatch events).
When triggered, empty the queue and process everything in it. If it's too much to process in a single invocation, immediately trigger an additional lambda.
This works well if processing is fairly cheap, and you want to minimize your number of Lambda invocations. The trigger schedule (how long you wait in between invocations) is determined by weighing how long you're willing to wait to process an event against how many invocations you're willing to pay for. Things to watch out for: 1. if you get no events at all, you will still be invoking your Lambda, and 2. if you get events faster than they can be processed, your queue will grow more and more and your processing will fall further and further behind.
I think you can achieve the batch operation by setting SQS queue as destination for S3 notification. Let's say you want to specify a batch size of 20, all your S3 events are going to SQS. You would create a CloudWatch rule to trigger a Lambda when your SQS have 20 items. Your Lambda would poll SQS for the batch of 20 items and process them.
You can also set SQS triggers but it has a limit of max batch size 10.
I’m working on figuring out the best way to have Lambda run one time tasks at a given time.
The system I’m envisioning will basically have events that will need to be sent out, either as soon as the event is received/created, at a specific time, or as a recurring action. And I’d like to use AWS as much as possible for this, due to the scalable nature.
My original idea was to have a AWS SQS queue for events to send. Then I’d have a DynamoDB table for future events. I’d also have two AWS Lambda functions, one setup to run on a cron job every few minutes to take the events that are scheduled in the next 15 minutes or so from the DynamoDB table and put them into that AWS SQS queue with a Message Timer setup to delay the message from being visible for that given time. The second Lambda function would be setup and have a trigger to be run from that AWS SQS queue. This function would be responsible for actually sending the event out.
From there I could either add the event to the SQS queue (with or without a message timer) if it’s gonna need to be sent out within the next 15 minutes. Or add it to the DynamoDB table if it’s gonna need to be sent out in the future (beyond 15 minutes).
The biggest problem I just figured out is that AWS SQS FIFO queues doesn’t support Message Timers on individual messages. I need a FIFO queue because I need to prevent these events from being sent out multiple times, or triggering my second Lambda function twice.
I've also looked into the AWS Lambda cron jobs, and although you can schedule invocations every say 5 minutes, I don't think this is what I'm looking for because I'm looking more for scheduling a 1 time invocation in the future, and having that be scalable. So I don't think this is what I'm looking for.
Any ideas on how I can achieve this, since it doesn’t look like Amazon SQS Message Timers will work for what I'm trying to do?
Have you considered Step Function? You could create a wait state before invoking the lambda.
I have 2 FIFO SQS queues which receives JSON messages that are to be indexed to elasticsearch. One queue is constantly adding delta changes to the database and adding them to the queue. The second queue is used for database re-indexing i.e. the entire 50Tb if data is to be indexing every couple of months (where everything is added to the queue). I have a lambda function that consumes the messages from the queues and places them into the appropriate queue (either the active index or the indexing being rebuilt).
How should I trigger the lambda function to best process the backlog of messages in SQS so it process both queues as quickly as possible?
A constraint I have is that the queue items need to be processed in order. If the lambda function could be run indefinitely without the 5 minute limit I could keep running one function that constantly processes messages.
Instead of pushing your messages directly into SQS you could publish the messages to a SNS Topic with 2 Subscriber registered.
Subscriber: SQS
Subscriber: Lambda Function
Has the benefit that your Lambda is invoked at the same time as the message is stored in SQS.
The standard way to do this is to use Cloudwatch Events that run periodically. This lets you pull data from the queue on a regular schedule.
Because you have to poll SQS this may not lead to the fastest processing of messages. Also, be careful if you constantly have messages to process - Lambda will end up being far more expensive than a small EC2 instance to handle the messages.
Not sure I fully understand your problem, but here are my 2 cents:
If you have a constant and real-time stream of data, consider using Kinesis Streams with 1 shard in order to preserve the FIFO. You may consume the data in batch of n items using lambda. Up to you to decide the batch size n and the memory size of lambda.
with this solution you pay a low constant price for Kinesis Streams and a variable price for Lambdas.
Should you really are in love with SQS and the real-time does not metter, you may consume items with Lambdas or EC2 or Batch. Either you trigger many lambdas with CloudWatch Events, either you keep alive an EC2, either you trigger on a regular basis an AWS Batch job.
there is an economic equation to explore, each solution is the best for one use case and the worst for another, make your choice ;)
I prefer SQS + Lambdas when there are few items to consume and SQS + Batch when there are a lot of items to consume.
You may probably also consider using SNS + SQS + Lambdas like #maikay says in his answer, but I wouldn't choose that solution.
Hope it helps. Feel free to ask for clarifications. Good luck!