I have a lambda that is subscribed to an SQS queue to process messages. The message volume is very high.
Problem: The queue grows very quickly, and the lambda function does not scale-out to process the messages fast enough. The concurrent lambda executions goes up to only 20 to 25, even though I have a remaining quota of 950 or more un-used lambda executions. Why is it not spinning up more lambda to process my queue faster? Is this configurable?
This is an issue in my application because I am using a standard SQS queue, which provides no ordering guarantee. So, sometimes I see unlucky messages get suck in the queue for hours, whereas some messages are processed in less than one minute. (As an aside, I'm quite shocked that the queue can be processed in such a random order. Even though there is no ordering guarantee, I would not have expected it to be this bad).
The problem was the memory allocation of the lambda function. I had naively left it as the default of 128MB. Changing this to 2048MB completely resolved the issue. The lambda now has no issues keeping up with high volumes of SQS messages.
Regarding SQS, you didn't say what region you are using but SQS does have a FIFO option in come regions.
FIFO queues are available in the US East (N. Virginia), US East (Ohio), US West (Oregon), EU (Ireland), Asia Pacific (Sydney), and Asia Pacific (Tokyo) regions. FIFO queues have all the capabilities of the standard queue.
Regarding Lambda concurrency, It sounds like you are running out of IP addresses in the subnet you're using. This would only apply if you're using a VPC.
If your function connects to VPC based resources, you must make sure your subnets have adequate address capacity to support the ENI scaling requirements of your function. You can estimate the approximate ENI capacity with the following formula:
Concurrent executions * (Memory in GB / 3 GB)
Where:
Concurrent execution – This is the projected concurrency of your workload. Use the information in Understanding Scaling Behavior to determine this value.
Memory in GB – The amount of memory you configured for your Lambda function.
You can set the concurrent execution limit for a function to match the subnet size limits you have.
References
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html
https://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
Related
I have a lambda function that is triggered by an sqs queue . I set batch size to 1 because I want each message to map to one lambda instance to benefit from concurrency and finish processing faster.
However,after some trials with 1000 messages available in sqs queue max concurrent execution only reachs 50 although I reserve 1000 concurrency for my function.
Is there a reason behind this behavior
One reason could be that your functions finish quickly. Thus there is no reason to span 1000 concurrent functions. Lambda polls the sqs at fixed intervals, so you can just span 1000 concurrent invocations in an instance. Similarly, lambda does not scale to 1000 in an instant. Please read the following for more details:
Understanding how AWS Lambda scales with Amazon SQS standard queues
I have setup an SQS queue where S3 paths are being pushed whenever there is a file upload.
I have also set up a Lambda with an SQS trigger and a batch size of 1.
In my scenario, I have to process n files at a time. Lets say (n = 10).
Say, there are 100 messages in the queue. In my current implementation I'm doing the following steps:
Whenever there is a message in the input queue, Lambda will be triggered
First I check the active number of concurrent executions I have. If am already running 10 executions, the code will simply return without doing anything. If it is less than 10, it reads one message from the queue and calls for processing.
Once the processing is done, the message will be manually deleted from the queue.
With the above mentioned approach, I'm able to process n files at a time. However, Say 100 files lands into S3 at the same time.
It leads to 100 lambda calls. Since we have a condition check in Lambda, the first 10 messages go for processing and the remaining 90 messages go to the in-flight mode.
Now, when some of my processing is done (say 3/10 got over), still the main queue is empty since the messages are still in-flight.
As per my understanding, if processing a file takes x minutes, the visibility timeout of the messages in the queue should be lesser than x (<x) . So that the message would once be available in the queue.
But it also leads to another problem. Say the batch took some more time to complete, message would come back to queue. Lambda would be triggered and once again it goes to the flight mode.
Is there any way, I can control the number of triggers made in lambda. For example: only first 10 messages should be processed however remaining 90 messages should remain visible in the queue. Or is there any other way I can make this design simple ?
I don't want to wait until 10 messages. Even if there are only 5 messages, it should trigger those files. And I don't want to call the Lambda in timely fashion (ex: calling it every 5 minutes).
There is a setting in Lambda called Reserved Concurrency, I'm going to quote from the docs (emphasis mine):
Reserved concurrency – Reserved concurrency creates a pool of requests that can only be used by its function, and also prevents its function from using unreserved concurrency.
[...]
To ensure that a function can always reach a certain level of concurrency, configure the function with reserved concurrency. When a function has reserved concurrency, no other function can use that concurrency. Reserved concurrency also limits the maximum concurrency for the function, and applies to the function as a whole, including versions and aliases.
For a deeper dive, check out this article from the documentation.
You can use this to limit how many Lambdas can be triggered in parallel - if no Lambda execution contexts are available, SQS invocations will wait.
This is only necessary if you want to limit how many files can be processed in parallel. If there is no actual need to limit this, it won't cost you more to let Lambda scale out for you.
You don't have to limit your concurrent Lambda execution. AWS already handling that for you. Here are the list of maximum concurrent per region from this document: https://docs.aws.amazon.com/lambda/latest/dg/invocation-scaling.html
Burst concurrency quotas
3000 – US West (Oregon), US East (N. Virginia), Europe (Ireland)
1000 – Asia Pacific (Tokyo), Europe (Frankfurt), US East (Ohio)
500 – Other Regions
In this document: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
Scaling and processing
For standard queues, Lambda uses long polling to poll a queue until it
becomes active. When messages are available, Lambda reads up to 5
batches and sends them to your function. If messages are still
available, Lambda increases the number of processes that are reading
batches by up to 60 more instances per minute. The maximum number of
batches that can be processed simultaneously by an event source
mapping is 1000.
For FIFO queues, Lambda sends messages to your function in the order
that it receives them. When you send a message to a FIFO queue, you
specify a message group ID. Amazon SQS ensures that messages in the
same group are delivered to Lambda in order. Lambda sorts the messages
into groups and sends only one batch at a time for a group. If the
function returns an error, all retries are attempted on the affected
messages before Lambda receives additional messages from the same
group.
Your function can scale in concurrency to the number of active message
groups. For more information, see SQS FIFO as an event source on the
AWS Compute Blog.
You can see that Lambda is handling the scaling up automatically. No need to artificially limit the number of Lambda running to 10.
The idea of Lambda is you want to run as many tasks as possible so that you can achieve parallel execution in the shortest time.
I've seen a number of SO questions on limiting Lambda concurrent execution but none on the inverse issue.
I need to increase my concurrent execution but am having issues. I've got a Lambda triggered off an SQS queue. I've published a version of the function and assigned it 3,000 concurrent execution (my limit has been increased to 5,000 from the default of 1,000).
Despite this, when I run my process I see hundreds of thousands of messages waiting in the queue while the Monitoring tab of my Lambda function shows my "Concurrent executions" never going above 1,250 and my "ProvisionedConcurrencyUtilization" never going above 50%. Moreover, the chart seems to imply a hard limit of 1,250.
I'd be inclined to suspect that there is some sort of limit preventing any single Lambda from using more than 25% of total provisioned capacity (1,250 is 25% of 5,000) but the AWS documentation states otherwise. I did see this SO question (AWS Lambda Triggered by SQS increases SQS request count) which discusses Labmda/SQS polling but it and the documentation it links to indicate my process should use 100% of the Provisioned Capacity. But perhaps it's the polling that's causing the issue.
In any event, these messages sit in the queue for over an hour to process ... with never more than 1,250 processing at the same time ... while the reset of that provisioned concurrency sits idle.
Any suggestions/ideas are greatly appreciated.
Jelly's suggestion was a good one.
Unfortunately, AWS says there is a hard limit of 1,250 Lambda concurrent executions when using Amazon SQS trigger.
I have a use case in which my system will send almost 40 million messages to SNS in span of 8-10 hours. I have a subscription from SNS to Lambda.
My question is how will the scaling work in this case? I understand there is a concurrency limit per account depending upon region. My region has 1000 concurrency, that means at one point in time 1000 executions will happen in parallel. When I send more than 1000 messages in SNS, how will lambda behave in this case? Will the messages go in some internal aws lambda queue and they will be executed as and when lambda has capacity to do so? Please note that there is no requirement of making it process in near real time.
I couldn’t find anything like this in documentation. Really appreciate if someone can point to me the correct documentation or explain the behaviour in this case.
Thanks
I have an AWS Lambda Function setup with a trigger from a SQS queue. Current the queue has about 1.3m messages available. According to CloudWatch the Lambda function has only ever reached 431 invocations in a given minute. I have read that Lambda supports 1000 concurrent functions running at a time, so I'm not sure why it would be maxing out at 431 in a given minute. As well it looks like my function only runs for about 5.55s or so on average, so each one of those 1000 available concurrent slots should be turning over multiple times per minute, therefor giving a much higher rate of invocations.
How can I figure out what is going on here and get my Lambda function to process through that SQS queue in a more timely manner?
The 1000 concurrent connection limit you mention assumes that you have provided enough capacity.
Take a look at this, particularly the last bit.
https://docs.aws.amazon.com/lambda/latest/dg/vpc.html
If your Lambda function accesses a VPC, you must make sure that your
VPC has sufficient ENI capacity to support the scale requirements of
your Lambda function. You can use the following formula to
approximately determine the ENI capacity.
Projected peak concurrent executions * (Memory in GB / 3GB)
Where:
Projected peak concurrent execution – Use the information in Managing Concurrency to determine this value.
Memory – The amount of memory you configured for your Lambda function.
The subnets you specify should have sufficient available IP addresses
to match the number of ENIs.
We also recommend that you specify at least one subnet in each
Availability Zone in your Lambda function configuration. By specifying
subnets in each of the Availability Zones, your Lambda function can
run in another Availability Zone if one goes down or runs out of IP
addresses.
Also read this article which points out many things that might be affecting you: https://read.iopipe.com/5-things-to-know-about-lambda-the-hidden-concerns-of-network-resources-6f863888f656
As a last note, make sure your SQS Lambda trigger has a batchSize of 10 (max available).