AWS allows a Lambda function to be triggered by an SQS queue. With respect to the documentation, Lambda polls the queue and invokes the function synchronously with a batch of messages.
AWS takes the execution time into account while charging for a Lambda function. Does the polling time counted as the execution time and be charged? Or is it free?
It is not counted as the lambda execution time but it is not free either.
When there is no data to be pulled from your SQS queue, your lambda function is not executed, therefore there is no cost associated with lambda execution time.
But when lambda service polls SQS queue, it is a normal SQS API request and it is charged based on the SQS pricing. No matter how the SQS API request is made, whether manually via CLI, programatically via SDK or automatically via lambda service, it will be charged.
For example, you pay $0.0000004 per SQS API request (Standard queue) in US East region (or $0.40 per one million requests) and $0.0000005 per request (FIFO queue).
Related
Will a standard SQS that I configured to invoke a lambda when it receives message invoke "many lambdas" or only 1 lambda at a time?
From Using AWS Lambda with Amazon SQS:
Lambda polls the queue and invokes your function synchronously with an event that contains queue messages.
It will invoke as many as required, depending on your reserved concurrency limits:
Lambda increases the number of processes that are reading batches by up to 60 more instances per minute. The maximum number of batches that can be processed simultaneously by an event source mapping is 1000.
I have a service which generates a burst of Cloud Watch Events once every hour. These Cloud Watch events (which could be in thousands) each will trigger an AWS Lambda function and ultimately number of concurrent lambdas running can cross the maximum limit. How can I modify my system such that these cloud watch events will be handled gracefully by Lambda functions or if possible somehow I can distribute all these cloud watch events over the rest period of the first service.
P.S. I do not want to modify the limit on concurrent running lambdas.
have you thought about adding these events to SQS instead of consuming Lambda directly and then configure the SQS to call the Lambda function?
This is how you can trigger a Lambda function by SQS queue
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-lambda-function-trigger.html
and you can define the delay in queue consumption by using this article
https://cloudaffaire.com/how-to-configure-delay-queue-in-sqs/
As prior stated you could use SQS, but I too would advise against this because you would still have the same concurrency issue (more here).
Depending on how quickly you require your processing, it may be a good idea to have lambda "poll" an SQS every couple of minutes, then send the batch of messages to a lambda to process via SNS (rather than have lambda trigger off of SQS PutMessage API Calls).
Why are some Amazon Web Services configured such that they can make direct calls to Lambda handlers with appropriate permissions, while for others like SQS, lambda needs to poll repeatedly? Why can't we have a provision for invoking Lambda as soon as a message is added to an SQS, instead of polling repeatedly?
I think this is related to scaling.
From Understanding Scaling Behavior - AWS Lambda:
Poll-based event sources that are not stream-based: For Lambda functions that process Amazon SQS queues, AWS Lambda will automatically scale the polling on the queue until the maximum concurrency level is reached, where each message batch can be considered a single concurrent unit. AWS Lambda's automatic scaling behavior is designed to keep polling costs low when a queue is empty while simultaneously enabling you to achieve high throughput when the queue is being used heavily.
When an Amazon SQS event source mapping is initially enabled, Lambda begins long-polling the Amazon SQS queue. Long polling helps reduce the cost of polling Amazon Simple Queue Service by reducing the number of empty responses, while providing optimal processing latency when messages arrive.
When messages are available, Lambda initially launches up to 5 instances of your function, to handle 5 batches simultaneously. Then, Lambda launches up to 60 more instances per minute, up to 1000 total, as long as you have concurrency available at the account and function level.
I have a AWS Lambda function which is triggered by SQS. This function is triggered approximately 100 times daily, but request count to the SQS queue is approximately 20.000 times daily. I don't understand why the number of requests made to the SQS is too high. My expectation is that the number of requests made to the SQS should be same with the Lambda invocation.
I have only one Lambda function and one SQS queue in my account.
Can be related with polling of SQS queue? I tried to change the polling interval of SQS from the queue configuration but nothing changed. Another possibility is to change polling interval from Lambda function configuration. However, I cannot find any related parameter.
Briefy, I want to reduce number of SQS request, how can i do that while invoking Lmabda function with SQS?
When using SQS as an event source for AWS Lambda, AWS Lambda regularly polls the configured SQS queue to fetch new messages. While the official documentation isn't clear really about that, the blog post announcing that feature goes into the details:
When an SQS event source mapping is initially created and enabled, or when messages first appear after a period with no traffic, then the Lambda service will begin polling the SQS queue using five parallel long-polling connections.
According to the AWS documentation, the default duration for a long poll from AWS Lambda to SQS is 20 seconds.
That results in five requests to SQS every 20 seconds for AWS Lambda functions without significant load, which sums up to the ~21600 per day, which is close to the 20000 you're experiencing.
While increasing the long poll duration seems like an easy way to decrease the number of requests, that's not possible, as the 20 seconds AWS Lambda is using by default is already the maximum possible duration for an SQS queue. I'm afraid there is no easy way to decrease the requests to SQS, when using it as event source for AWS Lambda. Instead depending it could be worth evaluating if another event source, like SNS, would fit your use case as well.
Here is how we originally implemented when there is no SQS trigger.
Create a SNS trigger with the SQS Cloudwatch Metric
ApproximateNumberOfMessagesVisible > 0
Trigger a Lambda from SNS, Read Messages from SQS and deliver it to whichever the lambda needs the message.
Alternatively, you can use Kinesis to deliver it to Lambda.
SQS --> Cloudwatch (Trigger Lambda) --> Lambda(Reads Messages) -->
Kinesis (Set Batch Size) --> Lambda (Handle Actual Message)
You can also use Kinesis directly but there is no delayed delivery.
Hope it helps.
My current AWS Lambda function invokes another AWS Lambda function but I want to make sure that the invoke succeeded. After looking at concurrent execution limits for AWS Lambda I am trying to figure out what would happen if the concurrent limit is hit and I tried to invoke the Lambda from another Lambda.
For now, I am solving this problem by putting messages in an SNS but I rather prefer invoking Lambda directly avoiding the indirection.
The best way to handle the concurrent limit is to use a Kinesis stream rather than SNS.
The number of shards will limit the number of lambda invoked. And if it pertinent for you, you can take several messages at once, which you can't do with SNS, and that can lead to hit the concurrent limit.
Can you elaborate a little? Not sure I Understand what you are trying to achieve.
Lambda limits can be viewed under AWS console / EC2 page, top left corner has menu item called Limits, there you should see the limit.
When you hit the limit, lambda will stop being Invoked, and if my memory serves me right you will see an error in the logs saying something about limit being hit.
From the AWS Lambda FAQs:
Q: What happens if my account exceeds the default throttle limit on concurrent executions?
On exceeding the throttle limit, AWS Lambda functions being invoked
synchronously will return a throttling error (429 error code). Lambda
functions being invoked asynchronously can absorb reasonable bursts of
traffic for approximately 15-30 minutes, after which incoming events
will be rejected as throttled. In case the Lambda function is being
invoked in response to Amazon S3 events, events rejected by AWS Lambda
may be retained and retried by S3 for 24 hours. Events from Amazon
Kinesis streams and Amazon DynamoDB streams are retried until the
Lambda function succeeds or the data expires. Amazon Kinesis and
Amazon DynamoDB Streams retain data for 24 hours.
Inside the AWS Console you can always create a Service Limit Increase for AWS Lambda concurrent executions at no additional cost. This answer explains this in more detail.
I believe you're handling it correctly currently. I was just reading an article that was explaining how you shouldn't invoke lambda from another lambda because:
"If you do, the first will run until the second is finished executing, and you’re double billing yourself. Instead, use SNS or SQS to send a message to the other Lambda."
http://web.archive.org/web/20160713113906/http://www.appliedsoftwaredesign.com/archives/aws-lambda-pro-tips/