I have an AWS lambda function that uploads a file to the dropbox using its api.
It works well in normal but when there are hundreds~thousands of requests in a short amount of time (within a few min, for example), dropbox file upload requests are throttled and get penalty to retry in 30/60/300 seconds. This obviously causes time-out in lambda functions (which is not efficient in pricing term as well).
Can someone help me how to control such a spike in lambda invocation? I tried to set the reserved concurrency for that function but it doesn't seem so reliable because lambda invocations are throttled and retries randomly.
Or is it something that I need to resolve with dropbox side? So that they do not put limit to their api calls?
Related
I am trying to use API Gateway as my API interface between Frontend and Lambda functions. Since API gateway has the maximum timout for 30 seconds and lambda take much time to do the computation, can we use the API Gateway Web socket to make this possible?
I currently am creating RESTful API's on API Gateway and found out about the Web sockets on API Gateway.
Do anyone has suggestions on how to make this possible?
Depending on what your Lambda function is doing, it may be worthwhile to increase the Lambda Memory configuration. Copied from the Lambda Developer Guide emphasis mine:
Memory – The amount of memory available to the function during execution. [...] Lambda allocates CPU power linearly in proportion to the amount of memory configured.
Thus, by increasing the amount of Lambda memory, you also increase the Lambda CPU performance. For computational intensive operations this configuration can significantly decrease the response time, hopefully less than the API Gateway 30s timeout.
Here is the situation I suffer.
Lambda has 1000 concurrency limit. (There is no reserved number)
100-200 Clients access to Lambda at a time.
Lambda still doesn't reach throttling in the figures(100-200).
However, Lambda returns a lot of 502 errors.
And I assume
The first time, any Lambda isn't up.
When Lambda receives a lot of requests, it starts scaling.
However, because of Lambda's cold start time, It takes time to execute enough concurrency to handle all requests and as the result, it returns the error(even if it is not reached to maximum concurrency execution number[1000])
Is my assumption correct? If so, is it inevitable situation?
I have read on warming Lambda by sending ping requests at regular interval to the Lambda.
However, It looks not to solve the above issue because the ping is sent to only one Lambda making only the Lambda is resued, causing the same issue when a lot of requests is received at a time.
------Edit------
About #M Mo's asking
How are the lambdas being invoked?
By API-getaway it is invoked.
If through api gateway, are you using proxy integration?
Yes, proxy integration.
Do the lambdas call any other resources?
Yes, The Lambda calls S3 resource to get objects.
What are the average response times of the lambdas?
it takes about 1 sec.
My lambda function works less than 1300 MS when I click the test button on the Lambda Page. (Lambda Page: https://eu-central-1.console.aws.amazon.com/lambda/home?region=eu-central-1#/functions/myfunc?tab=graph)
When I send a request to Lambda via API Gateway then I have to wait for 4300 ms.
The HTTP requests, which goes to Lambda via Gateway work 3-4 times slow.
I saw some similar forum posts. However, I couldn't find a solution for this issue.
How can I reduce the latency?
API Gateway is known to introduce a lot of latency.
Lambda is not ideal for synchronous requests + responses as you seem to be using it. It's more suited to asynchronous processes where the latency between invocation and execution are not as critical.
You should probably think about whether your system needs to be synchronous, and if it needs to be synchronous whether lambda is the best answer.
So I defined a fairly simple AWS Lambda. I created an HTTP GET URL for it using AWS API Gateway. I deployed it and tested the URL in the browser and it worked. I then created a desktop app to call the URL, it only takes one query string parameter. I ran the code serially to call the URL 100 times with a different query string input each time, and saw that the lambda executed an average of 500 milliseconds each time.
I then changed my desktop app to issue the requests in parallel. I expected the overall time to take maybe 1 second or so to complete, given that the longest execution time was like 950 milliseconds on average. However, when I did this, it took more than 30 seconds to complete all the requests.
I've done other tests to know the desktop app really is issuing all the URL requests in parallel, so that's not the issue. I just don't understand why it didn't spin up 100 lambdas to service each URL request so that they executed concurrently. It appears that the requests were buffered.
The only difference between each URL is the query string parameter. I am, at this point, considering creating 100 different lambdas, each built with the different value previously passed in the query string, but each with a different URL so I can achieve actual concurrent execution.
Am I missing something?
AWS lambda by default provided concurrent execution upto 75, i. e at a time 75 lambdas can be created.
EDIT: By default, AWS Lambda limits the total concurrent executions across all functions within a given region to 1000.
The previous limit was 75 per Lambdas. I didn't check the latest concurrent limit.
Here is the Documentation
If you need more concurrency you need to raise a case with AWS team.
Concurrent execution count will differ depending on whether or not your Lambda function is processing events from a stream-based event source.
Event sources that aren't stream-based – If you create a Lambda function to process events from event sources that aren't stream-based (for example, Amazon S3 or API Gateway), each published event is a unit of work. Therefore, the number of events (or requests) these event sources publish influences the concurrency.
You can use the following formula to estimate your concurrent Lambda function invocations:
events (or requests) per second * function duration
For example, consider a Lambda function that processes Amazon S3 events. Suppose that the Lambda function takes on average three seconds and Amazon S3 publishes 10 events per second. Then, you will have 30 concurrent executions of your Lambda function.
Request Rate
Request rate refers to the rate at which your Lambda function is invoked. For all services except the stream-based services, the request rate is the rate at which the event sources generate the events. For stream-based services, AWS Lambda calculates the request rate as follow:
request rate = number of concurrent executions / function duration
For example, if there are five active shards on a stream (that is, you have five Lambda functions running in parallel) and your Lambda function takes about two seconds, the request rate is 2.5 requests/second.
Source :- http://docs.aws.amazon.com/lambda/latest/dg/concurrent-executions.html
I've been playing with Lambda recently and am working on creating an API using API Gateway and Lambda. I have a lambda function in place that returns a JSON and an API Gateway endpoint that invokes the function. Everything works well with this simple setup.
I tried loadtesting the API gateway endpoint with the loadtest npm module. While Lambda processes the concurrent requests (albeit with an increase in mean latency over the course of execution), when I send it 40 requests per second or so, it starts throwing errors, only partially completing the requests.
I read in the documentation that by default, Lambda invocation is of type RequestResponse (which is what the API does right now) which is synchronous in nature, and it looks like it is non-blocking. For asynchronous invocation, the invocation type is Event. But lambda discards the return type for async invocations and the API returns nothing.
Is there something I am missing either with the sync, async or concurrency definitions in regards to AWS? Is there a better way to approach this problem? Any insight is helpful. Thank you!
You will have to use Synchronous execution if you want to get a return response from API Gateway. It doesn't make sense to use Async execution in this scenario. I think what you are missing is that while each Lambda execution is blocking, single threaded, there will be multiple instances of your function running in multiple Lambda server environments.
The default number of concurrent Lambda executions is fairly low, for safety reasons. This is to prevent you from accidentally writing a run-away Lambda process that would cost lots of money while you are still learning about Lambda. You need to request an increase in the Lambda concurrent execution limit on your account.