Cannot invoke more than 64 Lambda functions at one time? - amazon-web-services

Question

I cannot invoke more than 64 synchronous lambda functions without getting an Ops Limit: 64 exception, and I have no idea why.
There is no mention of this in the Lambda limits documentation. In fact, it says you can have up to 1000 concurrent executions. The lambda scaling docs additionally state that temporary bursts up to 3k invocations are supported.
So why is my paltry 65 invocations causing Lambda to reject things?
Reproducing:
On my account I have a dead simple Lambda which sits for 5 seconds (to simulate work) and then returns the default json blob.
import json
import time
def lambda_handler(event, context):
# TODO implement
time.sleep(5)
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
Excluding the time.sleep call, this is the exact code generated when you create a new Python Lambda function. So, no weirdness going on here.
Invoking
Now to invoke it, I just submit the invocation task into a thread pool.
(defn invoke-python [_]
(aws/invoke
lambda
{:op :Invoke
:request {:FunctionName "ExampleSlowPythonFunction"
:Payload (json/write-str payload)}})
Here's all the call is doing. Just a straight invoke call to AWS. (The library here is Cognitect's AWS API. But it just defers down to the REST APIs, so shouldn't matter.)
The thread pool is just one of Java's executors. I hand it a size and the tasks, and it executes said tasks in a pool of size n.
(defn call-it-a-bunch
[n tasks]
(let [pool (Executors/newFixedThreadPool n)]
(let [futures (.invokeAll pool tasks)]
(.shutdown pool)
(mapv #(.get %) futures))))
64 invocations: no problem
(def sixty-four-invoke-tasks (map invoke-python (range 64)))
(call-it-a-bunch 64 sixty-four-invoke-tasks)
A-OK. No problemo.
65 invocations: PROBLEM
(def sixty-FIVE-invoke-tasks (map invoke-python (range 65)))
(call-it-a-bunch 65 sixty-FIVE-invoke-tasks
I will get get an Ops limit reached: 64 on that 65th request.
I have no other Lambda's running on my account. I've tried dialing up the reserved instances on the Python Lambda Function to make double sure that the lambda IS available.
However, the Ops limit error remains.
Why can I not invoke my function more than 64 times concurrently despite having a bucket 1000 concurrency available on my account?

Answer 1

I'm unable to reproduce your problem. I'm using regular Java. I have your exact Python Lambda and have have a Runnable:
import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.AWSLambda;
import com.amazonaws.services.lambda.AWSLambdaClientBuilder;
import com.amazonaws.services.lambda.model.InvokeRequest;
import com.amazonaws.services.lambda.model.InvokeResult;
public class LambdaThread implements Runnable {
private String name;
public LambdaThread(String name) {
this.name = name;
}
#Override
public void run() {
System.out.println( "starting thread " + name);
Regions region = Regions.fromName("us-west-2");
AWSLambda client = AWSLambdaClientBuilder.standard()
.withRegion(region).build();
InvokeRequest req = new InvokeRequest()
.withFunctionName("test-load");
InvokeResult result = client.invoke(req);
System.out.println( "result from thread " + name + " is " + new String( result.getPayload().array()) );
}
}
And a runner:
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
public class LambdaRunner {
public static void main(String[] argv) {
long startTime = System.currentTimeMillis();
ExecutorService executor = Executors.newFixedThreadPool(999);
for (int i = 0; i < 999; i++) {
Runnable worker = new LambdaThread("thread " + i);
executor.execute(worker);
}
executor.shutdown();
while (!executor.isTerminated()) {
try { Thread.sleep(1000); } catch ( InterruptedException ie ) { /* ignored */ }
System.out.println( "waiting for thread termination...");
}
System.out.println("Finished all threads in " + (System.currentTimeMillis() - startTime) + "ms");
}
}
To me, the error is inline with some of the wording in Clojure in general. What version of the Cognitect library are you using? I can't see this message in the master branch.
If I run this with say 2000 threads I get:
Exception in thread "pool-1-thread-1990"
com.amazonaws.services.lambda.model.TooManyRequestsException: Rate
Exceeded. (Service: AWSLambda; Status Code: 429; Error Code:
TooManyRequestsException; Request ID:
b7f1426b-419a-4d40-902d-e0ed306ff120)
Nothing related to ops.