I have an AWS Lambda function, configured with only 128MB of memory, is triggered by SNS (which is itself triggered by S3) and will download the file from S3.
In my function, I have the following:
public class LambdaHandler {
private final AmazonS3Client s3Client = new AmazonS3Client();
public void gdeltHandler(SNSEvent event, Context context) {
System.out.println("Starting");
System.out.println("Found " + eventFiles.size() + " event files");
}
I've commented out and excluded from this post all of the logic because I am getting an OutOfMemoryError which I have isolated to the creation of the AmazonS3Client object. When I take that object out, I don't get the error. The exact above code results in the OutOfMemoryError.
I assigned 128MB of memory to the function, is that really not enough to simply grab the credentials and instantiate the AmazonS3Client object?
I've tried giving the AmazonS3Client constructor
new EnvironmentVariableCredentialsProvider()
as well as
new InstanceProfileCredentialsProvider()
with similar results.
Does the creation of the AmazonS3Client object simply require more memory?
Below is the stack trace:
Metaspace: java.lang.OutOfMemoryError java.lang.OutOfMemoryError:
Metaspace at
com.fasterxml.jackson.databind.deser.BeanDeserializerBuilder.build(BeanDeserializerBuilder.java:347)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:242)
at
com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143)
at
com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:409)
at
com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:358)
at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:265)
at
com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:245)
at
com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:143)
at
com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:439)
at
com.fasterxml.jackson.databind.ObjectReader._prefetchRootDeserializer(ObjectReader.java:1588)
at
com.fasterxml.jackson.databind.ObjectReader.(ObjectReader.java:185)
at
com.fasterxml.jackson.databind.ObjectMapper._newReader(ObjectMapper.java:558)
at
com.fasterxml.jackson.databind.ObjectMapper.reader(ObjectMapper.java:3108)
When I try providing the InstanceProfileCredentialsProvider or EnvironmentVariableCredentialsProvider, I get the following stack trace:
Exception in thread "main" java.lang.Error:
java.lang.OutOfMemoryError: Metaspace at
lambdainternal.AWSLambda.(AWSLambda.java:62) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:348) at
lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:94) Caused by:
java.lang.OutOfMemoryError: Metaspace at
java.lang.ClassLoader.defineClass1(Native Method) at
java.lang.ClassLoader.defineClass(ClassLoader.java:763) at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at
java.net.URLClassLoader.access$100(URLClassLoader.java:73) at
java.net.URLClassLoader$1.run(URLClassLoader.java:368) at
java.net.URLClassLoader$1.run(URLClassLoader.java:362) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:361) at
java.lang.ClassLoader.loadClass(ClassLoader.java:424) at
java.lang.ClassLoader.loadClass(ClassLoader.java:357) at
lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.makeRequestHandler(EventHandlerLoader.java:421)
at
lambdainternal.EventHandlerLoader.getTwoLengthHandler(EventHandlerLoader.java:777)
at
lambdainternal.EventHandlerLoader.getHandlerFromOverload(EventHandlerLoader.java:802)
at
lambdainternal.EventHandlerLoader.loadEventPojoHandler(EventHandlerLoader.java:888)
at
lambdainternal.EventHandlerLoader.loadEventHandler(EventHandlerLoader.java:740)
at
lambdainternal.AWSLambda.findUserMethodsImmediate(AWSLambda.java:126)
at lambdainternal.AWSLambda.findUserMethods(AWSLambda.java:71) at
lambdainternal.AWSLambda.startRuntime(AWSLambda.java:219) at
lambdainternal.AWSLambda.(AWSLambda.java:60) ... 3 more START
RequestId: 58837136-483e-11e6-9ed3-39246839616a Version: $LATEST END
RequestId: 58837136-483e-11e6-9ed3-39246839616a REPORT RequestId:
58837136-483e-11e6-9ed3-39246839616a Duration: 15002.92 ms Billed
Duration: 15000 ms Memory Size: 128 MB Max Memory Used: 50 MB
2016-07-12T14:40:28.048Z 58837136-483e-11e6-9ed3-39246839616a Task
timed out after 15.00 seconds
EDIT 1 If I increase the memory allocated to the function to even 192MB, it works just fine, though strangely enough, reports only using 59MB of memory in the cloudwatch logs. Am I simply losing the rest of the memory?
I have been observing this when using AWS Java SDK within the Lambda function.
It would seem when creating any of the AWS clients (Sync or Async) you may get out of Metaspace.
I believe this is due to things that the Amazon Client is performing upon instantiation, including AmazonHttpClient creation as well as dynamic loading of request handler chains (part of AmazonEc2Client#init() private method).
It is possible that the reported memory usage is for Heap itself, but may not include Metaspace. There are a few threads on AWS Forums but no responses from AWS on the matter.
One way to reduce cold start is setting the memory to 1536 mb and the timeout to 15 min. This will give dedicated host to run only your lambda instead of running your lambda on shared host + when a new instance has to be started, it will copy the code from cache on the host rather than copying from S3.
This though will be more expensive and if you don't want to do this, continue reading below.
How can I reduce my cold start times?
Follow the Lambda best practices
https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html
By choosing a larger memory setting for your function
Think of the memory as a "power" setting because it also dictates how much CPU your function will receive.
By reducing the size of your function ZIP
This likely means reducing the number of dependencies you include in your function ZIP.
Java JARs can be further reduced in size using ProGuard
[Java Only] Use the bytestream interface instead of the POJO interface.
The JSON serialization libraries that Lambda uses internally can take some time to start. It will take dev work on your end, but you may be able to improve on this by using the byte stream interface along with a lightweight JSON library. Here are some links that may help:
http://docs.aws.amazon.com/lambda/latest/dg/java-handler-io-type-stream.html
https://github.com/FasterXML/jackson-jr
[Java Only] Don't use Java 8 feature that replaces anonymous classes (lambdas, method references, constructor references, etc.)
We've noticed internally that Java 8 Lambda-related bytecode appears to result in sub-optimal startup performance. If your code is using any Java 8 feature that replaces anonymous classes (lambdas, method references, constructor references, etc.) you may get better startup time by moving back to anonymous classes.
By using a different runtime
Different runtimes have different cold start times, and different runtime performance. While NodeJS might be better for heavy IO work, Go might be better for code that does a lot of concurrent work. Customers have done some basic benchmarks to compare language performance on Lambda, and here is a more generic comparison of different programming languages performance. There is no one-size-fits-all answer, use what makes sense for your requirements.
basic benchmarks:https://read.acloud.guru/comparing-aws-lambda-performance-of-node-js-python-java-c-and-go-29c1163c2581
generic comparison : https://benchmarksgame-team.pages.debian.net/benchmarksgame/which-programs-are-fast.html
Try to increase the memory allocated to lambda from 128 to 256 MB
I use a tactic that helps for Java-based lambdas. Any class resources that only need a single (reusable) instance can be declared as static class members, and initialized inside a static initializer block. When the lambda creates a new instance of the class to handle an execution, those expensive resources are already initialized. Here is a simple example:
package com.mydomain.myapp.lambda.sqs;
import com.amazonaws.services.lambda.runtime.events.SQSEvent;
import com.amazonaws.services.sns.AmazonSNS;
import com.amazonaws.services.sns.AmazonSNSClientBuilder;
import com.amazonaws.services.sqs.AmazonSQS;
import com.amazonaws.services.sqs.AmazonSQSClientBuilder;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Objects;
public class MyLambdaFunctionHandler {
private static final Logger LOGGER = LoggerFactory.getLogger(MyLambdaFunctionHandler.class);
// These values come from the 'Environment' property for the lambda, defined in template.yaml
private static final String ENV_NAME = System.getenv("ENV_NAME");
// Declare these as static properties so they only need to be created once,
// rather than on each invocation of the lambda handler method that uses them
private static final ObjectMapper OBJECT_MAPPER;
private static final AmazonSNS SNS;
private static final AmazonSQS SQS;
static {
LOGGER.info("static initializer | START");
Objects.requireNonNull(ENV_NAME, "ENV_NAME cannot be null");
OBJECT_MAPPER = new ObjectMapper();
SNS = AmazonSNSClientBuilder.defaultClient();
SQS = AmazonSQSClientBuilder.defaultClient();
LOGGER.info("static initializer | END");
}
public MyLambdaFunctionHandler() {
LOGGER.info("constructor invoked");
}
public void handlerMethod(SQSEvent event) {
LOGGER.info("Received SQSEvent with {} messages", event.getRecords().size());
event.getRecords().forEach(message -> handleOneSQSMessage(message));
}
private void handleOneSQSMessage(SQSEvent.SQSMessage message) {
// your SQS message handling code here...
}
}
The properties I declared as static will stay in memory until the lambda instance is destroyed by AWS.
This isn't how I would normally write Java code. Lambda-based code is treated differently, so I think it is OK to break some traditional patterns here.
Related
I really don't understand why no one seems to have asked this question before, but is there a TIMEOUT environment variable which references the set timeout in the Lambda function in AWS?
It doesn't seem to be on the list of environment variables available, and that doesn't seem to make sense either: https://docs.aws.amazon.com/lambda/latest/dg/configuration-envvars.html
I'm not sure of your programming environment but ever Lambda environment I've seen includes a Context object. That has the ability to check how much time is left in a Lambda run. In Java, for example:
public class ShowTimeout implements RequestStreamHandler {
public void handleRequest(InputStream inputStream, OutputStream outputStream, Context context) throws IOException {
LambdaLogger logger = context.getLogger();
logger.log("there are " + context.getRemainingTimeInMillis() + "ms left to run");
}
}
logs how much time is left. By default this is 3 seconds but, of course, can change depending on how you configure the Lambda.
EDIT
This is not the total time set. But as the very first thing the Lambda does it's pretty close. For a 15 second timeout Lambda I got:
there are 14998ms left to run
and
there are 14999ms left to run
If you've started your Lambda and it looks like the number is too small then you can do something about it. If it's the first thing you do like my simple code then you'll be very close. I'd argue that a simple rounding would be accurate enough.
Am trying to get familiar with Apache beam Kafka IO and getting following error
Exception in thread "main" org.apache.beam.sdk.Pipeline$PipelineExecutionException: org.apache.kafka.common.errors.SerializationException: Size of data received by LongDeserializer is not 8
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:348)
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:318)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:213)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:317)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at com.andrewjones.KafkaConsumerExample.main(KafkaConsumerExample.java:58)
Caused by: org.apache.kafka.common.errors.SerializationException: Size of data received by LongDeserializer is not 8
Following is piece of code which reads messages from a kafka topic. Appreciate if you someone can provide some pointers.
public class KafkaConsumerJsonExample {
static final String TOKENIZER_PATTERN = "[^\p{L}]+";
public static void main(String[] args) {
PipelineOptions options = PipelineOptionsFactory.create();
// Create the Pipeline object with the options we defined above.
Pipeline p = Pipeline.create(options);
p.apply(KafkaIO.<Long, String>read()
.withBootstrapServers("localhost:9092")
.withTopic("myTopic2")
.withKeyDeserializer(LongDeserializer.class)
.withValueDeserializer(StringDeserializer.class)
.updateConsumerProperties(ImmutableMap.of("auto.offset.reset", (Object)"earliest"))
// We're writing to a file, which does not support unbounded data sources. This line makes it bounded to
// the first 5 records.
// In reality, we would likely be writing to a data source that supports unbounded data, such as BigQuery.
.withMaxNumRecords(5)
.withoutMetadata() // PCollection<KV<Long, String>>
)
.apply(Values.<String>create())
.apply(TextIO.write().to("wordcounts"));
System.out.println("running data pipeline");
p.run().waitUntilFinish();
}
}
The issue is caused by using LongDeserializer for keys that seems were serialised by other serialiser than Long and it depends how you produced the records.
So, you either can use a proper deserializer or, if keys don't matter, as a workaround, try to use StringDeserializer for keys as well.
There is a concern about potential problem with reusable variables in aws-lambda.
A user's locale is passed as
Browser cookies => AWS API Gateway => Lambda (NodeJS 6.10)
On the server side localization is implemented with a static variable in a class. Presenting typescript code for clarity but can be done in pure ECMAScript.
Module Language.ts
export default class Language
{
public static Current: LanguageCode = LanguageCode.es;
}
Static Language.Current variable is used across different parts of the application for manual localization and it works perfectly on the client side (react + redux).
Lambda function
import {APIGatewayEvent, Context, Callback} from 'aws-lambda';
import Language from './pathToModule/Language.ts';
export const api = function(event: APIGatewayEvent, context: Context, callback: Callback)
{
Language.Current = event.headers.cookie.locale;
// do the logic here
}
Potential problem
According to AWS documentation NodeJS instances can be reused for different requests. It means that famous concurrent problems have to be considered, e.g.
User 1 calls lambda function. The locale is set to English.
In parallel user 2 calls the same lambda instance. The local is changed to Spanish.
User 1 code continues and reads modified (wrong) locale variable from the shared module Language.
How do you resolve this problem?
For convenience it is good to have only one place for locale change. As I understand the same concern exists for all famous i18n npm packages (i18next, i18n, yahoo i18n, etc).
One of the best practices for Lambda functions is to try and not write code which maintains state.
Here you are initializing the locale based on an initial request and applying it to all future requests, which is inherently flawed even on server based code, forget server less.
To fix this, you will need to initialize the localization library for each request, or at least maintain an in memory lazy map, which you can make use of use the current request's locale to achieve the desired localization.
There are several solutions:
Node JS container is reused only after a function process is finished (callback or error is occurred) (thanks to #idbehold). Thus there is always a unique context per a function call.
Refactor code and pass a locale variable back and force (#Yeshodhan Kulkarni suggestion).
For example, return a function as an intermediate result and use it before calling the result back.
var localizableResult = ...;
var result = localizableResult.Localize(requestedLocale).
If there is a need to use a local stack (kind of a thread context) for other projects there is a npm package node-continuation-local-storage.
Case 1 makes it really simple to use global variables for current locale.
I'm trying to upload large files to a web application using the Spark framework, but I'm running into out of memory errors. It appears that spark is caching the request body in memory. I'd like either to cache file uploads on disk, or read the request as a stream.
I've tried using the streaming support of Apache Commons FileUpload, but it appears that calling request.raw().getInputStream() causes Spark to read the entire body into memory and return an InputStream view of that chunk of memory, as done by this code. Based on the comment in the file, this is so that getInputStream can be called multiple times. Is there any way to change this behavior?
I recently had the same problem and I figured out that you could bypass the caching. I do so with the following function:
public ServletInputStream getInputStream(Request request) throws IOException {
final HttpServletRequest raw = request.raw();
if (raw instanceof ServletRequestWrapper) {
return ((ServletRequestWrapper) raw).getRequest().getInputStream();
}
return raw.getInputStream();
}
This has been tested with Spark 2.4.
I'm not familiar with the inner workings of Spark so one potentiall, minor downside with this function is that you don't know if you get the cached InputStream or not, the cached version is reusable, the non-cached is not.
To get around this downside I suppose you could implement a function similar to the following:
public boolean hasCachedInputStream(Request request) {
return !(raw instanceof ServletRequestWrapper);
}
Short answer is not that I can see.
SparkServerFactory builds the JettyHandler, which has a private static class HttpRequestWrapper, than handles the InputStream into memory.
All that static stuff means no extending available.
I would like to increase the deploy time, in a stack layer that hosts many apps (AWS Opsworks).
Currenlty I get the following error:
Eror
[2014-05-05T22:27:51+00:00] ERROR: Running exception handlers
[2014-05-05T22:27:51+00:00] ERROR: Exception handlers complete
[2014-05-05T22:27:51+00:00] FATAL: Stacktrace dumped to /var/lib/aws/opsworks/cache/chef-stacktrace.out
[2014-05-05T22:27:51+00:00] ERROR: deploy[/srv/www/lakers_test] (opsworks_delayed_job::deploy line 65) had an error: Mixlib::ShellOut::CommandTimeout: Command timed out after 600s:
Thanks in advance.
First of all, as mentioned in this ticket reporting a similar issue, the Opsworks guys recommend trying to speed up the call first (there's always room for optimization).
If that doesn't work, we can go down the rabbit hole: this gets called, which in turn calls Mixlib::ShellOut.new, which happens to have a timeout option that you can pass in the initializer!
Now you can use an Opsworks custom cookbook to overwrite the initial method, and pass the corresponding timeout option. Opsworks merges the contents of its base cookbooks with the contents of your custom cookbook - therefore you only need to add & edit one single file to your custom cookbook: opsworks_commons/libraries/shellout.rb:
module OpsWorks
module ShellOut
extend self
# This would be your new default timeout.
DEFAULT_OPTIONS = { timeout: 900 }
def shellout(command, options = {})
cmd = Mixlib::ShellOut.new(command, DEFAULT_OPTIONS.merge(options))
cmd.run_command
cmd.error!
[cmd.stderr, cmd.stdout].join("\n")
end
end
end
Notice how the only additions are just DEFAULT_OPTIONS and merging these options in the Mixlib::ShellOut.new call.
An improvement to this method would be changing this timeout option via a chef attribute, that you could in turn update via your custom JSON in the Opsworks interface. This means passing the timeout attribute in the initial Opsworks::ShellOut.shellout call - not in the method definition. But this depends on how the shellout method actually gets called...