I have 8 different AWS Lambda functions that need to share some common data. (like common configuration for database, etc)
There is no in-built technique for sharing data between Lambda functions. Each function runs independently and there is no shared datastore.
You will need to use an external datastore -- that is, something outside of Lambda that can persist the data.
Some options include:
Amazon S3: You could store information in an S3 object, that is retrieved by your Lambda functions.
Amazon DynamoDB: A fully-managed NoSQL database that provides fast performance. Ideal if you are storing and retrieving a blog of data, such as a JSON object. Your Lambda function would access DynamoDB via standard API calls. For extreme performance, you could use DynamoDB Accelerator (DAX).
AWS Systems Manager Parameter Store: Provides secure, hierarchical storage for configuration data management and secrets management.
The above options are fully-managed services, so you don't need to run any extra infrastructure.
There are other options, such as Amazon ElastiCache, but they would require additional services to be running.
Depending on the nature of configuration you can decide on using different storage options.
If the configuration doesn't change often and mostly static you can use the following options,
Amazon Systems Manager Parameters Store
Embed configuration as a part of code
If they are changing often you can consider,
AWS DynamoDB
AWS S3
Note: Depending on your use case you can also consider these configurations as parameters to these Lambda functions.
I will add my 2 cents answer :
use lambda-layer (50mb) for persistent share data over function invocation
use EFS (ec2, within vpc) for persistente share data over any aws service
the lambda temporary memory (default 512mo extand recently to 10 gb) in ephemere only for current evocation.
Related
My AWS Lambda function needs to access data that is updated every hour and is going to be called very often via api. What is the most efficient and least expensive way?
The data that is already updated every hour is configured through Lambda batch, but I don't know where to store this data.
How about putting the latest data in the latest bucket of Amazon S3 every time? Or, even if there is a problem with the hot partition, how about storing it in Amazon DynamoDB because it is simple access? I considered the gateway cache, which is updated every hour, but at a cost. Please advise.
As you have mentioned "least expensive way" I will suggest to use Amazon DynamoDB because 25GB of space is free (always not free tier). Now if your data size is more than 25GB then also you can use DynamoDB over other services like RDS or S3 that comes at a cost.
The simplest option would be to use AWS Systems Manager Parameter Store. It is secured via IAM and is a great way to share parameters between AWS Lambda functions.
If your data is too big to store in Parameter Store, then consider storing it in Amazon S3. It is easily accessible and low-cost.
If there are problems using these services, then you could look at using databases but there is insufficient information in your question make an appropriate recommendation.
I'm building a Lambda function in AWS that need to load reference data from a mysql database. There is no real issue right now as it very limited amount of data. But what is best practice here? Is there away to keep this data within Lambda (or some other similar functionality) so that I don't need to request it for every invocation of the function? I'm using Node js though I don't think that affects this question.
Many thanks,
Marcus
There is no build-in persistent storage for lambda. Any data that you would like to keep reliably (not counting temporary persistence due to lambda execution context) between invocations is to store data outside of lambda itself.
You already store it in MySQL, but other popular choices are:
SSM Parameter Store
S3
EFS
DynamoDB
ElastiCache if you really need fast access to the data.
Since you already get the data from MySQL the only advantage of using SSM or DynamoDB would be that you can use AWS API to access and update them, or inspect/modify in AWS Console. You don't need to bundle any MySQL client with your function nor establish any connections to the database.
Does AWS provides any service for storing all the configs and we can get this config by just making a call to it? Here the config can be version controlled or available with less latency and so on?
Eg. I want to use some configs from the lambda function which I can easily change without changing the lambda function.
You can use AWS Systems Manager Parameter Store. It provides a centralized store to manage configuration data such as database strings, secrets or credentials.
https://aws.amazon.com/systems-manager/features/#Parameter_Store
DynamoDB is typically used for that purpose. The latency for a single GetItem request is typically around 5ms, and you can cache the results client-side to reduce the latency even further and to avoid a read io ever time.
I am using AWS to build an API, and deploy this to multiple stages.
When a call is made to a specific environment, I need to get a stage variable in Lambda and then data is recorded in a DynamoDB table such as "environment-Table".
Is this the best way to work with environments (like development, production etc) using AWS API Gateway, Lambda and DynamoDB?
It difficult to say what the best approach is for your specific situation, given the limited data in your post. Managing multiple environments such as development and production was one of the intended uses of stage and stage variables. I don't see any obvious problems with what your are proposing.
Depending on your use case, you can call a Lambda function to record data in DynamoDB, or you may be able to skip the Lambda function and record the data in DynamoDB directly using the AWS proxy integration type.
Objective: Using iPhone app, I would like the users store objects in DynamoDB and have Fine-Grained Access Control for the objects using IAM with TVM.
The objects will contain only Strings, no images/file storage -- I'm thinking I won't need an S3?
Question: Since there is no server-side application, do I still need an EC2 Instance? What all suite of AWS services will I have to subscribe to in order to accomplish my objective?
You can use either DynamoDB (or S3), and neither of them would require an EC2 instance - there is no dependency.
If it was me, I'd first see if I could get what I wanted down in S3(because you mentioned it as a possibility), and then go to DynamoDB if I couldn't (i.e. I wanted to be able to run agregation queries across my data set). S3 will be cheaper and depending on what your are doing, may even be faster and would allow you to globally distribute the stored data thru CloudFront easily, which if you have a globally diverse user base may be beneficial.