I have an AWS Lambda function that is fronted by an API gateway for access.
I need to store the last time this was executed so I can retrieve data from an external service since the last execution.
I had planned to use DynamoDB for this purpose.
Is this the simplest option for this scenario?
DynamoDB is a really good option for that. DynamoDB and AWS Lambda work really well together. I definitely recommend DynamoDB for this scenario.
With DynamoDB, you can create database tables that can store and retrieve any amount of data and serve any level of request traffic.For this case DynamoDB is the best option.
Related
I am looking to trigger code every 1 Hour in AWS.
The code should: Parse through a list of zip codes, fetch data for each of the zip codes, store that data somewhere in AWS.
Is there a specific AWS service would I use for parsing through the list of zip codes and call the api for each zip code? Would this be Lambda?
How could I schedule this service to run every X hours? Do I have to use another AWS Service to call my Lambda function (assuming that's the right answer to #1)?
Which AWS service could I use to store this data?
I tried looking up different approaches and services in AWS. I found I could write serverless code in Lambda which made me think it would be the answer to my first question. Then I tried to look into how that could be ran every x time, but that's where I was struggling to know if I could still use Lambda for that. Then knowing where my options were to store the data. I saw that Glue may be an option, but wasn't sure.
Yes, you can use Lambda to run your code (as long as the total run time is less than 15 minutes).
You can use Amazon EventBridge Scheduler to trigger the Lambda every 1 hour.
Which AWS service could I use to store this data?
That depends on the format of the data and how you will subsequently use it. Some options are
Amazon DynamoDB for key-value, noSQL data
Amazon Aurora for relational data
Amazon S3 for object storage
If you choose S3, you can still do SQL-like queries on the data using Amazon Athena
My AWS Lambda function needs to access data that is updated every hour and is going to be called very often via api. What is the most efficient and least expensive way?
The data that is already updated every hour is configured through Lambda batch, but I don't know where to store this data.
How about putting the latest data in the latest bucket of Amazon S3 every time? Or, even if there is a problem with the hot partition, how about storing it in Amazon DynamoDB because it is simple access? I considered the gateway cache, which is updated every hour, but at a cost. Please advise.
As you have mentioned "least expensive way" I will suggest to use Amazon DynamoDB because 25GB of space is free (always not free tier). Now if your data size is more than 25GB then also you can use DynamoDB over other services like RDS or S3 that comes at a cost.
The simplest option would be to use AWS Systems Manager Parameter Store. It is secured via IAM and is a great way to share parameters between AWS Lambda functions.
If your data is too big to store in Parameter Store, then consider storing it in Amazon S3. It is easily accessible and low-cost.
If there are problems using these services, then you could look at using databases but there is insufficient information in your question make an appropriate recommendation.
I'm building a Lambda function in AWS that need to load reference data from a mysql database. There is no real issue right now as it very limited amount of data. But what is best practice here? Is there away to keep this data within Lambda (or some other similar functionality) so that I don't need to request it for every invocation of the function? I'm using Node js though I don't think that affects this question.
Many thanks,
Marcus
There is no build-in persistent storage for lambda. Any data that you would like to keep reliably (not counting temporary persistence due to lambda execution context) between invocations is to store data outside of lambda itself.
You already store it in MySQL, but other popular choices are:
SSM Parameter Store
S3
EFS
DynamoDB
ElastiCache if you really need fast access to the data.
Since you already get the data from MySQL the only advantage of using SSM or DynamoDB would be that you can use AWS API to access and update them, or inspect/modify in AWS Console. You don't need to bundle any MySQL client with your function nor establish any connections to the database.
I have a project where I am building a simple single page app, that needs to pull data from an api only once a day. I have a backend that I am thinking of building with golang, where I need to do 2 things:
1) Have a scheduled job that would once a day update the DB with the new data.
2) Serve that data to the frontend. Since the data would only be updated once a day, I would like to cache it after each update.
Since, the number of options that AWS is offering is a bit overwhelming, I am wondering what would be the ideal solution for this scenario. Should I use lambda that connects to DB and updates it with a scheduled job? Should I create then a separate REST API lambda where I would pull that data from the DB and call it from the frontend?
I would really appreciate suggestions for this problem.
Her is my suggestion;
Create a lambda function
it will fetch required information from database
You may use S3 or DynamoDB to save your content. Both of the solutions may be free please check for free tier offers depending on your usage
it will save the fetched content to S3 or DynamoDB (you may check Dax for DynamoDB caching)
Create an Api gateway and integrate it to your lambda (Elastic LoadBalancer is another choice)
Create a Schedule Expressions on CloudWatch to trigger lambda daily
Make a request from your front end to Api Gateway or ELB.
You may use Route 53 for domain naming.
Your lambda should have two separate functions, one is to respond schedule expression, the other one is to serve your content via communicating with S3/DynamoDB.
Edit:
Here is the architecture
Edit:
If the content is going to be static, you may configure a S3 bucket for static site serving and your daily lambda may write it in there when it is triggered. Then you no longer need api gateway and DynamoDB.
here is the documentation for s3 static content
I am using AWS Dynamo DB and Elastic Search. I am looking for some way to keep Dynamo DB data in sync with Elastic Search if any of them fails.
Currently I use lambda to push my record into Elastic Search. I know there is plugin - Logstash available but I can't use that as it will require a lot of changes.
Also, I won't prefer scanning the DynamoDB table, as it is too expensive. Is there any other way I could achieve this?
You can make use of SQS. Move failed records to SQS and later you can schedule a lambda to read records from SQS and send the records to ElasticSearch.
If you don't want to go with the plugin solution, you can continue with a lambda but triggered by DynamoDb Streams. In this way you shouldn't have to scan the table since the stream will have the added item and you can reuse the part of sending it to ES.
Take a look at DynamoDB Streams and Lambda triggers.