I am trying to figure out what is the best way to read a JSON file from a specific endpoint and then save/post such object to AWS S3. I have created a mocked endpoint with a mocked response via https://www.mockable.io/ and I would like to know what is the best way to 'POST' it to a S3 bucket. New JSON files will be available weekly and I was thinking that perhaps a way to go about would be to use Lambda AWS and the API Gateway. Is this a viable way? I also would like to explore the possibility of enable an event trigger way to pull data or a scheduler. What would you recommend? I know that AWS SQS is an option, but how do you send the fetched JSON files to the queue?
Thank you, any resource or suggestion is more than welcome. I am looking for potential approaches.
There are quite a lot of different ways you could approach this, but if I understand correctly you want to retrieve a JSON response once a week from a fixed endpoint (which you have set up?), and then write that JSON to a file, or sequence of files, you store on S3.
If that is correct, then all you really need is Cloudwatch Events (to set up a weekly scheduled recurring event in cron format) which triggers a lambda function that makes the request and then writes it to S3. You can also use the same lambda function (or write another which is triggered by the same CloudWatch Event) to post a message to SQS with the JSON.
Depending on what language you are most comfortable writing it in, you can use the SDK to do all the things you want to do. Personally I like the python library boto3, and combined with a little file IO to get the JSON to a text file of some kind, and the requests library to make the actual HTTP request to your endpoint, you should be able to do all you need. Helpful functions in boto3 will be sending a SQS message and writing to S3.
I'm not sure why you'd necessarily need API Gateway to do anything here, unless instead of triggering the lambda via a scheduled event, you wanted to do it by making a separate HTTP request, but then you may as well just make the request to your original API!
Please consider using Lambda with NodeJS code to do GET from endpoint for invoking the lambda function use cloudwatch event
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
Related
I would normally handle the task of sending an email after a new DynamoDB entry with Lambda and SES but I'm required to not use Lambda for it.
There's a 'Contact us' section in the website and the email needs to be sent every time a new entry is made. We use API Gateway to post the data to DyanmoDB
Is there a way to carry this out without Lambda?
It's not possible without writing code. Furthermore you may probably want to tailor each email to make it more personal to the user, thus Lambda is a must.
You could design something using EventBridge Pipes which can trigger when a new user is added and can have SNS as a destination which could trigger an email. But that email may not be customizable nor may it send to people not subscribed to the topic
DynamoDB triggers Lambda functions by feeding the records of a DynamoDB Stream to the Lambda function. That is by far the easiest way to process updates to a DynamoDB table, but you can also write other code that processes a DynamoDB outside of Lambda: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html
If you really want to do it without lambda you can use the following pattern and glue some AWS services together.
You can consume the dynamodb stream with eventbridge pipes and send it to a stepfunction and there you use the sdk integration to call SES. Instead of stepfunction you can directly use SNS for simpler setups.
You will not have to use lambda for this setup. Further you can transform your message either in the pipe or in the stepfunction.
Be aware that eventbirdge pipes has currently no AWS CDK L2 construct which might make it harder to configure if you use CDK.
I read few articles but either I couldn't fully understand the solution or I felt they did not answer my problem at hand. So I am asking here again.
I need to make HTTP requests POST, PUT and GET calls to an external service.
I would like to batch up my POST calls in AWS SQS and execute them.
Currently, the POST calls via a lambda function is working fine. But I am not sure how to leverage AWS SQS for these callouts and also how to store the API responses and/or webhook responses.
I have created a lambda function to convert a file on S3 and put the new converted file back to S3, the lambda function is triggered by rest api on api gateway. What I want to do is to reply the client by that same converted file when accessing the same endpoint. The option that I have in my mind is to return the file from lambda, which may not be a time optimized solution. My question is there another possible solution ? like calling the lambda function and forward the request to the created object in S3 ? I thought of forwarding one api resource to another one which will read from S3 directly, is it possible somehow ?
You probably want to return a pre-signed S3 url for the file you've just created, which the client can then follow (at most once, and for a limited time), to download that file directly from S3. The general documentation is here, but there will probably be a helper for it in an SDK for whatever language you're using.
Remember that API Gateway has a maximum timeout of 29 seconds. This is probably fine if you're just moving small amounts of data around in S3, but it's worth being aware of!
We have a requirement to write custom logs for the application to capture the things like who did what and when.
To do that we have created a Lambda to insert the logs in DynamoDb database. We need this Lambda to be called from a common place every time we call an API from frontend of the application instead of invoking it in each and every individual lambdas.
We tried invoking this in the API Gateway Authorizer but it doesn't work because our gateway authorizer is of type 'Token'. So, it does not accept any other parameters than access token. We cannot change the type of custom authorizer to type 'Request' because we need access token to be present for authorizing user in Cognito.
Question:
Is there any place where we can invoke this Logs Lambda so that it executes when each API is called?
Your last paragraph makes no sense but typically the best way to do this is streaming, as this minimises the amount of Lambda invocations you need to make.
You can stream API Access logs which contain things like the path, current time, principal to a cloudwatch log streams, or a lambda.
In this lambda you can do your custom logging logic there. If you have other sources which will have different types of events you may need to use Kinesis directly for streaming.
try using a different event trigger. If your lambda can get triggered by a queue or cloudfront you won't have authorization problems. however your application has to assume a suitable role to use some of these. If you're using Java, you can intercept your request in many ways and make the lambda call via SDK before processing the API. Need more details to provide a holistic solution.
I am someone who is totally new to REST APIs, pardon the newbie-ish mistakes.
My requirement is:
The source Database people wants to send JSON data on an hourly basis to an API endpoint which I publish. I am not sure of what all do I need to build to make sure it happens seamlessly. My target is to receive the data and create CSV files and save in it AWS S3 for further downstream processing.
My plan is, creating an AWS API Gateway endpoint which will accept POST requests and whenever anyone sends data through POST, the API Gateway will trigger AWS Lambda Function which will run Python to parse the JSON data to CSV and store in AWS S3. Is this thought valid? What all am I missing out? Are there best practices which needs to be implemented?
This architecture seems to be what you wanna do.
You wanna make sure that your API is secured with a key or via Cognito (more complex) and that your Lambda have the IAM permissions needed in order to access your bucket.
This post will help you understand the Lambda blueprint that is triggered when an object is upload to s3. Just change the Lambda trigger and a little bit the Python code and you're done.
Yes,this is a simple, typical serverless stack and it works perfectly fine.
Additionally, you may also focus on the authentication on the API Gateway end point to make it secure.