I have a lambda handler returning response and I want to save the response as a JSON file to S3.
I went through some pages describing how to save information from AWS lambda to S3 using boto3 directly called from the lambda function. However, I'd like the lambda function to concentrate on calculating and making a response then let another lambda or module for creating such output on S3.
Is there any way offered by AWS? I guess the StepFunctions is a way to go but I'd like to know if there is another better way.
If you don't want to do it from the same lambda wit boto3 you have quite a few options. As you mentioned on your own, one option could be another lambda, for which you'll need a trigger, which on it's own could be:
Calling the second lambda from the first lambda directly with boto3
Sending the response to SNS which will trigger the second lambda which will then parse and save it to S3.
Sending the response as a message to SQS which will be then processed by the second lambda.
Sending it as an event to AWS EventBridge which will trigger the second lambda similar to the above.
As suggested by yourself and other answers/comments - Step Functions.
In general, it depends on your use case I'm not quite sure what your payload is, but the best option could actually be to push it directly from your first lambda either way.
Related
Required procedure:
Someone does an upload to an S3 bucket.
This triggers a Lambda function that does some processing on the uploaded file(s).
Processed objects are now copied into a "processed" folder within the same bucket.
The copy-operation in Step 3 should never re-trigger the initial Lambda function itself.
I know that the general guidance is to use a different bucket for storing the processed objects in a situation like this (but this is not possible in this case).
So my approach was to set up the S3 trigger to only listen to PUT/POST-Method and excluded the COPY-Method. The lambda function itself uses python-boto (S3_CLIENT.copy_object(..)). The approach seems to work (the lambda function seems to not be retriggered by the copy operation)
However I wanted to ask if this approach is really reliable - is it?
You can filter which events trigger the S3 notification.
There are 2 ways to trigger lambda from S3 event in general: bucket notifications and EventBridge.
Notifications: https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-filtering.html
EB: https://aws.amazon.com/blogs/aws/new-use-amazon-s3-event-notifications-with-amazon-eventbridge/
In your case, a quick search doesn't show me that you can setup a "negative" rule, so "everything which doesn't have processed prefix". But you can rework your bucket structure a bit and dump unprocessed items into unprocessed and setup filter based on that prefix only.
When setting up an S3 trigger for lambda function, there is the possibility, to define which kind of overarching S3-event should be listened to:
I have an AWS Lambda which is triggered by an S3 push event. The lambda will call an API which will trigger a long-running process. I recognize that I can configure S3 to invoke the lambda function asynchronously, and so S3 will not wait for a response, but I am interested to find out if I can configure lambda to call my API asynchronously as well. I don't want lambda waiting for several minutes while the process completes. Can anyone point me to some documentation which outlines this process? Thanks in advance.
I don't think Lambda can do this nor would I recommend a workaround. There is an article on SenseDeep which talks about this and specifically points out that points out "So what happens if we simply do not call "await" and thus not wait on the response from our HTTP request?" - "Strange things happen" - which is to say that invoking an async call in a lambda then returning immediately has unpredictable results.
Why do you need the Lambda to return quickly? If there is a valid reason (for example, you want a push notification that something in S3 has changed right away), then I'd recommend a different pattern.
Updating S3 triggers a lambda
That lambda writes to an SNS topic and does whatever fast action you want
There is a subscriber to the SNS topic that does the long running action you want
I would like to offload some functionality from my Lambda#Edge to speed up response time.
This would mean triggering another Lambda Function inside my Lambda#Edge.
Lambda#Edge distributes the application across all regions, so when a request is made it would execute the application in the region closest to the requester.
My current solution is to create an SNS with the same topic name on all regions, have an SQS in us-east-1 listen to all these SNS Topics, and the Lambda function to listen to the SQS.
However, creating an SNS on every region is quite a hassle to maintain.
Any other suggestions on how I can trigger another Lambda function inside my Lambda#Edge?
Thanks!
Within lambda you can simply make the call to another lambda. I don't know which language you are using, but here is an example in Python and the boto3 library with a sample payload of information you may want to pass on to the lambda being invoked (I used region and detail-type as example info to pass along):
payload = {'region': <the region>, 'detail-type': 'some other detail you care about'}
lambda_client = boto3.client('lambda', account_id=<your account ID>)
lambda_client.invoke(FunctionName=<ARN of the function you want to invoke>, InvocationType='Event', Payload=json.dumps(payload))
Similar options are available in other languages. More details for this call in Python are at https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/lambda.html#Lambda.Client.invoke
I have been researching AWS Documentation on how to invoke lambda functions, and I've come across different ways to do that. Mainly, Lambda invocation is done by calling Invoke() function which can be used to invoke lambda functions synchronously or asynchronously.
Currently I am invoking my Lambda functions via HTTP Request (as REST API), but, HTTP Request times out after 30 seconds, while asynchronous calls as far as I know times out after 15min.
What are the advantages, besides time that I have already mentioned, of asynchronous lambda invocation compared to invoking lambda with HTTP Request. Also, what are best (recommended) ways to invoke lambdas in production? On AWS docs (SDK for Go - https://docs.aws.amazon.com/sdk-for-go/api/service/lambda/#InvokeAsyncInput) I see that InvokeAsyncInput and InvokeAsyncOutput have been depricated. So I am wondering how async implementation would actually look like.
Lambda really is about event-driven-computing. This means Lambda always gets triggered in response to an event. This event can originate from a wide range of AWS Services as well as the AWS CLI and SDK.
All of these events invoke the Lambda function and pass some kind of information in the form of an event and context object. How this event looks like depends on the service that triggered lambda. You can find more information about the context in this documentation.
There is no real "best" way to invoke Lambda - this mostly depends on your use case - if you're building a webservice, let API Gateway invoke Lambda for you. If you want to process new files on S3 - let S3 trigger Lambda. If you're just testing the Lambda function you can invoke it via the CLI. If you have custom software that needs to trigger a Lambda function you can use the SDK. If you want to run Lambda on a schedule, configure CloudWatch events...
Please provide more information about your use case if you require a more detailed evaluation of the available options - right now this is very broad.
Quick question: Is it possible to trigger the execution of a Step Function after an SQS message was sent?, if so, how would you specify it into the cloudformation yaml file?
Thanks in advance.
The first think to consider is this: do you really need to use SQS to start a Step Functions state machine? Can you use API gateway instead? Or could you write your messages to a S3 bucket and use the CloudWatch events to start a state machine?
If you must use SQS, then you will need to have a lambda function to act as a proxy. You will need to set up the queue as a lambda trigger, and you will need to write a lambda that can parse the SQS message and make the appropriate call to the Step Functions StartExecution API.
I’m on mobile, so I can’t type up the yaml right now, but if you need it, I can try to update with it later. For now, here is detailed walkthrough of how to invoke a Step Functions state machine from Lambda (including example yaml), and here is walkthrough of how to use CloudFormation to set up SQS to trigger a Lambda.
EventBridge Pipes (launched at re:Invent 2022) allows you to trigger Step Functions State Machines without need for a Lambda function.
https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-pipes.html
You can find an example here:
https://github.com/aws-samples/aws-stepfunctions-examples/blob/main/sam/demo-trigger-stepfunctions-from-sqs/template.yaml