I'm looking to send a message to an SQS queue from a Lambda function written in Go, which in the AWS console I have configured so that the SQS queue is connected as a Destination.
The code sample I've seen (here) suggests I need to load a config, create a client and get a queueURL before issuing the message via a call to SendMsg. This approach works fine when running the code locally, but seems unnecessary within the Lambda given I'm connecting the SQS queue in the console. Do I need to tell my Lambda to find the queueURL when I've connected to it as a destination already?
As an aside, I tried this approach but was receiving the following error when running the code from within the Lambda. Invoking the code outside of a Lambda locally would find the queueURL without issue.
{
"errorMessage": "operation error SQS: GetQueueUrl, https response error StatusCode: 400, RequestID: 2f725471-ec4e-5380-99ed-0f708c1fcfa4, AWS.SimpleQueueService.NonExistentQueue: ",
"errorType": "OperationError"
}
This led me to look at the flip side, where I have a Lambda which is consuming from the queue and thereby has the SQS queue configured as a trigger. I simply need to iterate over the sqsEvent.Records without having to specify config, client or queueURL as I presume this is handled in the console behind the scenes, i.e.
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
)
func handler(ctx context.Context, sqsEvent events.SQSEvent) error {
for _, message := range sqsEvent.Records {
fmt.Printf("The message %s for event source %s = %s \n", message.MessageId, message.EventSource, message.Body)
}
return nil
}
func main() {
lambda.Start(handler)
}
Is there a way of sending a message to a queue from within the Lambda without having to find the queueURL first?
I have configured so that the SQS queue is connected as a Destination.
Sending a message doesn't need you to define a destination per se - destinations are for passing asynchronous Lambda function invocation results. Basically - if you don't explicitly need them (and you know when you do), they won't be what you need.
Give your Lambda permission to send messages to the queue, & use SendMessage using the AWS SDK to send messages. No destinations needed.
AWS.SimpleQueueService.NonExistentQueue
The error message suggests it's not finding the queue, maybe double-check if you're looking in the right region.
I simply need to iterate over the sqsEvent.Records without having to specify config, client or queueURL as I presume this is handled in the console behind the scenes, i.e.
Yes, this is abstracted away - ultimately, your Lambda function is invoked by events and setting Amazon SQS as a trigger allows it to pass events to your function.
Is there a way of sending a message to a queue from within the Lambda without having to find the queueURL first?
Let's rephrase it as - is there a way of sending a letter to someone without having to find their address? Of course not, as we need to know where the letter needs to go.
The same concept applies to SQS queues.
Related
I'm working on a solution where I have a SQS queue with Lambda trigger. My understanding is Lambda will receive messages in batches to be processed, and once Lambda function is successful, the messages in the SQS queue is automatically deleted. However, how do I only allow some of those messages to be deleted?
Let's assume this use case:
Lambda function receives a batch with 10 messages, and only 7 messages are valid and can be processed, and the other 3 messages needs to be reprocessed at later point.
My initial thought was I could update the visibility timeout via boto3.sqs.change_visibility_timeout for each of the 3 messages to have it reprocessed after the timeout, however, since overall lambda function execution is successful, all 10 messages are deleted from SQS queue.
Any suggestions?
Yes, by default, the Lambda function deletes all the messages upon success. You would need to handle this in your code, but not by changing the visibility timeout of the messages.
Add DLQ (dead-letter queue) that will actually handle the failed messages (messages go to DLQ after a certain number of failed attempts to be processed, depending on how you set it up)
You have few options here:
You can handle each item yourself, and delete messages that are processed successfully. In case of a message that's not successful, you can throw an error and it won't be deleted automatically by the lambda function
If you use JavaScript you can try with Middy
If you use Python, you can use Lambda Powertools Python
For AWS Lambdas with an SQS trigger, by default, when your function encounters an error processing one or more messages in a given batch, the entire batch is marked as a failure. All of the messages in the batch are made visible again in the queue. Depending on your redrive policy, you can end up repeatedly processing successful messages along with the failures.
Rather than change the visibility timeout, the simplest way to specify which messages should be retried later and which can safely be deleted from the queue is to change the function response type to ReportBatchItemFailures. This allows you to return a list of failed message ids, indicating that only those messages in the batch should be made visible again in the queue.
Here's what the reporting syntax looks like for a handler function in Node.js:
exports.handler = async (event) => {
// Process the event
const batchItemFailureResponse = {
batchItemFailures: [
{
itemIdentifier: "idFailedMessage1"
},
{
itemIdentifier: "idFailedMessage2"
},
{
itemIdentifier: "idFailedMessage3"
}
]
};
return batchItemFailureResponse;
};
There is more information to be found in the official documentation.
This response type is configured when setting a queue as an event source for the Lambda. If you're configuring from the console, navigate to the Lambda function page, select the Configuration tab, and then choose Triggers. Then choose Add Trigger and choose the SQS trigger type. In addition to providing the standard parameters, be sure to check the box under Report batch item failures after expanding Additional Settings. It should look something like this:
Add trigger with batch failure reporting
This parameter must be set when first creating the trigger.
This response type can also be defined if you use CloudFormation templates to provision your resources. See the AWS documentation for more information. Note that if you use AWS SAM event source mappings, the documentation suggests that adding FunctionResponseTypes to the YAML with ReportBatchItemFailures in the type list isn't supported. That is incorrect, the documentation is simply outdated. There is an open issue around addressing this oversight.
Finally, in addition to reporting batch item failures, you should provision a target DLQ (dead-letter queue) and determine a reasonable maximum receive count so that action can be taken on messages that fail repeatedly.
I have a single lambda function which responds to two different SQS queues invoking it (the second one is a back-off queue) and so I want to dynamically determine which queue the message came from, so that I can remove it once it has been processed.
I am able to create the EventSourceMapping to trigger the lambda from both queues, but I cannot see how I cleanly retrieve the QueueURL I need in order to perform a DeleteMessage call. I am using the AWS Golang SDK.
There is the EventSourceArn in the body of the event messages and it seems I could use GetQueueURL but that would be an additional API call.
The only way to avoid that (I can think of) is to pass both queue URLs in as environment variables and use the queue name from EventSourceArn to find it.
Is there a better way that this? Is there something hiding in the context that isn't documented?
If you are using lambda, SQS and an event-source-mapping, then you don't have to manually delete objects from sqs: When your function successfully processes a batch, Lambda deletes its messages from the queue
Just return a success code, and the message will automatically be deleted from the relevant SQS queue
You can rebuild the queue URL from its ARN which you have in the event as EventSourceARN.
package main
import (
"context"
"fmt"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/aws/arn"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/sqs"
)
var client *sqs.SQS
func main() {
client = sqs.New(session.Must(session.NewSessionWithOptions(session.Options{
SharedConfigState: session.SharedConfigEnable,
})))
lambda.Start(Handle)
}
func Handle(ctx context.Context, event events.SQSEvent) {
for _, record := range event.Records {
// Process message
fmt.Printf(record.Body)
// Rebuild Queue URL from ARN
queueArn, _ := arn.Parse(record.EventSourceARN)
queueUrl := fmt.Sprintf("https://sqs.%v.amazonaws.com/%v/%v", queueArn.Region, queueArn.AccountID, queueArn.Resource)
// Delete message from queue
client.DeleteMessage(&sqs.DeleteMessageInput{
QueueUrl: &queueUrl,
ReceiptHandle: &record.ReceiptHandle,
})
}
}
If you are using a non-standard AWS partition (e.g. China) you will also need to change the root domain of the URL to match.
I'm looking at the the AWS SQS documentation here: https://docs.aws.amazon.com/sdk-for-net/v3/developer-guide/ReceiveMessage.html#receive-sqs-message
My understanding is that we need to delete the message using AmazonSQSClient.DeleteMessage() once we're done processing it, but is this necessary when we're working with an SQS triggered Lambda?
I'm testing with a Lambda function that's triggered by an SQSEvent, and unless I'm mistaken, it appears that if the Lambda function runs to completion without throwing any errors, the message does NOT return to the SQS queue. If this is true, the I would rather avoid making that unnecessary call to AmazonSQSClient.DeleteMessage().
Here is a similar question from 2019 with the top answer saying that the SDK does not delete messages automatically and that they need to be explicitly deleted within the code. I'm wondering if anything has changed since then.
Thoughts?
The key here is that you are using the AWS Lambda integration with SQS. In that instance AWS Lambda handles retrieving the messages from the queue (making them available via the event object), and automatically deletes the message from the queue for you if the Lambda function returns a success status. It will not delete the message from the queue if the Lambda function throws an error.
When using AWS Lambda integration with SQS you should not be using the AWS SDK to interact with the SQS queue at all.
Update:
Lambda now supports partial batch failure for SQS whereby the Lambda function can return a list of failed messages and only those will become visible again.
Amazon SQS doesn't automatically delete a message after retrieving it for you, in case you don't successfully receive the message (for example, if the consumers fail or you lose connectivity). To delete a message, you must send a separate request which acknowledges that you've successfully received and processed the message.
This has not changed and likely won’t change in the future as there us no way for SQS to definitively know in all cases if messages have successfully been processed. If SQS started to “assume” what happens downstream it risk becoming unreliable in many scenarios.
Yes, otherwise the next time you ask for a set of messages, you will get the same messages back - maybe not on the next call, but eventually you will. You likely don't want to keep processing the same set of messages over and over.
I have setup an SQS, DeadLetterQue (DLQ) and a lambda. I am having some questions regarding the body format that the SQS receives.
After some tries, I can see some messages are ending up in a DLQ.
I have read many documentations, but really I still can't figure some basic things.
My task is fairly simple. I will use Axios to send some payload to external API. That API might give me an error, or not available at that time. This code is very simple, but at the end it will include Axios implementation:
exports.handler = async (event, context) => {
event.Records.forEach(record => {
const { body } = record;
const { messageAttributes } = record;
console.log(body)
console.log(messageAttributes);
});
let somethingWentWrongWithApi = new Error();
throw somethingWentWrongWithApi;
};
After this is run, after some time I see message in DLQ, and original SQS is empty, as I expected.
In real code, I will have catch block. And inside it I will throw the error, exactly as in example. Lambda will try to execute it three times (I am not sure where I got this information). Then, it will return it to SQS, where in turn the SQS will push it to DLQ.
I wonder, should I implement in Lambda retries, and after three times throw the error... or just throw it and rely on existing process (three times retrial)?
I am still reading about different setting on both SQS and Lambda.
To extend execution tries you have to go to:
SQS service;
Right click on your SQS;
Choose a configure Queue option;
Under the Dead Letter Queue Settings line you can see a
Maximum Receives property (which you can set between 1 and 1000).
About errors. If something going wrong on lambda side then lambda throwing an error, you can check what kind of errors throwed with cloudwatch.
Go to aws CloudWatch;
On the right panel find Logs;
Chose Log Group.
If you have any additional questions just summon me. I'll try to answer them and I'll extend that answer.
I know how to do this in java, but I just can't figure it out in Go at all.
All I want to do, is have a way to detect that an item got created in an S3 bucket, then have that trigger an SNS topic, which then notifies me of the file location in S3.
Has anybody got a working example of how I can do the go side of this for subscribing to the SNS topic or the SNS queue if I need one? Because all I seem to be able to find is Java and Node. I can find publish examples for go, but they are of little use to my use case.
To use SNS you will need a simple HTTP/HTTPS endpoint to receive SNS notifications. Which is divided into two parts (Confirm the subscription and Processing messages from HTTP/HTTPS endpoint)
1. Confirm the subscription
Do something as simple as this:
func confirmSubscription(subcribeURL string) {
response, err := http.Get(subcribeURL)
if err != nil {
fmt.Printf("Unbale to confirm subscriptions")
} else {
fmt.Printf("Subscription Confirmed sucessfully. %d", response.StatusCode)
}
}
2. Processing messages from HTTP/HTTPS endpoint
Parse the request's body, the documentations mentions how the body should be structured.
Sources:
https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
https://github.com/viveksyngh/aws-sns-subscriber/blob/master/subscriber/subscriber.go