When subscribing to a topic, I handle the ack/nack myself and can easily call message.nack(millisecondsToNextRetry).
I would like to do the same thing in cloud functions using nodejs, i.e. under certain circumstances retry the function after a specified time.
Anyone know a good solution or workaround when triggering a cloud function from pub/sub?
Cloud Functions will retry automatically if you enable that by configuration. Retries will happen if your function throws an exception, returns a rejected promise, or times out. You won't be able to control the schedule of retries.
Related
I'm developing my first microservice, I chose to deploy it in a Cloud Functions service with messaging done via Pub/Sub.
The Cloud Functions service is triggered by events (published messages) in a Pub/Sub topic, the microservice processes the message and so far so good. I know that Cloud Functions guarantees the acknowledgement and delivery of messages, and that's good.
The Cloud Functions service has automatic retrying: If I throw an exception in the code, a new program execution occurs. In order to avoid looping executions on consecutive failures, I introduced an if conditional that checks the 'age' of the message, and since I don't want to simply discard the message, I send/publish it to another Pub/Sub topic that I've named "my-dead-letter-queue" topic.
As I am unsure about everything now, I ask you: Is my approach good enough? What would you do instead considering Cloud Functions microservices and Pub/Sub messaging?
Yes your approach is good if you want to base on the message age the rule to send your bad messages in a dead letter topic.
If you want to do it on the number of fails (after 5 fails, put the message in the dead letter topic), you can't achieve that with a cloud function directly plugged on PubSub. You need to create an HTTP functions and then to create PubSub push subscription, on which you can set a dead letter topic (min 5 fails before sending the message automatically in a dead letter topic).
The advantage of that second solution is that you haven't to process the message and to push it to pubsub in your Cloud Functions (all processing time cost money), it's automatic and therefore you save money ;)
The approach you are trying to use is good every time you get an exception and insert it into your dead letter topic. It will work every time you try to achieve exception handling without any problem in the future, but if you would want to throw in there more exceptions, you should consider changing how you manage the exceptions.
Here you can see how you can publish the messages within gcloud command-line
I'm using a Google Cloud Function to run a function off an HTTP trigger. The code itself is non-idempotent, which is expected and desired for us as it's using an external API.
However, quick and repeated triggers of the cloud function yields the same output repeatedly (generally while there is 1 active instance), and we need the output to be different every time the function is triggered.
Unsure if this is due to instance caching or something else, but is there any known workaround to ensure that every time the cloud function is HTTP triggered we get a new output, even when triggered seconds apart?
Thanks.
The official documentation does mention that Google Cloud Pub / Sub resends messages to subscribers until subscribers acknowledge the message receipt when using official Cloud Pub / Sub Node.js client.
But it does not explicitly mention this for background functions if they return a callback error. Refer https://cloud.google.com/functions/docs/writing/background.
If it helps - My background function does not use the official Cloud Pub / Sub Node.js client since I get all the required info from the event arguement itself.
From documentation: https://cloud.google.com/functions/docs/bestpractices/retries
Cloud Functions guarantees at-least-once execution of a background
function for each event emitted by an event source. However, by
default, if a function invocation terminates with an error, the
function will not be invoked again, and the event will be dropped.
When you enable retries on a background function, Cloud Functions will
retry a failed function invocation until it completes successfully, or
the retry window expires.
And described below, you can enable retry on errors:
In any of the above cases, the function stops executing by default and
the event is discarded. If you want to retry the function when an
error occurs, you can change the default retry policy by setting the
"retry on failure" property. This causes the event to be retried
repeatedly for up to multiple days until the function successfully
completes.
I'm trying to implement an AWS Lambda function that should send an HTTP request. If that request fails (response is anything but status 200) I should wait another hour before retrying (longer that the Lambda stays hot). What the best way to implement this?
What comes to mind is to persist my HTTP request in some way and being able to trigger the Lambda function again in a specified amount of time in case of a persisted HTTP request. But I'm not completely sure which AWS service that would provide that functionality for me. Is SQS an option that can help here?
Or, can I dynamically schedule Lambda execution for this? Note that the request to be retried should be identical to the first one.
Any other suggestions? What's the best practice for this?
(Lambda function is my option. No EC2 or such things are possible)
You can't directly trigger Lambda functions from SQS (at the time of writing, anyhow).
You could potentially handle the non-200 errors by writing the request data (with appropriate timestamp) to a DynamoDB table that's configured for TTL. You can use DynamoDB Streams to detect when DynamoDB deletes a record and that can trigger a Lambda function from the stream.
This is obviously a roundabout way to achieve what you want but it should be simple to test.
As jarmod mentioned, you cannot trigger Lambda functions directly by SQS. But a workaround (one I've used personally) would be to do the following:
If the request fails, push an item to an SQS Delay Queue (docs)
This SQS message will only become visible on the queue after a certain delay (you mentioned an hour).
Then have a second scheduled lambda function which is triggered by a cron value of a smaller timeframe (I used a minute).
This second function would then scan the SQS queue and if an item is on the queue, call your first Lambda function (either by SNS or with the AWS SDK) to retry it.
PS: Note that you can put data in an SQS item, since you mentioned you needed the lambda functions to be identical you can store your first function's input in here to be reused after an hour.
I suggest that you take a closer look at the AWS Step Functions for this. Basically, Step Functions is a state machine that allows you to execute a Lambda function, i.e. a task in each step.
More information can be found if you log in to your AWS Console and choose the "Step Functions" from the "Services" menu. By pressing the Get Started button, several example implementations of different Step Functions are presented. First, I would take a closer look at the "Choice state" example (to determine wether or not the HTTP request was successful). If not, then proceed with the "Wait state" example.
I currently have a 3rd party application pushing messages to a Lambda function through API gateway. The Lambda function needs to serialize, log, and push the message to another ESB that I have very little control over.
I'm trying to ensure that there is some kind of recovery mechanism in the case that the Lambda function is either at max load or cannot communicate with the ESB. I've read about Kinesis being a good option for exactly this, but the ESB does not support batching for my use case.
This would cause me to run into the scenario where some messages might make it to ESB, while others don't, which would ultimately cause the batch to fail. Then, when the batch is retried, the messages would be duplicated in the ESB.
Is there a way I could utilize the functionality that Kinesis offers without the batching? Is there another AWS offering that better fits my use case? Ideally I would have one message being handled by the Lambda function that stays in the queue until it is successfully pushed into the ESB.
Any tips would be much appreciated.
Thanks,
Matt
Following might be of help to you:
1) setup api-gateway to log to sqs and 2) then set up a lambda function on that sqs queue to serialize, log, and push the message to the external endpoint.
For the first part: How to integrate API Gateway with SQS this will be of help. (as already mentioned in comments)
This article might help you more for second part: https://dzone.com/articles/integrate-sqs-and-lambda-serverless-architecture-f
Note that you can also choose what kind of trigger you would like (based on usecase)- cron based poll/ or event based, you also have control over when you are deleting from sqs in your lambda function. (you can also find the very basic code in lambda blueprint with name "sqs-poller").
Thanks!