I have a Google Cloud Pub/Sub subscription that is using a dead-letter topic. I recently had an outage that was preventing a number of messages from being processed & they ended up in the dead-letter topic.
The outage was resolved and I'd like to easily send the contents of dead letter subscription back to the original subscription. They're all present in the queue still (I have nothing consuming the dead letter sub) so I just need to route them somewhere.
This is an admin task so I'd like it to be manually initiated, if that makes any difference. Ideally via the UI but I can't see anything there.
You have a few options:
Use a Dataflow pipeline to move messages from dead letter topic to your topic.
Update existing pipeline to read from both original topic and dead letter topic based on configuration
Create a new system that, when enabled, moves messages from one topic to another.
The right answer might depend on your system design and requirements.
In case your use case for dead letter topic always includes moving the messages back to the primary topic after a delay, you might want to use configurable exponential backoff in Cloud Pub/Sub. This feature will have general availability towards the end of Q2 2020.
Related
Trying to understand what is the best way enabling Dead letter for a topic.
We have 20 topics and subscriptions for these topics. We see few failed messages on few topics. So, we are enabling Dead letter topic.
Below are steps I'm following:
Create a new topic - Dead-letter-topic-demo
Create new subscription - dead-letter-subs
enable dead letter on existing subscription and give the topic - dead-letter-topic-demo.
Can I use the same 'dead-letter-topic-demo' topic for all existing (20) topics?
Yes, you can use the same dead letter topic for all your topics. In reality, it's simply a publication from the main topic to a subsequent topic (i.e. dead letter).
However, you have to think wisely on what you want to do with the dead letter messages. If you simply want to log them (in a file or in BigQuery), no problem.
At the opposite, if you want to replay the message in the initial topic when you have fixed the problem that cause the dead-letter mechanism, it will be difficult to extract only the topic related messages and replay them. In that case, one dead letter topic per topic is a more appropriate design.
I am trying to configure dead letter topic for unack messages to handle failures gracefully, however, it doesn't seem to be working. I tried two approaches:
I created a dead letter topic and subscription through google cloud console but it didn't work, though all the permissions were granted and under dead lettering tab everything was blue tick. Also, the topic is a different topic from main topic.
Then, I tried to go completely through cloud shell. I created a new project and took a fresh start, still no luck. I followed each and every step mentioned in the documentation here. All the commands were successful and got all the blue ticks again under dead lettering tab.
In both approaches --max-delivery-attempts=5. My understanding is that after 5 attempts subscriber should forward the message to dead letter topic automatically. However, this isn't happening and the unack messages are hanging around in subscription only.
Any help would be highly appreciated.
You need to active this options in your suscriptions:
enter image description here
I'm developing my first microservice, I chose to deploy it in a Cloud Functions service with messaging done via Pub/Sub.
The Cloud Functions service is triggered by events (published messages) in a Pub/Sub topic, the microservice processes the message and so far so good. I know that Cloud Functions guarantees the acknowledgement and delivery of messages, and that's good.
The Cloud Functions service has automatic retrying: If I throw an exception in the code, a new program execution occurs. In order to avoid looping executions on consecutive failures, I introduced an if conditional that checks the 'age' of the message, and since I don't want to simply discard the message, I send/publish it to another Pub/Sub topic that I've named "my-dead-letter-queue" topic.
As I am unsure about everything now, I ask you: Is my approach good enough? What would you do instead considering Cloud Functions microservices and Pub/Sub messaging?
Yes your approach is good if you want to base on the message age the rule to send your bad messages in a dead letter topic.
If you want to do it on the number of fails (after 5 fails, put the message in the dead letter topic), you can't achieve that with a cloud function directly plugged on PubSub. You need to create an HTTP functions and then to create PubSub push subscription, on which you can set a dead letter topic (min 5 fails before sending the message automatically in a dead letter topic).
The advantage of that second solution is that you haven't to process the message and to push it to pubsub in your Cloud Functions (all processing time cost money), it's automatic and therefore you save money ;)
The approach you are trying to use is good every time you get an exception and insert it into your dead letter topic. It will work every time you try to achieve exception handling without any problem in the future, but if you would want to throw in there more exceptions, you should consider changing how you manage the exceptions.
Here you can see how you can publish the messages within gcloud command-line
I have a use-case. I want to read from SQS always, except when another event happens.
For instance, I have football news into SQS as messages. I want to retrieve them always, except for times when live matches are happening.
Is there any possibility to read unless there is another event does the job?
I scrolled the docs and Stack Overflow, but I don't see a solution.
COMMENT: I have a small and week service, and I cannot because of technical limitations increase it (memory/CPU, etc.), but I still want 2 "conflicting" flows to be in the service. They are both supposed to communicate to the same API, and I don't want them to send conflicting requests.
Is there a way to do it, or will I have to write a custom communicator with SQS?
You can't select which messages you want to read from SQS and which you'd rather not - there is no filtering in SQS.
If you have messages that need to be processed at all times and others that need to be processed only sometimes or in batches, you should put them in separate queues and read from the seperately.
You don't say anything about the infrastructure that reads from the queue, but if it's a process on EC2, you could just stop it while live matches are happening and restart it later. SQS is built for asynchronous messaging and will store the messages for up to 14 days (depending on your configuration) until a consumer is available to read them.
Problem Statement
Informal State
We have some scenarios where the integration layer (a combination of AWS SNS/SQS components and etc.) is also responsible for the data distribution to target systems. Those are mostly async flows. In this case, we send a confirmation to a caller that we have received the data and will take a responsibility for the data delivery. Here, although the data is not originated from the integration layer we are still holding it and need to make sure that the data is not lost, for example, if the consumers are down or if messages, on-error, are sent to the DLQs and hence being automatically deleted after the retention period.
Solution Design
Currently my idea was to proceed with a back-up of the SQS/DLQ queues based upon CloudWatch configured alerts using ApproximateAgeOfOldestMessage metric with some applied threshold (something like the below):
Msg Expiration Event if ApproximateAgeOfOldestMessage / Message retention > Threshold
Now, more I go forward with this idea and more I doubt that this might be actually the right approach…
In particular, I would like to build something unobtrusive that can be "attached" to our SQS queues and dump the messages that are about to expire in some repository, like for example the AWS S3. Then have a procedure to recover the messages from S3 to the same original queue.
The above procedure contains many challenges like: message identification and consumption (receive message is not design to "query" for specific messages), message dump in the repository with a reference to the source queue, etc. which would suggest to me that the above approach might be a complex over-kill.
That being said, I'm aware of other "alternatives" (such as this) but I would appreciate if you could answer to the specific technical details described above, without trying to challenge the "need" instead.
Similar to Mark B's suggestion, you can use the SQS extended client (https://github.com/awslabs/amazon-sqs-java-extended-client-lib) to send all your messages through S3 (which is a configuration knob: https://github.com/awslabs/amazon-sqs-java-extended-client-lib/blob/master/src/main/java/com/amazon/sqs/javamessaging/ExtendedClientConfiguration.java#L189).
The extended client is a drop-in replacement for the AmazonSQS interface so it minimizes the intrusion on business logic - usually it's a matter of just changing your dependency injection.