I'm using the AWS SDK v3 to push a message to SNS which is then subscribed to by an SQS Queue.
await snsClient.send(new PublishCommand({
Message: JSON.stringify(payload),
TopicArn: process.env[SNS_TOPIC_ARN],
}));
I want to delay individual messages. Is this possible if I'm pushing them via SNS or do I have to rework it and push directly to SQS?
You can only control the delay on individual messages when sending the messages to the Amazon SQS queue directly.
It is not possible to specify this value when sending the message via an Amazon SNS topic to the Amazon SQS queue.
Related
I am trying to create Pub/Sub microservice application using Amazon SQS. With a single publisher and multiple subscribers(consumers). Messages are consumed by the subscriber based on the message attributes. Also, a single message can be consumed by multiple subscribers.
Is my approach correct?
If it is show which consumer will be responsible for dequeuing message from the SQS ?
FYI - I am using Typescript / Express for this not using serverless stack.
Messages sent to an Amazon SQS queue wait until a consumer requests a message. When the message is retrieved, it is made 'invisible' on the queue so no other consumer will receive it. When a consumer has finished processing the message, it deletes the message from the queue.
Therefore, if you want multiple consumers to receive the same message, then Amazon SQS is not the correct service to use.
Since you want a publish/subscribe model, you should be using Amazon Simple Notification Service (Amazon SNS). Messages are published to a 'Topic' and multiple subscribers can receive messages sent to that topic. Subscribers can use Amazon SNS subscription filter policies - Amazon Simple Notification Service to limit which messages they receive.
Note that messages sent to an Amazon SNS topic are immediately sent to subscribers. If you do not wish to receive a message in real-time, it is possible to subscribe an Amazon SQS queue to an Amazon SNS topic. This way, the messages will be queued for later retrieval. The queue would work independently to other subscribers on the topic.
How to create in AWS SQS something like 'direct' exchange in RabbitMQ:
1 message -> N receivers queues.
Each client app connects to server and creates its own queue, a publisher sends one messages to exchange (direct+routing key) and its sent to all N queues, then each user reads its own queue and the queue is emptied.
This can be done with a 'fan-out' pattern combining Amazon SNS and Amazon SQS:
Create all desired Amazon SQS queues
Create an Amazon SNS Topic
Subscribe all of the Amazon SQS queues to the Amazon SNS Topic
Send a message to the Amazon SNS topic -- this will be sent to all subscribing queues. Each queue will have its own copy of the message.
Make sure you use Amazon SNS raw message delivery to preserve the format of the initial message as it goes from Amazon SNS to the SQS queues.
See also: How to Fan-Out to Different SQS Queues Using SNS Message Filtering | by Lorenz Vanthillo | Better Programming
I have requirement to publish messages from SNS to kinesis. I have found that, it is not possible directly by subscribing same as SNS/SQS. I will need to write lambda to fetch from SNS and publish it to kinesis.
Is there any other way to publish records from SNS to kinesis directly?
Thanks
Amazon SNS is a publish/subscribe model.
Messages sent to SNS can be subscribed from:
http/s: delivery of JSON-encoded message via HTTP POST
email: delivery of message via SMTP
email-json: delivery of JSON-encoded message via SMTP
sms: delivery of message via SMS
sqs: delivery of JSON-encoded message to an Amazon SQS queue
application: delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda: delivery of JSON-encoded message to an AWS Lambda function.
Other options: See Otavio's answer below!
See: Subscribe - Amazon Simple Notification Service
Of these, the only ones that could be used to send to Amazon Kinesis would be to use AWS Lambda. You would need to write a Lambda function that would send the message to a Kinesis stream.
To clarify: Your Lambda function will not "fetch from SNS". Rather, the Lambda function will be triggered by SNS, with the message being passed as input. Your Lambda function will then need to send the message to Kinesis.
Your only other alternative is to change the system that currently sends the message to SNS and have it send the message to Kinesis instead.
Good news! As of January 2021, Amazon SNS has added support for message archiving and analytics via Kinesis Data Firehose subscriptions. You can now load SNS messages into S3, Redshift, Elasticsearch, MongoDB, Datadog, Splunk, New Relic, and more. The SNS documentation has the details.
I am using Amazon SES for a project and have set up a Receipt rule to send messages from SES to SNS. The SNS has my API end point as a subscriber but to ensure that I do not miss any message I have also set up an SQS Queue and have subscribed the queue to SNS topic.
With this set up I receive each SES email twice. (One from SNS and one with a poll in SQS). Is there a way to send only the failed SNS messages to SQS queue so that I don't have to check for duplicates always ?
Rule which sends SES messages to SNS:
SQS queue subscribed to SNS topic:
With your setup, it's not possible to do so. But there are other approaches that you can try out. It might introduce complexity into your application, but it's worth trying. Some of the approaches are given below.
SES to SNS and send all messages from SNS to SQS, and poll SQS for the messages. If some fail, put them to a Deadletterqueue (similar to SQS), and poll that queue separately from time to time, to look for failed messages. This makes the messages more persistent, but slightly inefficient due to polling.
SES to SNS and let SNS use it's delivery policy as needed. You can avoid an SQS, and ask SNS to look for the delivery status, and retry if it is a failure. You can define the retry policy as needed and it's given in this link. After a lot of trying, the message can be discarded as a total failure. You can also write to AWS CloudWatch Logs on the status of the retries.
Short answer, no.
If you've subscribed an SQS queue to an SNS topic, it'll receive all messages published to the latter. SNS doesn't know which messages were successfully processed by your API, so can't selectively send to SQS.
I know I can configure an Amazon S3 bucket to publish events to a SQS topic and to a SNS topic.
But, is it possible to configure the bucket to publish the event to SQS first, and then, when the message has been sent to SQS, have the bucket publish the event to SNS (kind of publish these events synchronously)?
An Amazon S3 bucket can publish a notification to one of:
Amazon Simple Notification Service (SNS)
Amazon Simple Queue Service (SQS)
AWS Lambda
However, SNS can also send a message to SQS. (More accurately, SQS can be added as a subscriber to an SNS topic).
Therefore, you could choose to send the event to SNS, which can on-send the event to an SQS queue. This is a good way to "fork" the event, sending it to multiple SNS subscribers.