I am using Amazon SES for a project and have set up a Receipt rule to send messages from SES to SNS. The SNS has my API end point as a subscriber but to ensure that I do not miss any message I have also set up an SQS Queue and have subscribed the queue to SNS topic.
With this set up I receive each SES email twice. (One from SNS and one with a poll in SQS). Is there a way to send only the failed SNS messages to SQS queue so that I don't have to check for duplicates always ?
Rule which sends SES messages to SNS:
SQS queue subscribed to SNS topic:
With your setup, it's not possible to do so. But there are other approaches that you can try out. It might introduce complexity into your application, but it's worth trying. Some of the approaches are given below.
SES to SNS and send all messages from SNS to SQS, and poll SQS for the messages. If some fail, put them to a Deadletterqueue (similar to SQS), and poll that queue separately from time to time, to look for failed messages. This makes the messages more persistent, but slightly inefficient due to polling.
SES to SNS and let SNS use it's delivery policy as needed. You can avoid an SQS, and ask SNS to look for the delivery status, and retry if it is a failure. You can define the retry policy as needed and it's given in this link. After a lot of trying, the message can be discarded as a total failure. You can also write to AWS CloudWatch Logs on the status of the retries.
Short answer, no.
If you've subscribed an SQS queue to an SNS topic, it'll receive all messages published to the latter. SNS doesn't know which messages were successfully processed by your API, so can't selectively send to SQS.
Related
I am trying to create Pub/Sub microservice application using Amazon SQS. With a single publisher and multiple subscribers(consumers). Messages are consumed by the subscriber based on the message attributes. Also, a single message can be consumed by multiple subscribers.
Is my approach correct?
If it is show which consumer will be responsible for dequeuing message from the SQS ?
FYI - I am using Typescript / Express for this not using serverless stack.
Messages sent to an Amazon SQS queue wait until a consumer requests a message. When the message is retrieved, it is made 'invisible' on the queue so no other consumer will receive it. When a consumer has finished processing the message, it deletes the message from the queue.
Therefore, if you want multiple consumers to receive the same message, then Amazon SQS is not the correct service to use.
Since you want a publish/subscribe model, you should be using Amazon Simple Notification Service (Amazon SNS). Messages are published to a 'Topic' and multiple subscribers can receive messages sent to that topic. Subscribers can use Amazon SNS subscription filter policies - Amazon Simple Notification Service to limit which messages they receive.
Note that messages sent to an Amazon SNS topic are immediately sent to subscribers. If you do not wish to receive a message in real-time, it is possible to subscribe an Amazon SQS queue to an Amazon SNS topic. This way, the messages will be queued for later retrieval. The queue would work independently to other subscribers on the topic.
I'm using AWS SES for sending emails. To track email delivery status information I have followig pipeline:
SES Config set -> SNS -> SQS -> Lambda -> Our system
Sometimes the events comes in wrong order. For example the event Send comes 30 seconds later after event Delivery, so in our system is the status Delivery replaced by Send.
I tried to set the SNS topic and SQS queue as FIFO. But in SES config set it is not possible to add SNS FIFO topic as destination. AWS says: FIFO topics are not supported.
The SES is located in different account than SNS, but I believe, that this has no effect.
Maybe I can remove the SQS, the lambda is probably fast enough, so it does not need the queue. But anyway, the standard SNS is high-throughput, best-effort delivery, so it does not keep the order.
I would like to solve this issue on the AWS side. How can I configure the AWS, so the events (resp. lambda) are fired in correct order? Does the SES publish the events in specific order?
I have created a SNS Topic and subscribed a SQS Queue with to this Topic using CDK.
I would like to understand does SQS Queue automatically polls messages published through this SNS Topic?
SNS automatically "pushes" messages to subscribed SQS queues:
Docs: Amazon SNS allows applications to send time-critical messages to multiple subscribers through a “push” mechanism, eliminating the need to periodically check or “poll” for updates.
The integration works this way whether created with the CDK or not.
Can we have a poller for SNS Topic which would get all the messages from SNS Topic in regular intervals, like we can have for SQS?
Amazon SNS is a Publish/Subscribe system. When a message is sent to a topic, it is immediately sent out to all relevant subscribers. Messages are not retained in Amazon SNS.
If you wish to retain messages for later polling, you can use Amazon SQS.
However, Amazon SQS can subscribe to an Amazon SNS topic. This would allow you to store messages for later polling. The flow would be:
Message -> SNS Topic -> SQS Queue subscription -> Available for polling by worker
I have several services, and each service creates an SNS topic which sends an email to me. e.g., whenever code deploy fails, I receive an email from an SNS topic, whenever codepipeline fails I receive an email from another SNS topic, whenever a specific lambda fails, I receive another SNS topic. None of the above failures is urgent, so it does not need to be taken care immediately.
How can I receive only one email report once a day for all the above failures? Maybe I should let the corresponding service trigger SQS instead of SNS and then once every day I pull all messages from SQS?
The option you mentioned with an SQS queue subscribed to the SNS topic is probably the best way to go. You can then use a scheduled Lambda function that runs once a day to poll the messages in the queue and send you an email with the aggregated information.