I have developed a service that receives messages from AWS/Kinesis and process them. I'd like to have a watchdog service that will catch processing failures. Before the service starts message processing it sends a message to watchdog and after it finishes processing the message it notifies Watchdog that processing completed successfully. If there is no 'success' message after some timeout period Watchdog should fire some event.
The question is: which AWS service, if any, can do this job for me?
You can use SNS, making the service publish a notification to the subscriber (watchdog) for each event that you want to be notified about.
Amazon SNS is a fully managed pub/sub messaging service that makes it easy to decouple and scale microservices, distributed systems, and serverless applications. With SNS, you can use topics to decouple message publishers from subscribers, fan-out messages to multiple recipients at once, and eliminate polling in your applications. SNS supports a variety of subscription types, allowing you to push messages directly to Amazon Simple Queue Service (SQS) queues, AWS Lambda functions, and HTTP endpoints. AWS services, such as Amazon EC2, Amazon S3 and Amazon CloudWatch, can publish messages to your SNS topics to trigger event-driven computing and workflows. SNS works with SQS to provide a powerful messaging solution for building cloud applications that are fault tolerant and easy to scale.
https://aws.amazon.com/sns/
Related
I am trying to create Pub/Sub microservice application using Amazon SQS. With a single publisher and multiple subscribers(consumers). Messages are consumed by the subscriber based on the message attributes. Also, a single message can be consumed by multiple subscribers.
Is my approach correct?
If it is show which consumer will be responsible for dequeuing message from the SQS ?
FYI - I am using Typescript / Express for this not using serverless stack.
Messages sent to an Amazon SQS queue wait until a consumer requests a message. When the message is retrieved, it is made 'invisible' on the queue so no other consumer will receive it. When a consumer has finished processing the message, it deletes the message from the queue.
Therefore, if you want multiple consumers to receive the same message, then Amazon SQS is not the correct service to use.
Since you want a publish/subscribe model, you should be using Amazon Simple Notification Service (Amazon SNS). Messages are published to a 'Topic' and multiple subscribers can receive messages sent to that topic. Subscribers can use Amazon SNS subscription filter policies - Amazon Simple Notification Service to limit which messages they receive.
Note that messages sent to an Amazon SNS topic are immediately sent to subscribers. If you do not wish to receive a message in real-time, it is possible to subscribe an Amazon SQS queue to an Amazon SNS topic. This way, the messages will be queued for later retrieval. The queue would work independently to other subscribers on the topic.
Is it possible to receive notifications in a .NET Core application about bucket/object creation/deletion?
How to do it?
S3 bucket can generate SNS and SQS event notifications as well as trigger Lambda function on misc. events. Go to Bucket Properties->Events.
In your .NET code you'll need to react to these events, for instance by receiving SQS messages.
Amazon S3 Events can send a notification to:
An AWS Lambda function (Trigger): Does not appear relevant since your code is running elsewhere.
An Amazon SQS queue (Pull): Your application could regularly poll the Amazon SQS queue to retrieve a message, then act on that message.
An Amazon SNS topic (Push): Your application could subscribe to the Amazon SNS topic to receive the message via an HTTP endpoint. For example, this could point to your back-end web server.
If your application has a web server that is accessible from the Internet, then use the SNS push. Otherwise, your app will need to poll the SQS queue.
I am a newbie to AWS. As I understand, both AWS MQ and AWS SQS are Message Queue tools. The only noted difference is that SQS is fully managed.
When should we use SQS or MQ?
SQS is a simple queueing service. It doesn't support many higher level abstractions like message routing, fanouts, distribution lists etc. It is a queue - a message is produced, and a message is delivered. It is useful when you need a Queue with limited backing logic.
AWS MQ is a managed Apache ActiveMQ(or RabbitMQ) broker service.
This provides you a fully managed Apache ActiveMQ system in the cloud, with support for a variety of industry-standard queue and broadcast protocols like AMQP, JMS etc. It is useful when you have complicated delivery rules - or when you're migrating an existing system from outside AWS into AWS, and your systems happen to talk to one another with a standard queueing protocol.
From AWS Documentation, you can search "Q: When should I use Amazon MQ vs. Amazon SQS and SNS?":
Amazon MQ, Amazon SQS, and Amazon SNS are messaging services that are
suitable for anyone from startups to enterprises. If you're using
messaging with existing applications, and want to move your messaging
to the cloud quickly and easily, we recommend you consider Amazon MQ.
It supports industry-standard APIs and protocols so you can switch
from any standards-based message broker to Amazon MQ without rewriting
the messaging code in your applications. If you are building brand new
applications in the cloud, we recommend you consider Amazon SQS and
Amazon SNS. Amazon SQS and SNS are lightweight, fully managed message
queue and topic services that scale almost infinitely and provide
simple, easy-to-use APIs. You can use Amazon SQS and SNS to decouple
and scale microservices, distributed systems, and serverless
applications, and improve reliability.
Also, in this doc, you can check comparisons with other services.
I want to build a pub/sub messaging system into my services that are hosted on Amazon Web Services, and creating SQS queues that subscribe to SNS topics seems like the obvious direction to take, but I can't get it working at all.
So far my code looks for the topics and the queues at startup and creates anything that's missing. This seems to work, I can see the SNS topic and the SQS queues in the AWS management console, and I can see that the queue is subscribed to the topic, but when I publish messages to the topic nothing ends up in the queue.
Even if I manually publish a message using the 'Publish' button in the management console the queue is still empty.
I changed the permissions on both the topic and the queue to 'everyone can do everything' just to eliminate this possibility. My code receives a message ID in response to the publish and there are no errors, every API call returns a 200 (OK) status.
Where can I go from here to figure out why it's not working?
The SNS --> SQS link has a few gotchas:
"Amazon SNS isn't currently compatible with FIFO queues." per the note on their Subscribing an Amazon SQS Queue to an Amazon SNS Topic Tutorial
You have to fiddle with the IAM permissions (see the page on Sending Amazon SNS Messages to Amazon SQS Queues)
You can't send messages to encrypted queues (see their Server-Side Encryption page)
It would definitely have been easier to figure this out if all this info were consolidated into a single page. The killer for me was #3 - perhaps one of these will be the solution to your issue.
A couple of options -
Enable CloudTrail and monitor the logs
View the CloudWatch logs to identify any permissions issues
Open a ticket with AWS support.
Ideally, you wouldn't be creating the resources in your application but instead decouple those into CloudFormation or at a minimum CLI scripts. If you require the ability to dynamically create these resources, using the AWS IoT Message Broker may be a better option since it supports ephemeral messaging resources - http://docs.aws.amazon.com/iot/latest/developerguide/iot-message-broker.html
I had a similar issue with SQS subscriptions. It turned out that if I create the subscription from the SQS editor it works, but if I create it from the SNS creation screen it accepts the message but never forwards it to the queue.
To get more detailed information about specific failures you can turn on
SNS "Delivery status logging".
Can someone explain to me the advantage or disadvantage of using SNS -> Lambda vs. SNS -> SQS -> Lambda.
I'm looking to setup an architecture for pub/sub micro-service messaging, but having a queue in front of every Lambda seems excessive.
Unless something has changed, the question of whether to it makes more sense to deploy SNS → Lambda, or SNS → SQS → Lambda, is based on a premise with a significant flaw.
As is indicated in Supported Event Sources in the Lambda documentation, Lambda events can be sourced from S3, DynamoDB, Kinesis, SNS, SES, Cognito, CloudFormation, CloudWatch & Events (including Scheduled Events), AWS Config, Amazon Echo, and API Gateway.
And, of course, you can invoke them directly.
But SQS is not a supported Lambda event source.
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components—without requiring each component to be concurrently available. By using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time.
Untill you don't want to decouple sending and receiving components and just want to achieve your use case in the question it will work in both case SNS- Lambda and SNS - SQS - Lambda.