I am looking for a PubSub or Message Queue cloud service with the following requirements:
Pay per message
The ability to subscribe on-demand with a client sdk
Support for topics
Amazon MQ does not satisfy these requirements because it is not pay per message.
Amazon SNS does not satisfy these requirements because it doesn't allow on-demand subscription using a client sdk.
Amazon SQS does not satisfy these requirements because it doesn't support topics.
AWS IoT Core does satisfy these requirements.
However, I'm curious if there are other options - possibly even outside the AWS ecosystem. I'm hesitant to use AWS IoT since I wouldn't be using it for an IoT use case.
Should I move forward with AWS IoT Core or are there other options I should be considering?
Related
Amazon EventBridge enables developers to connect 3rd party event-driven applications with Amazon services. Amazon AppFlow offers event-driven integration with 3rd party apps as well.
What is the difference between the two services for the event-driven scenarios and when to use one over the other?
The AppFlow FAQ contains this info:
Amazon EventBridge enables developers to build event driven
applications that interact with SaaS applications and AWS services.
SaaS applications that have integrated with EventBridge emit events to
the customer’s event bus, which can then be routed to targets such as
Amazon EC2 instances or Lambda functions for processing. AppFlow
supports bi-directional transfer of data between SaaS applications and
AWS services that may be initiated by humans using a UI, a schedule,
or events - all with a point and click interface.
AppFlow is great for anyone who wants to connect together their applications without writing any code. EventBridge works with Step Functions and Lambda (and various other AWS services), and so is well suited for developers who need the extra flexibility that those services provide. Right now EventBridge is only one way, whereas AppFlow allows you to send data back to SaaS apps. The list of supported partners for each service is also different.
For some of AppFlow's partners, data is received via API polling (from the FAQ: "AppFlow is a fully managed API integration service that replaces custom connectors"), which is slightly different from EventBridge which receives data in an event-driven manner, where an event is sent via an HTTP call from the partner as soon as a change occurs. The pricing of the two services also differs: EventBridge charges $1/million events, and AppFlow charges $0.001 per flow (with an additional charge per GB of data processed).
I wanted to send data from IoT Rule to AWS Kinesis Stream/Firehose.
Both the IoT Core and Kinesis Steam is in different account.
Is there any way to do it ?
One way to do this is to have the IoT rule invoke a lambda in the other account. This lambda then publishes to an AWS Kinesis Stream in the same account.
An example of this cross-account rule is given at https://aws.amazon.com/blogs/iot/access-cross-account-resources-using-aws-iot-rules-engine/. This states:
Cross-account access, via AWS IoT Rules Engine, needs resource-based permissions. Hence, only destinations that support resource-based permission can be enabled for the cross-account access via AWS IoT Rules Engine. Following is the list of such destinations:
Amazon Simple Queue Service (SQS)
Amazon Simple Notification Service (SNS)
Amazon Simple Storage Service (S3)
AWS Lambda
I am a newbie to AWS. As I understand, both AWS MQ and AWS SQS are Message Queue tools. The only noted difference is that SQS is fully managed.
When should we use SQS or MQ?
SQS is a simple queueing service. It doesn't support many higher level abstractions like message routing, fanouts, distribution lists etc. It is a queue - a message is produced, and a message is delivered. It is useful when you need a Queue with limited backing logic.
AWS MQ is a managed Apache ActiveMQ(or RabbitMQ) broker service.
This provides you a fully managed Apache ActiveMQ system in the cloud, with support for a variety of industry-standard queue and broadcast protocols like AMQP, JMS etc. It is useful when you have complicated delivery rules - or when you're migrating an existing system from outside AWS into AWS, and your systems happen to talk to one another with a standard queueing protocol.
From AWS Documentation, you can search "Q: When should I use Amazon MQ vs. Amazon SQS and SNS?":
Amazon MQ, Amazon SQS, and Amazon SNS are messaging services that are
suitable for anyone from startups to enterprises. If you're using
messaging with existing applications, and want to move your messaging
to the cloud quickly and easily, we recommend you consider Amazon MQ.
It supports industry-standard APIs and protocols so you can switch
from any standards-based message broker to Amazon MQ without rewriting
the messaging code in your applications. If you are building brand new
applications in the cloud, we recommend you consider Amazon SQS and
Amazon SNS. Amazon SQS and SNS are lightweight, fully managed message
queue and topic services that scale almost infinitely and provide
simple, easy-to-use APIs. You can use Amazon SQS and SNS to decouple
and scale microservices, distributed systems, and serverless
applications, and improve reliability.
Also, in this doc, you can check comparisons with other services.
For the purpose of sending push notifications from the backend, if we need a Push Notification Platform, could you please suggest which of these is intended for that purpose – Amazon SNS or Pinpoint?
Both Amazon SNS and Amazon Pinpoint, supports sending push notification to various devices (e.g Android, iOS etc)
The major difference between Amazon SNS & Amazon Pinpoint is that :
with Amazon SNS you have to set up your application to manage each message's audience, content, and delivery schedule. On the other hand, with Amazon Pinpoint you do not have to code these features, most of them are already built in. With Amazon Pinpoint, you can collect data about your app usage, create highly-targeted segments and send full campaigns(either immediate or scheduled) plus many more features.
Can someone explain to me the advantage or disadvantage of using SNS -> Lambda vs. SNS -> SQS -> Lambda.
I'm looking to setup an architecture for pub/sub micro-service messaging, but having a queue in front of every Lambda seems excessive.
Unless something has changed, the question of whether to it makes more sense to deploy SNS → Lambda, or SNS → SQS → Lambda, is based on a premise with a significant flaw.
As is indicated in Supported Event Sources in the Lambda documentation, Lambda events can be sourced from S3, DynamoDB, Kinesis, SNS, SES, Cognito, CloudFormation, CloudWatch & Events (including Scheduled Events), AWS Config, Amazon Echo, and API Gateway.
And, of course, you can invoke them directly.
But SQS is not a supported Lambda event source.
Amazon SQS is a message queue service used by distributed applications to exchange messages through a polling model, and can be used to decouple sending and receiving components—without requiring each component to be concurrently available. By using Amazon SNS and Amazon SQS together, messages can be delivered to applications that require immediate notification of an event, and also persisted in an Amazon SQS queue for other applications to process at a later time.
Untill you don't want to decouple sending and receiving components and just want to achieve your use case in the question it will work in both case SNS- Lambda and SNS - SQS - Lambda.