Im writing an app using the AWS framework. It would be the perfect framework if the SQS had FIFO access!
I am using SNS to publish messages to queues, and the order of the messages is important. Does anyone know of a MQ that can be used, that is scalable etc.. like SQS that can be used?
I have been looking at using ironMQ but the SNS seems to have issues with receiving the messages?
(or even better yet,if you know of a good way of ordering messages with SQS)
Thanks,
Ben
These days, Amazon Kinesis would be a good choice. (It wasn't available when the question was raised.)
Kinesis captures data in streams, retains order and allows the data to be replayed and even consumed by multiple applications.
Related
SNS/SQS seems to do the job so far for communicating between microservices, I am looking at streaming events from different microservices now and would like to know at what scale SNS/SQS combination could start to become too expensive and when should I consider kinesis based on the scale?
I don't see any reasons other than scale to just communicate events from different microservices to events microservice.
To be honest, I don't think kinesis is cheaper at any scale. It's more related to if you need the features kinesis has to offer, like lower latency than the sqs/sns setup, data retention (e.g. for stream analytics) or message ordering or increased message sizes.
Since you are not using those features now, I'd think you don't need them and can stick to the current setup. It's also simpler to manage.
Kinesis and SQS serve different use-cases.
SQS is for when you want to process each message once, by a single consumer (recognizing that your actual guarantee is "at least once").
Kinesis is for when you have multiple consumers, all of them need to process the same message, and they may not all be running at the same time.
Prior to Kinesis, you could achieve the same thing by sending the message to SNS, and then subscribing an SQS queue for each consumer.
I am deciding if I should use MSK (managed kafka from AWS) or a combination of SQS + SNS to achieve a pub sub model?
Background
Currently, we have a micro service architecture but we don't use any messaging service and only use REST apis (dont ask why - related to some 3rd party vendors who designed the architecture). Now, I want to revamp it and start using messaging for communication between micro-services.
Initially, the plan is to start publishing entity events for any other micro service to consume - these events will also be stored in data lake in S3 which will also serve as a base for starting data team.
Later, I want to move certain features from REST to async communication.
Anyway, the main question I have is - should I decide to go with MSK or should I use SQS + SNS for the same? ( I already understand the basic concepts but wanted to understand from fellow community if there are some other pros and cons)?
Thanks in advance
MSK VS SQS+SNS is not really 1:1 comparison. The choice depends on various use cases. Please find out some of specific difference between two
Scalability ->
MSK has better scalability option because of inherent design of partitions that allow parallelism and ordering of message.
SNS has limitation of 300 publish/Second, to achieve same performance as MSK, there need to have higher number of SNS topic for same purpose.
Example : Topic: Order Service
in MSK -> one topic+ 10 Partitions
SNS -> 10 topics
if client/message producer use 10 SNS topic for same purpose, then client needs to have information of all 10 SNS topic and distribution of message.
In MSK, it's pretty straightforward, key needs to send in message and kafka will allocate the partition based on Key value.
Administration/Operation ->
SNS+SQS setup is much simpler compare to MSK.
Operational challenge is much more with MSK( even this is managed service). MSK needs more in depth skills to use optimally.
SNS +SQS VS SQS -> I believe you have multiple subscription(fanout) for same message thats why you have refer SNS +SQS.
If you have One Subscription for one message, then only SQS is also sufficient.
Replay of message -> MSK can be use for replaying the already processed message.
It will be tricky for SQS, though can be achieve by having duplicate queue so that can be use for replay.
I have a process which publish some data(json) onto a queue on Aws-Sqs. Another process reads from this queue. All this is working fine.
Now I want to create a topic which can be listened by mutiple processes and the data is delivered to all the processes. For example Activemq and many other messaging servers have this capability to create a topic. I could not find any such thing on AWS. The closest I could find is AWS SNS.
From what I understand AWS-SNS allows multiple clients to subscribe to a topic. But the type of subscription is either Email, Http, or Sms and so on ... This does not really serve my purpose. I want to recieve json data in all my clients just like Sqs.
Is that achievable? If so how?
You can subscribe multiple SQS into single SNS topic: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html
Then it will be distributed to all of them.
The other option is to use Kinesis - https://aws.amazon.com/kinesis/ but it is more difficult to set up. There you can also read from multiple clients from the stream.
amazon mq is a managed active mq service. maybe this will help with your needs?
I'm an AWS noob, I'm trying to figure out what the difference between Amazon's Kinesis Data Stream and EventBridge products. Can someone explain this for someone not familiar with the AWS tech stack?
Kinesis is a real-time stream processing service. Typically gets used for storing logs or end-user data coming from browser.
Event bridge is typically used to reliably communicate between apps / microservices, so it's quite similar to SQS, but has some added features.
Differences between SQS and Event Bridge are explained in the post below:
https://www.reddit.com/r/aws/comments/cjnw2l/what_makes_eventbridge_different_than_sqs_and/
I want to use an AWS lambda function to fan out and insert activity stream info to a firebase endpoint for every user.
Should I be using Kinesis, SQS or SNS to trigger the lambda function for this use case? The updates to the activity stream can be triggered from the server and clients should receive the update near real time (within 60 seconds or so).
I think I have a pretty good idea on what SQS is, and have used Kinesis in the past but not quite sure about SNS.
If we created an SNS topic for each user and then each follower subscribes to these topics with an AWS lambda function - would that work?
Does it make sense to programmatically create topics and subscriptions for every user and follow relationship respectively?
As usual, answer to such a question is mostly, 'it depends on your use-case'.
Kinesis vs SQS:
If your clients care about relative (timestamp-based, for e.g.) ordering between events, you'll almost certainly have to go with Kinesis. SQS is a best-effort FIFO queue, meaning events can arrive out of order and it would up to your client to manage relative ordering.
As far as latencies are concerned, I have seen that data ingested into Kinesis can become visible to its consumer in as less as 300 ms.
When can SNS be interesting to you?
(Even with SNS, you'd have to use SQS). If you use SNS, it will be easy to add a new application that can process your events. For example, if in future you decide to ingest all events into, say, an Elasticsearch to provide real-time analytics, all you'd have to do is add another SQS queue to your existing topic(s) and write a consumer.