We do have a system that is using Redis pub/sub features to communicate between different parts of the system. To keep it simple we used the pub/sub channel to implement different things. On both ends (producer and consumer), we do have Servers containing code that I see no way to convert into Lambda Functions.
We are migrating to AWS and among other changes, we are trying to replace the use of Redis with a managed pub/sub solution. The required solution is fairly simple: a managed broker that allows to publish a message from one node and to subscribe for its reception from 0 or more other nodes.
It seems impossible to achieve this with any of the available solutions:
Kinesis - It is a streaming solution for data ingestion (similar to Apache Pulsar)
SNS - From the documentation, it looks like exactly what we need until we realize that there is no solution to connect a server (not a Lambda) unless with a custom HTTP endpoint.
EventBridge - Same issue as with SNS
SQS - It is a queue, not a pub/sub.
Amazon MQ / Rabbit MQ - It is a queue, not a pub/sub. But also is not a SaaS solution but rather an installation to an owned node.
I see no reason to remove a feature such as subscribing from a server, this is why I was sure it will be present in one or more of the available solutions. But we went through the documentation and attempted to consume fro SNS and EventBridge without success. Are we missing something? How to achieve what we need?
Example
Assume we have an API server layer, deployed on ECS with a load balancer in front. The API has 2 endpoints, a PUT to update a document, and an SSE to listen for updates on documents.
Assuming a simple round-robin load balancer, an update for document1 may occur on node1 where a client may have an ongoing SSE request for the same document on node2. This can be done with a Redis backbone; node1 publishes on document1 topic and node2 is subscribed to the same topic. This solution is fast and efficient (in this case at-most-once delivery is perfectly acceptable).
Being this an example we will not consider WebSocket pub/sub API or other ready-made solutions for this specific use case.
Lambda
Subscriber side can not be a Lambda. Being two distinct contexts involved (the SSE HTTP Request one and the SNS event one) this will cause two distinct lambdas to fire and no way to 'stitch' them together.
SNS + SQS
We hesitate with SQS in conjunction with SNS being a solution that will add a lot of unneeded complexity:
Number of nodes is not known in advance and they scale, requiring an automated system to increase/reduce the number of SQS queues.
Persistence is not required
Additional latency is introduced
Additional infrastructure cost
HTTP Endpoint
This is the closest thing to a programmatic subscription but suffers from similar issues to the SNS-SQS solution:
Number of nodes is unknown requiring endpoint subscriptions to be automatically added.
Eiter we expose one endpoint for each node or have a particular configuration on the Load Balancer to route the message to the appropriate node.
Additional API endpoints must be exposed, maintained, and secured.
Related
I am deciding if I should use MSK (managed kafka from AWS) or a combination of SQS + SNS to achieve a pub sub model?
Background
Currently, we have a micro service architecture but we don't use any messaging service and only use REST apis (dont ask why - related to some 3rd party vendors who designed the architecture). Now, I want to revamp it and start using messaging for communication between micro-services.
Initially, the plan is to start publishing entity events for any other micro service to consume - these events will also be stored in data lake in S3 which will also serve as a base for starting data team.
Later, I want to move certain features from REST to async communication.
Anyway, the main question I have is - should I decide to go with MSK or should I use SQS + SNS for the same? ( I already understand the basic concepts but wanted to understand from fellow community if there are some other pros and cons)?
Thanks in advance
MSK VS SQS+SNS is not really 1:1 comparison. The choice depends on various use cases. Please find out some of specific difference between two
Scalability ->
MSK has better scalability option because of inherent design of partitions that allow parallelism and ordering of message.
SNS has limitation of 300 publish/Second, to achieve same performance as MSK, there need to have higher number of SNS topic for same purpose.
Example : Topic: Order Service
in MSK -> one topic+ 10 Partitions
SNS -> 10 topics
if client/message producer use 10 SNS topic for same purpose, then client needs to have information of all 10 SNS topic and distribution of message.
In MSK, it's pretty straightforward, key needs to send in message and kafka will allocate the partition based on Key value.
Administration/Operation ->
SNS+SQS setup is much simpler compare to MSK.
Operational challenge is much more with MSK( even this is managed service). MSK needs more in depth skills to use optimally.
SNS +SQS VS SQS -> I believe you have multiple subscription(fanout) for same message thats why you have refer SNS +SQS.
If you have One Subscription for one message, then only SQS is also sufficient.
Replay of message -> MSK can be use for replaying the already processed message.
It will be tricky for SQS, though can be achieve by having duplicate queue so that can be use for replay.
How to subscribe to AWS Event Bus events from client side applications ex: NodeJS app, Angular client or a Mobile client app ?
In December 2020, an email from AWS Marketing has presented the advantages to use the Event-Driven architecture . Following the documentation and the tutorials, soon I stumble into the wall of not finding a way to subscribe to this events from a client side application.
The email states:
4 Reasons to Care About Event-Driven Architectures
Are you looking to scale and build robust applications without delays and dependencies? We break down the basics of event-driven architectures, how they work, and show you ways to get started. Learn how event-driven architectures can help you:
Scale and fail independently - No more dependencies
Develop with agility - No custom polling code
Audit with ease - Use your event router to define policies
Cut costs - Stop paying for continuous polling
The disappointing part is that there is no example of libraries to be integrated in the client side code to subscribe to those event. Googling does not return any significant result and the only current library for node: #aws-sdk/client-eventbridge-node only expose a send and destroy methods.
There is no way to directly subscribe to an Amazon EventBridge bus, as it doesn't provide publish/subscribe functionality. In order to process events in EventBridge you create event rules that filter and send matching events to targets. You can find all targets available to EventBridge rules on this list: Amazon EventBridge Targets.
One of these targets could be an Amazon SNS topic, which provides pub/sub functionality, i.e. your client application can subscribe to the topic to automatically receive the respective events.
This may sound complicated at first, but the implementation is strictly following the principle of separating concerns. It provides simple building blocks--like Lego pieces--that you can put together in order to create truly loosely coupled architectures.
This diagram shows the functionality in scope of Amazon Event Bridge and how it communicates with other services and applications.
Services that allow you to subscribe as you want it (directly delivering subscribed messages to your code over a tcp connection such as a websocket) are:
AppSync - websocket
IOT Core - websocket, mqtt
SQS - long poll
Kafka
(From the top of my head)
So a straightforward serverless solution for you could be:
Event bridge —> SQS —> your code
I often use AppSync for this purpose. But eventbridge is cool too.
if you want to avoid polling and you cant/want use AWS Lambda then you can reverse the problem and call an api on your application from EventBridge with a rule.
You can create API Targets in EventBridge:
API destination
I am looking to replace hand rolled service, which reads messages of a queue and then sends them to outside endpoints via HTTP (basically outgoing webhooks).
I have been looking into SNS, but it feels kind of like trying to fit square peg into a round hole.
I think I could pull it off rolling out my own HTTP sender in Lambda and marrying it with SQS.
But is there any SaaS product in AWS that does it for me without need for custom code?
Like said in comments, there is no "turnkey" solution without need for a bit of coding.
Depending on the types of bandwidth / responsiveness / charge your application requires, i would go with one of these two approaches
SQS with Lambda : scalability from 0 to n virtual servers (no activity = no server = no $)
ELB Worker tiers : scalability from 1 to n virtual servers
SQS with Lambda
An SQS queue attached to a Lambda function seems a simple solution to me.
See https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html
One advantage is that you may log useful infos from your lambda function in addition to the http call to your outside endpoints.
With the use of a framework like serverless, it may be straightforward to setup.
See https://serverless.com/blog/aws-lambda-sqs-serverless-integration/
ELB Worker with SQS Daemon
You may also take a look at Elastic Beanstalk Worker environments. There is a turnkey worker environment with SQS daemon included.
See https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-daemon
I am using firebase to notify web browsers (javascript clients) about changes on specific topics. I am very happy with it. However I would really like to (only) use aws web services.
Unfortunately I am not able to determine whether it is possible to build such a service on aws. I am not talking about having EC2 instances running some firebase / fanout.io alternatives. I am talking about utilizing services such as lambda, dynamodb streams, SNS & SQS.
Are there any socket notification services available or is it possible to achieve something similar by using the provided services?
I looked into this very recently with the same idea, but eventually I came down on just using fanout. AWS does not provide server-push HTTP notification services out of the box.
Lambda functions are billed per 100 ms, so any long-polling against lambda will end up billing for the entirety of the time the client is connected.
SNS does not provide long polling to browsers; the available clients are geared towards mobile, email, HTTP/S, and other Amazon products like Lambda and SQS.
SQS would require a dedicated queue per client as it does not support broadcast.
Now, if the lambda pricing doesn't bother you, you could possibly do this:
Write a lambda function that is called via the API service that opens up a connection to SQS and waits for a message. The key is to start the lambda call from HTTP, but within the function wait on the queue (using Boto, for example, if you are writing this in Python). This code would need to create a queue dedicated to servicing one particular client, uniquely keyed by something like a GUID that is passed in by the client.
Link to the lambda function using the Amazon API service.
Call the lambda function via the API from the browser and wait for it to either receive a message on the dedicated SQS queue or timeout, probably using long-polling both in the API connection and the SQS connection. Fully draining the queue (or at least taking as many messages in a batch up to some limit) would be advisable here as well in order to reduce the number of calls to the API.
Publish your event to the dedicated SQS queue associated with the client. This will require the publisher to know the client's unique key.
Return the event read from SQS as the result of the lambda call.
Some problems with this approach:
Lambda pricing - not terribly expensive, but something like fanout is basically free
You would need a dedicated SQS queue per client; cleanup might become a problem
SQS bills on number of calls, which includes checking for a message. Long-polling SQS will alleviate some of this
You would need to write the JavaScript client to call the lambda API endpoint repeatedly in a long-polling fashion
Lambda is currently limited as to the number of concurrently running functions it supports (100 right now but you can contact support to bump that up)
Some benefits with this approach:
SQS queues are persistent, so unless a message is processed successfully it will go back on the queue after the visibility timeout
You can set up CloudWatch to monitor all of the API, Lambda, and SQS events
Other Notes
You could call the SQS APIs directly from the browser by using Lambda to issue temporary security credentials via STS. Receiving a message in JavaScript is documented here: http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-examples.html#Receiving_a_message I do not, however, know off the top of my head if you would run into cross-domain issues.
Your only other option, if it must be all AWS, is to use load-balanced EC2 instances running something like fanout as you mentioned.
Using fanout is very little work: it's both extremely affordable and already built and tested.
My project requires me to communicate with many devices outside the cloud. If successful, this means millions of devices. These devices will not be running Android or iOS, and will be running behind routers & firewalls (I cannot assume they have an external IP).
I am looking to use SQS to send messages to my users outside the cloud. To allow the servers to message individual users, I am designing the system to have one queue per client. This can potentially mean millions (billions?) of queues. While it states that SQS can support unlimited queues, I would like to make sure that I am not abusing the system. If successful, the probability of millions of users is very high.
I am aware that SQS can be expensive, but I am using it at this stage
for ease of administration.
As far as I can tell SNS requires either an IOS/Android client, or an
HTTP server running on the consumer. This is why I ruled out SNS, and I
am using SQS.
I am going to build a distributed cloud front-end over SQS to handle
client connections. This front-end will just be a wrapper, that will
authenticate clients, and relay them to the SQS queues.
Am I abusing the SQS "unlimited queues" policy (will SQS performance drop)? Is there a simpler design for per device messaging?
Let me break the answer by the parts of your question:
About your questions:
Am I abusing the SQS "unlimited queues" policy?
AWS services are designed to prevent abuse and you will pay exactly for what you use, so if you believe this is the right approach, go for it. To remove the uncertainty, i'd advise for a preliminary "proof of concept" implementation.
Is there a simpler design for per device messaging?
Probably yes, re-consider SNS and other messaging systems.
About your statements:
I am aware that SQS can be expensive, but I am using it at this stage
for ease of administration.
"Expensive" is a very context-depend classification, considering that a SQS message can cost $0.00000005.
As far as I can tell SNS requires either an IOS/Android client, or an
HTTP server running on the consumer. This is why I ruled out SNS, and
I am using SQS.
SNS is a push based messaging system (SQS is pull based) that can handle 5 types of subscriptions: smtp, sms, http, mobile push and SQS, so they are not mutually exclusive.
I am going to build a distributed cloud front-end over SQS to handle
client connections. This front-end will just be a wrapper, that will
authenticate clients, and relay them to the SQS queues.
Managing millions of queues can be a overwhelming task for your "distributed cloud front-end over SQS". Unless your project is exactly about queue management, this is probably undifferentiated heavy lifting.
This is about all i can say without knowing your case, but consider that you can use SNS/SQS together with each other and with other messaging software, such as Apache Camel and others, that may help you build your solution or proof of concept.
I think SQS (or SNS if you can eventually use them) are still your best bet, esp if you need "quick response" or "near real time"; however, for the sake of having "alternatives" just so you can compare...
You can consider a giant dynamoDB, with each device/client having it's own "device-id" and perhaps "message-id" as key. This way, your devices can query it's own keys for messages. DynamoDB is meant to handle billions of rows, so this won't stress it much. The querying part, you should be careful though, as you could use up provisioned queries, although at aggregate level, your devices may not all respond/query at the same time, so you may still be ok.
You can also consider having a giant S3 bucket, each folder key'ed to the device id and further keyed into message-id folders. This is a poor man's SQS but it's guaranteed to scale, both in message quantity and number of accesses to it.
In both #1 and #2, if your devices are registered with Cognito for credentials, there's a straightforward way to do policies, so the devices can only access their "own" stuff. However, both alternatives #1 and #2 is likely slower than SQS, esp if you use SQS long-poll -- in long poll, you get responses, as soon as SQS detects a message have been dropped off... These alternatives will require you to wait for next cycle-poll.