I want to use a native AWS queuing mechanism to implement a pub/sub process where there one publisher and multiple consumers, but for each message, there is only a single possible receiver. Is this possible using SQS or SNS, or do these always go to all consumers? Would I need to go to Amazon-MQ? Or maybe a combination of SNS and an ELB?
Thanks,
Stan
The default (simple) implementation sens all SNS messages to all subscribers (in your case SQS).
Since I don't know what exactly is your case, I would say this.
If the message content determines which subscriber should get the message, then use SNS filters.
If there is some logic or rules that need to be built, use MQ.
Related
I am currently building a microservices based backend for my E-Commerce Setup
I need to push all the transactions to a Queue Service, but AWS documentation says that I should publish my message to SNS and then subscribe my queue to a topic
but in SQS documentation there is also a way to send message directly to SQS
*PS: I have already searched stackoverflow but none of the question answers for my specific use case
so why are there two solutions for the same thing and need to use SNS and pay extra money
One reason for that is that you can very easily scale your architecture if you publish to SNS first. This is due to being able to implement fanout scenario:
Now you may only need your msg in a single SQS queue. But later you may want to add second one to process same messages, also maybe invoke some lambda functions independently from your queue, or send them as well to some HTTP endpoint.
I have this architectural dilemma, and thought, maybe this is solved problem, or solvable multiple ways. I've got an SQS queue (one of many), which is triggering a Lambda function (one of many). This queue requires slightly different processing of messages, based on one key in the payload. What would be the best way for sorting out the queue messages before actual processing by Lambda?
Should it be separate Lambda, which will check for the key, and then place the message into separate queue, which will trigger corresponding Lambda?
Should it be just bunch of if statements in primary Lambda?
Is there maybe automated way to deal with such situation?
Thanks!
It appears that your situation is:
An external process sends messages to an Amazon SNS topic
An Amazon SQS queue is subscribed to the SNS topic
An AWS Lambda function is subscribed to the SQS queue
There are various sub-types of messages, each of which should be processed slightly differently
One option is to use Amazon SNS Message Filtering, which can deliver messages differently based upon a message attribute. It could, for example, send a subset of messages to an SNS topic or Lambda function. However, this would require the messages being sent to the SNS topic to have a message attribute defined.
If this is not the case, then possible options are:
Use one Lambda function to process all message types (You can include quite complex code in multiple files within a Lambda function!), or
Use a Lambda function to determine the message sub-type and then send it to a specific Lambda function (or, send it via an SQS queue to a Lambda function for added resiliency)
guys need small help, I have a use case, where I want to set up a communication service.
using SQS, SQs is going to receive a different type of events to be communicated. Now we have a single lambda function which does a single communication. let's say one email Lambda, Slack lambda, etc.
how I can invoke different lambda based on queue attributes. I was planning to use SQS as an event source and something kind of this architecture link to sample architeture
here in the above, we can handle rate limiting and concurrency at the lambda service level
simplified works if event type is A invoke Lambda A if the event type is B invoke a lambda B
and both events are in same SQS
all suggestions are welcome
Your problem is a SQS message can only be read by one service at a time. When it is being read, it is invisible to anyone else. You can only have one Lambda consumer and there isn't any partitioning or routing in SQS besides setting up another SQS topic. Multiple consumers are implemented Kensis or AWS MSK (Kafka)
What you are trying to accomplish is called a fan out. This is a common cloud architecture. What you probably want to do is publish initially to SNS. Then with SNS you can filter and route to multiple SQS topics for each of the message types and each SQS topic would then be consumed by it's own Lambda.
Check out a tutorial here:
https://docs.aws.amazon.com/sns/latest/dg/sns-common-scenarios.html
I'm using Amazon SQS for my application in a producer/consumer context. I want to enable queue level logging where I can see items put on the queue and removed from it later. How can I do that?
I have read the following:
http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/logging-using-cloudtrail.html
However, that doesn't suffice for my use case. Are we not allowed to do this with AWS queues?
What you're trying to achieve is not possible with just SQS. Possible solutions include:
Implement some middleware API between you producer and SQS queue. API level would log requests from producer.
Use Kinesis instead of SQS. Kinesis allows you to replay/analyze records created in last 24 hours.
Implement logging in consumer.
Use Lambda function that will (with help of CloudWatch Event Rule triggers) read SQS queue once a minute, log records and put them in another SQS queue for later processing by consumer.
Use different type of queue that allows logging. For example, Redis has MONITOR command for that.
In addition to Sergey Kovalev answer, one now has the option for Lambda functions to be triggered by SQS events.
You simply:
select the SQS queue you want as the event source for your Lambda function
I understand your pain. Even I had the issue where SQS was not behaving as expected and I was looking for logs to understand the problem.
SQS don't publish logs, all SQS APIs are synchronous so the client get the appropriate response.
The solutions mentioned above are the workarounds
Among them having Loggin at produce and consumer might not help much. Because in my case I did had logging at produce and consumer, but still what exactly SQS ran into and when will not be visible.
In RabbitMQ, one can create an exchange, then bind it to multiple queues, each with a routing key. This enables messaging architectures like this:
message_x
/ | \
foo-msg_q bar-msg_q msg-logger_q
Clients publish messages to the message_x exchange, which routes only messages with routing key "foo" to the foo-msg_q queue, only messages with the routing key "bar" to the bar-msg_q queue, and all messages to msg-logger_q queue.
I'm having trouble figuring out how to do this in AWS. My first thought was to set up permissions on the individual queues to accept messages based on subject, but the only available fields for permission conditions are:
aws:CurrentTime
aws:EpochTime
aws:MultiFactorAuthAge
aws:principaltype
aws:SecureTransport
aws:SourceArn
aws:SourceIp
aws:UserAgent
aws:userid
aws:username
None of these seem like they can be influenced by any message I publish to the message_x topic.
Is it possible to doing something like this when using Amazon Simple Notification Service to fan out to multiple Simple Queue Service queues, with each queue receiving a subset of messages published to the topic?
This is possible by using message attribute filtering in SNS. After you subscribe different SQS queues to an SNS topic, you can specify attributes to filter on by using the SNS API SetSubscriptionAttributes. This will allow messages with different attributes to get routed to the correct SQS queue.
This is also not limited to SQS queues but any subscription sources on a SNS topic. For example, a single SNS topic can publishing one set of messages to Lambda, and another set to SQS.
SDK Reference:
http://docs.aws.amazon.com/sns/latest/api/API_SetSubscriptionAttributes.html
More details are given here with examples:
https://aws.amazon.com/blogs/compute/simplify-pubsub-messaging-with-amazon-sns-message-filtering/
EDIT
I cannot delete an accepted answer, so see the answer below for the now correct answer since this feature has been released.
Original (now incorrect) Answer (for posterity):
No it's not possible. Would be a great feature for them to add
though.
The only alternative I know is to create a topic for each routing rule
and then publish to the correct topic. It's not pretty, but it
accomplishes the task. If you have a lot of rules, you might need
more than the 3000 topics they allow. You can request an increase in
topic limit from AWS on their website by following the instructions
here
http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ses_quota.
AWS now supports filtering on SNS subscribers. Each subsriber can set its policy for filtering messages that it needs and discard others. If you don't set any policy on a subscriber it will get all messages. refer below
https://aws.amazon.com/getting-started/hands-on/filter-messages-published-to-topics/