I have a process which publish some data(json) onto a queue on Aws-Sqs. Another process reads from this queue. All this is working fine.
Now I want to create a topic which can be listened by mutiple processes and the data is delivered to all the processes. For example Activemq and many other messaging servers have this capability to create a topic. I could not find any such thing on AWS. The closest I could find is AWS SNS.
From what I understand AWS-SNS allows multiple clients to subscribe to a topic. But the type of subscription is either Email, Http, or Sms and so on ... This does not really serve my purpose. I want to recieve json data in all my clients just like Sqs.
Is that achievable? If so how?
You can subscribe multiple SQS into single SNS topic: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-subscribe-queue-sns-topic.html
Then it will be distributed to all of them.
The other option is to use Kinesis - https://aws.amazon.com/kinesis/ but it is more difficult to set up. There you can also read from multiple clients from the stream.
amazon mq is a managed active mq service. maybe this will help with your needs?
Related
I have 9000 AWS SNS topics with more than 1M subscribers in each topic. At the moment I am lopping to each topic to send a push message which is consuming lot of my system resources. Is there a way to send message to all the topics at once? what is the best approach to handle the scenario?
It is not possible to subscribe Amazon SNS queues to an Amazon SNS queue, so there is no out-of-the-box method for sending one message to multiple queues.
I would recommend creating an AWS Lambda function that will:
Retrieve a list of all relevant queues (based on tag?)
Loops through and sends a message to each queue
Thus, you would just trigger the Lambda function with one message and it would go to all other queues. It would not "consume system resources", but it is charged based upon run duration. Lambda functions can run for a maximum of 15 minutes, so as long as it sends 10+ messages per minute, it can send to 9000 topics.
Depending upon your use-case, you might also consider using Amazon Pinpoint:
Amazon Pinpoint is an AWS service that you can use to engage with your customers across multiple messaging channels. You can use Amazon Pinpoint to send push notifications, emails, SMS text messages, and voice messages.
I'd like to use AWS IoT to manage a grid of devices. Data by device must be sent to a queue service (RabbitMQ) hosted on an EC2 instance that is the starting point for a real time control application. I read how to make a rule to write data to other Service: Here
However there isn't an example for EC2. Using the AWS IoT service, how can I connect to a service on EC2?
Edit:
I have a real time application developed with storm that consume data from RabbitMQ and puts the result of computation in another RabbitMQ queue. RabbitMQ and storm are on EC2. I have devices producing data and connected to IoT. Data produced by devices must be redirected to the queue on EC2 that is the starting point of my application.
I'm sorry if I was not clear.
The AWS IoT supports pushing the data directly to other AWS services. As you have probably figured out by now publishing to third party APIs isn't directly supported.
From the choices AWS offers Lambda, SQS, SNS and Kinesis would probably work best for you.
With Lambda you could directly forward the incoming message using the one of Rabbit MQs APIs.
With SQS you would put it into an AWS queue first and than poll this queue transfering it to RabbitMQ.
Kinesis would allow more sophisticated processing, but is probably too complex.
I suggest you program a Lamba with the programming language of your choice using one of the numerous RabbitMQ APIs.
I have an external data source as an ActiveMQ topic. I can only connect and consume messages. They come pretty rarely, about 1 message per 10-30 seconds.
I want to collect all the messages and put them into the database.
Also I'd like to have an active web page that can receive the new messages over WebSockets and draw a chart.
I have a prototype built with Python/Flask/MongoDB/SocketIO, BUT...
I would like to use Amazon AWS cloud infrastructure to avoid processing the data on servers.
I believe that AWS Lambda can accept the messages and store them into the database (DynamoDB?) and also send a notification (maybe using SQS) being transformed into WebSocket message. (Not everything is clear there yet, maybe simple ajax polling will be enough).
Here is a question: how it would be possible to consume the messages from external ActiveMQ topic and process it with AWS Lambda?
I was looking on Kinesis, but it looks it only supports the data being pushed to it, but not polling for the data by some protocol...
You can use Lambda as a cron-like facility and poll on a schedule. See Lambda Scheduled Events.
In RabbitMQ, one can create an exchange, then bind it to multiple queues, each with a routing key. This enables messaging architectures like this:
message_x
/ | \
foo-msg_q bar-msg_q msg-logger_q
Clients publish messages to the message_x exchange, which routes only messages with routing key "foo" to the foo-msg_q queue, only messages with the routing key "bar" to the bar-msg_q queue, and all messages to msg-logger_q queue.
I'm having trouble figuring out how to do this in AWS. My first thought was to set up permissions on the individual queues to accept messages based on subject, but the only available fields for permission conditions are:
aws:CurrentTime
aws:EpochTime
aws:MultiFactorAuthAge
aws:principaltype
aws:SecureTransport
aws:SourceArn
aws:SourceIp
aws:UserAgent
aws:userid
aws:username
None of these seem like they can be influenced by any message I publish to the message_x topic.
Is it possible to doing something like this when using Amazon Simple Notification Service to fan out to multiple Simple Queue Service queues, with each queue receiving a subset of messages published to the topic?
This is possible by using message attribute filtering in SNS. After you subscribe different SQS queues to an SNS topic, you can specify attributes to filter on by using the SNS API SetSubscriptionAttributes. This will allow messages with different attributes to get routed to the correct SQS queue.
This is also not limited to SQS queues but any subscription sources on a SNS topic. For example, a single SNS topic can publishing one set of messages to Lambda, and another set to SQS.
SDK Reference:
http://docs.aws.amazon.com/sns/latest/api/API_SetSubscriptionAttributes.html
More details are given here with examples:
https://aws.amazon.com/blogs/compute/simplify-pubsub-messaging-with-amazon-sns-message-filtering/
EDIT
I cannot delete an accepted answer, so see the answer below for the now correct answer since this feature has been released.
Original (now incorrect) Answer (for posterity):
No it's not possible. Would be a great feature for them to add
though.
The only alternative I know is to create a topic for each routing rule
and then publish to the correct topic. It's not pretty, but it
accomplishes the task. If you have a lot of rules, you might need
more than the 3000 topics they allow. You can request an increase in
topic limit from AWS on their website by following the instructions
here
http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_ses_quota.
AWS now supports filtering on SNS subscribers. Each subsriber can set its policy for filtering messages that it needs and discard others. If you don't set any policy on a subscriber it will get all messages. refer below
https://aws.amazon.com/getting-started/hands-on/filter-messages-published-to-topics/
Im writing an app using the AWS framework. It would be the perfect framework if the SQS had FIFO access!
I am using SNS to publish messages to queues, and the order of the messages is important. Does anyone know of a MQ that can be used, that is scalable etc.. like SQS that can be used?
I have been looking at using ironMQ but the SNS seems to have issues with receiving the messages?
(or even better yet,if you know of a good way of ordering messages with SQS)
Thanks,
Ben
These days, Amazon Kinesis would be a good choice. (It wasn't available when the question was raised.)
Kinesis captures data in streams, retains order and allows the data to be replayed and even consumed by multiple applications.