We have 3 environments test, QA and production. Each with it own account.
This is what I am trying to accomplish. when a user uploads a new solr feed to AWS3 I want to send a SNS.
The queues in test, QA and production are subscribing to the SNS topic.
In this way I can have 3 services running and keeping my solr instances in sync.
It's super easy to setup AWS3 to send a SNS on a created event and also very easy for a SQS to subscribe to a SNS message.
But how do I subscribe to a SNS topic across all environments? / accounts.
If I understand your ask correctly, you'd like to subscribe SQS queues in different AWS accounts to one SNS topic. If so, that is totally possible, documentation for which can be found here.
Related
I am having an SNS topic name(Dev-Alert), I have created subscription to the topic and is receiving email, but i need to find out what services are publishing message to that topic. How can we back track it?
There is no 'breadcrumb' from an Amazon SNS topic pointing back to publishers.
In fact, a message can be published to an SNS topic with just a couple of lines of code from anywhere on the Internet (not just AWS services).
You will need to scan through the services themselves, looking for which ones are publishing to SNS rather than attempting to back-track.
I have industrial internet of things (IIoT) data coming from machines out in the wilderness. Currently the data are going from the machines to an Amazon Web Services (AWS) Simple Queueing Service (SQS) queue, for consumption by a third-party company.
I want to do two things now:
Put the data into the SQS queues for the third-party company, as before, and
Store the data in an Amazon Relational Database Service (RDS) database for analysis purposes
To accomplish this, should I send the data to an Amazon Simple Notification Service (SNS) topic first, and then push the data from there to both my database, and an SQS queue for the third-party company?
Is there a better way of accomplishing my objective, or am I on the right track?
I ask partly because it seems that Amazon is steering me toward a noSQL database called DynamoDB, but I think I want a structured relational database for my IIoT data.
Amazon’s IoT “message broker”, which communicates with my IoT devices using MQTT, is an SNS topic, which makes a lot of sense.
SNS message topics are different than SQS queues. Queues can only be read by one user, and then the message(s) get deleted from the queue—it’s just temporary storage. So we couldn’t share an SQS queue with the third-party company.
SNS topics, by contrast, send push notifications (and the messages) to multiple users instantly, and all “subscribed” users get the message(s). SNS is exactly like subscribing someone’s Twitter or Instagram feed, where all “subscribers” get notified of new messages from “publishers” to the message “topic”.
Long story short, we can use AWS IoT “rules” to send our devices' MQTT-secured “messages” to as many “subscribers” or AWS services as we want. So we’ll send one copy to the SQS queues, and another copy to an AWS database of some sort.
I want to build a pub/sub messaging system into my services that are hosted on Amazon Web Services, and creating SQS queues that subscribe to SNS topics seems like the obvious direction to take, but I can't get it working at all.
So far my code looks for the topics and the queues at startup and creates anything that's missing. This seems to work, I can see the SNS topic and the SQS queues in the AWS management console, and I can see that the queue is subscribed to the topic, but when I publish messages to the topic nothing ends up in the queue.
Even if I manually publish a message using the 'Publish' button in the management console the queue is still empty.
I changed the permissions on both the topic and the queue to 'everyone can do everything' just to eliminate this possibility. My code receives a message ID in response to the publish and there are no errors, every API call returns a 200 (OK) status.
Where can I go from here to figure out why it's not working?
The SNS --> SQS link has a few gotchas:
"Amazon SNS isn't currently compatible with FIFO queues." per the note on their Subscribing an Amazon SQS Queue to an Amazon SNS Topic Tutorial
You have to fiddle with the IAM permissions (see the page on Sending Amazon SNS Messages to Amazon SQS Queues)
You can't send messages to encrypted queues (see their Server-Side Encryption page)
It would definitely have been easier to figure this out if all this info were consolidated into a single page. The killer for me was #3 - perhaps one of these will be the solution to your issue.
A couple of options -
Enable CloudTrail and monitor the logs
View the CloudWatch logs to identify any permissions issues
Open a ticket with AWS support.
Ideally, you wouldn't be creating the resources in your application but instead decouple those into CloudFormation or at a minimum CLI scripts. If you require the ability to dynamically create these resources, using the AWS IoT Message Broker may be a better option since it supports ephemeral messaging resources - http://docs.aws.amazon.com/iot/latest/developerguide/iot-message-broker.html
I had a similar issue with SQS subscriptions. It turned out that if I create the subscription from the SQS editor it works, but if I create it from the SNS creation screen it accepts the message but never forwards it to the queue.
To get more detailed information about specific failures you can turn on
SNS "Delivery status logging".
We have got a strange requirement and we would like to send SMS to our clients based on the assets they are monitoring. Each asset can have 100s of subscribers and there are 1000s of assets so obviously, we can not create one SNS topic per asset. We have the assets and their list of subscribers in a RDS instance on AWS.
Is there anyway with SNS to make the list of its subscribers dynamic, each time we publish a message to it we also supply the list of subscriber this message should be sent to? What are my other options or another AWS service? Lambda maybe? Please advise. thanks
I finally used SNS for this purpose. Each time I need to sent an alert, I call a lambda and supple lambda with the list of subscribers and the message for that asset. Lambda will go ahead, create a new topic, add the subscribers to it, publish the message to it and when everything is done, removes the topic. Works great.
AWS SNS for sending sms had limitation like only for US or North America region. This point should be double checked before making such choice. If your users only from USA - so quite good. In other cases I would recommend to take a look on https://www.nexmo.com/, clickatell.com or Twilio. They provide nice API and can be managed very easy.
Is it possible to auto send/push the messages in Amazon SQS to DynamoDB? I wish to send my messages to SQS and for period of time I want to send this to DynamoDB. Another service should fetch the DynamoDB table and send it as email using SES.
Kindly help me out to achieve this. I will be using it for the User notification purpose from a Social networking site.
Thanks.
There is no AWS mechanism to automatically publish SQS messages to DynamoDB; but you can use an AWS Lambda event source mapping to automatically pull SQS messages and invoke a Lambda function, and it's pretty straightforward to write a Lambda function that writes those messages to DynamoDB. (Here's an example using Node.js: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/dynamodb-example-table-read-write.html.)
Yes I agree Hyangelo, you can do this with Simple Workflow Service (SWF).
SWF will give you a control feature over your application enabling you to distribute and execute different services or tasks when you want.
Here is the link to the documentation: http://aws.amazon.com/swf/
Sounds like a workflow system from how you describe what you want, have you considered Simple Workflow Service?
SQS can't be processed w/o pulling messages.
You can either use SWF to solve your use-case OR use SNS.
SNS<=>SQS binding is free by AWS.
Send your messages to SNS, bind your SNS with SQS & lambda-function.
On triggering lambda function - you can create dynamodb-record and send it to another SNS2.
Bind SNS2 <=> SES which will trigger the email.
checkout: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sns-ses-dynamodb/