I want to build a pub/sub messaging system into my services that are hosted on Amazon Web Services, and creating SQS queues that subscribe to SNS topics seems like the obvious direction to take, but I can't get it working at all.
So far my code looks for the topics and the queues at startup and creates anything that's missing. This seems to work, I can see the SNS topic and the SQS queues in the AWS management console, and I can see that the queue is subscribed to the topic, but when I publish messages to the topic nothing ends up in the queue.
Even if I manually publish a message using the 'Publish' button in the management console the queue is still empty.
I changed the permissions on both the topic and the queue to 'everyone can do everything' just to eliminate this possibility. My code receives a message ID in response to the publish and there are no errors, every API call returns a 200 (OK) status.
Where can I go from here to figure out why it's not working?
The SNS --> SQS link has a few gotchas:
"Amazon SNS isn't currently compatible with FIFO queues." per the note on their Subscribing an Amazon SQS Queue to an Amazon SNS Topic Tutorial
You have to fiddle with the IAM permissions (see the page on Sending Amazon SNS Messages to Amazon SQS Queues)
You can't send messages to encrypted queues (see their Server-Side Encryption page)
It would definitely have been easier to figure this out if all this info were consolidated into a single page. The killer for me was #3 - perhaps one of these will be the solution to your issue.
A couple of options -
Enable CloudTrail and monitor the logs
View the CloudWatch logs to identify any permissions issues
Open a ticket with AWS support.
Ideally, you wouldn't be creating the resources in your application but instead decouple those into CloudFormation or at a minimum CLI scripts. If you require the ability to dynamically create these resources, using the AWS IoT Message Broker may be a better option since it supports ephemeral messaging resources - http://docs.aws.amazon.com/iot/latest/developerguide/iot-message-broker.html
I had a similar issue with SQS subscriptions. It turned out that if I create the subscription from the SQS editor it works, but if I create it from the SNS creation screen it accepts the message but never forwards it to the queue.
To get more detailed information about specific failures you can turn on
SNS "Delivery status logging".
Related
2 weeks back, I saw a notification in AWS saying, the one of the 3 AWS EC2 instance will be stopped because of the underlying hardware cause. I almost missed the dead line as I didn't the notification via email. Is there a way get these notification via email?
It would be much better if I can get the notifications using cloud watch to my email.
If you're on Business or Enterprise Support, AWS provides a solution named "AWS Health Aware" built on top of AWS Health API, it does exactly what you're after sends you notifications via several methods including emails regarding outages and planning maintenance that affect your account(s).
If you're on a Basic support plan you can easily built something similar yourself using based on [AWS Health events][2]
This is an example of capturing AWS Health events via EventBridge so they can be further send to a lambda or SNS
https://docs.aws.amazon.com/health/latest/ug/cloudwatch-events-health.html
What I've done in the is following:
Capturing AWS Health events via EventBridge in spoke accounts and forward them to the event bus in my hub account
In the hub account I would subscribe a lambda function for all events with source "aws.health" and then send email or slack message whatever works for you.
I think Amazon EventBridge is what you are looking for. With EventBridge you can detect and react to different events such as EC2 instances scheduled for updates. You can, for example, create a rule for detecting a specific health event and in the target section choose to receive an email through SNS or invoke a CloudWatch log group, etc. Read more here.
On AWS I'm trying to create a cloudwatch alarm that sends a notification to a sns topic that in turn triggers a lambda function that is subscribed to the topic, but have run into a bit of a snag in which my lambda is never triggered. I've followed several tutorials on how to do this, I've manually tested the sns -lambda connection and this is working as is attaching the notification trigger onto cloudwatch.
I've done this all through cloudformation.
I have also tried to configure it manually from the cloudwatch console and then noticed something strange. Underneath the Send a notification to select box as shown in the image below, there is a message which says Only email lists for this account are available. So I'm guessing from that, that somewhere in this account, there is a weird setting that needs to be changed?
Updated answer:
Is the CloudWatch alarm actually in the "Alarm" state? Can you add 2 more notifications to be triggered by the "OK" and "Insufficient data" state respectively?
by reading the original post, SNS should have the right permission to invoke the Lambda function.
previous answer:
That is a general message, it has nothing to do with your SNS topic settings. I reckon it's meant to say only emails opt-in to that SNS topic will get emails.
Q: How does Amazon SNS validate a subscription request to ensure that notifications will not be sent to users as spam?
As part of the subscription registration, Amazon SNS will ensure that notifications are only sent to valid, registered subscribers/end-points. To prevent spam and ensure that a subscriber end-point is really interested in receiving notifications from a particular topic, Amazon SNS requires an explicit opt-in from subscribers using a 2-part handshake:
i. When a user first calls the Subscribe API and subscribes an end-point, Amazon SNS will send a confirmation message to the specified end-point.
ii. On receiving the confirmation message at the end-point, the subscriber should confirm the subscription request by sending a valid response. Only then will Amazon SNS consider the subscription request to be valid. If there is no response to the challenge, Amazon SNS will not send any notifications to that end-point. The exact mechanism of confirming the subscription varies by the transport protocol selected:
For HTTP/HTTPS notifications, Amazon SNS will first POST the confirmation message (containing a token) to the specified URL. The application monitoring the URL will have to call the ConfirmSubscription API with the token included token.
For Email and Email-JSON notifications, Amazon SNS will send an email to the specified address containing an embedded link. The user will need to click on the embedded link to confirm the subscription request.
For SQS notifications, Amazon SNS will enqueue a challenge message containing a token to the specified queue. The application monitoring the queue will have to call the ConfirmSubscription API with the token.
Note: The explicit “opt-in” steps described above are not required for the specific case where you subscribe your Amazon SQS queue to your Amazon SNS topic – and both are “owned” by the same AWS account.
We have 3 environments test, QA and production. Each with it own account.
This is what I am trying to accomplish. when a user uploads a new solr feed to AWS3 I want to send a SNS.
The queues in test, QA and production are subscribing to the SNS topic.
In this way I can have 3 services running and keeping my solr instances in sync.
It's super easy to setup AWS3 to send a SNS on a created event and also very easy for a SQS to subscribe to a SNS message.
But how do I subscribe to a SNS topic across all environments? / accounts.
If I understand your ask correctly, you'd like to subscribe SQS queues in different AWS accounts to one SNS topic. If so, that is totally possible, documentation for which can be found here.
I know that the AWS sends emails about any EC2 instances with Scheduled Events, but I would also like to have these notifications sent through HipChat and text message as well. Is there a default way of selecting a setting that can do this in AWS? Or would I have to create a more custom solution to achieve this?
With Scheduled Events in EC2 being manual tool for monitoring health of instances as of right now, there isn't many options. From the design point of view on AWS platform, this could be one possible solution:
Create AWS SES mailbox and enable it to receive emails,
Create a forwarding rule in your original e-mail inbox where you receive those e-mail notifications from AWS about Scheduled Events to forward e-mails of this type (based on filter you define) to the previously created AWS SES mailbox,
In your AWS SES mailbox create "Receipt Rule" with SNS Action, that will basically publish entire content of the e-mail to the SNS Topic using AWS SNS. (The SNS topic you choose must be in the same AWS region as the Amazon SES endpoint you use to receive email)
Once that e-mail content lands in AWS SNS Topic as an event, you can use AWS Lambda subscribed to your SNS topic as a handler of the event, parse what you need and forward customized notification to any of your third-party (outside of AWS) alerting tools (such as PagerDuty, Slack channel, HipChat, or whatever you use for SMS notifications).
This design does have one week point: relying onto your external mailbox forwarding system (if you are receiving Scheduled Events e-mails in inbox not based on AWS SES).
Ideally you could try to see in your AWS Account settings if it is possible to use AWS SES mailbox for Scheduled Events notifications, but I haven't tried that.
You can receive text messages as notification by setting alarm for that ec2 instance e.g if CPU usages goes above 80% then create an alarm and eventually you can set it to get emails or text messages. Thanks
Is it possible to auto send/push the messages in Amazon SQS to DynamoDB? I wish to send my messages to SQS and for period of time I want to send this to DynamoDB. Another service should fetch the DynamoDB table and send it as email using SES.
Kindly help me out to achieve this. I will be using it for the User notification purpose from a Social networking site.
Thanks.
There is no AWS mechanism to automatically publish SQS messages to DynamoDB; but you can use an AWS Lambda event source mapping to automatically pull SQS messages and invoke a Lambda function, and it's pretty straightforward to write a Lambda function that writes those messages to DynamoDB. (Here's an example using Node.js: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/dynamodb-example-table-read-write.html.)
Yes I agree Hyangelo, you can do this with Simple Workflow Service (SWF).
SWF will give you a control feature over your application enabling you to distribute and execute different services or tasks when you want.
Here is the link to the documentation: http://aws.amazon.com/swf/
Sounds like a workflow system from how you describe what you want, have you considered Simple Workflow Service?
SQS can't be processed w/o pulling messages.
You can either use SWF to solve your use-case OR use SNS.
SNS<=>SQS binding is free by AWS.
Send your messages to SNS, bind your SNS with SQS & lambda-function.
On triggering lambda function - you can create dynamodb-record and send it to another SNS2.
Bind SNS2 <=> SES which will trigger the email.
checkout: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sns-ses-dynamodb/