What's the easiest way to save/log every message published on a AWS SNS topic? I thought there might be a magic setting to automatically push them to S3 or a database, or maybe a database service supporting the HTTP destination automatically, but doesn't seem to be the case. Maybe it needs to be done via a Lambda function?
The purpose is just for basic diagnostics and debugging while setting up some SNS publishing. I don't really care about high scale or fast querying, just want to log and perform basic queries on all the activity for a few minutes at a time.
You can setup a trigger to push your SNS messages to SQS queue. Push is automatic and does not require any code.
According to the docs, SNS can publish to:
http – delivery of JSON-encoded message via HTTP POST
https – delivery of JSON-encoded message via HTTPS POST
email – delivery of message via SMTP
email-json – delivery of JSON-encoded message via SMTP
sms – delivery of message via SMS
sqs – delivery of JSON-encoded message to an Amazon SQS queue
application – delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda – delivery of JSON-encoded message to an AWS Lambda function.
So yes, a simple approach would be to trigger a lambda function to write to S3 or CloudWatch.
Related
On AWS I'm trying to create a cloudwatch alarm that sends a notification to a sns topic that in turn triggers a lambda function that is subscribed to the topic, but have run into a bit of a snag in which my lambda is never triggered. I've followed several tutorials on how to do this, I've manually tested the sns -lambda connection and this is working as is attaching the notification trigger onto cloudwatch.
I've done this all through cloudformation.
I have also tried to configure it manually from the cloudwatch console and then noticed something strange. Underneath the Send a notification to select box as shown in the image below, there is a message which says Only email lists for this account are available. So I'm guessing from that, that somewhere in this account, there is a weird setting that needs to be changed?
Updated answer:
Is the CloudWatch alarm actually in the "Alarm" state? Can you add 2 more notifications to be triggered by the "OK" and "Insufficient data" state respectively?
by reading the original post, SNS should have the right permission to invoke the Lambda function.
previous answer:
That is a general message, it has nothing to do with your SNS topic settings. I reckon it's meant to say only emails opt-in to that SNS topic will get emails.
Q: How does Amazon SNS validate a subscription request to ensure that notifications will not be sent to users as spam?
As part of the subscription registration, Amazon SNS will ensure that notifications are only sent to valid, registered subscribers/end-points. To prevent spam and ensure that a subscriber end-point is really interested in receiving notifications from a particular topic, Amazon SNS requires an explicit opt-in from subscribers using a 2-part handshake:
i. When a user first calls the Subscribe API and subscribes an end-point, Amazon SNS will send a confirmation message to the specified end-point.
ii. On receiving the confirmation message at the end-point, the subscriber should confirm the subscription request by sending a valid response. Only then will Amazon SNS consider the subscription request to be valid. If there is no response to the challenge, Amazon SNS will not send any notifications to that end-point. The exact mechanism of confirming the subscription varies by the transport protocol selected:
For HTTP/HTTPS notifications, Amazon SNS will first POST the confirmation message (containing a token) to the specified URL. The application monitoring the URL will have to call the ConfirmSubscription API with the token included token.
For Email and Email-JSON notifications, Amazon SNS will send an email to the specified address containing an embedded link. The user will need to click on the embedded link to confirm the subscription request.
For SQS notifications, Amazon SNS will enqueue a challenge message containing a token to the specified queue. The application monitoring the queue will have to call the ConfirmSubscription API with the token.
Note: The explicit “opt-in” steps described above are not required for the specific case where you subscribe your Amazon SQS queue to your Amazon SNS topic – and both are “owned” by the same AWS account.
As you know, we can publish to an SNS events via those channels:
HTTP
HTTPS
Email
Email-JSON
Amazon SQS
AWS Lambda
Platform application endpoint
SMS
Now let's say I want to test a published messsage.
According to the list, I can use a "request catcher service" for Http/s requests, but sometimes the info is too confidential to be published to third party services.
The email solution is too side effecty. Also writing lambda involves writing code.
Connecting an SQS to inspect messages is causing me to create a special SQS just for testing...
Question:
Is there any option to submit to an SNS and to inspect the message in logs or something like that? Or am I limited only to this list? I'm talking only for dev purposes.
I have tried to do some R&D but i couldn't find anything useful the only thing that i found is lambda functions is the only way. I want to write a simple application that execute when the Simple Queue Service receive any message , but i couldn't find a way to do that till now, since i don't want to use lambda. for example if i receive some message on Simple Queue Service and while receiving each messages i can trigger a event that is not lambda but instead of that any HTTP-request.
I think your choices are:
use lambda (which you said you didn't want to use, but its probably the best solution)
use your own app running on ec2 or even on premise to consume the message and invoke the http endpoint
use SNS instead of SQS for message delivery - SNS supports http endpoints.
You can use Amazon SNS to send notification messages to one or more
HTTP or HTTPS endpoints. When you subscribe an endpoint to a topic,
you can publish a notification to the topic and Amazon SNS sends an
HTTP POST request delivering the contents of the notification to the
subscribed endpoint. When you subscribe the endpoint, you select
whether Amazon SNS uses HTTP or HTTPS to send the POST request to the
endpoint.
from here: https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html
A short answer to your question is No, (until today)
Let me tell you the sineros i faced.In general Queue triggering lambda is widely used and for that u have to make sure about proper concurrency (minimum 5) in place and also database I/O if u are performing and any DB calls . But I've a scenario where we cannot use "lambda as a triggering service" as our DB is onprem "ORACLE" so the choices are .
Push to "SNS" and make http "what ever applies"(to a container we have custom Kubernetes routed through NLB ).Also make sure you push a batch of messages as it might make more http noise.
2.Poll the queue and perform the operations.
SQS triggering => Lambda and lambda invoking state machine (step functions)
I have requirement to publish messages from SNS to kinesis. I have found that, it is not possible directly by subscribing same as SNS/SQS. I will need to write lambda to fetch from SNS and publish it to kinesis.
Is there any other way to publish records from SNS to kinesis directly?
Thanks
Amazon SNS is a publish/subscribe model.
Messages sent to SNS can be subscribed from:
http/s: delivery of JSON-encoded message via HTTP POST
email: delivery of message via SMTP
email-json: delivery of JSON-encoded message via SMTP
sms: delivery of message via SMS
sqs: delivery of JSON-encoded message to an Amazon SQS queue
application: delivery of JSON-encoded message to an EndpointArn for a mobile app and device.
lambda: delivery of JSON-encoded message to an AWS Lambda function.
Other options: See Otavio's answer below!
See: Subscribe - Amazon Simple Notification Service
Of these, the only ones that could be used to send to Amazon Kinesis would be to use AWS Lambda. You would need to write a Lambda function that would send the message to a Kinesis stream.
To clarify: Your Lambda function will not "fetch from SNS". Rather, the Lambda function will be triggered by SNS, with the message being passed as input. Your Lambda function will then need to send the message to Kinesis.
Your only other alternative is to change the system that currently sends the message to SNS and have it send the message to Kinesis instead.
Good news! As of January 2021, Amazon SNS has added support for message archiving and analytics via Kinesis Data Firehose subscriptions. You can now load SNS messages into S3, Redshift, Elasticsearch, MongoDB, Datadog, Splunk, New Relic, and more. The SNS documentation has the details.
I want to get the delivery status of the mails sent through Amazon SES by hitting an API and saving the response in my database. I know of Amazon SNS and sending notification through email, but could not find a solution for this. Is there a solution to this?
There is no polling mechanism in SES for delivery status.
The SNS delivery/status notifications that SES provides, which you have indicated you are familiar with, is the standard mechanism for this.
Enable these and then subscribe an SQS queue, or a Lambda function, or even an HTTPS endpoint from your app, to the designated SNS topic. It should be relatively simple using any of these mechanisms to code a solution to store the reports in the database as they are provided to you by SES.