I have an SNS Topic that had no subscriptions for a period of time. So any messages published to that topic during that period when there were no subscriptions were essentially lost. I am wondering if there are any logs or other records I could access to glean any info on things that could have been published during the time when there were no subscriptions.
I saw from an AWS forum post that
"From [the AWS] perspective, a publish is successful if we
successfully process the publish API call. We do not enumerate the
subscriptions list until later. If there are no subscriptions to a
topic, the message is eventually dropped."
I believe I could create a CloudWatch alarm to monitor the NumberOfMessagesPublished metric, per this article. From that article, I understand that Amazon SNS automatically sends the NumberOfMessagesPublished metric to CloudWatch, so if that metric shows nothing (count = 0) during the time period, I at least would have evidence of the lack of anything published to the topic. But if something was published, is there any evidence that remains--in logs or elsewhere--of the message that was published?
Related
I have my application which processes messages from a pubsub topic and if it fails the message is send to a separate dlq topic. I want to be able to set an alarm in monitoring that when during a day there were 30k messages sent to the dlq it notifies me and I can check why my service is not wokring.
I tried to set up some polices in gcp but I don't know and couldn't find anywhere in the docs how to setup a metric of daily processed messages on a topic.
Can anyone help me ?
You can create a new alert policy like this
PubSub subscription/unacked messages.
You can add a filter on your subscription name if you have several subscriptions in your project.
Add the notification channel that you want, an email in my case. After few minutes, you can see the first alert
And the email
EDIT
For the acked messages, you can do this
I never tried an aggregation over 1 day, but it should be OK.
Please check the following GCP community tutorials which outline how to create an alert-based event archiver with Stackdriver and Cloud Pub/Sub
https://cloud.google.com/community/tutorials/cloud-pubsub-drainer
From AWS SNS FAQ page (Reliability section) I can see that SNS guarantees at-least-once message delivery to SQS, but it's not clear whether the same is applicable when notification is sent to Lambda.
So the question does SNS provide at-least-once message delivery when a message is sent to AWS Lambda?
Q. What happens to Amazon SNS messages if the subscribing endpoint is not available?
Lambda: If Lambda is not available, SNS will retry 2 times at 1
seconds apart, then 10 times exponentially backing off from 1 seconds
to 20 minutes and finally 38 times every 20 minutes for a total 50
attempts over more than 13 hours before the message is discarded from
SNS.
https://aws.amazon.com/sns/faqs/
The same FAQ page states that
When a message is published to a topic, Amazon SNS will attempt to deliver notifications to all subscribers registered for that topic. Due to potential Internet issues or Email delivery restrictions, sometimes the notification may not successfully reach an HTTP or Email end-point. In the case of HTTP, an SNS Delivery Policy can be used to control the retry pattern (linear, geometric, exponential backoff), maximum and minimum retry delays, and other parameters. If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).
This applies to all Non-SQS subscribers
I think your question is: will a Lambda function, subscribed to an SNS topic, be invoked at least once or only once for any message?
If so, the answer is: at least once. The following part from the FAQs helps answer this, with emphasis mine:
Q: How many times will a subscriber receive each message?
Although most of the time each message will be delivered to your application exactly once, the distributed nature of Amazon SNS and transient network conditions could result in occasional, duplicate messages at the subscriber end. Developers should design their applications such that processing a message more than once does not create any errors or inconsistencies.
Im studing aws pricing and I have two doubts about Amazon SNS and Amazon Cloudwatch.
About cloudwatch, Im using it to monitor sns topics and to monitor a dynamodb table. Im reading about cloudwatch pricing and it says that the basic monitoring metrics for amazon ec2, amazon ebs, elastic load balancers and amazon rds are free. So to monitor sns topics and dynamodb its not basic monitoring and we need to pay $0.50 per month for each metric?
About SNS it says that we pay based on the number of notifications we publish, the number of notifications we deliver, and any additional API calls for managing topics and subscriptions.
Im bit confuse about this pricing, we pay for any API call, for example create a new topic, getl all topics, etc, ok this part it is clear, but Im not understanding about the other two, for exmaple in my code I have :
message = "this is a test"
message_subject = "Message test"
publication = sns.publish(topicArn, message, subject=message_subject)
In this case we have the cost about publish, and also in this sns.publish we need to pay relative to API calls? And also we need to pay when we publish a message, and to this cost it is added the cost of deliver notifications? But its not the same thing? Publish a message or deliver a notification? Or notifications it is that confirmation subscriptions that we receive in email when we subscribe some email in the topic?
subscriptionEmail = sns.subscribe(topicArn, "email", email)
yup.
for cloud watch it's 0.5$ per metric. Pretty steep if you have a lot of metrics.
for sns: http://aws.amazon.com/sns/pricing/
you pay for api calls (what's called publishes in that page).
you also pay when SNS delivers messages to subscribers.
you are right that the operation of publishing a notification is different from the operation of sending the notification to subscribers and they fall in 2 different pricing buckets
what really matters is the number of notifications you push and who you push them to (deliveries). All the other numbers are a rounding error even for applications with a moderate amount of traffic.
to give an example.
Assume you have 1 SNS topic with 2 email address subscribed to it.
Assume you sent 5_000_000 notifications to the topic.
You would pay (this assumes you have exhausted the free tier already):
5_000_000 / 1_000_000 * 0.50$ = 2.5$ for publishing to the topic
5_000_000 / 100_000 * 2.00$ * 2 (2 email adresses) = 200 $ for email delivery
for a total of 202.5$
I have an SQS queue subscribed to an SNS topic. When I publish a message to the topic and subsequently receive it from the queue, the message body contains my published message wrapped in some metadata added by SNS, as per "Amazon SNS -> SQS message body". Unlike the poster of that question, I want the metadata, as it allows me to generate some latency metrics.
However, I could find nothing in Amazon's documentation or elsewhere on the Googleable web explaining precisely what the metadata means. In Sending Amazon SNS Messages to Amazon SQS Queues, the Amazon docs give an example of the metadata but do not explain it.
Does anyone know what time the Timestamp field in the metadata is representing? Please provide a citation if possible, because it is really important to me that my understanding is accurate.
Time stamp is the time when the notification was published.
Here is the model I want to build:
Unlimited topics (I know the AWS limit is currently 3000)
Unlimited subscribers per topic (AWS SNS documentation does not state a limit)
Each subscriber will have a different schedule that they wish to receive SMS or email messages.
If I have a topic with 10 subscribers where 5 want morning messages and 5 want evening messages. The schedules can be changed and will be administered by a web app (utilizing the API) by an administrator so building one topic per schedule is not the ideal solution and the variations would chew up a lot of the 3000 topics.
SNS does not support the ability to publish to a subset of topic subscribers, other than when using its Android/iOS push notification capability, which is neither email nor http nor sms, and is so different than everything else SNS does, I'm not entirely sure why Amazon didn't market it as an entirely different product.
At this time, direct addressing is only supported for push notifications to iOS, Kindle, and Android endpoints
http://aws.amazon.com/sns/faqs/#Does_SNS_support_direct_addressing_for_SMS_or_Email
The documented limit for the number of subscribers to a single topic is 10,000. This appears to be a hard limit; however, the limit of 3,000 topics per account is apparently not a hard limit, since there is a documented process for requesting an increase to that limit.
There's a big gap between 3,000 topics of 10,000 subscribers each and 10 subscribers split among 2 or 3 topics, but it still doesn't seem like SNS is the right platform for what you're trying to do -- limits or no limits -- because every time a subscriber is added to a topic, they have to confirm their subscription to that topic... so if whatever your admin does to manipulate topic subscribership will still result in a confirmation message from "AWS Notifications" being sent to each subscriber from Amazon SNS requiring them to opt-in before subsequent messages can be delivered, and that's a process that would be have to be repeated if a subscriber wanted to change their delivery schedule (which means adding them to a new topic). You can't programmatically skip this step.
Within AWS, Simple Email Service seems like a more appropriate (and recipient-friendly, assuming your recipients are from the general public) platform based on what it seems like you're contemplating, with the logic of determining which recipients are associated with each message being determined by logic from your database... it doesn't have the same pricing structure, and it doesn't do SMS (though SMS through SNS seems pretty pricey at any rate), and unlike SNS, SES doesn't have a 256k limit per message.
That does put more of a burden on your application, since you'd have to send copies of the message to SES for each subscriber, but if outgoing bandwidth is an issue, an instance deployed inside EC2 could easily take care of replicating and transmitting the message to SES. With SES, you also get bounce and complaint notifications on your messages.
This is the approach I would take.
But then again, it's hard to tell exactly what you're asking.