I have over 5M subscribers to a SNS topic. I want to slowly send the push notifications to these users say at the rate of 20000 per sec. AWS tries to deliver the message to 5M as fast as possible. Is there any way I can slow down the sends ?
There is no configuration setting for Amazon SNS to limit the speed of notifications.
You would probably need to create separate SNS topics and send to each in turn (with a delay).
If you are doing some form of marketing to mobile apps, you could consider using Amazon Pinpoint instead.
Related
I'm using AWS lambda to build my web app (next.js) and I'm successfully sending email using AWS SES.
I set a CloudWatch event to run every day, that runs a lambda function that sends personal email to some of the users.
The problem is that lambda has running time limit of max 30sec so I can't use it to send emails to a lot of users.
What can I do to solve this issue?
After looking in the CloudWatch logs, I saw it took ~5sec to complete this lambda for ONE user.
So, I'm guessing it will not handle 100 users or more.
At first, it is strange that sending an email takes 5 seconds unless there is a heavy data preparation to format each email.
As for the lambda timeout, to have a scalable design, I recommend that you make lambda A --> SQS ---> lambda B.
Lambda A just writes messages to sqs (recipient's email, template name, whatever necessary info to format the email). Please note that you can write a batch of messages (10 messages) to SQS.. based on one of my previous implementations, it is easy to write 5000+ sqs messages in 30 sec. You can set a longer time up to 15 minutes if lambda is not behind an API.
lambda B will be triggered by SQS to format the email and send it to SES. You can set batch size and sqs config the best suits your design and quota. This way, you can always send emails regardless of the volume as long as quotas of AWS Account allows. Please note that you can also introduce a random delay to each message written to SQS to make sure the limit of SES is not violated (14/sec <== can be raised easily).
you may consider a better design too based on your app architecture :)
I have 9000 AWS SNS topics with more than 1M subscribers in each topic. At the moment I am lopping to each topic to send a push message which is consuming lot of my system resources. Is there a way to send message to all the topics at once? what is the best approach to handle the scenario?
It is not possible to subscribe Amazon SNS queues to an Amazon SNS queue, so there is no out-of-the-box method for sending one message to multiple queues.
I would recommend creating an AWS Lambda function that will:
Retrieve a list of all relevant queues (based on tag?)
Loops through and sends a message to each queue
Thus, you would just trigger the Lambda function with one message and it would go to all other queues. It would not "consume system resources", but it is charged based upon run duration. Lambda functions can run for a maximum of 15 minutes, so as long as it sends 10+ messages per minute, it can send to 9000 topics.
Depending upon your use-case, you might also consider using Amazon Pinpoint:
Amazon Pinpoint is an AWS service that you can use to engage with your customers across multiple messaging channels. You can use Amazon Pinpoint to send push notifications, emails, SMS text messages, and voice messages.
I am willing to make a simple chat application in django.
After user sends message and the message in saved in Message model I would like to signal the publish event so that my mobile app gets message.
I went throught the SNS documentation and in its pricing I found Data transferred between Amazon SNS and Amazon EC2 within a single region is free of charge (i.e., $0.00 per GB)
My api is/will be hosted in ec2 in the same region lets say at Singapore the same region as sns. If I publish a event which I receive a message in my mobile application , will that cost me or it is considered as same region ? If it does than building a chat application using SNS will be very expensive.
If anyone can meke me clear about it that would be so helpful.
Thank you
Well I don't think it will be free even if they are in the same region but their prices for SNS are too low, if in free tier of Amazon you won't be charged for
1,000,000 Requests of Amazon Simple Queue Service every month**
1,000,000 Requests, 100,000 HTTP notifications and 1,000 email
notifications for Amazon Simple Notification Service every month**
as the SNS pricing suggests. Going above the limits will cost you :
It costs $1.00 to send one million mobile push notifications ($0.50 per million publishes, plus $0.50 per million mobile push notification deliveries). When you use SNS topics to broadcast identical messages to many recipients at once.
We need to be able to broadcast messages to all our users without overwhelming our database with a huge spike in users coming in all at once. I see there is a way under AWS SNS to rate limit messages going to a single subscribed endpoint, but I do not see a way to rate limit messages going to an application. Ideally, it would be great to have a way to say "Max Endpoints/Sec == 1000"
Here is the model I want to build:
Unlimited topics (I know the AWS limit is currently 3000)
Unlimited subscribers per topic (AWS SNS documentation does not state a limit)
Each subscriber will have a different schedule that they wish to receive SMS or email messages.
If I have a topic with 10 subscribers where 5 want morning messages and 5 want evening messages. The schedules can be changed and will be administered by a web app (utilizing the API) by an administrator so building one topic per schedule is not the ideal solution and the variations would chew up a lot of the 3000 topics.
SNS does not support the ability to publish to a subset of topic subscribers, other than when using its Android/iOS push notification capability, which is neither email nor http nor sms, and is so different than everything else SNS does, I'm not entirely sure why Amazon didn't market it as an entirely different product.
At this time, direct addressing is only supported for push notifications to iOS, Kindle, and Android endpoints
http://aws.amazon.com/sns/faqs/#Does_SNS_support_direct_addressing_for_SMS_or_Email
The documented limit for the number of subscribers to a single topic is 10,000. This appears to be a hard limit; however, the limit of 3,000 topics per account is apparently not a hard limit, since there is a documented process for requesting an increase to that limit.
There's a big gap between 3,000 topics of 10,000 subscribers each and 10 subscribers split among 2 or 3 topics, but it still doesn't seem like SNS is the right platform for what you're trying to do -- limits or no limits -- because every time a subscriber is added to a topic, they have to confirm their subscription to that topic... so if whatever your admin does to manipulate topic subscribership will still result in a confirmation message from "AWS Notifications" being sent to each subscriber from Amazon SNS requiring them to opt-in before subsequent messages can be delivered, and that's a process that would be have to be repeated if a subscriber wanted to change their delivery schedule (which means adding them to a new topic). You can't programmatically skip this step.
Within AWS, Simple Email Service seems like a more appropriate (and recipient-friendly, assuming your recipients are from the general public) platform based on what it seems like you're contemplating, with the logic of determining which recipients are associated with each message being determined by logic from your database... it doesn't have the same pricing structure, and it doesn't do SMS (though SMS through SNS seems pretty pricey at any rate), and unlike SNS, SES doesn't have a 256k limit per message.
That does put more of a burden on your application, since you'd have to send copies of the message to SES for each subscriber, but if outgoing bandwidth is an issue, an instance deployed inside EC2 could easily take care of replicating and transmitting the message to SES. With SES, you also get bounce and complaint notifications on your messages.
This is the approach I would take.
But then again, it's hard to tell exactly what you're asking.