Here is the model I want to build:
Unlimited topics (I know the AWS limit is currently 3000)
Unlimited subscribers per topic (AWS SNS documentation does not state a limit)
Each subscriber will have a different schedule that they wish to receive SMS or email messages.
If I have a topic with 10 subscribers where 5 want morning messages and 5 want evening messages. The schedules can be changed and will be administered by a web app (utilizing the API) by an administrator so building one topic per schedule is not the ideal solution and the variations would chew up a lot of the 3000 topics.
SNS does not support the ability to publish to a subset of topic subscribers, other than when using its Android/iOS push notification capability, which is neither email nor http nor sms, and is so different than everything else SNS does, I'm not entirely sure why Amazon didn't market it as an entirely different product.
At this time, direct addressing is only supported for push notifications to iOS, Kindle, and Android endpoints
http://aws.amazon.com/sns/faqs/#Does_SNS_support_direct_addressing_for_SMS_or_Email
The documented limit for the number of subscribers to a single topic is 10,000. This appears to be a hard limit; however, the limit of 3,000 topics per account is apparently not a hard limit, since there is a documented process for requesting an increase to that limit.
There's a big gap between 3,000 topics of 10,000 subscribers each and 10 subscribers split among 2 or 3 topics, but it still doesn't seem like SNS is the right platform for what you're trying to do -- limits or no limits -- because every time a subscriber is added to a topic, they have to confirm their subscription to that topic... so if whatever your admin does to manipulate topic subscribership will still result in a confirmation message from "AWS Notifications" being sent to each subscriber from Amazon SNS requiring them to opt-in before subsequent messages can be delivered, and that's a process that would be have to be repeated if a subscriber wanted to change their delivery schedule (which means adding them to a new topic). You can't programmatically skip this step.
Within AWS, Simple Email Service seems like a more appropriate (and recipient-friendly, assuming your recipients are from the general public) platform based on what it seems like you're contemplating, with the logic of determining which recipients are associated with each message being determined by logic from your database... it doesn't have the same pricing structure, and it doesn't do SMS (though SMS through SNS seems pretty pricey at any rate), and unlike SNS, SES doesn't have a 256k limit per message.
That does put more of a burden on your application, since you'd have to send copies of the message to SES for each subscriber, but if outgoing bandwidth is an issue, an instance deployed inside EC2 could easily take care of replicating and transmitting the message to SES. With SES, you also get bounce and complaint notifications on your messages.
This is the approach I would take.
But then again, it's hard to tell exactly what you're asking.
Related
I'm using AWS lambda to build my web app (next.js) and I'm successfully sending email using AWS SES.
I set a CloudWatch event to run every day, that runs a lambda function that sends personal email to some of the users.
The problem is that lambda has running time limit of max 30sec so I can't use it to send emails to a lot of users.
What can I do to solve this issue?
After looking in the CloudWatch logs, I saw it took ~5sec to complete this lambda for ONE user.
So, I'm guessing it will not handle 100 users or more.
At first, it is strange that sending an email takes 5 seconds unless there is a heavy data preparation to format each email.
As for the lambda timeout, to have a scalable design, I recommend that you make lambda A --> SQS ---> lambda B.
Lambda A just writes messages to sqs (recipient's email, template name, whatever necessary info to format the email). Please note that you can write a batch of messages (10 messages) to SQS.. based on one of my previous implementations, it is easy to write 5000+ sqs messages in 30 sec. You can set a longer time up to 15 minutes if lambda is not behind an API.
lambda B will be triggered by SQS to format the email and send it to SES. You can set batch size and sqs config the best suits your design and quota. This way, you can always send emails regardless of the volume as long as quotas of AWS Account allows. Please note that you can also introduce a random delay to each message written to SQS to make sure the limit of SES is not violated (14/sec <== can be raised easily).
you may consider a better design too based on your app architecture :)
I have over 5M subscribers to a SNS topic. I want to slowly send the push notifications to these users say at the rate of 20000 per sec. AWS tries to deliver the message to 5M as fast as possible. Is there any way I can slow down the sends ?
There is no configuration setting for Amazon SNS to limit the speed of notifications.
You would probably need to create separate SNS topics and send to each in turn (with a delay).
If you are doing some form of marketing to mobile apps, you could consider using Amazon Pinpoint instead.
I'll be needing to send out approximately 200-300k emails a week for our weekly newsletter. This number is expected to increase over time which is why I thought of letting lambda handle the scaling for us.
There will be a UI for employees to trigger the sending of these emails. My initial thought was to add the list of email addresses to a SNS topic and have the lambda function subscribed to it to trigger the actual sending of the emails. I've read that this may lead to some unexpected behavior and isn't the best for tracking down bugs.
Our current limit is 90 emails per second. This combined with the max time to execute a lambda function being 5 minutes makes me wonder how to split this up. Do I need to batch send these to SNS so multiple lambda containers can be spun up evenly to do the work? I understand that lambda can scale up for events like adding files to S3 but how would I explicitly tell lambda to spin up multiple to handle all the emails that need to be sent? Is there a better design pattern for this?
My other thought was to add the emails into an SQS queue so the emails can be persisted incase of errors but would need to do some workaround to make that play nicely with lambda.
Update
I'm a few years late, but Lambda worked like a charm. Here is blog post detailing the architecture: https://read.iopipe.com/how-to-use-aws-lambda-to-send-high-volume-emails-at-scale-10efe65b2f32
And a follow up post on more specifics and lessons learned: https://read.iopipe.com/6-lessons-learned-sending-mass-emails-with-aws-lambda-3c5e56e19571
On the surface Lambda might appear to be a good solution, but it is not.
You will either need to develop your own software or port / migrate an existing solution. Managing email lists and sending emails is a task that will fit just fine on a normal EC2 instance (sized appropriately) along with RDS. The process of sending a single email is slow (for a lot of SMTP servers several seconds per email) so overlapping is critical. With Lambda you will pay for all those seconds that Lambda waits for the SMTP server to accept each email.
There are a lot of issues in sending bulk (large quantity of) emails. This is why third party services are becoming so popular. They provide a managed service that you just plug into your app and move on to the next task.
If you are going to use AWS to send emails, consider reading this Best Practices document:
Amazon Simple Email Service Email Sending Best Practices
One thing to keep in mind when implementing a Lambda-based solution is that each lambda is guaranteed to run at least once. Occasionally there will be a glitch and some of your users will receive duplicate emails. Here it is estimated that around 0.02% of all invocations result in multiple executions.
I am developing an architecture for push notifications using AWS SNS with APNS and GCM. The model that I am following is
Each user (not device) will have an SNS topic corresponding to it
Each user can have multiple devices
Create an platform application endpoint for each device
Subscribe the platform application endpoint to the topic belonging to the device's user
This way, when we have to send a notification to all the devices of a user, we need to call the publish method using the user's topicArn and all its devices should get the message.
However, by default AWS has a limit of 100,000 topics. If we are anticipating higher number of users like 1 million or say 10 million to be optimistic, is there any workaround for this approach?
Should I ask AWS to raise the limit beforehand? Do they raise limit to a number like 100 million easily? Is there any cost implication of most of them are not used in the beginning few months?
disclosure: I used to work on Amazon SNS
The architecture you propose is a common pattern within Amazon SNS and is sound.
You should ask for a topic limit increase. There is no cost implication for having a higher topic limit, even if you create the topics and don't use them. However, you will pay $0.50/million requests for each CreateTopic call, with the first 1 million requests being free per month.
When you submit the support ticket, please document your use case. It helps the team expedite your limit increase request.
We followed a similar model to OP's with one exception - as the number of users grew, so did the number of dead accounts. Face it, user retention poses a challenge. So you may want to come up with a topic recycling strategy. Hint: Don't permanently bind a topic to a user but allow ARNs to expire, similar to a DHCP lease. Your app may periodically ask your web service what ARN it needs to subscribe to and expect to be re-assigned. If you build this into the client protocol early on, you won't need as many ARNs. After all, 100k active users is entirely different than 100k enrolled users :)
With Amazon SNS, there is no minimum fee and you pay only for what you use. Users pay $0.50 per 1 million Amazon SNS Requests, $0.06 per 100,000 Notification deliveries over HTTP, $0.75 per 100 Notification deliveries over SMS and $2.00 per 100,000 Notification deliveries over Email.
Amazon SNS also includes a Free Tier, where users can get started with Amazon SNS for free. Each month, Amazon SNS customers pay no charges for the first 1 million Amazon SNS Requests, no charges for the first 100,000 Notifications over HTTP, no charges for the first 100 Notifications over SMS and no charges for the first 1,000 Notifications over Email.
So I will suggest increase the limit before hand if that is anticipated by you. you don't get charged for increasing the limits. They will definitely increase to 100 Million.
Amazon SNS (Simple Notification Service) can break down my recipients to "Topics" which can then be used to send some messages to particular sub-segment of my app audience.
Let's say I have so many sub-segments of audiences that I can't use SNS Topics anymore (due to their limit).
Sending one by one is a heavy task.
Is it somehow possible to upload a list of pushIDs to SNS and then set the message and send the push message?
I don't think you can upload a list, bit you could ask AWS for limit increase or use multiple accounts.
You can request a topic limit increase here: https://aws.amazon.com/support/createCase
Unless you need some insane number of topics, this is probably your best bet.
You can use multiple micro ec2 instances to get over api limits. For this you have to divide workers into multiple instances. We implemented this a while ago and still using it without any problem.