I have the following use case scenario for which I am considering aws services to see how I can come up with a scalable solution. Thanks in advance for your help!
Scenario:
Users can sign up to an application(which is named say 'Let's Remind' or something else) using their email and phone.
The app does one thing that is to send email and sms alerts to user.
User can create n number of tasks for which he wants to be reminded. For instance he can set up a
monthly reminder for paying card dues. Currently the value of n is from 5 to 10 per user.
The notifications are flexible meaning it can be daily, weekly, monthly, bi-weekly. User can also
specify the start date of a notification. The end date is the date when the event is due (for instance
the day the card payment is due). Once this date is expired the notification is rendered inactive for
the current month.
For weekly,daily,bi-weekly notifications, the notifications are deleted once the event date is passed.
These are not recurring in nature.
For monthly recurring events such as payment of apartment rent etc, notification itself is not
deleted but rendered inactive after the event due date. Once the next event cycle (typically next
month billing cycle for payments use case) starts, the notification comes back to life and starts all
over again.
Use can delete any event anytime he wants. If an event is deleted, the notifications for that event
will be deleted as well.
First of all, I hope the use case is clear. Now here's my thoughts so far about solving this use case -
1) Use SNS since I need to send email and sms both. SES only supports emails.
2) When a user registers for the app, create 2 subscriptions(one for his email and one for his sms endpoint) and also create a topic for the user(maybe a dynamically generated random userid)
3) Once user creates an event (e.g. reminder for monthly apartment rental), store the event data such as userid, startdate, duedate, frequency, isactive in a dynamodb table.
4) Create a lambda function that will wake up when an entry is written to the dynamodb table (step 3); it will do the following -
i) it will read the event data from the dynamodb table
ii) determine the next date of the notification to be sent based on the current date and event
data
iii) For active events (check isActive column of the dynamodb record) create a scheduled cron
expression rule based on ii above in cloudwatch events and add the
target as the user's topic (created in step 2 above). For now, the notification message is
static.
I have some doubts/queries about step iii -
Is it possible to create cloudwatch event cron rule dynamically and add the user's topic as target dynamically as I described? Or is it better to trigger a second lambda function dedicated for sending messages to the user's topic using SNS notification? Which approach will be better for this use case?
If user base grows large, is it recommended to create one topic per user?
Am I on the right track with my approach above in general from a scalability point of view?
If not, can anyone suggest any better idea for implementing this use case?
Thanks in advance!
This will not work.
For SNS email subscriber to receive email notification sent via SNS it has to first confirm the subscription. You cannot just create subscriptions on the fly and send them email notification
I don't think SNS fits your use case. You would be better off sending email notifications using SES.
You can write your scheduling logic in Lambda though
Related
I'm trying to establish email alerts at a project level to send email alerts for when a certain number of query/job concurrency is reached e.g. 5 concurrent queries. We have a flat-rate pricing model.
I want to do a similar email notification when total slot Usage exceeds a certain threshold as well e.g. slot usage reaching 1000 slots
As a next step I would like to throttle new incoming queries based on the above mentioned thresholds. Meaning if there are already for example 5 queries actively running the 6th one will be put on hold until one of the 5 running earlier have completed.
You may create an Alert Policy in which you can set your desired metric type (eg. slots) and then configure your desired threshold similar to below.
In creating an Alert Policy you may also configure the notification channel to email notification which is also included on the same documentation.
For the available metric types for SLOTS in BigQuery, you may refer to this Google Cloud Metrics for BigQuery documentation.
For your next step, you may code (python, node.js, etc) using BigQuery API to count queries actively running (through JOB ID) and when the count hits 5, you may print "query queue is full" and then wait for the total JOBS to hit below 5 before running the next query. You may refer to this BigQuery Managing Jobs API Documentation.
Problem:
I know that Google Pub/Sub subscription cannot be patched to update the filter. I am just figuring out other ways to handle updates to filter in production.
Approach I could come up with:
Modify the push config to pull in existing subscription (old filter) so that it accumulates messages
Create a new subscription with latest filter
Transfer the messages from old subscription to a topic using dataflow
Detach the old subscription from the topic
Problems I see with the approach:
As both subscriptions exist at a point of time, I could end up processing duplicate messages
Any suggestions on the best way to handle this?
The timing is important to minimize the duplicates or the message loss.
Firstly, I will deploy a service (Cloud Run for exemple) that save the PubSub message as is somewhere (in a Cloud Storage file, in bigQuery, in Firestore,...)
Then, and in the same time, I will change the push of the old subscription to push to my Cloud Run service; and create the new push subscription with the new filter
Finally detach the subscription.
If you have the capacity, in your REAL app to detect the message already processed, you can remove them from your save location (it's easier with BigQuery for example) and then reprocess only the missing messages. With dataflow, or manually
However, it's recommended to have idempotent processing of your message. Keep in mind that PubSub is at least 1 delivery message and even with the same subscription you could have duplicates.
As you note, a filter expression cannot be changed once a subscription has been created. To effectively change a filter, please do the following steps:
Create a snapshot on the existing subscription (subscription A).
Create a new subscription (subscription B) with the new filter expression.
Seek subscription B to the snapshot created in step 1.
Change your subscribers to use subscription B instead of A.
Please note that during the time subscribers are switching from subscription A to B, there will be a high rate of duplicates delivered as messages are acked independently on the two subscriptions.
If you need to minimize the duplicates processed and can afford downtime, you can stop the all subscribing jobs/servers before step 1 and restart them configured to pull from the new subscription, B, after step 4. All steps must be completed well within the configured message_retention_duration to prevent message loss.
We have been trying to integrate authorize.net payment gateway in one of our clients project based on Asp.net web API. We have few queries that we came across while implementing Recurring Planning scenarios.
Query 1
We checked the API’s for Creating Subscription, Getting Subscription, Updating Subscription. However once we have created subscription, is there any way we can update the amount in the subscription.
Let’s say for example.
We have a created a subscription for our user for 50$ amount on 01st Jan 2021 with 30 days interval.
And on 15th Jan 2021, our user wishes to purchase 1 more license which will cost him 10$ more.
Hence can we increase his billing cycle of subscription by updating the subscription?
We checked in Update Subscription API, & it is only allowing to update credit card info hence is there any way to update amount.
Query 2
Is there any way to implement Autorenewal, hence when a user wishes he/she can set auto renewal on/off for recurring billing.
Query 3
If there is any way to switch off auto renewal of recurring billing, then is there any link that we can generate & send them through which they can pay there next due.
Query 1: You cannot update a subscription amount. If the amount needs to change you either need to cancel the current subscription and create a new one for the new amount (being sure to prorate credit from the previous subscription payment) or use CIM to manage your subscription service which allows you to charge against their card at your discretion but requires you to also manage the subscription yourself.
Query 2: Not through Authorize.Net. If you want a subscription to start or end you need to explicitly do so through their API.
Query 3: Not through Authorize.Net. That application logic and, once again, you would be responsible for managing.
I'm assuming you are using or are aware of the API provided for Authorize.net here: https://github.com/AuthorizeNet/sdk-dotnet/tree/master/Authorize.NET/Api/Controllers
Query 1: As of now, there is a way to update the amount for a given subscription. You can use ARBSubscriptionType class. There is an amount property there you can set. Then you can create the request ARBUpdateSubscriptionRequest, passing in the ARBSubscriptionType class and the subscription Id.
Note: You might have to handle pro-rating.
Query 2: There isn't a built in renewal feature in Authorize.Net as far as I know. It seems like you could potentially update the totalOccurrences by some amount to act as a "renewal", when technically its an extension of the subscription. The method in which you check when to update, either a Modulo operation or a date check is up to you. You can use paymentScheduleType class to update totalOccurrences, passing it along to a ARBUpdateSubscriptionRequest.
Query 3: Authorize.Net does not have any in house link generation.
Consider sample chat application where user purchase monthly/annual subscriptions (subscriptions like Amazon Prime, etc).
As soon as the subscription expires, user should not be able to send messages in app.
User can end their subscription before the original subscription end date.
One solution in my mind (Frontend) - to cache the end date in app and before every "send message" operation, compare the end date and current date.
But the problem is - if user ends the subscription early, user will still be able to send the message.
How can I push update the new subscription end date in cache.
Another solution was (Backend) - I have a table in backend storing subscription details like subscription_id, user_id, subscription_enddate. So before any "send message" operation, query the subscription table and compare the dates and then continue/cancel further operations.
Q1. Should I go with Backend solution or can you please share some improvements to frontend method or any best practice for this scenario?
Q2. Also is storing subscription details in separate table best practice or any good design instead. ?
PS- Sample chat app is based on AWS Amplify Datastore
Let me try to breakdown the answer and give my opinion. I would also like to mention solutions to such problems are determined by the scale and various tradeoffs.
Q1-
If sending messages has an adverse effect, you should never rely on the frontend solution only as it is easy to bypass them. You can use a mixture to ensure that the load is not very high on the backend.
Adding a Frontend Cache for subscription will ensure you will be able to filter most of the messages on the frontend if the cache is not tampered with.
Adding a service before the queue, that validates whether the user subscription has expired adds one more layer of security. If the user subscription is valid it pushes the message to Queue else throws an error. This way any bad actor can also not misuse the system.
Q2-
Depending on the use-cases and load, you can have a separate table or a separate micro-service for the subscription itself.
When to have a separate micro-service?
When the subscription data is required from multiple applications in your system and needs to have its own scalability independent of others, it can be beneficial to have a separate micro-service.
When to have a separate table?
In other cases, where you feel adding a service would be overkill. You can keep the data separate in a different table/DB giving you the flexibility to change subscription and even extract it easily in the future.
I am relatively new to GCP platform. I have a system which is publishing messages in pubsub topic. Messages has an attribute named country and based on this attribute different clients would like to subscribe to messages. Client X is interested only in messages for country a. Client Y is interested only in messages of country b.
Will I have to create topic for every country in google pub sub?
or there is a smart way to have subscription on single topic based on attribute value
I am referring to attribute passed by publisher as stated in google docs.
https://cloud.google.com/pubsub/docs/publisher
Thanks
Update June 2020: Filtering is now an available feature in Google Cloud Pub/Sub. When creating a subscription, one can specify a filter that looks at message attributes. If a message does not match the filter, the Pub/Sub service automatically acknowledges the message without delivering it to the subscriber.
In this specific case, you can create a single topic and two different subscriptions, one that is meant to get messages for country a and one that is meant to get messages for country b. The filters for each subscription would be:
attributes.country = a
attributes.country = b
Previous answer:
The feature you are talking about is called filtering: you want the ability for a subscription to specify that it wants to receive a subset of the messages based on the attributes provided in the message. At this time, that feature does not exist in Google Cloud Pub/Sub.
There are two ways to handle this right now:
Filter the messages in the subscribers themselves by looking at the attributes and immediately acking all messages that they are not interested in. This does mean you will pay for delivery of all of the messages to each subscriber which may not be desirable depending on the percentage of messages the subscriber is actually interested in.
Create separate topics and a subscription on each topic, publish messages to those individual topics based on the attributes, and then have subscribers get messages on the subscription for the appropriate topic.
We are exploring ways to add functionality that will make this use case easier in the future.