I'm working on an Ad-tech system which serves millions of users.
Basically users (non anonymous users) can see different Ads that are being created by the marketing team.
Our marketing team want to be able to set some Frequency caps on those Ads (among other targeting rules they already have)
For example:
"We should not show this ad for a user if he already seen/click this ad more than X times in the last Y days"
Also ads can be grouped to campaigns, so rules like that are also possibile:
"We should not show this for a user if he viewed more than X times ads in this campaign in the last Y days".
Also our marketing might wanna know how many people viewed/click a specific add in the last Y days.
We have roughly 200K RPM and our responses should be very fast.
The smallest unit of time for our queries is one day and it will not change.
Few questions and thoughts:
Is DynamoDB a good fit?
I thought about creating a table for each event type (Click/View/Close..)
What is the best way to configure the primary key?
I thought about settings the primary key as the user id and the sort key as a combination of the ad id and the current day {dd/mm/yyyy}
I thought about use "ADD" operation to increase the counter when a user click/view/.. an Ad in a specific date. are they expensive operations? do I have an alternative?
What is the best way I can use to also be able to query per ad and campaigns as well (for example: "all users views for all ads in campaign" or "get all ad views in the last 40 days) ) ?
What other considerations should I take in mind?
Thanks a lot
Related
I have an application being built using AWS AppSync with a primary focus of sending telemetry data from a mobile application. I am stuck on how to partition and structure the DynamoDB tables for this as the users of the application belong to different organizations, in those organizations there will be admins who are able to view the data specific to their organization.
OrganizationA
-->Admin # View all the telemetry data
---->User # Send the telemetry data from their mobile application
Based on some research from these resources,
Link 1.
Link 2.
The advised manner is to create tables for individual periods i.e., a table for every day with the telemetry readings.
Example(not sure what pk is in this example):
The way in which I am planning to separate the users using AWS Cognito is by attaching a custom attribute when the user signs up such as Organization and Role(Admin or User) as per this answer then use a Pre-Signup Lambda Trigger.
How should I achieve this?
Since you really don't need users from one organization to read data from another organization, and for all your access patterns you will always know the organization id, then that attribute should be a factor in partitioning: either at the table level, or at the partition key level.
Then you have to determine if you can simply use the organization id as a partition key, or you need to further partition -- say, by concatenating the organization id and the hour value for each sample. This will depend on the amount of data you expect to generate by each organization in a given day. The tradeoff being more granular partitioning vs. cost of querying for data.
If organizations generate small amounts of data each day (say, a few events an hour) then just use organization id as the partition key. Otherwise, partition the data further.
In all of the above, the sort key should probably be the timestamp of the events, either with second or millisecond precision depending on your needs. That way your queries can retrieve ordered time-series data.
Keep in mind that when you make queries, you may need to execute multiple queries and stick the results together in your application to fully represent the results as the range may span multiple partitions, or even multiple tables.
After my users log in the app makes too many requests to DynamoDB and I am thinking about different ways to reduce the number of calls.
The app allows user to trigger certain alerts that get sent to other users. For instance: "Shipment received, come to the deck", "Shipment completed", etc.
These are the calls made:
Get company's software license expiration date.
Get the computer's location in the building (i.e. "Office A").
Get the kinds of alerts that can be triggered (i.e. "Shipment received, come to the deck", "Shipment completed", etc).
Get information about the user (i.e. company teams the user belongs to, and admin level the user has (which can be 0, 1, 2, or 3).
Potential solutions I have though about:
Put the company's license expiration date as an attribute of each computer (This would reduce the number of queries by 1). However, if I need to update the company's license expiration date, then I need to update it for EVERY SINGLE computer I have in the system, which sounds impractical to me since I may have 200, 300 or perhaps even more computers in the database.
Add the company's license expiration date as an attribute of the alerts (This would reduce the number of queries by 1); which seems more reasonable because there are only about 15 different kinds of alerts, so if I need to change the license expiration date later on, it is not too bad.
Cache information on the user's device; however, I can't seem to find a good strategy to keep the information stored locally as updated as possible.
I still think these 3 options do not sound too good, so I am hoping someone can point me in the right direction. Is there a good way to reduce the number of calls? I am retrieving information about 4 different entities (license, computer, alert, user), should I leave those 4 calls after users log in?
here are few things that can be done wrt each component.
Get information about the user
keep it in session store and whenever details changes update the store. session stores are usually implemented using cache like redis.
Computer location
Keep it in a distributed cache like redis. lazily initialise it. and whenever new write happens to computer location (rare IMO) remove the entry from redis using dynamodb streams and aws lambda.
Kind of alerts
Same as Computer location
License expiration date
If possible don't allow license expiry date (issue a new one for these cases, so that traceability is maintained.) and cache licence expiry forever. OR same as Computer location.
In an application that has many 'shops' every registered admin user has a 'shop' entity, each shop sells items where each item belongs to a certain 'category'. Having multiple clients (100's in some cases) each client has an account to follow up on their purchases and past orders. Each shop generates invoices for their clients, clients pays the invoice.
Admin User -- > Shop
Shop ---> clients
|-> items Categories
|-> items
|-> invoices
|-> payments received
An admin page shows a report showing invoices within the year (from Jan to Dec) this page is a client requirement. The shop is able to manually generate a new invoice when a purchase is made, and records a payment when it is paid. Note: This all happens in the actual shop, there is no online client purchases.
As a single shop generates few invoices per month (~100's), and multiple payments per month (~100's), showing this per year easily goes to thousands entities to show on a single page.
To optimize loading the page and generating the sales year report (total sales, revenue, payment...etc.), we thought we'd structure the data in a way where each item category per year is also an entity. This means that whenever a purchase is made for an item in this category, we need to add the item's purchase price to the itemCategory at that year in this month.
itemCategory Model:
itemCategory(ndb.Model):
shopID = ndb.KeyProperty()
year = ndb.IntegerProperty()
monthly_sales = ndb.FloatProperty(repeated=True) #12 months
This way we can load the entire sales table by reading just the list of itemCategory for this shop for this year, instead of reading all individual purchases through the year. This would save lots of Datastore reads and decrease page load time on the expense of an extra read, sum & write to this summary like entity.
Category Jan Feb Mar ... Dec
--------------------------------------
Men's shoes 1000 1300 850 ... 1400
Kids shoes 600 850 650 ... 900
The challenge at this point is that strong consistency is quite essential, for individual purchases and for the itemCategory entities. Because if the shop tries to add multiple purchases in a successive short timed way, with eventual consistency itemCategory might have not been updated with the last purchase sum yet. Resulting in wrong sales values. Also the same for individual purchase if there was a requirement to edit one right after it was added, a query for the entity without its ID might have no results. So it seems that Ancestor queries is essential here with maybe the shop as the parent entity. Yet, this will result in a contention issue later on (at least until Datastore is migrated to Firestore) with all those entities (thousands in this case!) having one single parent!
The same goes for invoices, generating a new invoice means knowing the latest invoice number so that they are always in sequence without gaps. Querying invoice with eventual consistency may result in duplicate invoice numbers.
What is the optimum way to structure the data at this point for strong consistency? Unfortunately the project has been there for a few years, and was started using Google Datastore rather than Cloud SQL (which seem to be more appropriate for this kind of projects). Hopefully all these issues goes away after the migration to Firestore having Strong consistency for all reads
Consider exporting the data and then importing it into a Cloud Firestore in Datastore Mode project. No more eventual consistemcy issues.
There are certain ways you can achieve strong consistency.
Query using key. Whenever you try to read an object via its key it is strongly consistent.
Another approach would be to use NDB Asynchronous Operations. See related documentation here.
A really naive approach would be to provide a delay which could help you but the delay should be provided in such a way that it is sufficient for the object to get updated.
And the final approach could be to export data into Cloud Firestore. There you can achieve strong consistency always.
Hope this answers your question!!!
Let's say I'm creating a PWA (Progressive Web App) where products can be added by users.
Prices of these products are variable from 0,01 EUR to 1,00 EUR.
I'm using Stripe for payments.
The Stripe Order object do not support dynamic price, passed on the fly, without any reference (kind of foreign key).
To accept the Order, Stripe needs a reference to a SKU.
This SKU will be, in my case, a variation of the price, on the product.
It means that, to cover all variations, I need 100 SKUs, from 1 (0.01 EUR) to 100 (1,00 EUR).
So, for each product created in Stripe, I need to create 100 SKUs in Stripe.
I tried to insert a test dataset of 200 products, which means (200 products + (200 x 100 SKUs)) = 20200 requests.
I got a surprising "Request rate limit exceeded" error from Stripe.
Less than half of records where created... :(
That "Request rate limit exceeded" is the core of the problem.
Right now, the insertion process is the following (x 200):
Create product in Firestore.
Firebase cloud function listener :
OMG new product inserted in Firestore. Ok let's :
Import official nodejs Stripe & Algolia libraries
Create product in Stripe to make it billable
Create the 100 SKU related to the product in Stripe, with Promise.all (This is where, at some point, I end up with a rate limit error, because my concurrent cloud functions instances are using the same Stripe key, which means the same Stripe account)
Create product in Algolia to make it searchable
I need solutions to counter this Stripe API rate limit error.
I have several solutions in mind :
Solution 1 :
Be able to increase Stripe rate API limit for a given amount of time.
Not sure this is possible.
Solution 2 :
Be able to use differents Stripe keys, then rotate over them, to perform admin stuff, such inserting multiple products/SKUs in Stripe.
Ultimately on production, be able to create programmatically 1 Stripe key per user, so each user would have its own limit.
Not sure this is possible.
Solution 3 :
Slow down insertion process in javascript.
Don't know how to perform that.
Besides, Cloud functions have a budget/limit of 60 seconds for javascript execution. So I can't delay too much.
Solution 4 :
Delay work using Pub/Sub (?), or Firestore Triggers
For example, having an integer in Firestore, that each function call increments, and same function listen the write to re-increment he number, etc, etc, etc, until the number equals 100 for the 100th SKU. That solution would sequentialize the 100 SKUs writes in Stripe.
Not sure this will really slow down enough the work to be under the API rate limit. In addition, such a solution would cost lots of money : 100+ Firestore writes, and 100+ functions calls to perform these writes, for only one product, which means 20000+/20000+ for the 200 products. That would be expensive.
Solution 5 :
Perform Just-In-Time insertions, when user pays.
The server side algorithm, after a Payment Request API call, might look like this :
Create order in Stripe
If error 'No such sku...' catched {
For each SKU { // Ideally filter here SKUs to create (only those in error)
If price not between 1 and 100 {
continue // Bad price, not legit
}
Create SKU in Stripe
If error 'Already exists' {
continue // no creation needed for that SKU
}
If error 'No such product...' catched {
If productId does not exists in Firestore {
continue // Bad productId, not legit
}
Create product in Stripe
}
Create SKU in Stripe
}
}
Create order in Stripe
This last solution could do the job.
But it might comes with some delay for the user when it executes payment, which could increase stress. Plus it might increase Stripe calls during the business hours. Many purchases in same time could lead to a Stripe API rate limit error, especially with well furnished carts (let's say an average of 30 products in the cart, so in worst case 30+ HTTPS calls during payment, times 1000 users = 30000 calls => Stripe error). That problem might decrease over time for a given product, because once a SKU is created it is created definitively. Still, as there would be new products, so products with zero SKU at creation, every day, the problem remains.
What do you think ?
Do you have any other ideas ?
Solution 3 and Solution 5 with some tweaks will work best.
Solution 3: You can limit number of concurrent requests to Stripe using async module's forEachLimit or queue.
Solution 5 : Just in time insertions is also a good option as it won't put much load on Stripe server at same time. Regarding your concern of getting the same error during business hour, it will a very rare case as Stripe APIs are built to perform very well. But if you still have doubt regarding this what you can do is to have a Background process for adding SKUs during non-business hours, which will keep on creating SKUs for you without encountering Stripe API rate limit error.
Solution 6 (Modified Solution 5): Have just in time insertions but also create an extra API request to your server whenever a product is entered in the cart which will then check if the SKU exist in Stripe and if not then create it in the background before cart payment happens.
Solution 6 :
Same idea (JIT), but moving SKU creation from payment time to product selection time. Each time a product is selected, try to create the product and its current SKU (price variation) in Stripe. This way, Stripe calls should be more distributed in the time. Or maybe it will ends with more API calls, as we select products more often than we pay, because users can select & unselect products, so they might end with more products selected during their journey than the sum of products finally being paid in the cart ?
Solution 7 :
Same idea (JIT), but with SKU cached in Algolia or Firebase, so I can perform "does this SKU exist ?" calls without querying Stripe, which should reduces Stripe calls if the existence test is performed before the create call (we do not call Stripe.skus.create() blindly). The drawback is, that Firebase and Algolia are exposed in Front so the SKUs and prices will be too, and this is a potential source of threat, so another index, dedicated and only known by the server, has to be used.
I am looking for a generic method of filtering a series of sitecore items based on the users current profile, I found one promising example:
How do I trigger a profile in Sitecore DMS?
However a few critical references are missing which is a shame as it looks to be a suitably generic function
Resources.Settings.AnalyticsUserProfileEnableSwitch I assume to simply be a boolean switch
The killer is ApplyUserProfile(filter)
Please keep in mind that user profiles are NOT the same thing as profiles in DMS. In DMS this is in reference to Analytics profiles related not to the specific user, but in visiting profiles... i.e. Marketing personas.
If you want to filter items based on user profiles, you simply get the Sitecore.Context.User.Profile and get whatever the property is and implement your logic to how you want to filter.
If you want to filter items based on DMS profiles, then that's something that's going to be difficult to do due to the fact that personas are not entered into the Analytics database real time. Those really aren't something you'll even be aware of at run time and therefore it's going to be difficult to categorize the persona at run time. You could, however, use the rules system to do some filtering based on other criteria (such as using the Engagement plans or something else)... but without more information, that's about as much as can be said.