ex.
Let's say I'm trying to create an eCommerce platform with multiple sellers and I create a Table called Orders. The partition key will be storeID and the sort key will be orderNumber.
Stores can call API.get('Orders', {storeID}), which will return all the items with the partition key of storeID.
My application uses Amazon Cognito and each user is assigned a username which is a uuid. My question is can I use the uuid as the storeID in my DynamoDB table? The key assumption is that attackers won't be able to guess the uuid.
You should always validate access to resources server-side.
Not being able to guess a uuid isn't a safe assumption. Since you are using Amazon Cognito, there should be a way in your server code to get the logged-in user (the uuid). When you are making a query to DynamoDB, you shouldn't rely on a uuid passed by an HTTP query (client-side), but use instead the uuid value of the logged-in user.
The uuid could be used in a Global Secondary Index, so that you can quickly query the orders of a user.
you can use UUID to store the same users data in dynamo as well. But make sure while fetching the details in dynamo what detail an API is actually needed like whether a store API should request only orders and not users so to differentiate those stuffs just add an attribute called "entity_type" which will have
order
user
store
other resources
So while fetching add this entity_type as well one of the where condition to filter out rest of the unwanted stuffs.
Related
My DynamoDB table currently has the PK, SK and a GSI called "EmailIndex".
The idea behind this is that when users create their account, the username will be set as the PK and the email would be set as the EmailIndex GSI.
This should let me allow the user to login with either the username or the email.
Is it possible to reliably ensure there is no duplicated EmailIndex record? If for example, two different people enter the exact same email at the exact same time with different usernames, how can I ensure both users don't end up creating a record with the same GSI?
Right now I'm under the assumption that this isn't actually possible with DynamoDB. In which case, what would be the acceptable and recommended approach of allowing username or email logins instead of mandating one or the other?
There isn't a way to guarantee unique values on a GSI. What I've done in the past to solve this is to have two records for the user, one with the data you have today, and one that is keyed on the email address. When doing a Put operation you can use a transaction to be sure both records succeed. You can still put the email in the GSI on the main record and use that for querying, or use the second record instead, and duplicate the data (which is what the GSI would do if you have the projection set to ALL).
My Dynamo Tables have tenant_id as the partition key in my multi-tenant application but my partition key also has other types of entities in it in addition to tenant_id.
For example: (This is a small example, we are using this pattern throughout)
PK SK Att
Customer-4312a674-54a user-abc 672453782
user-abc user-abc 672453782
I would like to use dynamodb:LeadingKeys to ensure data of one tenant can never be accessed by another tenant. How can I go about that in this case when PK is overloaded and has other entities in it as well.
In a multi-tenant system my recommendation would be to add the tenant-id as a prefix to the partition key of all items belonging to the tenant. That way you can use the dynamodb:LeadingKeys condition for access control.
The tenant-id should be known at query time for every query anyway, my guess is that it's probably stored in the session information. This means you can add the tenant-id to every Key and still do partition key overloading.
I am trying to do something that would be relatively simple for a relational database but I don't know how to do it for a nonrelational one.
I am trying to make a simple task web app on AWS where people can post their tasks.
I have a table called tasks which uses the userid from the auth token provisioned by AWS Cognito. I am wondering how I can return the user information. I do not want to rely on Cognito by simply calling it every time a user sends a request. So, my thought would be to create another table to store all of the user information. That, however, is not a very nonrelational way of doing things since JOINS are so bad.
So, I was wondering if I should do any of the following
a) Using RDS instead
b) Not use Cognito and set up my own Auth system
c) Just doing the JOIN with a table containing all of the user info
d) Doing the request to Cognito each time
Although I personally like the idea of cognito, at this time it has some major drawbacks...
You can not backup / restore a user pool without loosing their password, also you have to implement your own backup/restore.
A way around is to save the user password in a cognito custom attribute.
I expected by using api gateway/lambda authorizer to have all the user data in the lambda context but its not there. Or am indoing something wrong with api gateway template mapping 😬
Good thing api gateway/lambda authorizer, can be cached by up to an hour, wont call the authorizer function again which seems like a top feature.
Does not work well with cloudformation, with every attribute update it recreates the user pool without restoring the users, thus loosing the users.
I used it only in one implementation and ended up duplicating the users in DynamoDB as well.
I'm avoiding it ever since. I wish they solve these issues as it looks like a service to be included with every project saving lot of time.
Reading your post I asked myself the same questions and not sure the answer either 😄
Pricing seems fair.
The default 5 requests/second to get user info seems strange as it woukd be consumed by one page load doing multiple ajax api requests .
For this in DynamoDB, there is no need for another table. If the access patterns dictate you store the information in another object, then so be it, but more than likely it should be in the same table. Sounds like you need two different item types in the same table.
For the task PK of userid and SK of task::your-task-id. This would allow you to get all of a user's tasks easily or even a specific task very easily if you knew the task ID. You might even have an attribute that is a timestamp and then have a GSI that is the userID as the PK and the timestamp as the SK. then you could use the begins_with operator on the SK and "paginate through all of the user's tasks that are in the month of 2019-04".
For the user information, have the userID be the PK and the SK be user_info and attributes be the user's information.
The one challenge for this is if you were to go to extremes and one single user is doing thousands of ops per second. e.g. "All tweets by very popular celebrity". If you have such a use case there are ways around that as well, e.g. write sharding. These are just examples for you to play with. Without knowing all your access patterns, I cannot model everything you might want to do. I highly recommend you go watch this presentation from reInvent 2018.
I am trying to build out a scalable smart home infrastructure on AWS using iot core, lambda, and dynamodb along with the serverless framework and subsequent Android/iOS app.
I am implementing locations and rooms in dynamodb. A user can have many locations, and locations can have many rooms. I am used to using Firebase Firestore, so the use of partition keys and sort keys (hash and range?) and the combination to query are a little confusing. I implemented my own hash to use as a primary (partition? hash?) id. Here is the structure I am thinking of:
Location
id
name
username
I also added a secondary index on username, so that a user could query all of their locations.
Room
id
name
locationId
I also added a secondary index on locationId, so that a user could query all rooms for a given location
Here is the code in which I create the id's:
// need a unique hash for the id
let hash = event.name + event.username + new Date().getTime();
let id = crypto.createHash('md5').update(hash).digest('hex');
let location = {
id: id,
name: event.name,
username: event.username
};
And for rooms:
// need a unique hash for the id
let hash = event.name + event.locationId + new Date().getTime();
Since I'm fairly new to Dynamo/AWS, I'm wondering if this is an acceptable solution. Obviously I would expand on this by adding multiple devices under rooms by associating via the roomId. I would also like to be able to share devices, so I'm not quite sure how that would work, as the association for a user is on location - so I assume I would have to share location, room(s), and device(s) (which I think is how Google Home does it)
Any suggestions would be greatly appreciated!
EDIT
The queries that I can think of would be:
Get Location by Id
Get all Locations by User
Get Room by Id
Get all Rooms by Location
However as the app expands in the future, I would want these queries to be flexible (share location, get shared locations, etc)
I would want these queries to be flexible
Then noSQL in general and Dynamo specifically may not be the right choice.
As #varnit alludes to, noSQL DB's are very flexible in what you store, but very inflexible in how you can query that data.
Dynamo for instance can only return a list (Query) if you use a sort key (SK) or if you do a full table scan (not recommended). Otherwise, it can only return a single record.
I don't understand what a "shared location" would entail.
But with multiple tenets in Dynamo, (each user is only looking at their data) the easy solution would be to use userID as the partition key (PK).
I'd use a composite sort key of location#room
Get Location by Id --> GetItem(PK = User, SK = location)
Get all Locations by User --> Query (PK = User)
Get all rooms by Location --> Query (PK = User, SK starts with Location)
This one is a little trickier...
- Get Room by Id -->
If you really need to get a room without having the location, then you'd want to have room as stand-a-lone attribute in addition to having it as part of the sort key. Then you can create a local secondary index over it and query (PK = User, Index SK = Room)
I suspect that finding a room via GetItem(PK = User, SK = location#room) might work for you instead.
Key point, the partition key comparision is always equal. There's no start with, ends with or contains for the partition key comparison.
If you haven't seen them, take a look at the following videos
AWS re:Invent 2018: Building with AWS Databases: Match Your Workload to the Right Database (DAT301)
AWS re:Invent 2018: Amazon DynamoDB Deep Dive: Advanced Design Patterns for DynamoDB (DAT401)
Also be sure to read the SaaS Storage Strategies - Building a Multitenant Storage Model on AWS whitepaper.
EDIT
"location" and "room" can be whatever makes the most sense to your application. GUID or a natural key such as "Home". In a noSQL db, GUIDs are useful when multiple nodes are adding records. But a natural key is good when that what the application user will have handy. Since you don't want to have to look up a guid by the natural key. RDBMS practices don't apply to noSQL DBs.
So yes, I'd use "Home" as the location, meaning the user won't be able to have multiple "Home"s. But I don't see that as a big deal, I'd use "Home" and "Vacation House" in real life.
EDIT2
Dynamo doesn't care if it's a GUID or a natural key. It internally hashes the whatever value you use for partition key. All that matters is the number of distinct values. Distinct is distinct, doesn't matter if the value is '0ae4ad25-5551-46a7-8e39-64619645bd58' or 'charles.wilt#mydomain.com'. If your authorization process returns a GUID, use that. Otherwise use the username.
I'm building a DynamoDB app that will eventually serve a large number (millions) of users. Currently the app's item schema is simple:
{
userId: "08074c7e0c0a4453b3c723685021d0b6", // partition key
email: "foo#foo.com",
... other attributes ...
}
When a new user signs up, or if a user wants to find another user by email address, we'll need to look up users by email instead of by userId. With the current schema that's easy: just use a global secondary index with email as the Partition Key.
But we want to enable multiple email addresses per user, and the DynamoDB Query operation doesn't support a List-typed KeyConditionExpression. So I'm weighing several options to avoid an expensive Scan operation every time a user signs up or wants to find another user by email address.
Below is what I'm planning to change to enable additional emails per user. Is this a good approach? Is there a better option?
Add a sort key column (e.g. itemTypeAndIndex) to allow multiple items per userId.
{
userId: "08074c7e0c0a4453b3c723685021d0b6", // partition key
itemTypeAndIndex: "main", // sort key
email: "foo#foo.com",
... other attributes ...
}
If the user adds a second, third, etc. email, then add a new item for each email, like this:
{
userId: "08074c7e0c0a4453b3c723685021d0b6", // partition key
itemTypeAndIndex: "Email-2", // sort key
email: "bar#bar.com"
// no more attributes
}
The same global secondary index (with email as the Partition Key) can still be used to find both primary and non-primary email addresses.
If a user wants to change their primary email address, we'd swap the email values in the "primary" and "non-primary" items. (Now that DynamoDB supports transactions, doing this will be safer than before!)
If we need to delete a user, we'd have to delete all the items for that userId. If we need to merge two users then we'd have to merge all items for that userId.
The same approach (new items with same userId but different sort keys) could be used for other 1-user-has-many-values data that needs to be Query-able
Is this a good way to do it? Is there a better way?
Justin, for searching on attributes I would strongly advise not to use DynamoDB. I am not saying, you can't achieve this. However, I see a few problems that will eventually come in your path if you will go this root.
Using sort-key on email-id will result in creating duplicate records for the same user i.e. if a user has registered 5 email, that implies 5 records in your table with the same schema and attribute except email-id attribute.
What if a new use-case comes in the future, where now you also want to search for a user based on some other attribute(for example cell phone number, assuming a user may have more then one cell phone number)
DynamoDB has a hard limit of the number of secondary indexes you can create for a table i.e. 5.
Thus with increasing use-case on search criteria, this solution will easily become a bottle-neck for your system. As a result, your system may not scale well.
To best of my knowledge, I can suggest a few options that you may choose based on your requirement/budget to address this problem using a combination of databases.
Option 1. DynamoDB as a primary store and AWS Elasticsearch as secondary storage [Preferred]
Store the user records in DynamoDB table(let's call it UserTable)as and when a user registers.
Enable DynamoDB table streams on UserTable table.
Build an AWS Lambda function that reads from the table's stream and persists the records in AWS Elasticsearch.
Now in your application, use DynamoDB for fetching user records from id. For all other search criteria(like searching on emailId, phone number, zip code, location etc) fetch the records from AWS Elasticsearch. AWS Elasticsearch by default indexes all the attributes of your record, so you can search on any field within millisecond of latency.
Option 2. Use AWS Aurora [Less preferred solution]
If your application has a relational use-case where data are related, you may consider this option. Just to call out, Aurora is a SQL database.
Since this is a relational storage, you can opt for organizing the records in multiple tables and join them based on the primary key of those tables.
I will suggest for 1st option as:
DynamoDB will provide you durable, highly available, low latency primary storage for your application.
AWS Elasticsearch will act as secondary storage, which is also durable, scalable and low latency storage.
With AWS Elasticsearch, you can run any search query on your table. You can also do analytics on data. Kibana UI is provided out of the box, that you may use to plot the analytical data on a dashboard like (how user growth is trending, how many users belong to a specific location, user distribution based on city/state/country etc)
With DynamoDB streams and AWS Lambda, you will be syncing these two databases in near real-time [within few milliseconds]
Your application will be scalable and the search feature can further be enhanced to do filtering on multi-level attributes. [One such example: search all users who belong to a given city]
Having said that, now I will leave this up to you to decide. 😊