Suppose I want to store some entities in DynamoDB, and each entity is defined by 3 crucial attributes:
group_id [string] : the id of the group the entity belongs to.
from [int] : the start of the range (inclusive).
to [int] : the end of the range (inclusive).
And the constraint is:
Within a group, the overlapping ranges are not allowed.
Across groups however, overlapping is allowed.
Here are few examples of the entries:
("abc",10,21)
("xyz",13,27)
("xyz",45,61)
("abc",39,57)
("abc",81,93)
As you can see, there are no overlapping ranges within a group in the above list. Now if we want to add an entry to the above list, then here are few examples as to what is allowed, and what is not:
("abc",19,27) is not allowed, as its overlapping with the first item.
("abc",23,27) is allowed.
("xyz",39,47) is not allowed, as its overlapping with the third item.
("xyz",39,55) is allowed.
Given this scenario, my question is, how to design the schema and how should it be used so that it can prevent users from inserting overlapping ranges for a given group?
If the solution requires some (meta) attributes to be added to the schema, I'm fine with it; I'm fine with anything as long as it solves the problem. Other related questions to ponder over: should we add each entity as a separate row? Or all entities belonging to a single group, should go to one row (with a list/map attribute)?
The possible queries on the table would be like:
Given a group_id and a super-range {from and to}, return all entries having the same group_id, and whose from and to fall within the limit defined by the super-range (inclusive).
Specifically what are the options to decide on partition key , range key , secondary index (local /secondary) based on the queries listed above ?
step 1 - Create a partition key (group id), range key on (from ,to)
step 2 - Restriction on from, to fields need to be done at application level. 'check constraints' to enable this functionality is not available at database level in dynamodb
One way to do this is to use group id as primary key, use "from" as range key and "to" as a secondary range key (Local Secondary Index).
Then every time you want check a new item with range between x and y, you need to do following checks:
Check that there is no "from" value between x and y
Check that there is no "to" value between x and y
Check that the item with largest value of "from" such that "from" is smaller than x also has "to" smaller than x
Check that the item with smallest value of "to" such that "to" is larger than y also has "from" larger than y
Given that all these checks are using range key queries it should be fairly fast.
Note that if your range options are limited (say range can only be between 1 and 365) then much simpler solutions are possible
Related
I'm somewhat confused as to what the proper secondary index would look like for DynamoDB.
I have Name, Date, Period, Data attributes and want an index that lets me efficiently lookup by Name, Date, and Period.
I also want to efficiently lookup all Names for a given Date.
I tried setting my secondary index partition key to Name since I want those to be grouped together on nodes. And added attribute projections for Date and Period. Is this the way to go?
Every single access pattern needs to be enumerated and you need to think about its corresponding retrieval mechanism. Your base table provides one access mechanism. You can use GSIs for the additional mechanisms.
The base table and each GSI provide a PK and SK for you to use. The PK must be an individual value (sometimes composed of several values concatenated together with a separator like hash). The SK can be a sortable value, used either as a value or range. Those are the tools at your disposal.
"All names for a given date" might use a GSI where the date is the PK and the names are the SK.
At reasonable scale you don't have to think too much about hot partitions. At high scale (more than 1,000 write units needed per second) you'll have to think harder before putting everything under a single date PK for the GSI.
I have a DyanmoDB table that for the sake of this question looks like this:
id (String partition key)
origin (String sort key)
I want to query the table for a subset of origins under a specific id.
From my understanding, the only operator DynamoDB allows on sort keys in a Query are 'between', 'begins_with', '=', '<=' and '>='.
The problem is that my query needs a form of 'CONTAINS' because the 'origins' list is not necessarily ordered (for a between operator).
If this was SQL it would be something like:
SELECT * from Table where id={id} AND origin IN {origin_list}
My exact question is: What do I need to do to achieve this functionality in the most efficient way? should I change my table structure? maybe add a GSI? Open to suggestions.
I am aware that this can be achieved with a Scan operation but I want to have an efficient query. Same goes for BatchGetItem, I would rather avoid that functionality unless absolutely necessary.
Thanks
This is a case for using Filter Expressions for Query. It has IN operator
Comparison Operator
a IN (b, c, d) — true if a is equal to any value in the list — for
example, any of b, c or d. The list can contain up to 100 values,
separated by commas.
However, you cannot use condition expressions on key attributes.
Filter Expressions for Query
A filter expression cannot contain partition key or sort key
attributes. You need to specify those attributes in the key condition
expression, not the filter expression.
So, what you could do is to use origin not as a sort key (or duplicate it with another attribute) to filter it after the query. Of course filter first reads all the items has that 'id' and filters later which consumes read capacity and less efficient but there is no other way to query that otherwise. Depending on your item sizes and query frequency and estimated number of returned items BatchGetItem could be a better choice.
Is it possible to Query a DynamoDB table using both the hash & range key AND a local secondary index?
I have three attributes I want to compare against in my query. Two are the main hash and range keys and the third is the range key of the local secondary index.
No, but that shouldn't be necessary based on your description of what you are trying to accomplish.
If you are trying to access an object based on the hash and range key (of the main table) as well as an additional attribute, selecting on only the hash and range of the main table (which is required to return a single record by definition) will return that record.
If your concern is that the third attribute may be a value that you want to ignore the entire record you can use a query filter to have that item filtered out by DynamoDB or you can use logic in your application to ignore that object.
I asked a question similar to this previously (How to use RecId as a foreign key in a form) but would like to explore it a bit further in a more complex scenario.
Replacement keys work great when you have indexes set up and allow duplicates set to no, but they don't seem to work at all with multiple-field indexes or when allow duplicates is set to yes.
Is there way, programmatically, to replace a foreign key in a grid with a translated value without using replacement keys? I tried writing a display method to override the field, but some odd behavior resulted--fields moving around in the grid, and the display method being unaware of which row to use, thus all values in the entire column were the same.
Table A: Bob:1, Sally:1, Sue:3
Table B: 1:Apples, 2:Apples, 3:Oranges
The "people" are tied to their favorite "foods" by the food RecId, refererenced in the People table. Assume there is additional data in other columns that make these records unique, so consolidating "1:Apples" and "2:Apples" is not possible.
It seems there should be a way to write a display method to overwrite a field value in a grid. Any suggestions? Sample code?
Thanks
Firstly, surrogate FK replacement does (or at least should) work with composite keys (e.g., {First Name, Last Name}).
Secondly, you state that there is "additonal data in other columns" that make these records unique...Then why aren't these columns being combined with the food's name to form an alternate key? The data model seems incorrect (or at least some metadata isn't being made consistent with the conditions you've stated)
Thirdly, any Field Group can be chosen as the ReplacementFieldGroup on a Reference Group control. That alone will allow you to do basically whatever you want. That said, I would strongly encourage you to use an alternate key as your replacement field group whenever possible due to the semantics of surrogate FK replacement.
Flow:
1) User types a value(s) into reference group.
2) User's tabs out.
3) User's typed value(s) are used to look up a record in the related table.
4a) If the user's typed in value(s) are uniquely mapped to a record that record is chosen, else,
4b) If the user's typed in values are not unique a lookup is presented to allow the user to pick which record they "meant". Note that the lookup must therefore present a collection of uniquely identifiable records so that the user knows which record to pick (if the records all look the same in the lookup then they'll have no idea what in the hell they should pick.)
5) Upon successful resolution of the typed values, the record is set back on the source form.
Given this flow, it is obvious that steps 3-5 will be broken if there is no unique index (key) on the table. (How is the user supposed to specify a unique reference to the record if the record has no means of being uniquely identified (assuming you don't want to display RecId to the user)???)
In the exceptional case that you decide that you still want to use a non-unique index as your replacement field group you must implement resolveReference and lookupReference to provide the user a unique resolution/lookup experience (to handle steps 3-5 in the above flow). Note: The common use case for this is wanting to effectively eliminate non-selective fields from being displayed in Reference Group and instead letting some outer context or mode implicitly set that value. E.g., if the alternate key was {Size, Color}, one could potentially make "Color" a global form context--perhaps by having the user pick a color at the top of the form--and only have the user enter Size into Reference Group...The Color could then be implicitly added back via the resolveReference and lookupReference overrides.
I thought this would be easy but I can't figure it out.
I have a DynamoDB table where all the items have the same attributes. One of the attributes is a numeric one named ytd. I simply want the first 5 items sorted by ytd.
you cant do it in a simple way.
dynamo db return ordered results of the same hash key
so if your hash key here is X, and range key will be 'ytd', then in order to get 'ytd' ordered items then X should be the same.
i dont know your exactly flow, but if you are not query X (you just need to get ordered 'ytd' no matter for X), then you can add a global secondary partition with hash key=partition, range key=ytd as described here:
How to choose a partition key in DynamoDB for a chat app