Is there any way to get sorted result out of Dynamodb when using Scan/Query APIs? I know in Query API you can sort by Rangekey and ScanIndexForward which sorts the result ascending if the value is true and descending if false;
+But as far as I understood you can have one range key, so how if I want to sort based on different fields?
+Also if I'm using scan, it seems there is no option to sort the result either!
Any help is appreciated!
For the first question about having only one range key, you can use Local secondary Index. You assign a normal attribute as the range key of the LSI and DynamoDB will sort your rows (with the same hashkey) by comparing that attribute.
So essentially LSI gives you "additional rangeKey". You can create up to 5 LSIs.
See here and here for example of querying LSI. You can treat an Index just like a regular table. You can do query & scan on index (but not put).
For your second question about sorting the rows globally instead of sorting items with the same hashkey, I don't think DynamoDB supports this feature out-of-the-box. You will have to
a) scan and sort the items on your own
b) or create a global secondary index with just one hash key and dump all your items into that hashkey. It is not recommended because this creates a hot partition in GSI.
c) or design your schema to avoid having to sort items globally.
Related
I need to get the rows by key (e.g. where status is "Active") but with sorting on multiple columns.
I'm using the pagination that's why I cannot sort the result after fetching it from the DynamoDB. (Just for the information, I'm using the serverless framework)
Expected Output is array of rows sorted (ordered) by multiple columns.
In DynamoDB you get "free" lexicographical sorting on the range keys.
When an item is being inserted first its partition is calculated based on the partition key then the item is inserted into a b-tree which keeps the partition lexicographically sorted at all times. This doesn't give you all of the features of SQLs Order By, which is not supported
So if your sort keys look something like this
Status#Active#UserId#0000004
You can do "begins_with" query with SK = "Status#Active"
This will give you all of the items that are in active status ordered by the UserId (that has to be zero-padded in order to enforce the lexicographical order).
You can't do that. Sorting can be only done on SK under the same PK. You could combine multiple columns into one value and query based on it. Something like column1-value1#column2-value2.
In that case you'll probably have issue in updating that field, dynamodb streams could help with it. You can trigger event on any modification and asynchronously update that sorting field.
I'd like to list records from my DDB table ordered by creation date.
My table has an attribute DateCreated.
All examples I can find describe ordering within some partition.
But I want global ordering.
Am I supposed to create an artificial attribute which will have the same value across all records, just to use it as a partition key? E.g. add new attribute GlobalPartition with value 1 to every record in the table, and create a GSI with partition key GlobalPartition and sort key DateCreated. Isn't there a better way?
Thx!
As you noticed, DynamoDB indeed does not have an option to sort items "globally". In other words, there is no way to Scan the database in sorted partition-key order. You can only sort items inside one partition, sorted by the "sort key".
When you have a small amount of data, you can indeed do what you said: Have a single partition with everything in this partition. However it's not clear how practical this approach becomes as your single partition grows - to gigabytes or terabytes, and how well DynamoDB can load-balance when you have just a single partition (I never saw any DynamoDB documentation which answer this question).
So another option is not to have a single partition but rather have a number of them. For example, consider that you want to sort items by date. Now insead of having a single partition, have a partition per month, i.e., the partition key is the month number. Now, if you want to sort everything within a month, you can do it directly, but if you want to get a sorted list of a full year, you need to Query twelve partitions, in order, getting a sorted list in each one and combining it to a sorted list for the full year. So-called time-series databases are often modeled this way.
If you want to sort any data in DynamoDB you need to add Sort Key index on that attribute. If value is not in attribute which maps to tables' sort key, or table does not have sort key, then you need to create GSI and put GSI's sort key on that attribute. You can use LSI too. Any attribute, which maps to "Sort Key" of any index. Table, LSI, GSI.
Check for more details "ScanIndexForward" param of the query request.
If ScanIndexForward is true, DynamoDB returns the results in the order in which they are stored (by sort key value). This is the default behavior. If ScanIndexForward is false, DynamoDB reads the results in reverse order by sort key value, and then returns the results to the client.
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#API_Query_RequestSyntax
UI has checkbox too for this:
"Global sort" is not possible, while "global" would mean scan operation and it just runs through all rows in database and filters by filters, yet it does not have sorting option. On query on attribute mapped to sort key has ScanIndexForward option to change sort direction.
I want to sort $ctx.result.items and reponse the sortedResultI don't want to manually write Velocity Template Language to sort $ctx.result.items in Response Mapping. Is there any better approach to response the sortedResult in AWS AppSync ?
What type of sorting are you looking to do? If it's ascending/descending using a DynamoDB resolver then you can set that on the ScanIndexForward argument for this on the request template: https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-dynamodb.html
( if you found a solution I hope this will help someone else )
It depends on how you designed GSI or LSI to your DynamoDB table.
As stated here "DynamoDB builds an unordered hash index on the hash primary key attribute, and a sorted range index on the range primary key attribute."
Here hash index is same as partition key, and range index is same as sort key (old and new terms).
Similar text is stated here - "All items with the same partition key value are stored together, in sorted order by sort key value."
So if you added a GSI or LSI to your DynamoDB table in a way stated above (e.g. all your Products IDs are hash / partition keys and creation times are range / sort keys and you need to sort Products by creation time) you can use something similar to example defined in this page of StackOverflow.
I know BatchGetItems allows for retrieval of multiple hash keys. To save on the read capacity, I like to know if Query provide same functionality via some "IN" keyword I can use? ie, all primary keys will be inserted into an array for Query to search "IN" in the array.
Query doesn't provide what you want. As per the documentation here:
KeyConditionExpression: The condition must perform an equality test on a single partition key value. The condition can also perform one of several comparison tests on a single sort key value. Query can use KeyConditionExpression to retrieve one item with a given partition key value and sort key value, or several items that have the same partition key value but different sort key values.
BatchGetItem is the only option that you have.
I thought this would be easy but I can't figure it out.
I have a DynamoDB table where all the items have the same attributes. One of the attributes is a numeric one named ytd. I simply want the first 5 items sorted by ytd.
you cant do it in a simple way.
dynamo db return ordered results of the same hash key
so if your hash key here is X, and range key will be 'ytd', then in order to get 'ytd' ordered items then X should be the same.
i dont know your exactly flow, but if you are not query X (you just need to get ordered 'ytd' no matter for X), then you can add a global secondary partition with hash key=partition, range key=ytd as described here:
How to choose a partition key in DynamoDB for a chat app