I have created list sub-partitions through sub-partition template on existing interval partition table, Now I want to drop the sub-partition template to go back to the original partition strategy i.e interval partition only.
Please suggest.
Related
With DynamoDB, there is simply no straightforward way to perform an indexed range query over a column. Primary key, local secondary index, and global secondary index all require a partition key to range query.
For example, suppose I have a high-scores table with a numerical score attribute. There is no way to get the top 10 scores or top scores 25 to 50 with an indexed range query
So, what is the idiomatic or preferred way to perform this incredibly common task?
Settle for a table scan.
Use a static partition key and take advantage of partition queries.
Use a fixed number of static partition keys and use multiple partition queries.
It's either 2) or 3) but it depends on the amount and structure of data as well as the read/write activity.
There's no generic answer here as it's use-case specific.
As long as you can get away with it, you probably want to use 2) as it only requires a single Query API call. If you have lots of data or heavy read/write activity, you'd use some bucketing-strategy (very close to your third option) to write to multiple partitions, then do multiple queries and aggregate the results.
DDB isn't suited for analytics. As Maurice said you can facilitate what you need via secondary index, but there are also other options to consider:
If you are providing this Top N to your customers consistently/frequently and N is fixed, then you can have dedicated item(s) that hold this information and you would update that/those item(s) upon writing an item to a table. You can have 1 item for the whole top N or you can apply some bucketing strat.
If your system needs this information infrequently (on some singular occasions), then scan might be also fine.
If this is for analytics/research, consider exporting the table to S3 and using Athena.
I'd like to list records from my DDB table ordered by creation date.
My table has an attribute DateCreated.
All examples I can find describe ordering within some partition.
But I want global ordering.
Am I supposed to create an artificial attribute which will have the same value across all records, just to use it as a partition key? E.g. add new attribute GlobalPartition with value 1 to every record in the table, and create a GSI with partition key GlobalPartition and sort key DateCreated. Isn't there a better way?
Thx!
As you noticed, DynamoDB indeed does not have an option to sort items "globally". In other words, there is no way to Scan the database in sorted partition-key order. You can only sort items inside one partition, sorted by the "sort key".
When you have a small amount of data, you can indeed do what you said: Have a single partition with everything in this partition. However it's not clear how practical this approach becomes as your single partition grows - to gigabytes or terabytes, and how well DynamoDB can load-balance when you have just a single partition (I never saw any DynamoDB documentation which answer this question).
So another option is not to have a single partition but rather have a number of them. For example, consider that you want to sort items by date. Now insead of having a single partition, have a partition per month, i.e., the partition key is the month number. Now, if you want to sort everything within a month, you can do it directly, but if you want to get a sorted list of a full year, you need to Query twelve partitions, in order, getting a sorted list in each one and combining it to a sorted list for the full year. So-called time-series databases are often modeled this way.
If you want to sort any data in DynamoDB you need to add Sort Key index on that attribute. If value is not in attribute which maps to tables' sort key, or table does not have sort key, then you need to create GSI and put GSI's sort key on that attribute. You can use LSI too. Any attribute, which maps to "Sort Key" of any index. Table, LSI, GSI.
Check for more details "ScanIndexForward" param of the query request.
If ScanIndexForward is true, DynamoDB returns the results in the order in which they are stored (by sort key value). This is the default behavior. If ScanIndexForward is false, DynamoDB reads the results in reverse order by sort key value, and then returns the results to the client.
https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Query.html#API_Query_RequestSyntax
UI has checkbox too for this:
"Global sort" is not possible, while "global" would mean scan operation and it just runs through all rows in database and filters by filters, yet it does not have sorting option. On query on attribute mapped to sort key has ScanIndexForward option to change sort direction.
I have a scenario in which I will have to list the incoming requests of a user sorted based on creation time and priority(High, Medium, Low) along with pagination. Is there a way to achieve this in dynamoDb ?
Right now I'm using a secondary Index like userId-createdAt-index which sorts data based on creation time and further sorting the request based on priority separately in the frontend. Somebody please provide a right solution for this.
You're correct to use an index with a sort key. This could also be your primary index, thus reducing how many indexes you need, but that of course depends on whether you already have a sort key on your primary.
DDB guarantees the order of a sorted index, so paging will correctly page by date for you, if you want to reverse the order, add the ScanIndexForward to your query and set it to false.
Your model of query / sort by date at the DB level, then sort by other fields at the application level is normal and correct.
Depending very much on your use-case, another option to consider is querying by priority by using KeyConditions and adding the condition #priority EQ :priority, but I doubt this is what you want.
I am new to DynamoDB schema designing. We have a table that stores metadata information for a customer with HashKey being CustomerId. The table also includes an attribute called "isActive" which is not a boolean. If customer unregisters, we plan to set the 'isActive' attribute to be empty.
We wish to pull list of all customerIds that are active. I read about 'sparseIndexes' wherein we can create a GSI on the 'isActive' attribute and only records with 'non-empty' values will be populated in the GSI.
However, it appears scanning is the only way to retrieve list of active customerIds. We can either
a) Scan entire table and filter only active customerIds at application layer
b) Scan the GSI which will be smaller than base table, but not necessarily very small (I would expect at least 1000+ records in it).
Are there any better design approaches to solve this by achieving high cardinality?
Sounds like you have a fairly good understanding of your options. Using GSIs to create a sparse index is fairly common for the access pattern you describe. Keep in mind that you can run a query operation against the index (as opposed to a scan), which will make the operation very fast. In the event you have many items, you could always paginate through the results.
Keep in mind you can add/remove the GSI Primary Key for the item to include/exclude the item from the index. For example, lets say your table has a GSI with a Partition (Hash) key named GSI1PK. Here's what it could look like with 4 customer items defined:
Notice that only Joe and Jill have a GSI1PK value defined, while Sue and Sam do not. Since I defined a global secondary index on GSI1PK, only items with that attribute defined will get projected into that index. Logically, that index would look like this:
If you want to remove Joe or Jill from GSI1, simply update the item to REMOVE GSI1PK from those items. Likewise, if you want to add Sue or Sam to the index, update the item to ADD the GSI1PK attribute to those items.
As primary key I have an id for a recipe and the sort key is the type of food (breakfast, meal, snack, etc).
Is there a way with scan or query to get all the items with a given sort key?
As others have pointed in the comments, you can't query a sort key in the sense that there is no operation that gives a list of items that have the same sort key.
In fact, the whole reason for a sort key is generally to order items in a particular partition.
Putting the two together, what you need is a way to partition the items by the food type and then query on that. Enter the Global Secondary Index (GSI).
With the help of a GSI you can index the data in your table in a way that the food type becomes the partition key, and some other attribute becomes the sort key. Then, getting all the items that match a particular food type becomes possible with a Query.
There are a few things to keep in mind:
a GSI is like another table: it consumes capacity that you will be charged for
a GSI is eventually consistent, meaning changes in the table could take a bit of time before being reflected in the GSI
if you end up creating a GSI where the choice of partition key results in very large partitions, it can lead to throttling (reduced throughput) if any one partition receives a lot of requests
Some more guidelines: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes-general.html
But before you start creating GSIs, consider for a moment the schema of your table: your choice of partition key seems less than ideal. On the one hand, using the recipe id as the partition key is great because it probably results in very good spread of data but on the other hand, you have no ability to use queries on your table without creating GSIs.
Instead of recipe id as the partition key, consider creating a partition key composed of food type, and perhaps another attribute. This way, you can actually query on food type, or perhaps issue several queries to retrieve all items of a particular food type.