I think I'm misunderstanding DynamoDb. I would like to query for all items, with a child field of the json, which match an identifier I'm passing. The structure is something like -
{
"messageId": "ced96cab-767e-509198be5-3d2896a3efeb",
"identifier": {
"primary": "9927fd47-5d33-4f51-a5bb-f292a0c733b1",
"secondary": "none",
"tertiary": "cfd96cab-767e-5091-8be5-3d2896a3efeb"
},
"attributes": {
"MyID": {
"Type": "String",
"Value": "9927fd47-5c33-4f51-a5bb-f292a0c733b1"
}
}
}
I would like to query for all items in DynamoDb that has a value of MyID that I'm passing. Everything I've read seems to say you need to use the key which in my case is the messageId, this is unique for each entry and not a value I can use.
Hope this makes sense.
The DynamoDB Query API can be used only if you know the value of Partition key. Otherwise, you may need to scan the whole table using FilterExpression to find the item.
Scanning tables
You can create GSI on scalar attribute only. In the above case, it is a document data type (i.e. MAP). So, GSI can't be created.
Related
It is a jobPosts schema that has a posted_date as one of the attributes. The goal is to query all the job posts between two dates.
Here is the schema for your reference:
{
'job_id': {S: jobInfo.job_id},
'company': {S: jobInfo.company},
'title': {S: jobInfo.title},
'posted_on': {S: jobInfo.posted_on},
}
posted_on' is based on ISO string (2019-11-10T10:52:38.013Z). job_id is the primary key (partition key) and since I need to query the dates, I created GSI(partition key) on posted_on. Now here is the query:
const params = {
TableName : "jobPosts",
IndexName: 'date_for_filter_purpose-index',
ProjectionExpression:"job_id, company, title, posted_on",
KeyConditionExpression: "posted_on BETWEEN :startDate AND :endDate",
ExpressionAttributeValues: {
":startDate": {S: "2019-10-10T10:52:38.013Z"},
":endDate": {S: "2019-11-10T10:52:38.013Z"}
}
};
I have one document in dynamoDB and here it is:
{
job_id:,
company: "xyz",
title: "abc",
posted_on: "2019-11-01T10:52:38.013Z"
}
Now, on executing this, I get the following error:
{
"message": "Query key condition not supported",
"code": "ValidationException",
"time": "2019-11-11T06:15:37.231Z",
"requestId": "J078NON3L8KSJE5E8I3IP9N0IBVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 12.382362030893768
}
I don't know what is wrong with the above query.
Update after Tommy Answer:
I removed the GSI on posted_on and re-created the table with job_id as partition key and posted_on as sort key. I get the following error:
{
"message": "Query condition missed key schema element: job_id",
"code": "ValidationException",
"time": "2019-11-12T11:01:48.682Z",
"requestId": "M9E793UQNJHPN5ULQFJI2NR0BVVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 42.52613025785952
}
As per this SO answer, GSI should be able to query the dates using BETWEEN keyword.
The answer you refer to relates to a query where the partition key has a specific value and the sort key is in a given range. It's analagous to select * from table where status=Z and date between X and Y. That's not what you're trying to do, if I read your question correctly. You want select * from table where date between X and Y. You cannot do this with DynamoDB query - you cannot query a partition key by range.
If you knew that your max range of query dates was on a given day then you could create a GSI with a partition key set to the computed YYYYMMDD value of the date/time and whose sort key was the full date/time. Then you could query with a key condition expression for a partition key of the computed YYYYMMDD and a sort key between X and Y. For this to work, the YYYYMMDD of X and Y would have to be the same.
If you knew that your max range of query dates was a month then you could create a GSI with partition key set to the computed YYYYMM of the date/time and whose sort key was the full date/time. For this to work, the YYYYMM of X and Y would have to be the same.
I guess it's a little counter-intuitive but DynamoDB supports only .eq condition on partition key attributes.
As per KeyConditions Documentation
You must provide the index partition key name and value as an EQ condition. You can optionally provide a second condition, referring to the index sort key.
Furthermore, in Query API Documentation you can find the following
The condition must perform an equality test on a single partition key value.
The condition can optionally perform one of several comparison tests on a single sort key value. This allows Query to retrieve one item with a given partition key value and sort key value, or several items that have the same partition key value but different sort key values.
That explains the error message you are getting.
One of the solutions might be to create a composite primary key with posted_on attribute as the sort key, instead of the GSI. Then, depending on your use case and access pattern, you'll need to figure out which attribute would work best as the partition key.
This blog should help you to choose the right partition key for your schema.
I want to query a DDB GSI with key condition, and apply filter on returned result using contains function.
Data I have in DDB table:
{
"lookupType": "PRODUCT_GROUP",
"name": "Spring framework study set",
"structureJson": {
"productIds": [
"FBCUPOQsrp",
"Y4LDaiViLY",
"J6N3UWq9CK"
]
},
"ownerId": "mT9R9y6zGO"
}
{
"lookupType": "PRODUCT_GROUP",
"name": "Relational databases study set",
"structureJson": {
"productIds": [
"QWQWQWQWQW",
"XZXZXZXZXZ"
]
},
"ownerId": "mT9R9y6zGO"
}
...
I have a compound GSI (ownerId - HASH, lookupType - RANGE).
When I try to query the DDB (query structure is in "2" field) I get the result (the result is in "3"):
{
"0":[
],
"2":{
"TableName":"Products",
"IndexName":"ProductsOwnerIdLookupTypeIndex",
"KeyConditionExpression":"#ownerId = :ownerId and #lookupType = :lookupType",
"FilterExpression":"contains(#structureMember, :memberId)",
"ExpressionAttributeNames":{
"#ownerId":"ownerId",
"#lookupType":"lookupType",
"#structureMember":"structureJson.productIds"
},
"ExpressionAttributeValues":{
":ownerId":"mT9R9y6zGO",
":lookupType":"PRODUCT_GROUP",
":memberId":"FBCUPOQsrp"
}
},
"3":{
"Items":[
],
"Count":0,
"ScannedCount":2
}
}
The returned result set is empty, despite I have data with given field values.
How I see the query (or what I want to achieve):
When I query the GSI with ownerId = mT9R9y6zGO and lookupType = PRODUCT_GROUP it will find 2 items - Spring and Relational DB sets
As the second step DDB will scan the returned query result with contains (structureJson.productIds, FBCUPOQsrp) filter expression and it should return one result to me, but I get empty set
Is something wrong with the query or do I miss some point in DDB query workflow?
I have an AWS DynamoDb cart table with the following item structure -
{
"cart_id": "5e4d0f9f-f08c-45ae-986a-f1b5ac7b7c13",
"user_id": 1234,
"type": "OTHER",
"currency": "INR",
"created_date": 132432423,
"expiry": 132432425,
"total_amount": 90000,
"total_quantity": 2,
"items": [
{
"amount": 90000,
"category": "Laptops",
"name": "Apple MacBook Pro",
"quantity": 1
}
]
}
-
{
"cart_id": "12340f9f-f08c-45ae-986a-f1b5ac7b1234",
"user_id": 1234,
"type": "SPECIAL",
"currency": "INR",
"created_date": 132432423,
"expiry": 132432425,
"total_amount": 1000,
"total_quantity": 2,
"items": [
{
"amount": 1000,
"category": "Special",
"name": "Special Item",
"quantity": 1
}
]
}
The table will have cart_id as Primary key,
user_id as an Index or GSI,
type as an Index or GSI.
I want to be able to query the cart table,
to find the items which have user_id = 1234 AND type != "SPECIAL".
I don't know if this means for the query -
--key-condition-expression "user_id = 1234 AND type != 'SPECIAL'"
I understand that an AWS DynamoDb table cannot be queried using multiple indexes at the same time,
I came across the following question, it has a similar use case and the answer is recommending creating a composite key,
Querying with multiple local Secondary Index Dynamodb
Does it mean that while putting a new item in the table,
I will need to maintain another column like user_id_type,
with its value as 1234SPECIAL and create an Index / GSI for user_id_type ?
Sample item structure -
{
"cart_id": "5e4d0f9f-f08c-45ae-986a-f1b5ac7b7c13",
"user_id": 1234,
"type": "OTHER",
"user_id_type" : "1234OTHER",
"currency": "INR",
"created_date": 132432423,
"expiry": 132432425,
"total_amount": 90000,
"total_quantity": 2,
"items": [
{
"amount": 90000,
"category": "Laptops",
"name": "Apple MacBook Pro",
"quantity": 1
}
]
}
References -
1. Querying with multiple local Secondary Index Dynamodb
2. Is there a way to query multiple hash keys in DynamoDB?
Your assumption is correct. Maybe you can add into that a delimitter field1_field2 or hash them if either of them is too big in size hashOfField1_hashOfField2
That mean spending some more processing power on your side, however. As DynamoDB does not natively support It.
Composite key in DynamoDB with more than 2 columns?
Dynamodb: query using more than two attributes
Additional info on your use case
KeyConditionExpression only allowed for the hash key.
You can put it in the FilterExpression
Why is there no **not equal** comparison in DynamoDB queries?
Does it mean that while putting a new item in the table,
I will need to maintain another column like user_id_type,
with its value as 1234SPECIAL and create an Index / GSI for user_id_type?
The answer is it depends on how many columns (dynamodb is schema-less, by a column I mean data field) you need and are you happy with 2 round trips to DB.
your query:
user_id = 1234 AND type != "SPECIAL"
1- if you need all information in the cart but you are happy with two round trips:
Solution: Create a GSI with user_id (HASH) and type (RANGE), then add cart_id (base table Hash key) as projection.
Explanation: so, you need one query on index table to get the cart_id given user_id and type
--key-condition-expression "user_id = 1234 AND type != 'SPECIAL'"
then you need to use cart_id(s) from the result and make another query to the base table
2- if you do not need all of cart information.
Solution: you need to create a GSI and make user_id HASH and type as RANGE and add more columns (columns you need) to projections.
Explanation: projection is additional columns you want to have in your index table. So, add some extra columns, which are more likely to be used as a result of the query, to avoid an extra round trip to the base table
Note: adding too many extra columns can double your costs, as any update on base table results in updates in GSI tables projection fields)
3- if you want just one round trip and you need all data
then you need to manage it by yourself and your suggestion can be applied
One possible answer is to create a single index with a sort key. Then you can do this:
{
TableName: "...",
IndexName: "UserIdAndTypeIndex",
KeyConditionExpression: "user_id = :user_id AND type != :type",
ExpressionAttributeValues: {
":user_id": 1234,
":type": "SPECIAL"
}
}
You can build GraphQL schema with AWS AppSync from your DynamoDB table and than query it in your app with GraphQL. Link
Lets suppose I have the following JSON
{
"id": "ads",
"model": "PS-19213",
"trips": [
{
"time": {
"startTime": "2016-11-08T14:59:07.198Z",
"endtime": "2016-11-08T15:01:01.132Z"
}
}
]
}
I have taken property 'id' as a partition key. My question is can I take the property of a map as a sort/range key in dynamoDB i.e. trips[0].time.startTime as a sort key. Although I know whole map can't be a sort key but property of map is itself scalar.
The Partition/Sort Key must be a a scalar element (String, Number, or Binary).
A KeySchemaElement must be a scalar, top-level attribute (not a nested
attribute). The data type must be one of String, Number, or Binary.
You can't create indexes on Document (i.e. Map and List) data type as well.
Link
I have a table in DynamoDB that looks like this:
I added a global secondary index on "Category" to the table and it worked fine and gave me the number of items in the table under item count.
I then realized that i actually needed to be able to search for in a particular "Category" but sorted by "UserRating"
So I deleted the GSI and made a new one like this:
This all worked fine I thought, the names where correct the types (string) for Category and (number) for UserRating was correct.
But then after it finished creating the GSI I looked at the console and it is showing item count 0 even though there should be 13 in this testing table as pictured below:
Thanks for your help.
As per Amazon documentation this is being updated approx every 6 hours.
ItemCount - The number of items in the global secondary index. DynamoDB updates this value approximately every six hours. Recent changes might not be reflected in this value.
In my case, even though the console was still showing ItemCount zero and was returning no results for scan/query of the index, I was able to successfully query it from my code.
I think this is more likely to happen when you have a simple type mis-match between the key-schema types and the concrete item types.
From Managing Global Secondary Indexes -
Note
In some cases, DynamoDB will not be able to write data from the table to the index due to index key violations. This can occur if the data type of an attribute value does not match the data type of an index key schema data type, or if the size of an attribute exceeds the maximum length for an index key attribute. Index key violations do not interfere with global secondary index creation; however, when the index becomes ACTIVE, the violating keys will not be present in the index.
DynamoDB provides a standalone tool for finding and resolving these issues. For more information, see Detecting and Correcting Index Key Violations.
By Example
Item looks like:
"Items": [
{
"Timestamp": {
"N": "1542475507"
},
"DevID": {
"S": "slfhioh1234oi23lk23kl4h235pjpo235lnsfvuwerfj2roin2l3rn9fj9f8hwen"
},
"UID": {
"S": "1"
}
}
],
index looks like:
"GlobalSecondaryIndexes": [
{
"IndexName": "UID-Timestamp-index",
"Projection": {
"ProjectionType": "KEYS_ONLY"
},
"ProvisionedThroughput": {
"WriteCapacityUnits": 1,
"ReadCapacityUnits": 1
},
"KeySchema": [
{
"KeyType": "HASH",
"AttributeName": "UID"
},
{
"KeyType": "RANGE",
"AttributeName": "Timestamp"
}
],
}
]
Table has the attribute definitions:
"AttributeDefinitions": [
{
"AttributeName": "Timestamp",
"AttributeType": "S"
},
{
"AttributeName": "UID",
"AttributeType": "S"
}
]
That item will NOT appear in your new index.
It is completely possible to have a mismatch in type ( in this case "S" != "N" ) without it being flagged when created. This makes sense. You may want to do this sort of thing on purpose, but when you do it accidentally - it's not great.
I also had strange behavior (no results) when the index name contains dashes, as in the OP's screenshots. Replacing the dashes with underscores fixed my problem.
Found the answer. I had my read write capacity set to 1 unit for everything while testing as soon as I increased it fixed the error and I could see the items.