I am trying query the values in dynamoDB and I have still the error.
Date is ISO-8601 (String) = createdAt and is it sort key.
My params:
{
TableName: 'Pool',
ExpressionAttributeValues: { ':oin': 'lol', ':from': '2017-12-16T20:26:02.594Z' },
KeyConditionExpression: 'oin = :oin',
ConditionExpression: 'createdAt >= :from',
ProjectionExpression: 'createdAt, h10m, h30m, h1h, h24h, accepted, stale, dupl, oth',
ScanIndexForward: false
}
I try GE with same result.
I generate the date with this code in Node.js:
var date = new Date();
date.setHours(date.getHours()-24);
var dateiso = date.toISOString();
I get following error:
ValidationException: Value provided in ExpressionAttributeValues unused in expressions: keys: {:from}
Any idea how to solve ConditionExpression? Thank you
The error message seems to indicate that you have an unused ExpressionAttributeValue of :from.
The ConditionExpression attribute that you specified can only be used for an Update, Delete and PutItem operation. If the createdAt attribute is a sort key, you want to specify that in the KeyConditionExpression along with oin.
For example:
KeyConditionExpression: 'oin = :oin AND createdAt >= :from'
Related
I have a lambda function that make some requests on DynamoDB.
var ddb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
const lookupminutes = 10;
var LookupDate = new Date(Date.now() - 1000 * lookupminutes);
params = {
TableName: TableName,
IndexName: "requestdate-index",
KeyConditionExpression: "requestdate > :startdate",
ExpressionAttributeValues: {":startdate": {S: LookupDate.toISOString()}
},
ProjectionExpression: "id, requestdate"
};
var results = await ddb.query(params).promise();
When running the lambda function, I'm getting the error : "Query key condition not supported" in the line that runs the query against DynamoDB
The field requestdate is stored in the table as a string.
Does anyone know what am I doing wrong please ?
Thanks.
You cannot use anything other than an equals operator on a partition key:
params = {
TableName: TableName,
IndexName: "requestdate-index",
KeyConditionExpression: "requestdate = :startdate",
ExpressionAttributeValues: {":startdate": {S: LookupDate.toISOString()}},
ProjectionExpression: "id, requestdate"
};
If you need all of the data back within the last 10 mins then they you have two choices, both of which are not very scalable, unless you shard your key (1a):
Put all the data in your index under the same partition key with sort key being timestamp. Then use KeyConditionExpression like:
gsipk=1 AND timestamp> 10mins
As all of the items are under the same partition key, the query will be efficient but at the cost of scalability as you will essentially bottleneck your throughput to 1000WCU.
1a. And probably the best option if you need scale beyond 1000 WCU is to do just as above except use a random number for the partition key (within a range). For example range = 0-9. That would give us 10 unique partition keys allowing us to scale to 10k WCU, however would require us to request 10 Query in parallel to retrieve the data.
Use a Scan with FilterExpression on the base table. If you do not want to place everything under the same key on the GSI then you can just Scan and add a filter. This becomes slow and expensive as the table grows.
So I have this notifications table with the following columns:
PK: (which stores the userId)
sentAt: (which stores the date the notifications was sent)
data: (which stores the data of the notification)
Read: (a boolean value which tells if the user has read the specific notification)
I wanted to create a GSI to get all the notification from a specific user that are not read (Read: False)
So the partition key would be userId and the sort key would be Read but the issue here is that I cannot give a boolean value to the sort key to be able to query the users that have not read the notifications.
This works with scan but that is not the result I am trying to achieve. Can anyone help me on this? Thanks
const params ={
TableName: await this.configService.get('NOTIFICATION_TABLE'),
FilterExpression: '#PK = :PK AND #Read = :Read',
ExpressionAttributeNames: {
'#PK': 'PK',
'#Read': 'Read',
},
ExpressionAttributeValues: {
':PK': 'NOTIFICATION#a8a8e4c7-cab0-431e-8e08-1bcf962358b8',
':Read': true, *//this is causing the error*
},
};
const response = await this.dynamoDB.scan(params).promise();
Yes, we cannot have bool type value to be used as DynamoDB Partition Key or Sort Key.
Some alternatives you could actually consider:
Create a GSI with only Partition Key, gsi-userId. When you do the query, you can query with userId and filter by Read. This will at least help you in saving some costs as you do not need to scan the whole table. However, be aware of Hot Partitions. Link
Consider changing the Read data type to string instead. E.g. It could be values such as Y or N only. As such, you will be able to create a GSI with gsi-userId-Read and this would fulfill what you need.
I have a DynamoDB table where each Item has a key of the name 'DataType'.
Also there is a GSI on this table with this 'DataType' as the HashKey and 'timestamp' as rangeKey.
Around 10 per cent of the table items have the 'DataType' value as 'A'.
I want to scan all the items of this GSI with HashKey fixed as 'A'. Is there any way to perform this using scan/parallel scan? Or do i need to use query on GSI itself to perform this operation?
As per the documentation,
https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/document/spec/ScanSpec.html
i could not find any way to specify a GSI on which i can scan with HashKey fixed.
Given that you want to only look at items with the Hash key "A", you'll need to use the Query API rather than the Scan API, provide the index name, and query for items in the index that have that partition key.
Here's some sample code using the Node AWS SDK V3 that creates three items in a table with a Global Secondary Index (called GSI1). Two of the items have a GSI1PK value of "orange", while the other has a GSI1PK value of "gold". The query returns the two matches:
const tableName = getFromEnv(`TABLE_NAME`);
const client = new DynamoDBClient({});
async function createItem(
name: string,
PK: string,
SK: string,
GSI1PK: string,
GSI1SK: string,
): Promise<void> {
const item = { PK, SK, GSI1PK, GSI1SK, name };
const putCommand = new PutItemCommand({
TableName: tableName,
Item: marshall(item)
});
await client.send(putCommand);
log(`Created item: ${name} with GSI1PK ${GSI1PK}`);
}
await createItem(`foo`, `fooPK`, `fooSK`, `orange`, `blue`);
await createItem(`bar`, `barPK`, `barSK`, `orange`, `white`);
await createItem(`baz`, `bazPK`, `bazSK`, `gold`, `garnet`);
log(`Waiting 5 seconds, as GSIs don't support consistent reads`)
await wait(5);
const query: QueryCommandInput = {
TableName: tableName,
IndexName: `GSI1`,
KeyConditionExpression: `#pk = :pk`,
ExpressionAttributeNames: {
'#pk': `GSI1PK`,
},
ExpressionAttributeValues: {
':pk': { S: `orange` },
},
}
const result = await client.send(new QueryCommand(query));
log(`Querying GSI1 for "orange"`);
result.Items.forEach((entry) => {
log(`Received: `, unmarshall(entry).name);
});
This produces the output of:
Created item: foo with GSI1PK orange
Created item: bar with GSI1PK orange
Created item: baz with GSI1PK gold
Waiting 5 seconds, as GSIs don't support consistent reads
Querying GSI1 for "orange"
Received: foo
Received: bar
One thing worth noting from this example is that GSIs don't allow consistent reads. So if your use case requires immediate consistency, you'll need to find another solution.
It is a jobPosts schema that has a posted_date as one of the attributes. The goal is to query all the job posts between two dates.
Here is the schema for your reference:
{
'job_id': {S: jobInfo.job_id},
'company': {S: jobInfo.company},
'title': {S: jobInfo.title},
'posted_on': {S: jobInfo.posted_on},
}
posted_on' is based on ISO string (2019-11-10T10:52:38.013Z). job_id is the primary key (partition key) and since I need to query the dates, I created GSI(partition key) on posted_on. Now here is the query:
const params = {
TableName : "jobPosts",
IndexName: 'date_for_filter_purpose-index',
ProjectionExpression:"job_id, company, title, posted_on",
KeyConditionExpression: "posted_on BETWEEN :startDate AND :endDate",
ExpressionAttributeValues: {
":startDate": {S: "2019-10-10T10:52:38.013Z"},
":endDate": {S: "2019-11-10T10:52:38.013Z"}
}
};
I have one document in dynamoDB and here it is:
{
job_id:,
company: "xyz",
title: "abc",
posted_on: "2019-11-01T10:52:38.013Z"
}
Now, on executing this, I get the following error:
{
"message": "Query key condition not supported",
"code": "ValidationException",
"time": "2019-11-11T06:15:37.231Z",
"requestId": "J078NON3L8KSJE5E8I3IP9N0IBVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 12.382362030893768
}
I don't know what is wrong with the above query.
Update after Tommy Answer:
I removed the GSI on posted_on and re-created the table with job_id as partition key and posted_on as sort key. I get the following error:
{
"message": "Query condition missed key schema element: job_id",
"code": "ValidationException",
"time": "2019-11-12T11:01:48.682Z",
"requestId": "M9E793UQNJHPN5ULQFJI2NR0BVVV4KQNSO5AEMVJF66Q9ASUAAJG",
"statusCode": 400,
"retryable": false,
"retryDelay": 42.52613025785952
}
As per this SO answer, GSI should be able to query the dates using BETWEEN keyword.
The answer you refer to relates to a query where the partition key has a specific value and the sort key is in a given range. It's analagous to select * from table where status=Z and date between X and Y. That's not what you're trying to do, if I read your question correctly. You want select * from table where date between X and Y. You cannot do this with DynamoDB query - you cannot query a partition key by range.
If you knew that your max range of query dates was on a given day then you could create a GSI with a partition key set to the computed YYYYMMDD value of the date/time and whose sort key was the full date/time. Then you could query with a key condition expression for a partition key of the computed YYYYMMDD and a sort key between X and Y. For this to work, the YYYYMMDD of X and Y would have to be the same.
If you knew that your max range of query dates was a month then you could create a GSI with partition key set to the computed YYYYMM of the date/time and whose sort key was the full date/time. For this to work, the YYYYMM of X and Y would have to be the same.
I guess it's a little counter-intuitive but DynamoDB supports only .eq condition on partition key attributes.
As per KeyConditions Documentation
You must provide the index partition key name and value as an EQ condition. You can optionally provide a second condition, referring to the index sort key.
Furthermore, in Query API Documentation you can find the following
The condition must perform an equality test on a single partition key value.
The condition can optionally perform one of several comparison tests on a single sort key value. This allows Query to retrieve one item with a given partition key value and sort key value, or several items that have the same partition key value but different sort key values.
That explains the error message you are getting.
One of the solutions might be to create a composite primary key with posted_on attribute as the sort key, instead of the GSI. Then, depending on your use case and access pattern, you'll need to figure out which attribute would work best as the partition key.
This blog should help you to choose the right partition key for your schema.
I have setup a TTL attribute in my dynamoDB table. when i push records in a get the current date (using js sdk in node) and add a value to it (like 5000). It is my understanding that when that date is reached aws will purge the record but only within 48 hours. during that time the record could be returned as the result of a query.
I want to filter out the expired items so that if they are expired but not deleted they won't be returned as part of the query.
here is what i am using to try to do that:
var epoch = Math.floor(Date.now() / 1000);
console.log("ttl epoch is ", epoch);
var queryTTLParams = {
TableName : table,
KeyConditionExpression: "id = :idval",
ExpressionAttributeNames:{
"#theTTL": "TTL"
},
FilterExpression: "#theTTL < :ttl",
ExpressionAttributeValues: {
":idval": {S: "1234"},
":ttl": {S: epoch.toString()}
}
};
i do not get any results. I believe the issue has to do with the TTL attribute being a string and me trying to do a < on it. But i didn't get to decide on the datatype for the TTL field - aws did that for me.
How can i remedy this?
According to the Enabling Time to Live AWS documentation, the TTL should be set to a Number attribute:
TTL is a mechanism to set a specific timestamp for expiring items from your table. The timestamp should be expressed as an attribute on the items in the table. The attribute should be a Number data type containing time in epoch format. Once the timestamp expires, the corresponding item is deleted from the table in the background.
You probably just need to create a new Number attribute and set the TTL attribute to that one.