I have a simple query to show a given users photos:
SELECT pid FROM photo_tag WHERE subject in "12345" LIMIT 400,500
It returns 500 results... shouldn't it return 100?
No. Your query is asking for 500 records starting with record 401.
To return the records 400-500 your query should read
SELECT pid FROM photo_tag WHERE subject in "12345" LIMIT 400, 100
Related
I've following dynamoDB table
user_id
date
game
user1
2021-12-06 14:36:46
game1
user1
2021-12-06 15:36:46
game1
user1
2021-12-07 11:36:46
game2
user1
2021-12-07 12:36:46
game2
partition key: user_id
sort key: date
I want to Query the latest entry of user for game game1
(Which is the second item from table with date 2021-12-06 15:36:46). I can achieve this from code as follows;
expr, _ := expression.NewBuilder().
WithKeyCondition(expression.Key("user_id").Equal(expression.Value("user1"))).
WithFilter(expression.Name("game").Equal(expression.Value("game1"))).
Build()
var queryInput = &dynamodb.QueryInput{
KeyConditionExpression: expr.KeyCondition(),
ExpressionAttributeNames: expr.Names(),
ExpressionAttributeValues: expr.Values(),
FilterExpression: expr.Filter(),
ScanIndexForward: aws.Bool(false),
TableName: aws.String(table),
}
This returns me all items of user user1 for game game1. Problem occurs when I apply limit=1 Limit: aws.Int64(1) in QueryInput, it returns nothing. Could someone explain why is that so ?
When I change Limit: aws.Int64(4) (total number of items in table), only then the query returns single expected item. How is this limit working ?
Do I need to use game as GSI ?
The limit on a DDB parameter is applied before your filter expressions.
Essentially with a limit of 1, it retrieves 1 record, then applies the filters and returns you the items that match (0).
See https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Query.html#Query.Limit for more details, copied in case link breaks
Limiting the Number of Items in the Result Set
The Query operation allows you to limit the number of items that it reads. To do this, set the Limit parameter to the maximum number of items that you want.
For example, suppose that you Query a table, with a Limit value of 6, and without a filter expression. The Query result contains the first six items from the table that match the key condition expression from the request.
Now suppose that you add a filter expression to the Query. In this case, DynamoDB reads up to six items, and then returns only those that match the filter expression. The final Query result contains six items or fewer, even if more items would have matched the filter expression if DynamoDB had kept reading more items.
I'm making a Letter/pageNum pagination for my site, so that this way you can get into an index, and query for items that begin with the letter "A" and move between pages of registers that begin with the letter "A".
For this, I've created a python process executed in AWS Lambda that queries for these items, with a begins_with filter on a field, so that I can sort them out.
This process aims to query for (In example) 1000 items and keep the last evaluated key (if exists) to set it as exclusiveStartKey for the next page.
Let's imagine a letter "A" with 3 pages:
Page 1 has no exclusiveStartKey because I query for all registers that begin with the letter A.
Page 2 will have exclusiveStartKey taken from the query that has been done to get page 1 registers.
And so on...
This is gonna be used to limit between pages in real-time without losing any registers.
If a user asks me for the first page of the letter "A" I will query all items that exist between the first register of the letter "A" while gsi4sk is less than the second page's exclusiveStartKey's gsi4sk.
Same operation for Page 2, exclusiveStartKey taken from Page 1 query, while gsi4sk is less than Page 3 exclusiveStartKey's gsi4sk. If Page 3 does not exist, I will query all the items to the end.
I'm using boto3 from a lambda process, some code example can be this one:
if lastEvaluatedKey == "":
response = ddb_client.query(
TableName = singleTable,
IndexName = "gsi4-index",
KeyConditionExpression='gsi4pk = :gsi4pk and begins_with(gsi4sk, :gsi4sk)',
ExpressionAttributeValues={
':gsi4pk': gsi4pk,
':gsi4sk': gsi4sk
},
ProjectionExpression="pk, sk, gsi4pk, gsi4sk",
Limit=current_limit
)
else:
response = ddb_client.query(
TableName = singleTable,
IndexName = "gsi4-index",
ExclusiveStartKey = lastEvaluatedKey,
KeyConditionExpression='gsi4pk = :gsi4pk and begins_with(gsi4sk, :gsi4sk)',
ExpressionAttributeValues={
':gsi4pk': gsi4pk,
':gsi4sk': gsi4sk
},
ProjectionExpression="pk, sk, gsi4pk, gsi4sk",
Limit=current_limit
)
This code queries up to ~200 registers when a current_limit is set to 1000.
I have cloudwatch entries that may be group with respect to a certain field. To be clear assume that field is city. I would like to count the entries with respect to cities. This is the easy part.
fields city
|stats count(*) by city
However I also want to get maximum minimum and average of this count, but I can not. Is it possible to have such queries i.e:
fields city
|stats avg(count(*) by city)
The console return an error for such query : mismatched input 'by' expecting {SYM_COMMA, SYM_RParen}
Here's how you'd do it: You'd first get the count (that you already figured) and then get the metrics you want by calling the relevant functions like so:
fields city
|stats count(*) as cityCount, avg(cityCount), max(cityCount), min(cityCount)
From the DynamoDB documentation
The Query operation allows you to limit the number of items that it
returns in the result. To do this, set the Limit parameter to the
maximum number of items that you want.
For example, suppose you Query a table, with a Limit value of 6, and
without a filter expression. The Query result will contain the first
six items from the table that match the key condition expression from
the request.
Now suppose you add a filter expression to the Query. In this case,
DynamoDB will apply the filter expression to the six items that were
returned, discarding those that do not match. The final Query result
will contain 6 items or fewer, depending on the number of items that
were filtered.
Looks like the following query should return (at least sometimes) 0 records.
In summary, I have a UserLogins table. A simplified version is:
1. UserId - HashKey
2. DeviceId - RangeKey
3. ActiveLogin - Boolean
4. TimeToLive - ...
Now, let's say UserId = X has 10,000 inactive logins in different DeviceIds and 1 active login.
However, when I run this query against my DynamoDB table:
QuerySpec{
hashKey: null,
rangeKeyCondition: null,
queryFilters: null,
nameMap: {"#0" -> "UserId"}, {"#1" -> "ActiveLogin"}
valueMap: {":0" -> "X"}, {":1" -> "true"}
exclusiveStartKey: null,
maxPageSize: null,
maxResultSize: 10,
req: {TableName: UserLogins,ConsistentRead: true,ReturnConsumedCapacity: TOTAL,FilterExpression: #1 = :1,KeyConditionExpression: #0 = :0,ExpressionAttributeNames: {#0=UserId, #1=ActiveLogin},ExpressionAttributeValues: {:0={S: X,}, :1={BOOL: true}}}
I always get 1 row. The 1 active login for UserId=X. And it's not happening just for 1 user, it's happening for multiple users in a similar situation.
Are my results contradicting the DynamoDB documentation?
It looks like a contradiction because if maxResultSize=10, means that DynamoDB will only read the first 10 items (out of 10,001) and then it will apply the filter active=true only (which might return 0 results). It seems very unlikely that the record with active=true happened to be in the first 10 records that DynamoDB read.
This is happening to hundreds of customers that are running similar queries. It works great, when according to the documentation it shouldn't be working.
I can't see any obvious problem with the Query. Are you sure about your premise that users have 10,000 items each?
Your keys are UserId and DeviceId. That seems to mean that if your user logs in with the same device it would overwrite the existing item. Or put another way, I think you are saying your users having 10,000 different devices each (unless the DeviceId rotates in some way).
In your shoes I would just remove the filterexpression and print the results to the log to see what you're getting in your 10 results. Then remove the limit too and see what results you get with that.
I want to get records from Google Health within a range. Every time I send a new query, I want to get ten records.
I.e. Start: 1 to 10
Second query: start 11 to 20
....
I used the following query :
PROFILE_URL + selectedProfileId
+ "/-/labtest?start-index="+index+"&max-results=10";
This retrieved records, but when it reached the end where the list had only five records and it queries for ten results, the application crashes.
How can I get the total count of results, or partial results?
One can use the above query and it will return records as many as it has even if it has records less than max-results.
But this will throw exception when try to query for the invalid count. So catch the exception there where u add the values into your listView.