Can page size be set with dynamodb.ScanPages? - amazon-web-services

Question

The documentation for working with dynamodb scans, found here, makes reference to a page-size parameter for the AWS CLI.
In looking at the documentation for the go AWS SDK, found here, there is function ScanPages. There is an example of how to use the function, but no where in the documentation is there a way to specify something like page-size as the AWS CLI has. I can't determine how the paging occurs other than assuming if the results exceed 1MB, then that would be considered a page based on the go documentation and the general scan documentation.
I'm also aware of the Limit value that can be set on the ScanInput, but the documentation indicates that value would function as a page size only if every item processed matched the filter expression of the scan:
The maximum number of items to evaluate (not necessarily the number of matching items)
Is there a way to set something equivalent to page-size with the go SDK?

Answer 1

How Pagination Works in AWS?
DynamoDB paginates the results from Scan operations. With pagination,
the Scan results are divided into "pages" of data that are 1 MB in
size (or less). An application can process the first page of results,
then the second page, and so on.
So for each request if you have more items in the result you will always get the LastEvaluatedKey. You will have re-issue scan request using this LastEvaluatedKey to get the complete result.
For example for a sample query you have 400 results and each result fetches to the upper limit 100 results, you will have to re-issue the scan request till the lastEvaluatedKey is returned empty. You will do something like below. documentation
var result *ScanOutput
for{
if(len(resultLastEvaluatedKey) == 0){
break;
}
input := & ScanInput{
ExclusiveStartKey= LastEvaluatedKey
// Copying all parameters of original scanInput request
}
output = dynamoClient.Scan(input)
}
What page-size on AWS-CLI does?
The scan operation scan's all the dynamoDB and returns result according to filter. Ordinarily, the AWS CLI handles pagination automatically.The AWS CLI keeps on re-issuing scan request for us. This request and response pattern continues, until the final response.
The page-size tells specifically to scan only the page-size number of rows in the DB table at a time and filter on those. If the complete table is not scanned or the result is more than 1MB the result will send out lastEvaluatedKey and cli will re-issue the request.
Here is a sample request response from documentation.
aws dynamodb scan \
--table-name Movies \
--projection-expression "title" \
--filter-expression 'contains(info.genres,:gen)' \
--expression-attribute-values '{":gen":{"S":"Sci-Fi"}}' \
--page-size 100 \
--debug
b'{"Count":7,"Items":[{"title":{"S":"Monster on the Campus"}},{"title":{"S":"+1"}},
{"title":{"S":"100 Degrees Below Zero"}},{"title":{"S":"About Time"}},{"title":{"S":"After Earth"}},
{"title":{"S":"Age of Dinosaurs"}},{"title":{"S":"Cloudy with a Chance of Meatballs 2"}}],
"LastEvaluatedKey":{"year":{"N":"2013"},"title":{"S":"Curse of Chucky"}},"ScannedCount":100}'
We can clearly see that the scannedCount:100 and the filtered count Count:7, so out of 100 items scanned only 7 items are filtered. documentation
From Limit's Documentation
// The maximum number of items to evaluate (not necessarily the number of matching
// items). If DynamoDB processes the number of items up to the limit while processing
// the results, it stops the operation and returns the matching values up to
// that point, and a key in LastEvaluatedKey to apply in a subsequent operation,
// so that you can pick up where you left off.
So basically, page-size and limit are same. Limit will limit the number of rows to scan in one Scan request.