How can I get faraday/scan to walk an entire DynamoDB table? - clojure

Question

I am struggling for the Faraday incantation to scan an entire DDB table. The following function produces output, but returns far fewer results than the 18M records I know are in the table.
(far/scan
common/client-opts
v2-index/layer-table-name
{:return #{:layer-key :range-key}})
=>
[{:range-key "soil&2015-07-22T15:13:09.101Z&ssurgo&v1", :layer-key "886985&886985"}
{:range-key "soil&2015-07-29T19:20:09.973Z&ssurgo&v1", :layer-key "886985&886985"}
...
{:range-key "veg&2014-05-29T16:16:31.000Z&true-color&v1", :layer-key "1674603&1674603"}
{:range-key "veg&2014-06-14T16:16:39.000Z&abs&v1", :layer-key "1674603&1674603"}]
What can I do to get faraday deal up all the records? The source-code suggests that there is some :last-prim-kvs option but its not clear to me what would go in there? The Primary Key on this DDB table is a composite primary key composed of :layer-key and :range-key.

Answer 1

If it'll fit in memory, this works...
The key to the whole scheme is the getting the opts map set up with a :limit 99 mapping as well as some :span-reqs {:max 1} mapping. The :span-reqs mapping is totally obscure to me, but it seems to be the real driver behind what is conceptually "page size". I have set up a 10 element table like...
;; This only works on the whole table because the table is small!!!!
(far/scan
common/client-opts
"users.robert.kuhar.wtf_far"
{:return #{:part_key :sort_key :note}})
=>
[{:part_key "456", :sort_key "fha.abs", :note "\"456\",\"fha.abs\" created 2016-12-08T21:32:20.789Z."}
{:part_key "456", :sort_key "fha.rank", :note "\"456\",\"fha.rank\" created 2016-12-08T21:32:20.789Z."}
{:part_key "456", :sort_key "fha.raw", :note "\"456\",\"fha.raw\" created 2016-12-08T21:32:20.789Z."}
{:part_key "456", :sort_key "fha.true-color", :note "\"456\",\"fha.true-color\" created 2016-12-08T21:32:20.789Z."}
{:part_key "456", :sort_key "soil.ssurgo", :note "\"456\",\"soil.ssurgo\" created 2016-12-08T21:32:20.789Z."}
{:part_key "123", :sort_key "fha.abs", :note "\"123\",\"fha.abs\" created 2016-12-08T21:24:30.139Z."}
{:part_key "123", :sort_key "fha.rank", :note "\"123\",\"fha.rank\" created 2016-12-08T21:24:30.139Z"}
{:part_key "123", :sort_key "fha.raw", :note "\"123\",\"fha.raw\" created 2016-12-08T21:24:30.139Z."}
{:part_key "123", :sort_key "fha.true-color", :note "\"123\",\"fha.true-color\" created 2016-12-08T21:24:30.139Z."}
{:part_key "123", :sort_key "soil.ssurgo", :note "\"123\",\"soil.ssurgo\" created 2016-12-08T21:24:30.139Z."}]
If I want to move through this 4 elements at a time, the initial call is...
(far/scan
common/client-opts
"users.robert.kuhar.wtf_far"
{:return #{:part_key :sort_key :note}
:limit 4
:span-reqs {:max 1}})
=>
[{:part_key "456", :sort_key "fha.abs", :note "\"456\",\"fha.abs\" created 2016-12-08T21:32:20.789Z."}
{:part_key "456", :sort_key "fha.rank", :note "\"456\",\"fha.rank\" created 2016-12-08T21:32:20.789Z."}
{:part_key "456", :sort_key "fha.raw", :note "\"456\",\"fha.raw\" created 2016-12-08T21:32:20.789Z."}
{:part_key "456", :sort_key "fha.true-color", :note "\"456\",\"fha.true-color\" created 2016-12-08T21:32:20.789Z."}]
An all subsequent calls need to set a :last-prim-kvs {:part_key "xxx" :sort_key "yyy"} into that opts map to tell faraday where to pick up. For the 2nd page the call is like...
(far/scan
common/client-opts
"users.robert.kuhar.wtf_far"
{:return #{:part_key :sort_key :note}
:limit 4
:span-reqs {:max 1}
:last-prim-kvs {:part_key "456" :sort_key "fha.true-color"}})
=>
[{:part_key "456", :sort_key "soil.ssurgo", :note "\"456\",\"soil.ssurgo\" created 2016-12-08T21:32:20.789Z."}
{:part_key "123", :sort_key "fha.abs", :note "\"123\",\"fha.abs\" created 2016-12-08T21:24:30.139Z."}
{:part_key "123", :sort_key "fha.rank", :note "\"123\",\"fha.rank\" created 2016-12-08T21:24:30.139Z"}
{:part_key "123", :sort_key "fha.raw", :note "\"123\",\"fha.raw\" created 2016-12-08T21:24:30.139Z."}]
The last page of my 10 element table is...
(far/scan
common/client-opts
"users.robert.kuhar.wtf_far"
{:return #{:part_key :sort_key :note}
:limit 4
:span-reqs {:max 1}
:last-prim-kvs {:part_key "123" :sort_key "fha.raw"}})
=>
[{:part_key "123", :sort_key "fha.true-color", :note "\"123\",\"fha.true-color\" created 2016-12-08T21:24:30.139Z."}
{:part_key "123", :sort_key "soil.ssurgo", :note "\"123\",\"soil.ssurgo\" created 2016-12-08T21:24:30.139Z."}]
Just 2 elements even though I asked for 4. Trying to far/scan beyond that is always empty.
(far/scan
common/client-opts
"users.robert.kuhar.wtf_far"
{:return #{:part_key :sort_key :note}
:limit 4
:span-reqs {:max 1}
:last-prim-kvs {:part_key "123" :sort_key "soil.ssurgo"}})
=> []
So this does it end-to-end, provided everything will fit in memory.
(loop [accum []
page (far/scan
client-opts
"users.robert.kuhar.wtf_far"
{:limit 4
:span-reqs {:max 1}})]
(if (empty? page)
accum
(let [last-on-page (last page)
last-part-key (:part_key last-on-page)
last-sort-key (:sort_key last-on-page)]
(recur
(into accum page)
(far/scan
client-opts
"users.robert.kuhar.wtf_far"
{:limit 4
:span-reqs {:max 1}
:last-prim-kvs {:part_key last-part-key :sort_key last-sort-key}})))))
=>
[{:part_key "456", :sort_key "fha.abs", :note "\"456\",\"fha.abs\" created 2016-12-08T21:32:20.789Z."}
...
{:part_key "123", :sort_key "soil.ssurgo", :note "\"123\",\"soil.ssurgo\" created 2016-12-08T21:24:30.139Z."}]
I think the sad final answer in the case of "How can I get faraday/scan to walk an entire DynamoDB table?" is that it can't. You need to build it by hand.