I would like to be able to build the where clauses for a query. I would like to input a array of where conditions and build the query using korma, as seen below:
(defn ^:private fetch-by
"Append conditions to the query."
[query ^clojure.lang.PersistentVector conditions]
(for [condition conditions]
(if (instance? clojure.lang.PersistentArrayMap condition)
(korma/where query condition) query)))
However, the for loop here will duplicate the query object. Is there away to merge these objects or another approach you can recommend that would achieve the desired output?
The merging of the conditions into one query map can be done with reduce:
(defn ^:private fetch-by
"Append conditions to the query."
[query conditions]
(->> (filter map? conditions)
(reduce (fn [query condition]
(korma/where query condition))
query)))
Here your initial reduce state is whatever query you pass in, and the reducing function (and korma/where) take care of merging each condition into the query map.
(-> (korma/select* :my-table)
(fetch-by [{:active true}
{:deleted false}])
(korma/as-sql))
=> "SELECT \"my-table\".* FROM \"my-table\" WHERE (\"my-table\".\"active\" = ?) AND (\"my-table\".\"deleted\" = ?)"
However this is not really different than just passing a single map with multiple entries to korma/where:
(-> (korma/select* :my-table)
(korma/where {:active true
:deleted false})
(korma/as-sql))
which I would suggest as long as the conditions' keys are unique.
Related
I need to apply additional logic (like mapping, conditionals, aggregating) to entities I get from Datomic. I had hard time translating it to Datomic query (I'm not sure if it's even possible in my case), which is why I used datomic's raw index access instead, so the most work and logic is done in Clojure.
It worked fine until I got to ~500K entries and the whole approach is getting very slow.
The relevant code:
(defn e->entry
"Map e into entry"
[e]
{:id (:entry/uuid e)
;; each flat field increases mapping time (seems linearly)
:date (:entry/date e)
:summ (:entry/summ e)
;; although when using nested fields, mapping time rises significantly
:groups (map #(-> % :dimension/group :group/name)
(:entry/dimensions e))})
;; query code:
(->> (d/datoms db :aevt :entry/uuid)
(map #(->> %
:e
(d/entity db)
e->entry))))
;; TODO: other actions on mapped entries ...
It takes about 30 seconds to run query code just to map entities and the more fields I need in my query, the more it takes.
Is this an expected behavior? Is there a way I can speed things up or am I missing something and this is bad approach?
To fully answer this question would require more information, please feel free to ask on the forum or open a support ticket.
I ended up with following optimizations, in case someone will need it:
(defn eid->entry
"Mapping via :eavt index"
[db eid]
(->> (d/datoms db :eavt eid) ; access all datoms by eid once
(seq)
(reduce (fn [m dtm]
(let [attr-key (d/ident db (:a dtm))
v (:v dtm)]
(assoc m attr-key v))))))
;; new query code
(->> (d/datoms db :aevt :entry/uuid)
(pmap #(->> %
:e
(eid->entry db))))
I used pmap instead of map and resorted to :eavt index to get all attributes and values of entity instead of accessing fields directly with d/entity
I would like to be able to pass a user-defined array of fields which contains a list of all the columns that need fetching. Can korma/fields take an array of columns?
What I am essentially wanting to create is something like this:
(defn fetch [fields]
(->
(korma/select* foo)
(as-> query
(if (not-empty? fields)
(korma/fields query fields)
query))
(korma/select)))
I am assuming this is because I have specified entity-fields in defentity. I am therefore wondering, if it is possible to override the entity-fields specified in defentity?
I think your assumption is correct, and it's also possible to override this. If you look at the map generated by (defentity foo) it has a :fields key with all the fields. korma.core/entity-fields doesn't replace what's already there, but this will:
(-> foo
(assoc :fields [:first])
(korma/select*)
(korma/as-sql))
=> "SELECT \"foo\".\"first\" FROM \"foo\""
I would like to be able to pass a user-defined array of fields which contains a list of all the columns that need fetching.
(defn fetch [& fields]
(-> (korma/select* foo)
(as-> query
(if (seq fields)
(assoc query :fields fields)
query))
(korma/select)))
I used variadic args in this example to mirror korma.core/entity-fields (and made fetch return the SQL string for my testing rather than execute the query):
(fetch :first :last)
=> "SELECT \"foo\".\"first\", \"foo\".\"last\" FROM \"foo\""
(fetch)
=> "SELECT \"foo\".\"id\", \"foo\".\"first\", \"foo\".\"last\" FROM \"foo\""
According to Om Next's documentation:
query->ast
(om.next/query->ast '[(:foo {:bar 1})])
Given a query expression return the AST.
ast->query
(om.next/ast->query ast)
Given a query expression AST, unparse it into a query expression.
Question: Why would one need these functions? That is, why would one need to directly manipulate a query abstract syntax tree (which I'm assuming are clojure maps that represent a query tree, along with some meta data) in om next?
There are some scenarios where you need to manipulate the query ast directly. In remote parsing mode, the parser expects your read functions to return either {:remote-name true } or a (possibly modified) {:remote-name AST-node} (which comes in as :ast in env). Most often you'll have to modify the AST to restructure it or add some data.
Example 1:
You have a query: [{:widget {:list [:name :created]}}]
The :widget part is pure UI related, your server doesn't need to know it exists, it only cares/knows about the :list.
Basically you'll have to modify the AST in the parser:
(defmethod read :list
[{:keys [ast query state]} key _ ]
(let [st #state]
{:value (om/db->tree query (get st key) st)
:remote (assoc ast :query-root true)}))
If you use om/process-rootsin your send function, it'll pick up the :query-root out of the ast and rewrite the query from [{:widget {:list [:name :created]}}] to [{:list [:name :created]}].
Example 2:
Another example would be when you want to mutate something at a remote:
(defmethod mutate 'item/update
[{:keys [state ast]} key {:keys [id title]}]
{:remote (assoc ast :params {:data {:id id :title title })})
Here you need to explicitly tell Om to include the data you want to send in the AST. At your remote you then pick apart :data to update the title at the given id
Most of the time you won't use the functions you described in your questions directly. The env available in every method of the parser has the ast in it.
Something I stumbled on, while trying to use Compassus:
Let's say you have a complex union/join query that includes parametric sub-queries. Something like this:
`[({:foo/info
{:foo/header [:foo-id :name]
:foo/details [:id :description :title]}} {:foo-id ~'?foo-id
:foo-desc ~'?foo-desc})]
Now let's say you want to set parameters so on the server you can parse it with om/parser and see those params as 3rd argument of read dispatch. Of course it's possible to write a function that would find all necessary parameters in the query and set the values. That's not easy though, and as I said - imagine your queries can be quite complex.
So what you can do - is to modify ast, ast includes :children :params key. So let's say the actual values for :foo-id and :foo-desc are in the state atom under :route-params key:
(defn set-ast-params [children params]
"traverses given vector of `children' in an AST and sets `params`"
(mapv
(fn [c]
(let [ks (clojure.set/intersection (-> params keys set)
(-> c :params keys set))]
(update-in c [:params] #(merge % (select-keys params (vec ks))))))
children))
(defmethod readf :foo/info
[{:keys [state query ast] :as env} k params]
(let [{:keys [route-params] :as st} #state
ast' (-> ast
(update :children #(set-ast-params % route-params))
om/ast->query
om.next.impl.parser/expr->ast)]
{:value (get st k)
:remote ast'}))
So basically you are:
- grabbing ast
- modifying it with actual values
you think maybe you can send it to server right then. Alas, no! Not yet. Thing is - when you do {:remote ast}, Om takes :query part of the ast, composes ast out of it and then sends it to the server. So you actually need to: turn your modified ast into query and then convert it back to ast again.
Notes:
set-ast-params function in this example would only work for the first level (if you have nested parametrized queries - it won't work),
make it recursive - it's not difficult
there are two different ways to turn ast to query and vice-versa:
(om/ast->query) ;; retrieves query from ast and sets the params based
;; of `:params` key of the ast, BUT. it modifies the query,
;; if you have a join query it takes only the first item in it. e.g. :
[({:foo/foo [:id]
:bar/bar [:id]} {:id ~'?id})]
;; will lose its `:bar` part
(om.next.impl.parser/ast->expr) ;; retrieves query from an ast,
;; but doesn't set query params based on `:params` keys of the ast.
;; there are also
(om/query->ast) ;; and
(om.next.impl.parser/expr->ast)
Below is my attempt to iterate over a result set and get its values
(sql/with-connection db
(sql/with-query-results rs ["select * from user where UserID=?" 10000]
(doseq [rec rs
s rec]
(println (val s))
)))
But how do you extract one particular value from it; i need only the user name field.
Can anyone please demonstarte how to do this?
The result set is a sequence of maps, so if you wanted to obtain one field (e.g. one called name) then:
(sql/with-connection db
(sql/with-query-results rs ["select * from user where UserID=?" 10000]
(doseq [rec rs]
(let [name (:name rec)]
(println "User name:" name)
(println "Full record (including name):" rec)))))
But as mentioned in the comments, if you only want name, then select name from would be the more efficient option. The code above is useful when you need the full row for something else.
The with-connection / with-query-results syntax is deprecated as of clojure.java.jdbc 3.0. Filtering results can be done much easier with the new query syntax and additional :row-fn and :result-set-fn parameters.
(query db ["select * from user"]
:row-fn :name
:result-set-fn #(doall (take 1000 (drop 10000 %))))
Be sure to make the result-set-fn realize all values, it shouldn't return a lazy sequence (hence the doall in this example).
I'm running two select statements against Cassandra, so instead of having a join I need to join them in code. Being relatively new to Clojure, I'm having a hard time doing this without resorting to really ugly nested loops. Furthermore, if table-b is missing a matching entry from table-a, it should add default table-b values.
The two selects each result in a list of maps (each "row" is one map). The id key is a UUID, not string.
Here's how the selects look if I def something with the same structure.
(def table-a (list {:id "105421db-eca4-4500-9a2c-08f1e09a35ca" :col-b "b-one"}
{:id "768af3f3-3981-4e3f-a93d-9758cd53a056" :col-b "b-two"}))
(def table-b (list {:id "105421db-eca4-4500-9a2c-08f1e09a35ca" :col-c "c-one"}))
I want the end result to be this:
({:id "105421db-eca4-4500-9a2c-08f1e09a35ca" :col-b "b-one" :col-c "c-one"}
{:id "768af3f3-3981-4e3f-a93d-9758cd53a056" :col-b "b-two" :col-c "default-value"})
Thanks for any help.
This can be done by splitting it into groups with the same key, merging all the like-keyed maps and then filling in the default values:
user> (->> (concat table-a table-b) ;; stat with all the data
(sort-by :id) ;; split it into groups
(partition-by :id) ;; by id
(map (partial apply merge)) ;; merge each group into a single map.
(map #(assoc % ;; fill in the missing default values.
:col-c (or (:col-c %) "default value")
:col-b (or (:col-b %) "default value"))))
({:col-c "c-one",
:col-b "b-one",
:id "105421db-eca4-4500-9a2c-08f1e09a35ca"}
{:col-c "default value",
:col-b "b-two",
:id "768af3f3-3981-4e3f-a93d-9758cd53a056"})
Using the thread-last macro ->> makes this a lot easier for me to read, though that is just my opinion. There is also likely a more elegant way to supply the default keys.