I'm writing some tests on my datomic code. I create a transaction map:
(def tx-map
{:db/id #db/id[:db.part/user -1000235]
:some-ns/my-attr {:db/id 17592186045421}}
It transacts fine. I query for my newly added entity:
(def ent
"Query result."
{:db/id 17592186045425})
Test to see if all is well:
(= (:some-ns/my-attr tx-map) ; {:db/id 17592186045421}
(:some-ns/my-attr ent)) ; {:db/id 17592186045421}
=> false ; All is not well
The reason being?
(type (:some-ns/my-attr tx-map)) => clojure.lang.PersistentArrayMap
(type (:some-ns/my-attr ent)) => datomic.query.EntityMap
How do I correctly test for equality between a clojure PersistentArrayMap and a datomic EntityMap?
Entities are lazy maps, so they cannot be compared with maps.
http://docs.datomic.com/entities.html
But they are associative and we can get the values for each attributes. So you can compare the ids :
(== (:db/id (:some-ns/my-attr tx-map)) ; 17592186045421
(:db/id ent)) ; 17592186045421
Beware the ==, we compare two numbers... ;)
So only the :db/id will be selected. Perhaps the ent function should give you the id.
(def ent
":db/id of query result."
17592186045425)
And the test becomes :
(== (:db/id (:some-ns/my-attr tx-map)) ; 17592186045421
ent) ; 17592186045421
Sorry for the late answer, I was very buzy.
Warning, it's only according to the documentation, I don't have an runnable installed datomic version now...
Assoc'ing and the other Clojure map interface capabilities aren't currently supported on Datomic EntityMaps, but this feature is under consideration.
To convert an entity to a Clojure map, simply use (into).
(into {} some-entity)
Reference : https://groups.google.com/d/msg/datomic/IzRSK9e7VJo/YGejFLl3vxYJ
But it's not the problem here, as this gives the whole attributes that would be returned if we realize the whole map.
Related
I want to use Datalevin as a database for my app. The README mentions :db/id as the identifier of entities, and I see these do get an autoincremented integer value on insertion.
However, there are also multiple mentions of :db/ident in the source code, for example for implicit constants.
What are the purposes of the two keywords and what are the differences?
:db/ident are called Entity
Identifiers
in Datomic.
They are used to allow for easier "pointing" to other, well known
things; like an enum. So you can have the enum values as datums, but
still be able to reference them via a keyword (no need to look them up
every time before using them).
E.g.
(def conn (d/create-conn "./tst" {}))
(d/transact! conn
[{:db/id 1, :customer-type/name "Fictional", :db/ident :fictional}
{:db/id 2, :customer/name "ACME", :customer/type :fictional}])
(d/pull #conn
[:db/id :customer/name {:customer/type [:db/id :customer-type/name]}]
2)
; ⇒ {:db/id 2, :customer/name "ACME",
; :customer/type {:db/id :fictional,
; :customer-type/name "Fictional"}}
I need to apply additional logic (like mapping, conditionals, aggregating) to entities I get from Datomic. I had hard time translating it to Datomic query (I'm not sure if it's even possible in my case), which is why I used datomic's raw index access instead, so the most work and logic is done in Clojure.
It worked fine until I got to ~500K entries and the whole approach is getting very slow.
The relevant code:
(defn e->entry
"Map e into entry"
[e]
{:id (:entry/uuid e)
;; each flat field increases mapping time (seems linearly)
:date (:entry/date e)
:summ (:entry/summ e)
;; although when using nested fields, mapping time rises significantly
:groups (map #(-> % :dimension/group :group/name)
(:entry/dimensions e))})
;; query code:
(->> (d/datoms db :aevt :entry/uuid)
(map #(->> %
:e
(d/entity db)
e->entry))))
;; TODO: other actions on mapped entries ...
It takes about 30 seconds to run query code just to map entities and the more fields I need in my query, the more it takes.
Is this an expected behavior? Is there a way I can speed things up or am I missing something and this is bad approach?
To fully answer this question would require more information, please feel free to ask on the forum or open a support ticket.
I ended up with following optimizations, in case someone will need it:
(defn eid->entry
"Mapping via :eavt index"
[db eid]
(->> (d/datoms db :eavt eid) ; access all datoms by eid once
(seq)
(reduce (fn [m dtm]
(let [attr-key (d/ident db (:a dtm))
v (:v dtm)]
(assoc m attr-key v))))))
;; new query code
(->> (d/datoms db :aevt :entry/uuid)
(pmap #(->> %
:e
(eid->entry db))))
I used pmap instead of map and resorted to :eavt index to get all attributes and values of entity instead of accessing fields directly with d/entity
I would like to be able to build the where clauses for a query. I would like to input a array of where conditions and build the query using korma, as seen below:
(defn ^:private fetch-by
"Append conditions to the query."
[query ^clojure.lang.PersistentVector conditions]
(for [condition conditions]
(if (instance? clojure.lang.PersistentArrayMap condition)
(korma/where query condition) query)))
However, the for loop here will duplicate the query object. Is there away to merge these objects or another approach you can recommend that would achieve the desired output?
The merging of the conditions into one query map can be done with reduce:
(defn ^:private fetch-by
"Append conditions to the query."
[query conditions]
(->> (filter map? conditions)
(reduce (fn [query condition]
(korma/where query condition))
query)))
Here your initial reduce state is whatever query you pass in, and the reducing function (and korma/where) take care of merging each condition into the query map.
(-> (korma/select* :my-table)
(fetch-by [{:active true}
{:deleted false}])
(korma/as-sql))
=> "SELECT \"my-table\".* FROM \"my-table\" WHERE (\"my-table\".\"active\" = ?) AND (\"my-table\".\"deleted\" = ?)"
However this is not really different than just passing a single map with multiple entries to korma/where:
(-> (korma/select* :my-table)
(korma/where {:active true
:deleted false})
(korma/as-sql))
which I would suggest as long as the conditions' keys are unique.
According to Om Next's documentation:
query->ast
(om.next/query->ast '[(:foo {:bar 1})])
Given a query expression return the AST.
ast->query
(om.next/ast->query ast)
Given a query expression AST, unparse it into a query expression.
Question: Why would one need these functions? That is, why would one need to directly manipulate a query abstract syntax tree (which I'm assuming are clojure maps that represent a query tree, along with some meta data) in om next?
There are some scenarios where you need to manipulate the query ast directly. In remote parsing mode, the parser expects your read functions to return either {:remote-name true } or a (possibly modified) {:remote-name AST-node} (which comes in as :ast in env). Most often you'll have to modify the AST to restructure it or add some data.
Example 1:
You have a query: [{:widget {:list [:name :created]}}]
The :widget part is pure UI related, your server doesn't need to know it exists, it only cares/knows about the :list.
Basically you'll have to modify the AST in the parser:
(defmethod read :list
[{:keys [ast query state]} key _ ]
(let [st #state]
{:value (om/db->tree query (get st key) st)
:remote (assoc ast :query-root true)}))
If you use om/process-rootsin your send function, it'll pick up the :query-root out of the ast and rewrite the query from [{:widget {:list [:name :created]}}] to [{:list [:name :created]}].
Example 2:
Another example would be when you want to mutate something at a remote:
(defmethod mutate 'item/update
[{:keys [state ast]} key {:keys [id title]}]
{:remote (assoc ast :params {:data {:id id :title title })})
Here you need to explicitly tell Om to include the data you want to send in the AST. At your remote you then pick apart :data to update the title at the given id
Most of the time you won't use the functions you described in your questions directly. The env available in every method of the parser has the ast in it.
Something I stumbled on, while trying to use Compassus:
Let's say you have a complex union/join query that includes parametric sub-queries. Something like this:
`[({:foo/info
{:foo/header [:foo-id :name]
:foo/details [:id :description :title]}} {:foo-id ~'?foo-id
:foo-desc ~'?foo-desc})]
Now let's say you want to set parameters so on the server you can parse it with om/parser and see those params as 3rd argument of read dispatch. Of course it's possible to write a function that would find all necessary parameters in the query and set the values. That's not easy though, and as I said - imagine your queries can be quite complex.
So what you can do - is to modify ast, ast includes :children :params key. So let's say the actual values for :foo-id and :foo-desc are in the state atom under :route-params key:
(defn set-ast-params [children params]
"traverses given vector of `children' in an AST and sets `params`"
(mapv
(fn [c]
(let [ks (clojure.set/intersection (-> params keys set)
(-> c :params keys set))]
(update-in c [:params] #(merge % (select-keys params (vec ks))))))
children))
(defmethod readf :foo/info
[{:keys [state query ast] :as env} k params]
(let [{:keys [route-params] :as st} #state
ast' (-> ast
(update :children #(set-ast-params % route-params))
om/ast->query
om.next.impl.parser/expr->ast)]
{:value (get st k)
:remote ast'}))
So basically you are:
- grabbing ast
- modifying it with actual values
you think maybe you can send it to server right then. Alas, no! Not yet. Thing is - when you do {:remote ast}, Om takes :query part of the ast, composes ast out of it and then sends it to the server. So you actually need to: turn your modified ast into query and then convert it back to ast again.
Notes:
set-ast-params function in this example would only work for the first level (if you have nested parametrized queries - it won't work),
make it recursive - it's not difficult
there are two different ways to turn ast to query and vice-versa:
(om/ast->query) ;; retrieves query from ast and sets the params based
;; of `:params` key of the ast, BUT. it modifies the query,
;; if you have a join query it takes only the first item in it. e.g. :
[({:foo/foo [:id]
:bar/bar [:id]} {:id ~'?id})]
;; will lose its `:bar` part
(om.next.impl.parser/ast->expr) ;; retrieves query from an ast,
;; but doesn't set query params based on `:params` keys of the ast.
;; there are also
(om/query->ast) ;; and
(om.next.impl.parser/expr->ast)
After I run a transaction in datomic to insert a value, how I can use the return value of the transaction to get the ids of any entities that were created?
Here is a sample of the return value I get after an insert:
#<promise$settable_future$reify__4841#7c92b2e9: {:db-before datomic.db.Db#62d0401f, :db-after datomic.db.Db#bba61dfc,
:tx-data [#Datum{:e 13194139534331 :a 50
:v #inst "2013-06-19T11:38:08.025-00:00"
:tx 13194139534331 :added true} #Datum{:e 17592186045436 .....
I can see the underlying datums...how can I extract their values?
Use d/resolve-tempid. If you were to transact a single entity, looking at :tx-data would work but if your transaction contained more than one entity, then you wouldn't know the order in which they appear in :tx-data.
What you should do is give temporary ids to your entities (before transacting them) using either (d/tempid) or its literal representation #db/id[:db.part/user _negativeId_] and then use d/resolve-tempid to go from your temporary id to the real id given by the database. The code would look something like:
(d/resolve-tempid (d/db conn) (:tempids tx) (d/tempid :db.part/user _negativeId_))
For a full code sample, see this gist.
Ah, figured it out.
I had to deref the Clojure promise, and then I was able to yank out the values I wanted:
(:e (second (:tx-data #(transact! conn query))))
Wrote a quick function based on a2ndrade's answer. The naming isn't ideal and I may be committing idiomatic faux pas; suggestions are very much welcome.
(ns my.datomic.util
(:require [datomic.api :as d]))
(defn transact-and-get-id
"Transact tx and return entity id."
[conn tx]
(let [tempid (:db/id tx)
post-tx #(d/transact conn [tx])
db (:db-after post-tx)
entid (d/resolve-tempid db (:tempids post-tx) tempid)]
entid))
Example usage:
(def my-conn
(d/connect (str "datomic:sql://datomic?jdbc:postgresql://"
"127.0.1:5432/datomic?user=datomic&password=somepw")
(defn thing-tx
"Create transaction for new thing."
[name]
{:db/id (d/tempid :db.part/user)
:thing/name name})
(transact-and-get-id my-conn (thing-tx "Bob")) ;; => 17592186045502
The Tupelo Datomic library has a function (td/eids tx-result) to easily extract the EIDs created in a transaction. For example:
; Create Honey Rider and add her to the :people partition
(let [tx-result #(td/transact *conn*
(td/new-entity :people ; <- partition is first arg (optional) to td/new-entity
{ :person/name "Honey Rider" :location "Caribbean" :weapon/type #{:weapon/knife} } ))
[honey-eid] (td/eids tx-result) ; retrieve Honey Rider's EID from the seq (destructuring)
]