How to get all ref attribute values? - clojure

I touch an entity and get many entity ids. I want all the attribute values instead of the ids while keeping the nested structure.
(d/touch (d/entity (get-db) (ffirst (find-all-families))))
=> {:family/parent #{{:db/id 17592186045423}
{:db/id 17592186045424}
{:db/id 17592186045426}
{:db/id 17592186045427}},
:family/child #{{:db/id 17592186045420}
{:db/id 17592186045421}},
:family/address {:db/id 17592186045428},
:family/email "someemail#gmail.com",
:db/id 17592186045429}
Thought about using something like simply touching all the entity ids but seems like complexity creeps up if I want all of them:
(map d/touch (:family/parent (d/touch (d/entity (get-db) (ffirst (find-all-families))))))
Not sure what the idiomatic approach is: finding a way to do it more through the querying side or through clojure.

The idiomatic way to do this in Datomic is to declare components in your schema. touch will touch all of the attributes of the entity, including any components recursively

You will probably wish to use the Datomic Pull API for this purpose. It can recursively return the attr/value pairs for all sub-entities that the user designates as "component". An example:
(def dark-side-of-the-moon [:release/gid #uuid "24824319-9bb8-3d1e-a2c5-b8b864dafd1b"])
(d/pull db [:release/media] dark-side-of-the-moon)
; result
{:release/media
[{:db/id 17592186121277,
:medium/format {:db/id 17592186045741},
:medium/position 1,
:medium/trackCount 10,
:medium/tracks
[{:db/id 17592186121278,
:track/duration 68346,
:track/name "Speak to Me",
:track/position 1,
:track/artists [{:db/id 17592186046909}]}
{:db/id 17592186121279,
:track/duration 168720,
:track/name "Breathe",
:track/position 2,
:track/artists [{:db/id 17592186046909}]}
{:db/id 17592186121280,
:track/duration 230600,
:track/name "On the Run",
:track/position 3,
:track/artists [{:db/id 17592186046909}]}
...]}]}
You may also use the Tupelo Datomic Pull API, which I think is nicer. As an example:
; If you wish to retain duplicate results on output, you must use td/query-pull and the Datomic
; Pull API to return a list of results (instead of a set).
(let [result-pull (td/query-pull :let [$ (live-db)] ; $ is the implicit db name
:find [ (pull ?eid [:location]) ] ; output :location for each ?eid found
:where [ [?eid :location] ] ) ; find any ?eid with a :location attr
result-sort (sort-by #(-> % first :location) result-pull)
]
(is (s/validate [ts/TupleMap] result-pull)) ; a list of tuples of maps
(is (= result-sort [ [ {:location "Caribbean"} ]
[ {:location "London" } ]
[ {:location "London" } ] ] )))

Related

Does anyone use Datomic to get the structure and entities separately?

So I use queries to filter data and then use pull to get the information out from the Datomic database.
(def rules
[[[search ?txt ?id] [(fulltext $ :artist/name ?txt) [[?id]]]]
[[search ?txt ?id] [(fulltext $ :track/name ?txt) [[?id]]]]])
(d/q
'[:find [(pull ?id [* {:track/artists [:db/id :track/name] :track/_artists [:db/id :artist/name] }]) ...]
:in $ % ?query
:where [search ?query ?id]]
db rules "John Lennon")
And sometimes these queries can get recursive, so for example I can change the pull to:
(d/q
'[:find [(pull ?id [* {:track/artists [:db/id :track/name] :track/_artists [* {:track/artists [:db/id :track/name]}]}]) ...]
:in $ % ?query
:where [search ?query ?id]]
db rules "John Lennon")
Now what I'd like to do is ensure that unique entities are being returned along with the :db/id structure as I don't want to return duplicate data as much as possible.
For example: (results elided with ...)
{:entities [{:db/id 1 :track/name "..." ...} {:db/id 2 :track/name "..." ...} {:db/id 3 :artist/name "..." ...}]
:structure [{:db/id 1 :track/artists [{:db/id 3}]} {:db/id 2 :track/artists [{:db/id 3}]}]}
Can this be done at the query level? Or do I need to walk the structure after the query returns and modify it? I'm happy to walk the structure at present, I'm just wondering if anyone has worked out a better approach?

How to update/overwrite a ref attribute with cardinality many in datomic?

Let’s say I have a schema that includes an attribute :x/value, where :x/value is a component, is a ref, and has cardinality many. The schema also has an id for x :x/id.
Now let’s say I say I transact the following:
(d/transact conn [{:x/id "1234" :x/value [{:text "test"}]}])
Then later I want to update the value, meaning really that I want to replace :x/value, so that in the end I have an entity like this:
{:db/id <some eid>
:x/id "1234"
:x/value [{:text "replacement"}]}
How would I do this?
So far, I've tried the following:
(d/transact conn [{:x/id "1234" :x/value [{:text "replacement"}]}])
But this simply added a new ref, so I got an entity looking like:
{:db/id <some eid>
:x/id "1234"
:x/value [{:text "test"} {:text "replacement"}]}
One way to achieve what I want here, I think, would be to manually retract both of the :text attributes by their entity id, and then do a new add transaction for the x entity.
But I wonder if there's a better way to do this. Any ideas?
You need to retract the old value and then update it with a new value:
[:db/retract entity-id attribute old-value]
[:db/add entity-id attribute new-value]
See http://docs.datomic.com/transactions.html
You can see more details in the James Bond example from Tupelo Datomic. Here is how the attributes are created:
(td/transact *conn* ; required required zero-or-more
; <attr name> <attr value type> <optional specs ...>
(td/new-attribute :person/name :db.type/string :db.unique/value) ; each name is unique
(td/new-attribute :person/secret-id :db.type/long :db.unique/value) ; each secret-id is unique
(td/new-attribute :weapon/type :db.type/ref :db.cardinality/many) ; one may have many weapons
(td/new-attribute :location :db.type/string) ; all default values
(td/new-attribute :favorite-weapon :db.type/keyword )) ; all default values
Suppose James throws his knife at a villan. We need to remove it from the DB.
(td/transact *conn*
(td/retract-value james-eid :weapon/type :weapon/knife))
(is (= (td/entity-map (live-db) james-eid) ; lookup by EID
{:person/name "James Bond" :location "London" :weapon/type #{:weapon/wit :weapon/gun} :person/secret-id 7 } ))
Once James has defeated Dr No, we need to remove him (& everything he possesses) from the database.
; We see that Dr No is in the DB...
(let [tuple-set (td/find :let [$ (live-db)]
:find [?name ?loc] ; <- shape of output tuples
:where {:person/name ?name :location ?loc} ) ]
(is (= tuple-set #{ ["James Bond" "London"]
["M" "London"]
["Dr No" "Caribbean"]
["Honey Rider" "Caribbean"] } )))
; we do the retraction...
(td/transact *conn*
(td/retract-entity [:person/name "Dr No"] ))
; ...and now he's gone!
(let [tuple-set (td/find :let [$ (live-db)]
:find [?name ?loc]
:where {:person/name ?name :location ?loc} ) ]
(is (= tuple-set #{ ["James Bond" "London"]
["M" "London"]
["Honey Rider" "Caribbean"] } )))
Update: Native Datomic Solution
Using native datomic is almost identical, just not quite as sweet as Tupelo:
; Dr No is no match for James. He gives up trying to use guile...
; Remove it using native Datomic.
(spy :before (td/entity-map (live-db) [:person/name "Dr No"]))
(d/transact *conn*
[[:db/retract [:person/name "Dr No"] :weapon/type :weapon/guile]])
(is (= (spy :after (td/entity-map (live-db) [:person/name "Dr No"])) ; LookupRef
{:person/name "Dr No"
:location "Caribbean"
:weapon/type #{:weapon/knife :weapon/gun}}))
:before => {:person/name "Dr No",
:weapon/type #{:weapon/guile :weapon/knife :weapon/gun},
:location "Caribbean"}
:after => {:person/name "Dr No",
:weapon/type #{:weapon/knife :weapon/gun},
:location "Caribbean"}
Update #2:
Side Note: I noticed that your example shows :arb/value [{:db/id 17592186045435, :content/text "tester"}], which is a list of maps of length 1. This is different than my example where :weapon/type is just a plain set of N items. This output difference is because you are using the pull API of Datomic. However, this won't affect your original problem.
We all know that James has had many Bond girls over the years. Here is an example of how to add in some honeys and them demote one of them:
(defn get-bond-girl-names []
(let [result-pull (d/pull (live-db) [:bond-girl] [:person/name "James Bond"])
bond-girl-names (forv [girl-entity (grab :bond-girl result-pull) ]
(grab :person/name (td/entity-map (live-db) (grab :db/id girl-entity))))
]
bond-girl-names))
(td/transact *conn*
(td/new-attribute :bond-girl :db.type/ref :db.cardinality/many)) ; there are many Bond girls
(let [tx-result #(td/transact *conn*
(td/new-entity {:person/name "Sylvia Trench"})
(td/new-entity {:person/name "Tatiana Romanova"})
(td/new-entity {:person/name "Pussy Galore"})
(td/new-entity {:person/name "Bibi Dahl"})
(td/new-entity {:person/name "Octopussy"})
(td/new-entity {:person/name "Paris Carver"})
(td/new-entity {:person/name "Christmas Jones"}))
tx-datoms (td/tx-datoms (live-db) tx-result)
girl-datoms (vec (remove #(= :db/txInstant (grab :a %)) tx-datoms))
girl-eids (mapv :e girl-datoms)
txr-2 (td/transact *conn*
(td/update [:person/name "James Bond"] ; update using a LookupRef
{:bond-girl girl-eids})
(td/update [:person/name "James Bond"] ; don't forget to add Honey Rider!
{:bond-girl #{[:person/name "Honey Rider"]}}))
]
(is (= (get-bond-girl-names)
["Sylvia Trench" "Tatiana Romanova" "Pussy Galore" "Bibi Dahl"
"Octopussy" "Paris Carver" "Christmas Jones" "Honey Rider"]))
; Suppose Bibi Dahl is just not refined enough for James. Give her a demotion.
(td/transact *conn*
(td/retract-value [:person/name "James Bond"] :bond-girl [:person/name "Bibi Dahl"]))
(newline)
(is (= (get-bond-girl-names) ; Note that Bibi Dahl is no longer listed
["Sylvia Trench" "Tatiana Romanova" "Pussy Galore"
"Octopussy" "Paris Carver" "Christmas Jones" "Honey Rider"] ))
)
Note that you can only use a LookupRef like [:person/name "Honey Rider"] since the attribute :person/name has :db.unique/value. If your :content/text is not :db.unique/value you'll have to use an EID to detach it from the parent entity.

How to construct a query that matches exactly a vector of refs in DataScript?

Setup Consider the following DataScript database of films and cast, with data stolen from learndatalogtoday.org: the following code can be executed in a JVM/Clojure REPL or a ClojureScript REPL, as long as project.clj contains [datascript "0.15.0"] as a dependency.
(ns user
(:require [datascript.core :as d]))
(def data
[["First Blood" ["Sylvester Stallone" "Brian Dennehy" "Richard Crenna"]]
["Terminator 2: Judgment Day" ["Linda Hamilton" "Arnold Schwarzenegger" "Edward Furlong" "Robert Patrick"]]
["The Terminator" ["Arnold Schwarzenegger" "Linda Hamilton" "Michael Biehn"]]
["Rambo III" ["Richard Crenna" "Sylvester Stallone" "Marc de Jonge"]]
["Predator 2" ["Gary Busey" "Danny Glover" "Ruben Blades"]]
["Lethal Weapon" ["Gary Busey" "Mel Gibson" "Danny Glover"]]
["Lethal Weapon 2" ["Mel Gibson" "Joe Pesci" "Danny Glover"]]
["Lethal Weapon 3" ["Joe Pesci" "Danny Glover" "Mel Gibson"]]
["Alien" ["Tom Skerritt" "Veronica Cartwright" "Sigourney Weaver"]]
["Aliens" ["Carrie Henn" "Sigourney Weaver" "Michael Biehn"]]
["Die Hard" ["Alan Rickman" "Bruce Willis" "Alexander Godunov"]]
["Rambo: First Blood Part II" ["Richard Crenna" "Sylvester Stallone" "Charles Napier"]]
["Commando" ["Arnold Schwarzenegger" "Alyssa Milano" "Rae Dawn Chong"]]
["Mad Max 2" ["Bruce Spence" "Mel Gibson" "Michael Preston"]]
["Mad Max" ["Joanne Samuel" "Steve Bisley" "Mel Gibson"]]
["RoboCop" ["Nancy Allen" "Peter Weller" "Ronny Cox"]]
["Braveheart" ["Sophie Marceau" "Mel Gibson"]]
["Mad Max Beyond Thunderdome" ["Mel Gibson" "Tina Turner"]]
["Predator" ["Carl Weathers" "Elpidia Carrillo" "Arnold Schwarzenegger"]]
["Terminator 3: Rise of the Machines" ["Nick Stahl" "Arnold Schwarzenegger" "Claire Danes"]]])
(def conn (d/create-conn {:film/cast {:db/valueType :db.type/ref
:db/cardinality :db.cardinality/many}
:film/name {:db/unique :db.unique/identity
:db/cardinality :db.cardinality/one}
:actor/name {:db/unique :db.unique/identity
:db/cardinality :db.cardinality/one}}))
(def all-datoms (mapcat (fn [[film actors]]
(into [{:film/name film}]
(map #(hash-map :actor/name %) actors)))
data))
(def all-relations (mapv (fn [[film actors]]
{:db/id [:film/name film]
:film/cast (mapv #(vector :actor/name %) actors)}) data))
(d/transact! conn all-datoms)
(d/transact! conn all-relations)
Description In a nutshell, there are two kinds of entities in this database—films and actors (word intended to be ungendered)—and three kinds of datoms:
film entity: :film/name (a unique string)
film entity: :film/cast (multiple refs)
actor entity: :actor/name (unique string)
Question I would like to construct a query which asks: which films have these N actors, and these N actors alone, appeared as the sole stars, for N>=2?
E.g., RoboCop starred Nancy Allen, Peter Weller, Ronny Cox, but no film starred solely the first two of these, Allen and Weller. Therefore, I would expect the following query to produce the empty set:
(d/q '[:find ?film-name
:where
[?film :film/name ?film-name]
[?film :film/cast ?actor-1]
[?film :film/cast ?actor-2]
[?actor-1 :actor/name "Nancy Allen"]
[?actor-2 :actor/name "Peter Weller"]]
#conn)
; => #{["RoboCop"]}
However, the query is flawed because I don't know how to express that any matches should exclude any actors who are not Allen or Weller—again, I want to find the movies where only Allen and Weller have collaborated without any other actors, so I want to adapt the above query to produce the empty set. How can I adjust this query to enforce this requirement?
Because DataScript doesn't have negation (as of May 2016), I don't believe that's possible with one static query in 'pure' Datalog.
My way to go would be:
build the query programmatically to add the N clauses that state that the cast must contain the N actors
Add a predicate function which, given a movie, the database, and the set of actors ids, uses the EAVT index to find if each movie has an actor that is not in the set.
Here's a basic implementation
(defn only-those-actors? [db movie actors]
(->> (datoms db :eavt movie :film/cast) seq
(every? (fn [[_ _ actor]]
(contains? actors actor)))
))
(defn find-movies-with-exact-cast [db actors-names]
(let [actors (set (d/q '[:find [?actor ...] :in $ [?name ...] ?only-those-actors :where
[?actor :actor/name ?name]]
db actors-names))
query {:find '[[?movie ...]]
:in '[$ ?actors ?db]
:where
(concat
(for [actor actors]
['?movie :film/cast actor])
[['(only-those-actors? ?db ?movie ?actors)]])}]
(d/q query db actors db only-those-actors?)))
You can use predicate fun and d/entity together for filtering datoms by :film/cast field of an entity. This approach looks much more straightforward until Datascript doesn't support negation (not operator and so on).
Look at the row (= a (:age (d/entity db e)) in the test case of the Datascript here
[{:db/id 1 :name "Ivan" :age 10}
{:db/id 2 :name "Ivan" :age 20}
{:db/id 3 :name "Oleg" :age 10}
{:db/id 4 :name "Oleg" :age 20}]
...
(let [pred (fn [db e a]
(= a (:age (d/entity db e))))]
(is (= (q/q '[:find ?e
:in $ ?pred
:where [?e :age ?a]
[(?pred $ ?e 10)]]
db pred)
#{[1] [3]})))))
In your case, the predicate body could look something like this
(clojure.set/subset? actors (:film/cast (d/entity db e))
In regards to performance, the d/entity call is fast because it is a lookup by index.

Possible to get enum value via Datomic pull syntax?

In the mbrainz sample data, the :artist/type is an enum. Is it possible to pull the value of the enum out of :db/ident and associate it as the value of the :artist/type key using pull syntax?
This is as close as I could get:
[:find (pull ?e [:artist/name {:artist/type [:db/ident]}])
:where
[?e :artist/name "Ray Charles"]
]
;;=> [[{:artist/name "Ray Charles", :artist/type {:db/ident :artist.type/person}}]]
Is it possible to use pull syntax to reshape the result into something like this?
;;=> [[{:artist/name "Ray Charles", :artist/type :artist.type/person}]]
I don't think you can do it using the Pull API the way you are seeking. You may find that it is easier to use the Tupelo Datomic library:
(require '[tupelo.datomic :as td]
'[tupelo.core :refer [spyx]] )
(let [x1 (td/query-scalar :let [$ db-val]
:find [ ?e ]
:where [ [?e :artist/name "Ray Charles"] ] )
x2 (td/entity-map db-val x1)
]
(spyx x1)
(spyx x2)
)
which gives the result:
x1 => 17592186049074
x2 => {:artist/sortName "Charles, Ray", :artist/name "Ray Charles",
:artist/type :artist.type/person, :artist/country :country/US,
:artist/gid #uuid "2ce02909-598b-44ef-a456-151ba0a3bd70",
:artist/startDay 23, :artist/endDay 10, :artist/startYear 1930,
:artist/endMonth 6, :artist/endYear 2004, :artist/startMonth 9,
:artist/gender :artist.gender/male}
So :artist/type is already converted into the :db/ident value and you can just pull it out of the map.
You can use specter on the result that the pull expression returns:
(->> pull-result
(sp/transform (sp/walker :db/ident) :db/ident))
The value of key :db/ident is extracted for every map that has that key.
Was quite easy to do with postwalk
for any pulled :db/ident you can transform with this function
(defn flatten-ident [coll]
(clojure.walk/postwalk
(fn [item] (get item :db/ident item)) coll))

Mapping a list of datomic ids to entity maps

When I query for a list of Datomic entities, e.g like in the example below:
'[:find ?e
:where
[?e :category/name]]
Usually, I'd like to create a list of maps that represent the full entities, i.e
#{[1234] [2223]} => [{:category/name "x" :db/id 1234}, {:category/name "y" :db/id 2223}]
Here is my approach at the moment, in the form of a helper function.
(defn- db-ids->entity-maps
"Takes a list of datomic entity ids retrieves and returns
a list of hydrated entities in the form of a list of maps."
[db-conn db-ids]
(->>
db-ids
seq
flatten
(map #(->>
%
;; id -> lazy entity map
(d/entity (d/db db-conn))
;; realize all values, except for db/id
d/touch
(into {:db/id %})))))
Is there a better way?
With the pull api, this is pretty easy now.
'[:find [(pull ?e [*]) ...]
:in $ [[?e] ...]
:where [?e]]
I used to take this approach to save queries to the DB, the code is probably less reusable but it depends on what is more critical in your current scenario. I haven't a Datomic instance configured as I am not working with it right now so it may contain syntax error but I hope you get the idea.
(def query-result '[:find ?cat-name ?id
:where
[?cat-name :category/name
[?id :db/id]])
=>
#{["x" 1234] ["x" 2223]}
(defn- describe-values
"Adds proper keys to the given values."
[keys-vec query-result]
(vec (map #(zipmap keys-vec %) query-result))
(describe-values [:category/name :db/id] query-result)
=>
[{:db/id 2223, :category/name "x"} {:db/id 1234, :category/name "x"}]