Getting a seemingly bizarre problem with Datascript. For some reason when I run this query without it being wrapped in a function, everything works. But once I wrap it in a function, it returns the value for :block/content for every entity in the database. I'm confused because I haven't encountered any issues with wrapping other Datascript queries in the past. Does anyone more experienced than me with Datascript see any issues?
;; Works fine and returns the correct value
(ds/q '[:find ?block-text
:where
[?id :block/id "l_63xa4m1"]
[?id :block/content ?block-text]]
#conn)
;; Returns every value for `:block/content` in the db
(defn content-find
[id-passed]
(ds/q '[:find ?block-text
:where
[?id :block/id ?id-passed]
[?id :block/content ?block-text]]
#conn))
(content-find "l_63xa4m1")
EDIT: Solved over here
In your defn version you are using the query clause [?id :block/id ?id-passed]. This doesn't actually use the id-passed parameter you passed to the function.
I'm not sure how to pass parameters correctly. I believe there is an :in clause or so?
Related
I'd like to get the history of values for a particular field in Datomic.
My intuition is to use (d/history) like
(d/q '[:find ?entity ?field-val ?date ?tx
:in $
:where
[?entity :namespace/field ?field-val ?tx]
[?tx :db/txInstant ?date]]
(d/history (db/get-db)))
However, this query will duplicate most values because it lists every retraction as well as every value update (every db/add and db/retract).
I thought maybe I could query the datoms with the transaction, then check the operations. But I can't find a way to query the datoms.
(d/pull db '[*] tx-id) doesn't include datoms.
search engine results were not helpful for keywords like "query datomic transaction datoms"
searching for datomic transaction schema is not fruitful
I can use tx-range, but that seems unweildly.
Any better approaches?
I was looking in the wrong place. History queries offer an extra hidden positional value described in the history docs.
So any where clause can include ?entity ?attribute ?value ?transaction ?operation.
?operation is true for :db/add and false for :db/retract
So, the query I want looks like
(d/q '[:find ?entity ?field-val ?date ?tx
:in $
:where
[?entity :namespace/field ?field-val ?tx true] ;;ADDED TRUE
[?tx :db/txInstant ?date]]
(d/history (db/get-db)))
I'm using Datomic and would like to pull entire entities from any number of points in time based on my query. The Datomic docs have some decent examples about how I can perform queries from two different database instances if I know those instances before the query is performed. However, I'd like my query to determine the number of "as-of" type database instances I need and then use those instances when pulling the entities. Here's what I have so far:
(defn pull-entities-at-change-points [entity-id]
(->>
(d/q
'[:find ?tx (pull ?dbs ?client [*])
:in $ [?dbs ...] ?client
:where
[?client ?attr-id ?value ?tx true]
[(datomic.api/ident $ ?attr-id) ?attr]
[(contains? #{:client/attr1 :client/attr2 :client/attr3} ?attr)]
[(datomic.api/tx->t ?tx) ?t]
[?tx :db/txInstant ?inst]]
(d/history (d/db db/conn))
(map #(d/as-of (d/db db/conn) %) [1009 1018])
entity-id)
(sort-by first)))
I'm trying to find all transactions wherein certain attributes on a :client entity changed and then pull the entity as it existed at those points in time. The line: (map #(d/as-of (d/db db/conn) %) [1009 1018]) is my attempt to created a sequence of database instances at two specific transactions where I know the client's attributes changed. Ideally, I'd like to do all of this in one query, but I'm not sure if that's possible.
Hopefully this makes sense, but let me know if you need more details.
I would split out the pull calls to be separate API calls instead of using them in the query. I would keep the query itself limited to getting the transactions of interest. One example solution for approaching this would be:
(defn pull-entities-at-change-points
[db eid]
(let
[hdb (d/history db)
txs (d/q '[:find [?tx ...]
:in $ [?attr ...] ?eid
:where
[?eid ?attr _ ?tx true]]
hdb
[:person/firstName :person/friends]
eid)
as-of-dbs (map #(d/as-of db %) txs)
pull-w-t (fn [as-of-db]
[(d/as-of-t as-of-db)
(d/pull as-of-db '[*] eid)])]
(map pull-w-t as-of-dbs)))
This function against a db I built with a toy schema would return results like:
([1010
{:db/id 17592186045418
:person/firstName "Gerry"
:person/friends [{:db/id 17592186045419} {:db/id 17592186045420}]}]
[1001
{:db/id 17592186045418
:person/firstName "Jerry"
:person/friends [{:db/id 17592186045419} {:db/id 17592186045420}]}])
A few points I'll comment on:
the function above takes a database value instead of getting databases from the ambient/global conn.
we map pull for each of various time t's.
using the pull API as an entry point rather than query is appropriate for cases where we have the entity and other information on hand and just want attributes or to traverse references.
the impetus to get everything done in one big query doesn't really exist in Datomic since the relevant segments will have been realized in the peer's cache. You're not, i.e., saving a round trip in using one query.
the collection binding form is preferred over contains and leverages query caching.
Given that Datomic does not support pagination I'm wondering how to efficiently support a query such as:
Take the first 30 entities on :history/body, find entities whose
:history/body matches some regex.
Here's how I'd do regex matching alone:
{:find [?e]
:where [[?e :history/body ?body]
[(re-find #"foo.*bar$" ?body)]]}
Observations:
I could then (take ...) from those, but that is not the same as matching against the first 30 entities.
I could get all entities, take 30 then manually filter with re-find, but if I have 30M entities, getting all of them just to take 30 seems wildly inefficient. Additionally: what if I wanted to take 20M out of my 30M entities and filter them via re-find?
Datomic docs talk about how queries are executed locally, but I've tried doing in-memory transformations on a set of 52913 entities (granted, they're fully touched) and it takes ~5 seconds. Imagine how bad it'd be in the millions or 10s of millions.
(Just brainstorming, here)
First of all, if you're ever using regexp, you may want to consider a fulltext index on :history/body so that you can do:
[(fulltext $ :history/body "foo*bar") [[?e]]]
(Note: You can't change :db/fulltext true/false on an existing entity schema)
Sorting is something you have to do outside the query. But depending on your data, you may be able to constrain your query to a single "page" and then apply your predicate to just those entities.
For example, if we were only paginating :history entities by an auto-incrementing :history/id, then we'd know beforehand that "Page 3" is :history/id 61 to 90.
[:find ?e
:in $ ?min-id ?max-id
:where
[?e :history/id ?id]
(<= ?min-id ?id ?max-id)
(fulltext $ :history/body "foo*bar") [[?e]]]
Maybe something like this:
(defn get-filtered-history-page [page-n match]
(let [per-page 30
min-id (inc (* (dec page-n) per-page))
max-id (+ min-id per-page)]
(d/q '[:find ?e
:in $ ?min-id ?max-id ?match
:where
[?e :history/id ?id]
[(<= ?min-id ?id ?max-id)]
[(fulltext $ :history/body ?match) [[?e]]]]
(get-db) min-id max-id match)))
But, of course, the problem is that constraining the paginated set is usually based on an ordering you don't know in advance, so this isn't very helpful.
Using 'load-data' below from the Clojure repl (using 'util.clj' from the tutorial https://github.com/swannodette/om/wiki/Intermediate-Tutorial with a modified schema and initial data set) to load data into a new Datomic database, the data does not show up in the Datomic console.
However, I get no error message when performing the 'load-data' action from the repl.
The schema shows up as expected in the Datomic console. Using code unmodified from the tutorial, I can see both the schema and the data.
I must have a problem in the code that sets the initial data. But I don't know where it is since there is no error message.
How can I get error messages and other detail from an init transaction on a Datomic database?
Code:
(defn transact-all [conn f]
(doseq [txd (read-all f)]
(d/transact conn txd))
:done)
(defn load-schema []
(transact-all (get-conn) (io/resource "data/schema.edn")))
(defn load-data []
(transact-all (get-conn) (io/resource "data/initial.edn")))
;; Logging provides some comparison with the known working scenario, but so far I only can log entity id's:
(defn read-log []
(d/q '[:find ?e
:in ?log ?t1 ?t2
:where [(tx-ids ?log ?t1 ?t2) [?tx ...]]
[(tx-data ?log ?tx) [[?e]]]]
(d/log (get-conn)) #inst "2014-07-14" #inst "2015-07-01")
)
In Clojure you can use # or deref to get a transaction's results, e.g.:
#(d/transact conn txd)
The map it returns is described in the docs for d/transact:
http://docs.datomic.com/clojure/#datomic.api/transact
See in particular:
If the transaction aborts, attempts to get the future's value throw a java.util.concurrent.ExecutionException, wrapping a java.lang.Error containing error information. If the transaction times out, the call to transact itself will throw a RuntimeException. The transaction timeout can be set via the system property datomic.txTimeoutMsec, and defaults to 10000 (10 seconds).
Invalid transactions will also throw an IllegalArgumentException (or some other exception).
This will produce two different ids, which is great:
#db/id[:db.part/user]
#db/id[:db.part/user]
but anything like the following (I tried a lot of ideas so far) will produce the same id twice, which is not what I want:
(repeatedly 2 (fn [] #db/id[:db.part/user]))
(for [n [1 2]] #db/id[:db.part/user])
All yield something like
(#db/id[:db.part/user -1000774] #db/id[:db.part/user -1000774])
where the number produced is the same for each call.
What I actually want is for the calls to NOT produce a number at all, so that I can just add the produced data via a transaction.
Any ideas?
Just to be clear, the documentation says, "Each call to tempid produces a unique temporary id."
[Edited after comment by #maxthoursie that repeat would be having this problem in any case.]
Use
(require '[datomic.api :as d])
(repeatedly 2 #(d/tempid :db.part/user))
;; => (#db/id[:db.part/user -1000118] #db/id[:db.part/user -1000119])
Consider that #... are reader macros meaning that their value will be resolved when the expression is read which naturally happens only once. Use the #... macro only when you are writing literal transaction data (like a schema). Use datomic.api/tempid to generate tempids in runtime.
Because repeat is repeating the value it got from calling id once.
Use repeatedly instead.
See examples at http://clojuredocs.org/clojure_core/clojure.core/repeatedly