datomic: transact function not really writing to database - clojure

I'm running the dev db in datomic-free
However, I thought that it would write something to the database but when I do a pull, it's not there. this is my peer.clj
(ns dank.peer
(:require [datomic.api :as d :refer (q)]
[clojure.pprint :as pp]))
; Name the database as a uri
(def uri "datomic:mem://dank")
; Read the schema and seed data as strings
(def schema-tx (read-string (slurp "resources/dank/schema.edn")))
(def data-tx (read-string (slurp "resources/dank/seed-data.edn")))
; Initialize the db with above vars
(defn init-db
[]
(when (d/create-database uri)
(let
[conn (d/connect uri)]
#(d/transact conn schema-tx)
#(d/transact conn data-tx))))
(init-db)
(def conn (d/connect uri))
(def db (d/db conn))
(defn create
"Creates a new note"
[notes]
#(d/transact
conn
[{:db/id #db/id[:db.part/user]
:item/author "default"
:item/notes notes}]))
(defn get-notes
[author]
(d/pull-many db '[*]
(flatten (into []
(q '[:find ?e
:in $ ?author
:where
[?e :item/author ?author]]
db
author
)))))
(create "some note") ----> shows the whole transaction data structure with before and after.
(get-notes "default") ---> [] it is empty!
Am I missing something here?

mavbozo is correct in his response that when you call your get-notes function you are using a value of the database from before you transacted your new note.
If you were to call (def db (d/db conn)) again prior to get-notes you will see the new data in the database.
However, it's generally preferred to avoid dealing with connections or databases as global variables, as it prevents function behaviors from being determined strictly from their parameters. As you noticed, the behavior of get-notes depends on the state of the db variable in your global state. As a rule of thumb, it's better to pass a database value to functions that query or pull from the database and pass a connection to functions that transact against a database or monitor the state of a database over time (see http://docs.datomic.com/best-practices.html#consistent-db-value-for-unit-of-work for more discussion of individual values of a Datomic database).
In this case, your get-notes function could be something like:
(defn get-notes
[db author]
(d/q '[:find (pull ?e [*])
:in $ ?author
:where
[?e :item/author ?author]]
db
author))
Which you can then invoke via:
(get-notes (d/db conn) "default")
To query the most current version of the database. This approach also has the advantage that you can call get-notes with any database value (i.e. the most recent, one from a given time in the past, or a history db).
Similarly, I would recommend changing your create function to take a connection instead of using the globally defined conn.
As far as managing your connection, again it is best to avoid using a global variable whenever possible. Some preferred options might be to put the connection into an application state map or an atom that is passed throughout your program as appropriate. Another good option that is used frequently is to leverage Stuart Sierra's component library (https://github.com/stuartsierra/component) to manage the lifecycle of the connection.

t0, t1, t2 denote increasing time t where t0 precedes t1, t1 precedes t2
(d/db conn) returns current snapshot of datomic database when the call is made.
let's say in t0 you called (def db (d/db conn)). At time t0 the notes are empty. the db here is a snapshot of db time t0.
then you create something during time t1 using (create "some note"). At time t1, the notes at least contain 1 note.
then during time t2 you call (get-notes "default"). At time t2, the notes at least contain 1 note.
the problem is, the db inside get-notes still refers to database snapshot during time t0 that is the db you retrieve using (def db (d/db conn)) at t0. At time t0, there are still no notes. That's why get-notes return empty.
when you query a datomic database, you must specify which db you want to query: the db as of now, yesterday db, db from 1 hour ago, etc. Of course, it's quite easy to get db as of now, just call (d/db conn).
I find it useful to make db explicit for query functions such as changing your get-notes function declaration parameter to (defn get-notes [db author] ....).

Related

Datomic: How do I query across any number of database inside of a query?

I'm using Datomic and would like to pull entire entities from any number of points in time based on my query. The Datomic docs have some decent examples about how I can perform queries from two different database instances if I know those instances before the query is performed. However, I'd like my query to determine the number of "as-of" type database instances I need and then use those instances when pulling the entities. Here's what I have so far:
(defn pull-entities-at-change-points [entity-id]
(->>
(d/q
'[:find ?tx (pull ?dbs ?client [*])
:in $ [?dbs ...] ?client
:where
[?client ?attr-id ?value ?tx true]
[(datomic.api/ident $ ?attr-id) ?attr]
[(contains? #{:client/attr1 :client/attr2 :client/attr3} ?attr)]
[(datomic.api/tx->t ?tx) ?t]
[?tx :db/txInstant ?inst]]
(d/history (d/db db/conn))
(map #(d/as-of (d/db db/conn) %) [1009 1018])
entity-id)
(sort-by first)))
I'm trying to find all transactions wherein certain attributes on a :client entity changed and then pull the entity as it existed at those points in time. The line: (map #(d/as-of (d/db db/conn) %) [1009 1018]) is my attempt to created a sequence of database instances at two specific transactions where I know the client's attributes changed. Ideally, I'd like to do all of this in one query, but I'm not sure if that's possible.
Hopefully this makes sense, but let me know if you need more details.
I would split out the pull calls to be separate API calls instead of using them in the query. I would keep the query itself limited to getting the transactions of interest. One example solution for approaching this would be:
(defn pull-entities-at-change-points
[db eid]
(let
[hdb (d/history db)
txs (d/q '[:find [?tx ...]
:in $ [?attr ...] ?eid
:where
[?eid ?attr _ ?tx true]]
hdb
[:person/firstName :person/friends]
eid)
as-of-dbs (map #(d/as-of db %) txs)
pull-w-t (fn [as-of-db]
[(d/as-of-t as-of-db)
(d/pull as-of-db '[*] eid)])]
(map pull-w-t as-of-dbs)))
This function against a db I built with a toy schema would return results like:
([1010
{:db/id 17592186045418
:person/firstName "Gerry"
:person/friends [{:db/id 17592186045419} {:db/id 17592186045420}]}]
[1001
{:db/id 17592186045418
:person/firstName "Jerry"
:person/friends [{:db/id 17592186045419} {:db/id 17592186045420}]}])
A few points I'll comment on:
the function above takes a database value instead of getting databases from the ambient/global conn.
we map pull for each of various time t's.
using the pull API as an entry point rather than query is appropriate for cases where we have the entity and other information on hand and just want attributes or to traverse references.
the impetus to get everything done in one big query doesn't really exist in Datomic since the relevant segments will have been realized in the peer's cache. You're not, i.e., saving a round trip in using one query.
the collection binding form is preferred over contains and leverages query caching.

Inconsistency between in (mem) and out (dev,sql,etc) of process Transactor in Datomic:

I've discovered an inconsistency between the way db.type/instant values are handled in the in (mem) and out (dev,sql...) of process transactor.
When using the in process mem transactor the underlying type of a db.type/instant value is preserved when transacting and retrieving. For example, (type-printer uri) below shows that a transacted java.util.TimeStamp is retrieved as a java.sql.TimeStamp. However in the out of process transactors it is retrieved as a java.util.Date.
In most cases this inconsistency does not cause problems, however when dealing with bad Date implementations it can cause all sorts of problems. In my case the fact that java.sql.TimeStamp does not have a symmetric equals method, meant that a bug in my code only appeared when I ran it in an out of process transactor...
Obviously Datomic does not have any control over external implementations of java.util.Date, so I think the 'Date sanitisation' approach of the out of process transactors is correct.
Really all I want is a sanity check before I raise it as an issue with the Datomic guys so if someone could take a look and see if you agree it would be greatly appreciated.
Thanks,
Matt.
(require '[datomic.api :as d])
(import '(java.sql Timestamp))
(defn type-printer [uri]
"Prints the type of a transacted java.sql.TimeStamp when retrieved from the databse"
(d/create-database uri)
(let [ts-att-schema {:db/id (d/tempid :db.part/db)
:db/ident :entity/ts
:db/valueType :db.type/instant
:db/cardinality :db.cardinality/one
:db.install/_attribute :db.part/db}
ts (Timestamp. 1000000000000)
entity {:db/id (d/tempid :db.part/user)
:entity/ts ts}
conn (d/connect uri)]
;transact schema and then entity with associated TimeStamp value
#(d/transact conn [ts-att-schema])
#(d/transact conn [entity])
(let [type (->> (d/db conn)
(d/q '[:find ?ts :where[?e :entity/ts ?ts]])
(first)
(first)
(type)
)]
(println "java.sql.TimeStamp is retrived as type" type))))
;TimeStamp type is preserved when being transacted and then retrieved
(type-printer "datomic:mem://test-db")
;TimeStamp is sanitised in the database and retrieved as a java.util.Date. This case obviously relies on a
;dev transactor running on localhost:4334. Have only verified for dev and sql transactors.
(type-printer "datomic:dev://localhost:4334/test-db")

how to get the datetime of the last transaction in a datomic db?

I want to find the most recent transaction made to a connection. The following does not seem to give the correct date:
(require '[datomic.api :as datomic])
(-> conn datomic/db datomic/basis-t datomic/t->tx (java.util.Date.))
I figured it out:
(defn last-transaction-time [db]
(let [t (-> db datomic/basis-t)]
[t (ffirst (datomic/q '[:find ?t
:in $ ?tx
:where [?tx :db/txInstant ?t]]
db
(datomic/t->tx t)))]))
You would probably want to let the result of your thread function up to the datomic/t->tx. Then use that to query for the transaction entity (implicitly created entity for each transaction). Each transaction entity has a :db/txInstant attribute that is implicitly added during a transaction. The value of that attribute is what you would want to pass to the java.util.Date. static method.
Should be much easier
(d/q
'[:find (max 1 ?tx)
:where
[?tx :db/txInstant]]
db)

How to get details/error messages from Datomic's' transact'

Using 'load-data' below from the Clojure repl (using 'util.clj' from the tutorial https://github.com/swannodette/om/wiki/Intermediate-Tutorial with a modified schema and initial data set) to load data into a new Datomic database, the data does not show up in the Datomic console.
However, I get no error message when performing the 'load-data' action from the repl.
The schema shows up as expected in the Datomic console. Using code unmodified from the tutorial, I can see both the schema and the data.
I must have a problem in the code that sets the initial data. But I don't know where it is since there is no error message.
How can I get error messages and other detail from an init transaction on a Datomic database?
Code:
(defn transact-all [conn f]
(doseq [txd (read-all f)]
(d/transact conn txd))
:done)
(defn load-schema []
(transact-all (get-conn) (io/resource "data/schema.edn")))
(defn load-data []
(transact-all (get-conn) (io/resource "data/initial.edn")))
;; Logging provides some comparison with the known working scenario, but so far I only can log entity id's:
(defn read-log []
(d/q '[:find ?e
:in ?log ?t1 ?t2
:where [(tx-ids ?log ?t1 ?t2) [?tx ...]]
[(tx-data ?log ?tx) [[?e]]]]
(d/log (get-conn)) #inst "2014-07-14" #inst "2015-07-01")
)
In Clojure you can use # or deref to get a transaction's results, e.g.:
#(d/transact conn txd)
The map it returns is described in the docs for d/transact:
http://docs.datomic.com/clojure/#datomic.api/transact
See in particular:
If the transaction aborts, attempts to get the future's value throw a java.util.concurrent.ExecutionException, wrapping a java.lang.Error containing error information. If the transaction times out, the call to transact itself will throw a RuntimeException. The transaction timeout can be set via the system property datomic.txTimeoutMsec, and defaults to 10000 (10 seconds).
Invalid transactions will also throw an IllegalArgumentException (or some other exception).

Getting the id of an inserted entity in datomic?

After I run a transaction in datomic to insert a value, how I can use the return value of the transaction to get the ids of any entities that were created?
Here is a sample of the return value I get after an insert:
#<promise$settable_future$reify__4841#7c92b2e9: {:db-before datomic.db.Db#62d0401f, :db-after datomic.db.Db#bba61dfc,
:tx-data [#Datum{:e 13194139534331 :a 50
:v #inst "2013-06-19T11:38:08.025-00:00"
:tx 13194139534331 :added true} #Datum{:e 17592186045436 .....
I can see the underlying datums...how can I extract their values?
Use d/resolve-tempid. If you were to transact a single entity, looking at :tx-data would work but if your transaction contained more than one entity, then you wouldn't know the order in which they appear in :tx-data.
What you should do is give temporary ids to your entities (before transacting them) using either (d/tempid) or its literal representation #db/id[:db.part/user _negativeId_] and then use d/resolve-tempid to go from your temporary id to the real id given by the database. The code would look something like:
(d/resolve-tempid (d/db conn) (:tempids tx) (d/tempid :db.part/user _negativeId_))
For a full code sample, see this gist.
Ah, figured it out.
I had to deref the Clojure promise, and then I was able to yank out the values I wanted:
(:e (second (:tx-data #(transact! conn query))))
Wrote a quick function based on a2ndrade's answer. The naming isn't ideal and I may be committing idiomatic faux pas; suggestions are very much welcome.
(ns my.datomic.util
(:require [datomic.api :as d]))
(defn transact-and-get-id
"Transact tx and return entity id."
[conn tx]
(let [tempid (:db/id tx)
post-tx #(d/transact conn [tx])
db (:db-after post-tx)
entid (d/resolve-tempid db (:tempids post-tx) tempid)]
entid))
Example usage:
(def my-conn
(d/connect (str "datomic:sql://datomic?jdbc:postgresql://"
"127.0.1:5432/datomic?user=datomic&password=somepw")
(defn thing-tx
"Create transaction for new thing."
[name]
{:db/id (d/tempid :db.part/user)
:thing/name name})
(transact-and-get-id my-conn (thing-tx "Bob")) ;; => 17592186045502
The Tupelo Datomic library has a function (td/eids tx-result) to easily extract the EIDs created in a transaction. For example:
; Create Honey Rider and add her to the :people partition
(let [tx-result #(td/transact *conn*
(td/new-entity :people ; <- partition is first arg (optional) to td/new-entity
{ :person/name "Honey Rider" :location "Caribbean" :weapon/type #{:weapon/knife} } ))
[honey-eid] (td/eids tx-result) ; retrieve Honey Rider's EID from the seq (destructuring)
]