Clojure function to realise a sequence - clojure

I have the following function to collect datomic transaction reports, output of this earlier question.
(def recreate-database
"Defines a function which when called will call each of the four given
functions and return a collection containing the output of each."
(juxt pt1-transact-schema
pt1-transact-data
pt2-transact-schema
pt2-transact-data))
The next step is that each of these functions may itself call not one but a series of transactions, so the complete result desired will be a nested collection of transaction reports for the initial database build.
I had the following, which was unsuitable, because when called from recreate-database above, only the last transaction report comes through, a function of course only returning its final form.
(defn pt1-transact-data []
(d/transact conn {:tx-data pt1-user-data})
(d/transact conn {:tx-data pt1-path-data})
(d/transact conn {:tx-data pt1-series-data}))
So I arrived at the following
(defn pt1-transact-data []
(map identity
[(d/transact conn {:tx-data pt1-user-data})
(d/transact conn {:tx-data pt1-path-data})
(d/transact conn {:tx-data pt1-series-data})
(d/transact conn {:tx-data pt1-path-series-data})]))
This works fine, except there's still incidental complexity. map identity is linguistically low level in relation to intended meaning here.
Hence thinking was, we just need to "realize the sequence", so I extracted:
(defn realize-all [coll] (map identity coll))
Then I remembered doall.
Just using that works here too.
So, is this the same as what doall does?
I note from (source doall) it doesn't appear the same, that leads to dorun, which leads to... recur.
So map identity or doall or dorun appear to serve similar purpose here.
Are these equivalent? Is there more to say?

why don't you just wrap it in a vector? that seems to be the easiest solution:
(defn pt1-transact-data []
[(d/transact conn {:tx-data pt1-user-data})
(d/transact conn {:tx-data pt1-path-data})
(d/transact conn {:tx-data pt1-series-data})])
or else you could use mapv:
(defn pt1-transact-data []
(mapv (partial d/transact conn)
[{:tx-data pt1-user-data}
{:tx-data pt1-path-data}
{:tx-data pt1-series-data}))

Related

Clojure : force evaluation of code with return value

I have had an issue with my code due to a very twisted behaviour of a function.
I use google-api to stream data in BigQuery. In Java, you create an object called Bigquery.Tabledata.InsertAll (a request) and then you execute it
TableDataInsertAllResponse response = request.execute();
(sample code from Google)
But as you can see, the execution is something that has side effect while returning a response.
I reproduced it in Clojure (in a chunked fashion)
(defn load-bq
[client project-id dataset-id table-id data & {:keys [chunk-size] :or {chunk-size 500}}]
(let [chunks (partition-all chunk-size data)
_ (str "Now streaming data into table : [" project-id ":" dataset-id "." table-id "]")]
(map (partial atomic-load-bq client project-id dataset-id table-id) chunks)))
If I try to stream in repl, it works fine. But surpinsingly a doall does not work in code like in a let or with a do.
Here a function to illustrate the principle
(def load-this-table
[... data]
(let [_ (doall (load-bq ... data))]
(load-bq ... data)
(do (load-bq ...data))))
Here, nothing will be loaded.
I found a trick that works though it is a bit far-fetched :
(def load-this-table
[... data]
(let [_ (println (load-bq ... data))]
(println (load-bq ... data))))
Here both line will execute. of course i need only one streaming so I chose a single solution here.
How to force evaluation of this code without having to use println ?
I could use what force evaluation behing println or any more general core function.
I have the impression that the difference is not really linked to lazyness but more to a more fundamental difference bewteen Clojure and Java. And maybe that the response have to be "taken" by the client.
Thanks !

get agent content and convert it to JSON

I'm still newbie in clojure and I'm trying to build application which read two files and write the diffrence on JSON file
(defn read-csv
"reads data."
[]
(with-open [rdr (
io/reader "resources/staples_data.csv")]
(doseq [line (rest(line-seq rdr))]
(println(vec(re-seq #"[^,]+" line))))))
(defn read-psv
"reads data."
[]
(with-open [rdr (
io/reader "resources/external_data.psv")]
(doseq [line (rest(line-seq rdr))]
; (print(vec(re-seq #"[^|]+" line))))))
(doall(vec(re-seq #"[^|]+" line))))))
(defn process-content []
(let [csv-records (agent read-csv)
psv-records (agent read-psv)]
(json/write-str {"my-data" #csv-records "other-data" #psv-records}))
)
Im getting an exception: Exception Don't know how to write JSON of class $read_csv clojure.data.json/write-generic (json.clj:385)
Please some help with some explanation, thanks in advance!
You are giving the agent a function as its initial value. Perhaps you meant to do an asynchronous call to that function instead? In that case, a future is a better match for your scenario as shown. agent is synchronous, it's send and send-off that are async, and they assume you are propagating some state across calls which doesn't match your usage here.
(defn process-content []
(let [csv-records (future-call read-csv)
psv-records (future-call read-psv)]
(json/write-str {"my-data" #csv-records "other-data" #psv-records})))
The problem after that is that doseq is only for side effects, and always returns nil. If you want the results read from the csv files (evaluating eagerly so you are still in the scope of the with-open call), use (doall (for ...)) as a replacement for (doseq ...). Also, the println in read-csv will need to be removed, or replaced with (doto (vec (re-seq #"[^,]+" line)) println) because println always returns nil, and I assume you want the actual data from the file, not a list of nils.

Getting the id of an inserted entity in datomic?

After I run a transaction in datomic to insert a value, how I can use the return value of the transaction to get the ids of any entities that were created?
Here is a sample of the return value I get after an insert:
#<promise$settable_future$reify__4841#7c92b2e9: {:db-before datomic.db.Db#62d0401f, :db-after datomic.db.Db#bba61dfc,
:tx-data [#Datum{:e 13194139534331 :a 50
:v #inst "2013-06-19T11:38:08.025-00:00"
:tx 13194139534331 :added true} #Datum{:e 17592186045436 .....
I can see the underlying datums...how can I extract their values?
Use d/resolve-tempid. If you were to transact a single entity, looking at :tx-data would work but if your transaction contained more than one entity, then you wouldn't know the order in which they appear in :tx-data.
What you should do is give temporary ids to your entities (before transacting them) using either (d/tempid) or its literal representation #db/id[:db.part/user _negativeId_] and then use d/resolve-tempid to go from your temporary id to the real id given by the database. The code would look something like:
(d/resolve-tempid (d/db conn) (:tempids tx) (d/tempid :db.part/user _negativeId_))
For a full code sample, see this gist.
Ah, figured it out.
I had to deref the Clojure promise, and then I was able to yank out the values I wanted:
(:e (second (:tx-data #(transact! conn query))))
Wrote a quick function based on a2ndrade's answer. The naming isn't ideal and I may be committing idiomatic faux pas; suggestions are very much welcome.
(ns my.datomic.util
(:require [datomic.api :as d]))
(defn transact-and-get-id
"Transact tx and return entity id."
[conn tx]
(let [tempid (:db/id tx)
post-tx #(d/transact conn [tx])
db (:db-after post-tx)
entid (d/resolve-tempid db (:tempids post-tx) tempid)]
entid))
Example usage:
(def my-conn
(d/connect (str "datomic:sql://datomic?jdbc:postgresql://"
"127.0.1:5432/datomic?user=datomic&password=somepw")
(defn thing-tx
"Create transaction for new thing."
[name]
{:db/id (d/tempid :db.part/user)
:thing/name name})
(transact-and-get-id my-conn (thing-tx "Bob")) ;; => 17592186045502
The Tupelo Datomic library has a function (td/eids tx-result) to easily extract the EIDs created in a transaction. For example:
; Create Honey Rider and add her to the :people partition
(let [tx-result #(td/transact *conn*
(td/new-entity :people ; <- partition is first arg (optional) to td/new-entity
{ :person/name "Honey Rider" :location "Caribbean" :weapon/type #{:weapon/knife} } ))
[honey-eid] (td/eids tx-result) ; retrieve Honey Rider's EID from the seq (destructuring)
]

"No reader function" error using Datomic in Light Table

When I eval this code in lighttable:
(ns app.core
(:require [datomic.api :refer [q] :as d]
:reload-all))
(defn add-person
[conn id]
(d/transact conn [{:db/id #db/id[:db.part/user -1000001]
:person/id id}]))
I get:
clojure.lang.ExceptionInfo: No reader function for tag id
core.clj:4327 clojure.core/ex-info
Anyone knows what is going on?
This tutorial is attributed to stuart halloway and Bobby Calderwood:
(use :reload 'datomic.samples.repl)
(easy!)
(def conn (scratch-conn))
;; in data, use data literals for tempids
(def tx-data [{:db/id #db/id[:db.part/user]
:db/doc "Example 1"}])
(transact conn tx-data)
;; in code, call tempid to create tempids
(let [id (tempid :db.part/user)
doc "Example 2"]
(transact conn [{:db/id id :db/doc doc}]))
;; same argument applies to functions:
;; use #db/fn literals in data
;; use Peer.function or d/function in code
;; broken, uses db/fn literal in code
(transact conn [{:db/id #db/id [:db.part/user]
:db/ident :hello
:db/fn #db/fn {:lang "clojure"
:params []
:code '(println :hello)}}])
;; corrected: used d/function to construct function
(transact conn [{:db/id (d/tempid :db.part/user)
:db/ident :hello
:db/fn (d/function {:lang "clojure"
:params []
:code '(println :hello)})}])
(d/invoke (db conn) :hello)
Source: https://github.com/Datomic/day-of-datomic/blob/master/samples/literals_vs_code.clj
It looks like there's an issue with trying to set :person/id. After the #db/id[:db.part/user -1000001] part, you've got a temporary id for adding data.
You should be able to start setting attributes for the entity using things like things like :person/name name.
If you're trying to create a "public id" type of thing, this blog post may be helpful.
It's a problem in nREPL. The way I solved this is to start the REPL at the command line with:
lein repl
This will start a process that you can connect to from LightTable or Emacs. It will print information like:
nREPL server started on port 51395 on host 127.0.0.1
^^^^^
Now in LightTable, Add a Connection -> Clojure Remote -> 127.0.0.1:XXXXX
The XXXXX should equal the port printed out by lein repl.
If you're in Emacs, cider has the same issue. Follow the same steps of starting lein repl, then use M-x cider-connect (it's default keybinding is C-c M-c).

How Should I Iterate a Sequence?

Below is my attempt to iterate a sequence of maps; the code fails due to the casting error: Exception in thread "main" java.lang.RuntimeException: java.lang.ClassCastException: clojure.lang.Cons cannot be cast to java.util.Map$Entry.
Can anyone explain/demonstrate how I should iterate the result-set? Thanks.
(with-connection db
(with-query-results rs ["select category from users group by category"]
(doall
(for [s [rs]]
(do (println (val s)))))))
You wrapped the rs into a vector. So s will be bound to the whole sequence, not the individual map entries. So when you call val it doesn't know what to do with a sequence. Hence the exception. This should work:
(with-connection db
(with-query-results rs ["select category from users group by category"]
(doall
(for [rec rs
s rec]
(do
(println (val s)))))))
However the ugly doall and do around the for should ring a bell, that something could be improved. And indeed for is used to construct another lazy sequence. This does not work well with side-effects as you intend in your example. You should use doseq in this case.
(with-connection db
(with-query-results rs ["select category from users group by category"]
(doseq [rec rs
s rec]
(println (val s)))))
The interface for the bindings of doseq is identical to that of for. However it executes things immediatelly, and thusly realises any side-effects immediatelly. If you put multiple expressions in the body of a for, you have to wrap it into a do. This is a reminder that the body should produce a value. Multiple expressions however indicate side-effects. doseq therefore wraps the body into a do for you. So you can easily have multiple expressions. For illustration:
(doall
(for [s seq-of-maps]
(do
(println (key s))
(println (val s)))))
(doseq [s seq-of-maps]
(println (key s))
(println (val s)))))
As a rule of thumb: you need side-effects? Look for things starting in do!
As a rule of thumb 2: if something looks ugly (see above comparison), this should ring a bell.
OK, so it sounds like you are trying to do a DB query from Clojure. You may have to supply more information about the "users" table for instance and what your query result set looks like.
At any rate, something like this may work
(def a (with-query-results rs ["select category from users group by category"]
(doall rs)))
(map #(:category %) a)