I'm playing around with clojure/datomic and I setup a really simple website using datomic on-prem about two weeks ago.
When I wanted to check how much data I had been colecting I started a repl and made a connection to datomic. The result showed something like this:
#object[datomic.peer.Connection 0xfbe8fd3
"{:unsent-updates-queue 0, :pending-txes 0,
:next-t 12021, :basis-t 12020, :index-rev 0,
:db-id \"my-website-b2e771e0-c8bb-4397-a6c0-0981516c0a0d\"}"]
And there was no data other than the tests I made.
Then I checked some of the logs I had and the transactions look like this:
post insertion resp #object[datomic.promise$settable_future$reify__7373 0x56e2a1c5 {:status :ready,
:val {:db-before datomic.db.Db#5ddcffae,
:db-after datomic.db.Db#63d04810,
:tx-data #object[java.util.Arrays$ArrayList 0x3be44815 [datomic.db.Datum#fbfd09b6]],
:tempids {-9223301668109598143 17592186045431}}}]
post insertion resp #object[datomic.promise$settable_future$reify__7373 0x70fe2f2c {:status :ready,
:val {:db-before datomic.db.Db#63d04810,
:db-after datomic.db.Db#72e1eed2,
:tx-data #object[java.util.Arrays$ArrayList 0x58541f8 [datomic.db.Datum#7502796a]],
:tempids {-9223301668109598142 17592186045432}}}]
So I did some research and now I know that I must dereference when I make the transaction to the db. But I would like to know if there's a way to get the data from all those promises that weren't dereferenced or if they are gone.
Thanks.
When you send off a transaction, whether or not it succeeds is not related to whether or not you dereference it in the client code. If the transaction fails, you might not notice that if you don't deref it, since your code will just send it off and not care too much about how it all went.
But, if you did send off a bunch of transactions that the transactor for some reason wasn't able to apply, and you didn't log them anyhwhere, they're gone forever. I don't believe the transactor keeps a log of failed transactions.
Related
(defn my-func [opts]
(assoc opts :something :else))
What i want to be able to do, is serialize a reference to the function (maybe via #'my-func ?) to a string in such a way that i can upon deserializing it, invoke it with args.
How does this work?
Edit-- Why This is Not a Duplicate
The other question asked how to serialize a function body-- the entire function code. I am not asking how to do that. I am asking how to serialize a reference.
Imagine a cluster of servers all running the same jar, attached to a MQ. The MQ pubs in fn-reference and fn-args for functions in the jar, and the server in the cluster runs it and acks it. That's what i'm trying to do-- not pass function bodies around.
In some ways, this is like building a "serverless" engine in clojure.
Weirdly, a commit for serializing var identity was just added to Clojure yesterday: https://github.com/clojure/clojure/commit/a26dfc1390c53ca10dba750b8d5e6b93e846c067
So as of the latest master snapshot version, you can serialize a Var (like #'clojure.core/conj) and deserialize it on another JVM with access to the same loaded code, and invoke it.
(import [java.io File FileOutputStream FileInputStream ObjectOutputStream ObjectInputStream])
(defn write-obj [o f]
(let [oos (ObjectOutputStream. (FileOutputStream. (File. f)))]
(.writeObject oos o)
(.close oos)))
(defn read-obj [f]
(let [ois (ObjectInputStream. (FileInputStream. (File. f)))
o (.readObject ois)]
(.close ois)
o))
;; in one JVM
(write-obj #'clojure.core/conj "var.ser")
;; in another JVM
(read-obj "var.ser")
As suggested on the comments, if you can just serialize a keyword label for the function and store/retrieve that, you are finished.
If you need to transmit the function from one place to another, you essentially need to send the function source code as a string and then have it compiled via eval on the other end. This is what Datomic does when a Database Function is stored in the DB and automatically run by Datomic for any new additions/changes to the DB (these can perform automatic data validation, for example). See:
http://docs.datomic.com/database-functions.html
http://docs.datomic.com/clojure/index.html#datomic.api/function
As similar technique is used in the book Clojure in Action (1st Edition) for the distributed compute engine example using RabbitMQ.
I'm checking if YeSQL may help in my Clojure project, but I cannot find any example of YeSQL using a connection pool.
does this mean that YeSQL creates a new connection to every statement?
I also found an example on how to use transactions using clojure.java.jdbc/with-db-transaction, but I feel it is outdated (I haven't tried yet).
does this mean that YeSQL depends on clojure.java.jdbc to commit/rollback control? in this case, shouldn't I just use clojure.java.jdbc alone, since YeSQL doesn't offer too much more (other than naming my queries and externalizing them)?
thanks in advance
YeSQL doesn't handle connections or connection pooling. You need to handle it externally and provide a connection instance to the query function.
As you can see from the YeSQL example from README:
; Define a database connection spec. (This is standard clojure.java.jdbc.)
(def db-spec {:classname "org.postgresql.Driver"
:subprotocol "postgresql"
:subname "//localhost:5432/demo"
:user "me"})
; Use it standalone. Note that the first argument is the db-spec.
(users-by-country db-spec "GB")
;=> ({:count 58})
; Use it in a clojure.java.jdbc transaction.
(require '[clojure.java.jdbc :as jdbc])
(jdbc/with-db-transaction [connection db-spec]
{:limeys (users-by-country connection "GB")
:yanks (users-by-country connection "US")})
If you ask how to add connection pool handling you might check an example from Clojure Cookbook.
As for the transaction handling, YeSQL documentation is none, but source is quite easy to understand:
(defn- emit-query-fn
"Emit function to run a query.
- If the query name ends in `!` it will call `clojure.java.jdbc/execute!`,
- If the query name ends in `<!` it will call `clojure.java.jdbc/insert!`,
- otherwise `clojure.java.jdbc/query` will be used."
[{:keys [name docstring statement]}]
(let [split-query (split-at-parameters statement)
{:keys [query-args display-args function-args]} (split-query->args split-query)
jdbc-fn (cond
(= [\< \!] (take-last 2 name)) `insert-handler
(= \! (last name)) `execute-handler
:else `jdbc/query)]
`(def ~(fn-symbol (symbol name) docstring statement display-args)
(fn [db# ~#function-args]
(~jdbc-fn db#
(reassemble-query '~split-query
~query-args))))))
So it will just generate a function that will either call clojure.java.jdbc/execute! or clojure.java.jdbc/insert! with the generated query. You might need to refer to those functions' documentation for further details.
When doing transactions using YesQL, I wrap the YesQL calls in a clojure.java.jdbc/with-db-transation invocation, and pass the generated connection details in to the YesQL function call, which they'll use instead of the standard non-transaction connection if supplied. For example:
;; supply a name for the transaction connection, t-con, based on the existing db connection
(jdbc/with-db-transaction [t-con db-spec]
;; this is the yesql call with a second map specifying the transaction connection
(create-client-order<! {...} {:connection t-con}))
All wrapped YesQL calls using the {:connection t-con} map will be part of the same transaction.
I have been reading through Programming in Clojure and found thing text
(defn get-document [id]
; ... do some work to retrieve the identified document's metadata ... {:url "http://www.mozilla.org/about/manifesto.en.html"
:title "The Mozilla Manifesto"
:mime "text/html"
:content (delay (slurp "http://www.mozilla.org/about/manifesto.en.html"))})
if callers are likely to always require that data, the change of replacing future over delay can prove to be a significant improvement in throughput.
I didn't got this part completely, can someone please explain a bit.
Simple answer future is background execution of body, delay is on-demand execution of body. Example: if you have list of 100 delay-ed code, and trying to loop through it - code will block while evaluating each list item (e.g. doing HTTP request) and first iteration will be slow. Same with future-d code - it'll evaluate all content in background thread(s) and results will be available instantly in your loop.
Rule of thumb - if there is good chance that some or most of content will not be needed at all - use delay, otherwise use future.
https://clojuredocs.org/clojure.core/delay
https://clojuredocs.org/clojure.core/future
future creates a Future and schedules it for execution immediately, therefore calling
(get-document "id")
will cause a future to be created which fetches the document immediately and then caches the result in the future.
In contrast, delay creates a lazy operation which will not be executed until dereferenced. In this case, calling
(get-document "id")
will not cause the document to be fetched. This will only happen when dereferencing e.g.
(let [{:keys [content]} (get-document "id")]
(println #content))
I was looking at https://github.com/juxt/dirwatch library. The example from the front page is:
(require '[juxt.dirwatch :refer (watch-dir)])
(watch-dir println (clojure.java.io/file "/tmp"))
That works fine. Let's say the above is executed in REPL:
user=> (watch-dir println (clojure.java.io/file "/tmp"))
#<Agent#16824c93: #<LinuxWatchService sun.nio.fs.LinuxWatchService#17ece9ac>>
Now, I have an agent that will print events when I modify files in /tmp:
{:file #<File /tmp/1>, :count 1, :action :modify}
so all is fine.
I know I can reference the agent by using previous expression references (*1, *2 and *3). However, I don't know how to, without restarting the REPL itself:
Unbind an implicit var created like this - i.e. how to remove the binding completely, so that agent gets GCed and stops working
Access it in case I lost it in cases where I did not bind it, such as the above. If I'm not mistaken, in REPL only the last three results are available (*3 is, but *4 and further are not), at least per http://clojure.org/repl_and_main
Any suggestions?
Did you take a look at the code? The documentation to watch-dir has this: "The watcher returned by this function is a resource which
should be closed with close-watcher."
Looking at the code, it watch-dir uses send-off, which "Dispatch a potentially blocking action to an agent. Returns the agent immediately.". In other words, to address your first question, there is no implicit var created. If you want to get rid of the agent, you should bind the returned agent to some var and call close-watcher on it afterwards.
To address the second question, take a look at the canonical documentation for agents. Specifically, you can call shutdown-agents, which will shut-down the thread pool (potentially killing other agents as well).
I'm confused by how calls with carmine should be done. I found the wcar macro described in carmine's docs:
(defmacro wcar [& body] `(car/with-conn pool spec-server1 ~#body))
Do I really have to call wcar every time I want to talk to redis in addition to the redis command? Or can I just call it once at the beginning? If so how?
This is what some code with tavisrudd's redis library looked like (from my toy url shortener project's testsuite):
(deftest test_shorten_doesnt_exist_create_new_next
(redis/with-server test-server
(redis/set "url_counter" 51)
(shorten test-url)
(is (= "1g" (redis/get (str "urls|" test-url))))
(is (= test-url (redis/get "shorts|1g")))))
And now I can only get it working with carmine by writing it like this:
(deftest test_shorten_doesnt_exist_create_new_next
(wcar (car/set "url_counter" 51))
(shorten test-url)
(is (= "1g" (wcar (car/get (str "urls|" test-url)))))
(is (= test-url (wcar (car/get "shorts|1g")))))
So what's the right way of using it and what underlying concept am I not getting?
Dan's explanation is correct.
Carmine uses response pipelining by default, whereas redis-clojure requires you to ask for pipelining when you want it (using the pipeline macro).
The main reason you'd want pipelining is for performance. Redis is so fast that the bottleneck in using it is often the time it takes for the request+response to travel over the network.
Clojure destructuring provides a convenient way of dealing with the pipelined response, but it does require writing your code differently to redis-clojure. The way I'd write your example is something like this (I'm assuming your shorten fn has side effects and needs to be called before the GETs):
(deftest test_shorten_doesnt_exist_create_new_next
(wcar (car/set "url_counter" 51))
(shorten test-url)
(let [[response1 response2] (wcar (car/get (str "urls|" test-url))
(car/get "shorts|1g"))]
(is (= "1g" response1))
(is (= test-url response2))))
So we're sending the first (SET) request to Redis and waiting for the reply (I'm not certain if that's actually necessary here). We then send the next two (GET) requests at once, allow Redis to queue the responses, then receive them all back at once as a vector that we'll destructure.
At first this may seem like unnecessary extra effort because it requires you to be explicit about when to receive queued responses, but it brings a lot of benefits including performance, clarity, and composable commands.
I'd check out Touchstone on GitHub if you're looking for an example of what I'd consider idiomatic Carmine use (just search for the wcar calls). (Sorry, SO is preventing me from including another link).
Otherwise just pop me an email (or file a GitHub issue) if you have any other questions.
Don't worry, you're using it the correct way already.
The Redis request functions (such as the get and set that you're using above) are all routed through another function send-request! that relies on a dynamically bound *context* to provide the connection. Attempting to call any of these Redis commands without that context will fail with a "no context" error. The with-conn macro (used in wcar) sets that context and provides the connection.
The wcar macro is then just a thin wrapper around with-conn making the assumption that you will be using the same connection details for all Redis requests.
So far this is all very similar to how Tavis Rudd's redis-clojure works.
So, the question now is why does Carmine need multiple wcar's when Redis-Clojure only required a single with-server?
And the answer is, it doesn't. Apart from sometimes, when it does. Carmine's with-conn uses Redis's "Pipelining" to send multiple requests with the same connection and then package the responses together in a vector. The example from the README shows this in action.
(wcar (car/ping)
(car/set "foo" "bar")
(car/get "foo"))
=> ["PONG" "OK" "bar"]
Here you will see that ping, set and get are only concerned with sending the request, leaving the receiving of response up to wcar. This precludes asserts (or any result access) from inside of wcar and leads to the separation of requests and multiple wcar calls that you have.