Core.async <! channel deadlock - clojure

Why does Alpha stop early, when I expect it to behave like Beta? The only difference between Alpha and Beta is >! and put!, as commented below.
Alpha:
user=> (def q (chan))
#'user/q
user=> (def counter (atom 0))
#'user/counter
user=> (defn mg [event-queue]
#_=> (go-loop [event (<! event-queue)]
#_=> (swap! counter inc)
#_=> (when (< #counter 4)
#_=> (println "counter: " #counter)
#_=> (>! event-queue {:a #counter}) ;; Here's the only difference
#_=> (println "event: " event)
#_=> (recur (<! event-queue)))))
#'user/mg
user=> (mg q)
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x3a1ffd56 "clojure.core.async.impl.channels.ManyToManyChannel#3a1ffd56"]
user=> (put! q "hi")
counter: true
1
user=>
Beta:
user=> (def q (chan))
#'user/q
user=> (def counter (atom 0))
#'user/counter
user=> (defn mg [event-queue]
#_=> (go-loop [event (<! event-queue)]
#_=> (swap! counter inc)
#_=> (when (< #counter 4)
#_=> (println "counter: " #counter)
#_=> (put! event-queue {:a #counter}) ;; Here's the only difference
#_=> (println "event: " event)
#_=> (recur (<! event-queue)))))
#'user/mg
user=> (mg q)
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x72c9b65a "clojure.core.async.impl.channels.ManyToManyChannel#72c9b65a"]
user=> (put! q "hi")
true
counter: 1
event: hi
counter: 2
event: {:a 1}
counter: 3
event: {:a 2}
user=>
It's also interesting that, after executing Alpha, the channel #'user/q was properly enqueued:
user=> (take! q println)
event: hi
{:a 1}
nil
user=>
The same results occur in both Clojure and Clojurescript. Is this some sort of deadlock, or is the suppose to happen?

This is expected.
The channel q is created without a buffer, so when a value is placed with >!, it will block (park) the go-loop until another thread is ready to consume the value with <!.
One way to work around this is to give q a 1-slot buffer with (def q (chan 1)). The buffer allows 1 value to be placed in the channel without blocking the sender.
Beta behaves differently because put! is asynchronous wrt. the caller -- it uses a separate thread to place the new value in the channel. This avoids blocking the current go-loop, allowing the channel to be read and progress to continue.

Related

Why doesn't 'for' work inside 'go'?

Why calling this function doesn't print anything?
(defn test-go-loop []
(go (for [a (cycle [:a :b :c])]
(do (println a) (<! (timeout 1000))))))
for is lazily evaluated, and nothing in your code is asking for the result of that for. Try doseq:
(defn test-go-loop []
(go (doseq [a (cycle [:a :b :c])]
(println a)
(<! (timeout 1000)))))

Sleeping barber in Clojure

I'm implementing Sleeping barber using core.async. My current code is:
(def workingtime 10000)
(defn barber [in waiting-room]
(go-loop [served-customers 0]
(let [[v] (alts! [waiting-room in])]
(if (= v :close)
served-customers
(do (Thread/sleep 20)
(recur (inc served-customers)))))))
(defn customers [in waiting-room]
(go-loop [customers-overall 0]
(let [customer-arrival-interval (timeout (+ 10 (rand-int 20)))
[v] (alts! [in customer-arrival-interval])]
(if (= v :close)
customers-overall
(do (>! waiting-room :customer)
(recur (inc customers-overall)))))))
(defn -main [& args]
(let [in (chan)
waiting-room (chan (dropping-buffer 3))
barber-ch (barber in waiting-room)
customers-ch (customers in waiting-room)]
(println "opening the shop for 10 seconds...")
(Thread/sleep workingtime)
(>!! in :close)
(>!! in :close)
(println "closing the shop...")
(println (str "Served " (<!! barber-ch) " customers"))
(println (str "Overall " (<!! customers-ch) " customers came"))))
Is it a correct solution? Can it be improved to make it more Clojure-like?
I wanted to use alt! instead of alts! which makes code easier to read:
(defn barber [in]
(go-loop [served-customers 0]
(alt!
waiting-room (do (Thread/sleep 20)
(recur (inc served-customers)))
in served-customers)))
Runtime throws an exception: Can only recur from tail position. Can I still use alt!?
You could solve the alt!/recur problem by rewriting to:
(defn barber [in]
(go-loop [served-customers 0]
(if (= :waiting-room
(a/alt!
waiting-room ([result] :waiting-room) ;; you could also use result if needed
in ([result] :in))) ;; same here
(do (Thread/sleep 20)
(recur (inc served-customers)))
served-customers)))

How to model Rx's `withLatestFrom` with core.async channels?

For example given a channel with operations and another channel with data, how to write a go block that will apply the operation on whatever was the last value on the data channel?
(go-loop []
(let [op (<! op-ch)
data (<! data-ch)]
(put! result-ch (op data))))
Obviously that doesn't work because it would require both channels to have the same frequency.
(see http://rxmarbles.com/#withLatestFrom)
Using alts! you could accomplish what you want.
The with-latest-from shown below implements the same behavior found in the withLatestFrom from RxJS (I think :P).
(require '[clojure.core.async :as async])
(def op-ch (async/chan))
(def data-ch (async/chan))
(defn with-latest-from [chs f]
(let [result-ch (async/chan)
latest (vec (repeat (count chs) nil))
index (into {} (map vector chs (range)))]
(async/go-loop [latest latest]
(let [[value ch] (async/alts! chs)
latest (assoc latest (index ch) value)]
(when-not (some nil? latest)
(async/put! result-ch (apply f latest)))
(when value (recur latest))))
result-ch))
(def result-ch (with-latest-from [op-ch data-ch] str))
(async/go-loop []
(prn (async/<! result-ch))
(recur))
(async/put! op-ch :+)
;= true
(async/put! data-ch 1)
;= true
; ":+1"
(async/put! data-ch 2)
;= true
; ":+2"
(async/put! op-ch :-)
;= true
; ":-2"
There's an :priority true option for the alts!.
An expression which always returns the latest seen value in some channel would look something like this:
(def in-chan (chan))
(def mem (chan))
(go (let [[ch value] (alts! [in-chan mem] :priority true)]
(take! mem) ;; clear mem (take! is non-blocking)
(>! mem value) ;; put the new (or old) value in the mem
value ;; return a chan with the value in
It's untested, it's probably not efficient (a volatile variable is probably better). The go-block returns a channel with only the value, but the idea could be expanded to some "memoized" channel.

Can one monitor STM's contention level?

Is there any way to poll whether Clojure's STM transactions are being retried, and at what rate?
You can observe the history count of a ref which will indicate that there is contention on it:
user=> (def my-ref (ref 0 :min-history 1))
#'user/my-ref
user=> (ref-history-count my-ref)
0
user=> (dosync (alter my-ref inc))
1
user=> (ref-history-count my-ref)
1
The history count does not directly represent contention. Instead it represents the number of past values that have been maintained in order to service concurrent reads.
The size of the history is limited by min and max values. By default those are 0 and 10, respectively, but you can change them when creating the ref (see above). Since min-history is 0 by default, you won't usually see ref-history-count return non-zero values, unless there is contention on the ref.
See more discussion on history count here: https://groups.google.com/forum/?fromgroups#!topic/clojure/n_MKCoa870o
I don't think there is any way, provided by clojure.core, to observe the rate of STM transactions at the moment. You can of course do something similar to what #Chouser did in his history stress test:
(dosync
(swap! try-count inc)
...)
i.e. increment a counter inside the transaction. The increment will happen every time the transaction is tried. If try-count is larger than 1, the transaction was retried.
By introducing named dosync blocks and commit counts (the times a named dosync has succeeded), one can quite easily keep track of the times threads have retried a given transaction.
(def ^{:doc "ThreadLocal<Map<TxName, Map<CommitNumber, TriesCount>>>"}
local-tries (let [l (ThreadLocal.)]
(.set l {})
l))
(def ^{:doc "Map<TxName, Int>"}
commit-number (ref {}))
(def history ^{:doc "Map<ThreadId, Map<TxName, Map<CommitNumber, TriesCount>>>"}
(atom {}))
(defn report [_ thread-id tries]
(swap! history assoc thread-id tries))
(def reporter (agent nil))
(defmacro dosync [tx-name & body]
`(clojure.core/dosync
(let [cno# (#commit-number ~tx-name 0)
tries# (update-in (.get local-tries) [~tx-name] update-in [cno#] (fnil inc 0))]
(.set local-tries tries#)
(send reporter report (.getId (Thread/currentThread)) tries#))
~#body
(alter commit-number update-in [~tx-name] (fnil inc 0))))
Given the following example...
(def foo (ref {}))
(def bar (ref {}))
(defn x []
(dosync :x ;; `:x`: the tx-name.
(let [r (rand-int 2)]
(alter foo assoc r (rand))
(Thread/sleep (rand-int 400))
(alter bar assoc (rand-int 2) (#foo r)))))
(dotimes [i 4]
(future
(dotimes [i 10]
(x))))
...#history evaluates to:
;; {thread-id {tx-name {commit-number tries-count}}}
{40 {:x {3 1, 2 4, 1 3, 0 1}}, 39 {:x {2 1, 1 3, 0 1}}, ...}
This additional implementation is substantially simpler.
;; {thread-id retries-of-latest-tx}
(def tries (atom {}))
;; The max amount of tries any thread has performed
(def max-tries (atom 0))
(def ninc (fnil inc 0))
(def reporter (agent nil))
(defn report [_ tid]
(swap! max-tries #(max % (get #tries tid 0)))
(swap! tries update-in [tid] (constantly 0)))
(defmacro dosync [& body]
`(clojure.core/dosync
(swap! tries update-in [(.getId (Thread/currentThread))] ninc)
(commute commit-id inc)
(send reporter report (.getId (Thread/currentThread)))
~#body))

How to launch two threads and wait for them

I can launch two threads and they work, but synchronously. What am I missing to get these threads independently launched?
main, thread, and output
(defn -main
[& args]
(do
(let [grid-dim-in [0 5]
mr1-pos [\N 2 4]
mr2-pos [\N 1 5]
mr1-movs "LMLMMRMM"
mr2-movs "RMRMMMLM"]
(reset! grid-dim grid-dim-in)
(reset! mr1-id {:mr1 mr1-pos})
(reset! mr2-id {:mr2 mr2-pos})
(.start (Thread. (rover-thread mr1-id mr1-movs update-work-block)))
(.start (Thread. (rover-thread mr2-id mr2-movs update-work-block))))))
(defn rover-thread [id movs update-ref]
(let [id-key (keys #id)
id-vals (vals #id)]
(doseq [mov movs]
(println "Rover " id-key " is moving ")
(let [new-mov (determine-rover-move (first id-vals) mov)]
(move-rover id new-mov update-ref)
(print "Rover ")
(print (first id-key))
(print " is at ")
(println new-mov)
(Thread/sleep (rand 1000)))))
Rover :mr1 is at [E 2 4]
Rover (:mr1) is moving
Rover :mr1 is at [N 2 5]
Rover (:mr1) is moving
Rover :mr1 is at [N 2 5]
Finished on Thread[main,5,main]
Rover (:mr2) is moving
Rover :mr2 is at [E 1 5]
Rover (:mr2) is moving
Rover :mr2 is at [N 1 6]
Take a close look at these two lines:
(.start (Thread. (rover-thread mr1-id mr1-movs update-work-block)))
(.start (Thread. (rover-thread mr2-id mr2-movs update-work-block))))))
This code evaluates the (rover-thread mr1-id mr1-movs update-work-block) first, and passes the result of that to the constructor of Thread, which is not what you want.
Here's a simple function to illustrate the principle. This doesn't work, because the (f ...) is evaluated before its result it passed to the Thread constructor:
(defn run-thread-thing-wrong []
(let [f (fn [n s]
(doseq [i (range n)]
(prn s i)
(Thread/sleep (rand 1000))))]
(.start (Thread. (f 10 "A")))
(.start (Thread. (f 10 "B"))))
nil)
Here's a version that does work. A function is passed to the Thread constructor instead:
(defn run-thread-thing []
(let [f (fn [n s]
(doseq [i (range n)]
(prn s i)
(Thread/sleep (rand 1000))))]
(.start (Thread. (fn [] (f 10 "A"))))
(.start (Thread. (fn [] (f 10 "B")))))
nil)
Note: instead of (fn [] ....) you can use the short form #(....) for anonymous functions.
Here's another version that does the same, but with a future instead of manually creating threads:
(defn run-thread-thing []
(let [f (fn [n s]
(doseq [i (range n)]
(prn s i)
(Thread/sleep (rand 1000))))]
(future (f 10 "A"))
(future (f 10 "B")))
nil)
Note that in this case, you pass a form to future instead of a function.
This seems like a really good place to use Clojure's agent feature. I am not qualified to fully explain how to use them, but a really good example of their usage can be found here. Starting threads using agents is dead-easy, and I think it is more idiomatic.
The code would look something like,
(def rover1 (agent [mr1-posn mr1-movs mr1-id]))
(def rover2 (agent [mr2-posn mr2-movs mr2-id]))
(defn rover-behave [[posn movs id]]
(send-off *agent* #'rover-behave)
(. Thread (sleep 1000))
(let [new-mov (determine-rover-move posn movs id)
new-posn (posn-after-move posn new-mov)]
;return value updates state of agent
[new-posn movs id]
)
)
(send-off rover1 rover-behave)
(send-off rover2 rover-behave)