I'm at Ch. 6 of Paul Butcher's 7 Concurrency Models in 7 Weeks, which is focussing on core.async.
We have the following function
(defn map-chan [f from]
(let [to (chan)]
(go-loop []
(when-let [x (<! from)]
(>! to (f x))
(println "parking channel write.")
(recur))
(close! to))
(println "map-chan done.")
to))
I added the printlns myself, to explore exact order of computation, which I want to ask about here.
We can run it like this
(def ch (to-chan (range 10))) ; [1]
(def mapped (map-chan (partial * 2) ch)) ; [2]
(<!! (async/into [] mapped)) ; [3]
;; [1] Create & rtn a channel from els of seq, closing it when seq fin.
;; [2] map-chan returns immediately, with blocked go blocks inside of it.
;; [3] calling async/into finally triggers the parked channel writes, as seen below.
in the repl:
channels.core=> (def ch (to-chan (range 10)))
#'channels.core/ch
channels.core=> (def mapped (map-chan (partial * 2) ch))
map-chan done.
#'channels.core/mapped
channels.core=> (<!! (async/into [] mapped))
parking channel write.
parking channel write.
parking channel write.
parking channel write.
parking channel write.
parking channel write.
parking channel write.
parking channel write.
parking channel write.
parking channel write.
[0 2 4 6 8 10 12 14 16 18]
channels.core=>
Question
We have a (sync) (i.e. unbuffered) channel here that has both writer and reader to it ready to go. Why is my "parking channel write" above not triggered until async/into is called?
(It's not the channel read with <!! that triggers it, it's async/into itself - easy to check). I'm not complaining about this, just seeking to understand why the trace is the way it is.
Are channels actually somehow lazy as well? He hasn't mentioned this in the book yet.
Note the dependency on this code is org.clojure/core.async "0.1.267.0-0d7780-alpha", if that makes any difference.
Also, in the book he has used a buffered channel of length 10.
Yet, I also tried it with an unbuffered (sync) channel and the result seems the same.
Your output channel to has a size of zero, and so the write cannot take place until a corresponding take is requested. Look at a modified version of your code:
(ns tst.demo.core
(:use tupelo.core tupelo.test )
(:require
[clojure.core.async :as async]
))
(defn map-chan [f from]
(let [to (async/chan)]
(async/go
(loop []
(when-let [x (async/<! from)]
(println "put - pre")
(async/>! to (f x))
(println "put - post")
(recur)))
(async/close! to))
(println "map-chan returns output buffer")
to))
(dotest
(println :1)
(spyx
(def ch (async/to-chan (range 10)))) ; [1]
(Thread/sleep 2000) (println :2)
(spyx
(def mapped (map-chan (partial * 2) ch))) ; [2]
(Thread/sleep 2000) (println :3)
(spyx
(async/<!! (async/into [] mapped))) ; [3]
)
with results:
-------------------------------
Clojure 1.10.1 Java 13
-------------------------------
lein test tst.demo.core
:1
(def ch (async/to-chan (range 10))) => #'tst.demo.core/ch
:2
map-chan returns output buffer
(def mapped (map-chan (partial * 2) ch)) => #'tst.demo.core/mapped
put - pre
:3
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
(async/<!! (async/into [] mapped)) => [0 2 4 6 8 10 12 14 16 18]
So, the go loop does start running immediately, but the first put operation blocks until the async/into at step [3] occurs.
If we use a buffered output channel of length 20, we see the go loop running before step [3] occurs:
...
(let [to (async/chan 20)]
...
with result:
:1
(def ch (async/to-chan (range 10))) => #'tst.demo.core/ch
:2
map-chan returns output buffer
(def mapped (map-chan (partial * 2) ch)) => #'tst.demo.core/mapped
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
put - pre
put - post
:3
(async/<!! (async/into [] mapped)) => [0 2 4 6 8 10 12 14 16 18]
Related
Take the following program as an example:
(defn echo-ints []
(doseq [i (->> (BufferedReader. *in*)
(line-seq)
(map read-string)
(take-while integer?))]
(println i)))
The idea is to prompt the user for input and then echo it back if it's an integer. However, in this particular program almost every second input won't be echoed immediately. Instead the program will wait for additional input before processing two inputs at once.
Presumably this a consequence of some performance tweaks happening behind the scenes. However in this instance I'd really like to have an immediate feedback loop. Is there an easy way to accomplish this, or does the logic of the program have to be significantly altered?
(The main motivation here is to pass the infinite sequence of user inputs to another function f that transforms lazy sequences to other lazy sequences. If I wrote some kind of while-loop, I wouldn't be able to use f.)
It is generally not good to mix lazyness with side-effect (printing in this case), since most sequence functions have built-in optimizations that cause unintended effects while still being functionally correct.
Here's a good write up: https://stuartsierra.com/2015/08/25/clojure-donts-lazy-effects
What you are trying to do seems like a good fit for core.async channels. I would think as the problem as 'a stream of user input' instead of 'infinite sequence of user inputs', and 'f transforms lazy sequences to lazy sequences' becomes 'f transform a stream into another stream'. This will allow you to write f as transducers which you can arbitrarily compose.
I would do it like the following. Note we use spyx and spyxx from the Tupelo library to display some results.
First, write a simple version with canned test data:
(ns tst.demo.core
(:use tupelo.test)
(:require
[tupelo.core :as t] )
(:import [java.io BufferedReader StringReader]))
(t/refer-tupelo)
(def user-input
"hello
there
and
a
1
and-a
2
and
a
3.14159
and-a
4
bye" )
(defn echo-ints
[str]
(let [lines (line-seq (BufferedReader. (StringReader. str)))
data (map read-string lines)
nums (filter integer? data) ]
(doseq [it data]
(spyxx it))
(spyx nums)))
(newline)
(echo-ints user-input)
This gives us the results:
it => <#clojure.lang.Symbol hello>
it => <#clojure.lang.Symbol there>
it => <#clojure.lang.Symbol and>
it => <#clojure.lang.Symbol a>
it => <#java.lang.Long 1>
it => <#clojure.lang.Symbol and-a>
it => <#java.lang.Long 2>
it => <#clojure.lang.Symbol and>
it => <#clojure.lang.Symbol a>
it => <#java.lang.Double 3.14159>
it => <#clojure.lang.Symbol and-a>
it => <#java.lang.Long 4>
it => <#clojure.lang.Symbol bye>
nums => (1 2 4)
So, we see that it works and gives us the numbers we want.
Next, write a looping version. We make it terminate gracefully when our test data runs out.
(defn echo-ints-loop
[str]
(loop [lines (line-seq (BufferedReader. (StringReader. str)))]
(let [line (first lines)
remaining (rest lines)
data (read-string line)]
(when (integer? data)
(println "found:" data))
(when (not-empty? remaining)
(recur remaining)))))
(newline)
(echo-ints-loop user-input)
found: 1
found: 2
found: 4
Next, we write an infinite loop to read the keyboard. You need to terminate this one with CRTL-C at the keyboard:
(ns demo.core
(:require [tupelo.core :as t])
(:import [java.io BufferedReader StringReader]))
(t/refer-tupelo)
(defn echo-ints-inf
[]
(loop [lines (line-seq (BufferedReader. *in*))]
(let [line (first lines)
remaining (rest lines)
data (read-string line)]
(when (integer? data)
(println "found:" data))
(when (not-empty? remaining)
(recur remaining)))))
(defn -main []
(println "main - enter")
(newline)
(echo-ints-inf))
And we run it manually:
~/clj > lein run
main - enter
hello
there
1
found: 1
and
a
2
found: 2
and-a
3
found: 3
further more
4
found: 4
^C
~/clj >
~/clj >
I have a channel where I am putting values into inside a doseq loop.
This code reads from a list of isbns and for each isbn, does an amazon search to return contents of a book, and then calls another function to get the title and rank
(def book_channel (chan 10))
make sure you use clojure.core.async/into rather than clojure.core/into. Here is an example of a round trip from collection to channel and back to collection:
user> (require '[clojure.core.async :as async :refer [<! <!! >!! >! chan go]])
nil
user> (def book-chan (async/to-chan [:book1 :book2 :book3]))
#'user/book-chan
user> (<!! (clojure.core.async/into [] book-chan))
[:book1 :book2 :book3]
clojure.core.async/into returns a channel that will have exactly one item written to it. That one item will be written once it's input channel closes. This keeps the whole thing asynchronous and it does require that the code putting things into the book-channel close the chan to signal that all the books are there.
You need to do some type of coordination to determine when all of your work is finished. You can pull that coordination out into the main thread fairly easily:
(def book_channel (chan 10))
(defn concurrency_test
[list_of_isbns]
(doseq [isbn list_of_isbns]
(go (>! book_channel
(get_title_and_rank_for_one_isbn
(amazon_search isbn)))))
(prn (loop [results []]
(if (= (count results) (count list_of_isbns))
results
(recur (conj results (<!! book_channel)))))))
Here, I used a loop that keeps waiting for results and adding them to the vector until we have as many results as we do isbns. You'll want to make sure that get_title_and_rank_for_one_isbn always generates a result that can be put on a channel, otherwise the loop will wait forever.
You should close! the book_channel after you finish pushing stuff into it. Per async/into documentation - "ch must close before into produces a result."
(let [book> (chan)]
(go
(doseq [e (range 8)]
(>! book> e))
(close! book>))
(<!! (async/into [] book>)))
Alternatively, you can use async/onto-chan which will close the channel for you:
(let [book> (chan)]
(async/onto-chan book> (range 8))
(<!! (async/into [] book>)))
My situation is that the same channel is shared between different functions and I need to close the channel when all of this functions signal they are done with it. This is what I've come up with but it's not ideal because I need to handle tow channels and need to send a special keyword instead of using the close! function. Is there another, better way to do this? Is there some feature of core.async to do this?
(defn shared-chan [n]
(let [in (chan)
out (chan)]
(go-loop [n n]
(if (= n 0)
(do
(async/close! in)
(async/close! out))
(let [in-msg (<! in)]
(if (not= :close in-msg)
(do
(>! out in-msg)
(recur n))
(recur (dec n))))))
[in out]))
merge may help you. According to the doc, merge
Takes a collection of source channels and returns a channel which
contains all values taken from them. ... The channel
will close after all the source channels have closed.
So, basically you need to create a channel for each of your functions and then merge them into one.
you can generate this shared channel using take transducer. For example if you want an auto closable channel for 3 items:
user> (require '[clojure.core.async :as a])
nil
user> (def shared-ch (a/chan 3 (take 3)))
#'user/shared-ch
user> (a/go-loop []
(when-let [val (a/<! shared-ch)]
(println :reading val)
(recur)))
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x6bb1dee5 "clojure.core.async.impl.channels.ManyToManyChannel#6bb1dee5"]
user> (a/>!! shared-ch 1)
true:reading 1
user> (a/>!! shared-ch 2)
true:reading 2
user> (a/>!! shared-ch 3)
true:reading 3
user> (a/>!! shared-ch 4)
false
user> (a/>!! shared-ch 5)
false
you can see, that the channel has been closed as soon as the transducer has exhausted (as false is returned after the writing attempt)
What's best way in clojure to implement something like an actor or agent (asynchronously updated, uncoordinated reference) that does the following?
gets sent messages/data
executes some function on that data to obtain new state; something like (fn [state new-msgs] ...)
continues to receive messages/data during that update
once done with that update, runs the same update function against all messages that have been sent in the interim
An agent doesn't seem quite right here. One must simultaneously send function and data to agents, which doesn't leave room for a function which operates on all data that has come in during the last update. The goal implicitly requires a decoupling of function and data.
The actor model seems generally better suited in that there is a decoupling of function and data. However, all actor frameworks I'm aware of seem to assume each message sent will be processed separately. It's not clear how one would turn this on it's head without adding extra machinery. I know Pulsar's actors accept a :lifecycle-handle function which can be used to make actors do "special tricks" but there isn't a lot of documentation around this so it's unclear whether the functionality would be helpful.
I do have a solution to this problem using agents, core.async channels, and watch functions, but it's a bit messy, and I'm hoping there is a better solution. I'll post it as a solution in case others find it helpful, but I'd like to see what other's come up with.
Here's the solution I came up with using agents, core.async channels, and watch functions. Again, it's a bit messy, but it does what I need it to for now. Here it is, in broad strokes:
(require '[clojure.core.async :as async :refer [>!! <!! >! <! chan go]])
; We'll call this thing a queued-agent
(defprotocol IQueuedAgent
(enqueue [this message])
(ping [this]))
(defrecord QueuedAgent [agent queue]
IQueuedAgent
(enqueue [_ message]
(go (>! queue message)))
(ping [_]
(send agent identity)))
; Need a function for draining a core async channel of all messages
(defn drain! [c]
(let [cc (chan 1)]
(go (>! cc ::queue-empty))
(letfn
; This fn does all the hard work, but closes over cc to avoid reconstruction
[(drainer! [c]
(let [[v _] (<!! (go (async/alts! [c cc] :priority true)))]
(if (= v ::queue-empty)
(lazy-seq [])
(lazy-seq (cons v (drainer! c))))))]
(drainer! c))))
; Constructor function
(defn queued-agent [& {:keys [buffer update-fn init-fn error-handler-builder] :or {:buffer 100}}]
(let [q (chan buffer)
a (agent (if init-fn (init-fn) {}))
error-handler-fn (error-handler-builder q a)]
; Set up the queue, and watcher which runs the update function when there is new data
(add-watch
a
:update-conv
(fn [k r o n]
(let [queued (drain! q)]
(when-not (empty? queued)
(send a update-fn queued error-handler-fn)))))
(QueuedAgent. a q)))
; Now we can use these like this
(def a (queued-agent
:init-fn (fn [] {:some "initial value"})
:update-fn (fn [a queued-data error-handler-fn]
(println "Receiving data" queued-data)
; Simulate some work/load on data
(Thread/sleep 2000)
(println "Done with work; ready to queue more up!"))
; This is a little warty at the moment, but closing over the queue and agent lets you requeue work on
; failure so you can try again.
:error-handler-builder
(fn [q a] (println "do something with errors"))))
(defn -main []
(doseq [i (range 10)]
(enqueue a (str "data" i))
(Thread/sleep 500) ; simulate things happening
; This part stinks... have to manually let the queued agent know that we've queued some things up for it
(ping a)))
As you'll notice, having to ping the queued-agent here every time new data is added is pretty warty. It definitely feels like things are being twisted out of typical usage.
Agents are the inverse of what you want here - they are a value that gets sent updating functions. This easiest with a queue and a Thread. For convenience I am using future to construct the thread.
user> (def q (java.util.concurrent.LinkedBlockingDeque.))
#'user/q
user> (defn accumulate
[summary input]
(let [{vowels true consonents false}
(group-by #(contains? (set "aeiouAEIOU") %) input)]
(-> summary
(update-in [:vowels] + (count vowels))
(update-in [:consonents] + (count consonents)))))
#'user/accumulate
user> (def worker
(future (loop [summary {:vowels 0 :consonents 0} in-string (.take q)]
(if (not in-string)
summary
(recur (accumulate summary in-string)
(.take q))))))
#'user/worker
user> (.add q "hello")
true
user> (.add q "goodbye")
true
user> (.add q false)
true
user> #worker
{:vowels 5, :consonents 7}
I came up with something closer to an actor, inspired by Tim Baldridge's cast on actors (Episode 16). I think this addresses the problem much more cleanly.
(defmacro take-all! [c]
`(loop [acc# []]
(let [[v# ~c] (alts! [~c] :default nil)]
(if (not= ~c :default)
(recur (conj acc# v#))
acc#))))
(defn eager-actor [f]
(let [msgbox (chan 1024)]
(go (loop [f f]
(let [first-msg (<! msgbox) ; do this so we park efficiently, and only
; run when there are actually messages
msgs (take-all! msgbox)
msgs (concat [first-msg] msgs)]
(recur (f msgs)))))
msgbox))
(let [a (eager-actor (fn f [ms]
(Thread/sleep 1000) ; simulate work
(println "doing something with" ms)
f))]
(doseq [i (range 20)]
(Thread/sleep 300)
(put! a i)))
;; =>
;; doing something with (0)
;; doing something with (1 2 3)
;; doing something with (4 5 6)
;; doing something with (7 8 9 10)
;; doing something with (11 12 13)
Starting with Clojure I discovered a talk by Rich Hickey where he demonstrates some of Clojure's strengths on a basic Ant-Simulator.
Can this code still be considered as a good reference for Clojure? Especially the parts when he recursively sends off functions to agents to simulate a game loop.
Example:
(defn animation [x]
(when b/running
(send-off *agent* #'animation))
(. panel (repaint))
(. Thread (sleep defs/animation-sleep-ms))
nil)
Edit:
I am not interested in the #' reader macro but more wether it is idiomatic/good Clojure to
recursively call a function on a agent or not.
This snippet is current in Clojure 1.4. Is it idiomatic for a function to submit a task back to the agent that called it? Yes.
Here is an example that uses a similar approach to recursively calculate a factorial:
(defn fac [n limit total]
(if (< n limit)
(let [next-n (inc n)]
(send-off *agent* fac limit (* total next-n))
next-n)
total))
(def a (agent 1))
(await (send-off a fac 5 1))
; => nil
#a
;=> 120
Update
The above is a contrived example and actually not a good one, as there is a race condition between the various recursive send-off calls and the later await. There may be some send-off calls yet to be added to the agent's task queue.
I re-wrote the above as follows:
(defn factorial-using-agent-recursive [x]
(let [a (agent 1)]
(letfn [(calc [n limit total]
(if (< n limit)
(let [next-n (inc n)]
(send-off *agent* calc limit (* total next-n))
next-n)
total))]
(await (send-off a calc x 1)))
#a))
and observed the following behavior:
user=> (for [x (range 10)] (factorial-using-agent-recursive 5))
(2 4 3 120 2 120 120 120 120 2)
user=> (for [x (range 10)] (factorial-using-agent-recursive 5))
(2 2 2 3 2 2 3 2 120 2)
user=> (for [x (range 10)] (factorial-using-agent-recursive 5))
(120 120 120 120 120 120 120 120 120 120)
Moral of the story is: don't use agents for synchronous calculations. Use them for asynchronous independent tasks - like updating animations displayed to a user :)