As the title asks, is there any difference between (chan n) and (chan (buffer n)) when in use?
The question originates from the idea that I want to pull messages from a db (yeah, I do not want to use Kafka or RabbitMQ) and process them in order.
So the code segment looks like:
(defn processer [id]
(let [ch (chan (buffer 1000))] ;;(chan 1000) or (chan (buffer 1000))?
(go-loop []
(when-let [msg (<! ch)]
(process-msg msg))
(recur))
ch))
(defn enqueue [id]
(let [ch (processer id)]
(go-loop []
(let [msgs (take-100-msg-from-db id)]
(if (seq msgs)
(doseq [msg msgs]
(>! ch msg))
(<! (timeout 100))))
(recur))))
In my testing, their behaviours do not differ.
Yes, they are the same.
You can always look at the source:
(defn chan
([] (chan nil))
([buf-or-n] (chan buf-or-n nil))
([buf-or-n xform] (chan buf-or-n xform nil))
([buf-or-n xform ex-handler]
(channels/chan (if (number? buf-or-n) (buffer buf-or-n) buf-or-n) xform ex-handler)))
Related
I am using [org.clojure/clojure "1.10.1"],[org.clojure/core.async "1.2.603"] and the latest Amazon Corretto 11 JVM if there was anything to do with them.
The following code is a simplified version of the code used in production and it does cause memory leak. I have no idea why that happened but suspect it might due to sub/unsub of channels. Can anyone help point out where my code may go wrong or how I can fix the memory leak?
(ns test-gc.core
(:require [clojure.core.async :as a :refer [chan put! close! <! go >! go-loop timeout]])
(:import [java.util UUID]))
(def global-msg-ch (chan (a/sliding-buffer 200)))
(def global-msg-pub (a/pub global-msg-ch :id))
(defn io-promise []
(let [id (UUID/randomUUID)
ch (chan)]
(a/sub global-msg-pub id ch)
[id (go
(let [x (<! ch)]
(a/unsub global-msg-pub id ch)
(:data x)))]))
(defn -main []
(go-loop []
(<! (timeout 1))
(let [[pid pch] (io-promise)
cmd {:id pid
:data (rand-int 1E5)}]
(>! global-msg-ch cmd)
(println (<! pch)))
(recur))
(while true
(Thread/yield)))
A quick heap dump gives the following statistics for example:
Class by number of instances
java.util.LinkedList 5,157,128 (14.4%)
java.util.concurrent.atomic.AtomicReference 3,698,382 (10.3%)
clojure.lang.Atom 3,094,279 (8.6%)
...
Class by size of instances
java.lang.Object[] 210,061,752 B (13.8%)
java.util.LinkedList 206,285,120 B (13.6%)
clojure.lang.Atom 148,525,392 B (9.8%)
clojure.core.async.impl.channels.ManyToManyChannel 132,022,336 B (8.7%)
...
I finally figured out why. By looking at the source code, we get the following segment:
(defn pub
"Creates and returns a pub(lication) of the supplied channel, ..."
...
(let [mults (atom {}) ;;topic->mult
ensure-mult (fn [topic]
(or (get #mults topic)
(get (swap! mults
#(if (% topic) % (assoc % topic (mult (chan (buf-fn topic))))))
topic)))
p (reify
Mux
(muxch* [_] ch)
Pub
(sub* [p topic ch close?]
(let [m (ensure-mult topic)]
(tap m ch close?)))
(unsub* [p topic ch]
(when-let [m (get #mults topic)]
(untap m ch)))
(unsub-all* [_] (reset! mults {}))
(unsub-all* [_ topic] (swap! mults dissoc topic)))]
...
p)))
We can see mults stores all topic hence shall increase monotonically if we do not clear it. We may add something like (a/unsub-all* global-msg-pub pid) to fix that.
I'm implementing Sleeping barber using core.async. My current code is:
(def workingtime 10000)
(defn barber [in waiting-room]
(go-loop [served-customers 0]
(let [[v] (alts! [waiting-room in])]
(if (= v :close)
served-customers
(do (Thread/sleep 20)
(recur (inc served-customers)))))))
(defn customers [in waiting-room]
(go-loop [customers-overall 0]
(let [customer-arrival-interval (timeout (+ 10 (rand-int 20)))
[v] (alts! [in customer-arrival-interval])]
(if (= v :close)
customers-overall
(do (>! waiting-room :customer)
(recur (inc customers-overall)))))))
(defn -main [& args]
(let [in (chan)
waiting-room (chan (dropping-buffer 3))
barber-ch (barber in waiting-room)
customers-ch (customers in waiting-room)]
(println "opening the shop for 10 seconds...")
(Thread/sleep workingtime)
(>!! in :close)
(>!! in :close)
(println "closing the shop...")
(println (str "Served " (<!! barber-ch) " customers"))
(println (str "Overall " (<!! customers-ch) " customers came"))))
Is it a correct solution? Can it be improved to make it more Clojure-like?
I wanted to use alt! instead of alts! which makes code easier to read:
(defn barber [in]
(go-loop [served-customers 0]
(alt!
waiting-room (do (Thread/sleep 20)
(recur (inc served-customers)))
in served-customers)))
Runtime throws an exception: Can only recur from tail position. Can I still use alt!?
You could solve the alt!/recur problem by rewriting to:
(defn barber [in]
(go-loop [served-customers 0]
(if (= :waiting-room
(a/alt!
waiting-room ([result] :waiting-room) ;; you could also use result if needed
in ([result] :in))) ;; same here
(do (Thread/sleep 20)
(recur (inc served-customers)))
served-customers)))
The number of possible executions of a function should be throttled. So after calling a function, any repeated call should be ignored within a time period. If there where calls in the meantime, the last one should be executed after the time period.
Here's my approach with core.async. The problem here is, that additional calls are summing up in the channel c. I'd need a channel with only one position inside, which will be overridden by put! everytime.
(defn throttle [f time]
(let [c (chan 1)]
(go-loop []
(apply f (<! c))
(<! (timeout time))
(recur))
(fn [& args]
(put! c (if args args [])))))
usage:
(def throttled (throttle #(print %) 4000))
(doseq [x (range 10)]
(throttled x))
; 0
;... after 4 seconds
; 9
Does anyone have an idea how to fix this?
Solution
(defn throttle [f time]
(let [c (chan (sliding-buffer 1))]
(go-loop []
(apply f (<! c))
(<! (timeout time))
(recur))
(fn [& args]
(put! c (or args [])))))
To solve your channel question you can use a chan with a sliding buffer:
user> (require '[clojure.core.async :as async])
nil
user> (def c (async/chan (async/sliding-buffer 1)))
#'user/c
user> (async/>!! c 1)
true
user> (async/>!! c 2)
true
user> (async/>!! c 3)
true
user> (async/<!! c)
3
that way only the last value put into the channel will be computed at the next interval.
You can use a debounce function.
I'll copy it out here:
(defn debounce [in ms]
(let [out (chan)]
(go-loop [last-val nil]
(let [val (if (nil? last-val) (<! in) last-val)
timer (timeout ms)
[new-val ch] (alts! [in timer])]
(condp = ch
timer (do (>! out val) (recur nil))
in (recur new-val))))
out))
Here only when in has not emitted a message for ms is the last value it emitted forwarded onto the out channel. While in continues to emit without a long enough pause between emits then all-but-the-last-message are continuously discarded.
I've tested this function. It waits 4 seconds and then prints out 9, which is nearly what you asked for - some tweaking required!
(defn my-sender [to-chan values]
(go-loop [[x & xs] values]
(>! to-chan x)
(when (seq xs) (recur xs))))
(defn my-receiver [from-chan f]
(go-loop []
(let [res (<! from-chan)]
(f res)
(recur))))
(defn setup-and-go []
(let [in (chan)
ch (debounce in 4000)
sender (my-sender in (range 10))
receiver (my-receiver ch #(log %))]))
And this is the version of debounce that will output as required by the question, which is 0 immediately, then wait four seconds, then 9:
(defn debounce [in ms]
(let [out (chan)]
(go-loop [last-val nil
first-time true]
(let [val (if (nil? last-val) (<! in) last-val)
timer (timeout (if first-time 0 ms))
[new-val ch] (alts! [in timer])]
(condp = ch
timer (do (>! out val) (recur nil false))
in (recur new-val false))))
out))
I've used log rather than print as you did. You can't rely on ordinary println/print functions with core.async. See here for an explanation.
This is taken from David Nolens blog's source code:
(defn throttle*
([in msecs]
(throttle* in msecs (chan)))
([in msecs out]
(throttle* in msecs out (chan)))
([in msecs out control]
(go
(loop [state ::init last nil cs [in control]]
(let [[_ _ sync] cs]
(let [[v sc] (alts! cs)]
(condp = sc
in (condp = state
::init (do (>! out v)
(>! out [::throttle v])
(recur ::throttling last
(conj cs (timeout msecs))))
::throttling (do (>! out v)
(recur state v cs)))
sync (if last
(do (>! out [::throttle last])
(recur state nil
(conj (pop cs) (timeout msecs))))
(recur ::init last (pop cs)))
control (recur ::init nil
(if (= (count cs) 3)
(pop cs)
cs)))))))
out))
(defn throttle-msg? [x]
(and (vector? x)
(= (first x) ::throttle)))
(defn throttle
([in msecs] (throttle in msecs (chan)))
([in msecs out]
(->> (throttle* in msecs out)
(filter #(and (vector? %) (= (first %) ::throttle)))
(map second))))
You probably also want to add a dedupe transducer to the channel.
I needed to pass a function to capture the args because I was using it for an input event and it was passing a mutable object. 🤷
(defn throttle-for-mutable-args [time f arg-capture-fn]
(let [c (async/chan (async/sliding-buffer 1))]
(async-m/go-loop []
(f (async/<! c))
(async/<! (async/timeout time))
(recur))
(fn [& args]
(async/put! c (apply arg-capture-fn (or args []))))))
And I use like
[:input
{:onChange (util/throttle-for-mutable-args
500
#(really-use-arg %)
#(-> % .-target .-value))}]
Here is an example from http://www.braveclojure.com/core-async/ :
(defn hotdog-machine-v2 [hotdog-count]
(let [in (chan) out (chan)]
(go (loop [hc hotdog-count]
(if (> hc 0)
(let [input (<! in)]
(if (= 3 input)
(do
(>! out "hotdog")
(recur (dec hc)) )
(do
(>! out (Exception. "Not enough payment!"))
(recur hc))))
(do
(close! in)
(close! out)))))
[in out]))
(let [[in out] (hotdog-machine-v2 2)]
(>!! in "pocket lint")
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
)
If I leave out the channel-closing part, then this code hangs forever, why?
If I run this in the normal case with the close it prints this:
user> (let [[in out] (hotdog-machine-v2 2)]
(>!! in "pocket lint")
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out)))
#error {
:cause Not enough payment!
:via
[{:type java.lang.Exception
:message Not enough payment!
... stack trace here ...
[java.lang.Thread run Thread.java 724]]}
hotdog
hotdog
nil ;; <---- look here
That last nil is the result of the last read from out reading the value nil, which is being sent as a result of the channel closing because the machine is out of hotdogs. Without the close that last put (>!! in 3) blocks waiting for somthing to read from the chan which nobody is going to do. By default writes to core.async chans don't succeed in writing until someone is ready to read that value
If I take out the close, and take out the last write it does not print that nil, and does not block:
user> (let [[in out] (hotdog-machine-v2 2)]
(>!! in "pocket lint")
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
)
#error {
:cause Not enough payment!
:via
[{:type java.lang.Exception
:message Not enough payment!
:at ... stack trace here ...}
hotdog
hotdog
I'm trying to make all print statements in a clojure program put their strings on a core.async channel. Is there a way I can do this?
(I'm trying to get cljsbuild to send its compiler messages to a frontend.)
(def is (java.io.PipedInputStream.))
(def os (java.io.PipedOutputStream. is))
(def channel (chan))
(.start (Thread. (fn []
(let [buffer (byte-array 1024)]
(while true
(let [n (.read is buffer 0 1024)]
(>!! channel (vec (take n buffer)))))))))
(binding [*out* (java.io.OutputStreamWriter. os)]
(println "Hello, world!"))
gfredericks gave me this reference in the #clojure irc. He recommended I adapt what nREPL does:
(defn- session-out
"Returns a PrintWriter suitable for binding as *out* or *err*. All of
the content written to that PrintWriter will (when .flush-ed) be sent on the
given transport in messages specifying the given session-id.
`channel-type` should be :out or :err, as appropriate."
[channel-type session-id transport]
(let [buf (clojure.tools.nrepl.StdOutBuffer.)]
(PrintWriter. (proxy [Writer] []
(close [] (.flush ^Writer this))
(write [& [x ^Integer off ^Integer len]]
(locking buf
(cond
(number? x) (.append buf (char x))
(not off) (.append buf x)
; the CharSequence overload of append takes an *end* idx, not length!
(instance? CharSequence x) (.append buf ^CharSequence x (int off) (int (+ len off)))
:else (.append buf ^chars x off len))
(when (<= *out-limit* (.length buf))
(.flush ^Writer this))))
(flush []
(let [text (locking buf (let [text (str buf)]
(.setLength buf 0)
text))]
(when (pos? (count text))
(t/send (or (:transport *msg*) transport)
(response-for *msg* :session session-id
channel-type text))))))
true)))
from nREPL source here