Trying to understand how channels work; I don't get why my take! only works once in this REPL sequence, even though I try multiple puts along the way:
cplay.core> (def h (chan))
#'cplay.core/h
cplay.core> (go (put! h "hello"))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#4afdd6ba>
cplay.core> (go (take! h (fn [x] (println x))))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#9fe39a1>
cplay.core> hello
cplay.core> (go (take! h (fn [x] (println x))))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#206564e9>
cplay.core> (go (put! h "hello"))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#6c0ec468>
cplay.core> (go (take! h (fn [x] (println x))))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#60c85184>
cplay.core> (go (take! h (fn [x] (println x))))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#3edc08c3>
cplay.core> (go (take! h (fn [x] (println x))))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#6a7b295f>
cplay.core> (go (put! h "hello"))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#60331d8b>
cplay.core> (go (take! h (fn [x] (println x))))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#557c3bce>
cplay.core> (go (put! h "hello"))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#203fdcfb>
cplay.core> (go (put! h "hello"))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#ed11c1>
cplay.core> (go (take! h (fn [x] (println x))))
#<ManyToManyChannel clojure.core.async.impl.channels.ManyToManyChannel#6c7ea146>
cplay.core>
Is it because I then take from the channel several times in succession when nothing is on the channel, thereby leaving "voids" that are filled by subsequent puts until the channel somehow gets back to and even number of puts and takes?
The println behavior is quiet weird in this async scenario.
If your last invocation is the "take!" fn then you'll end up with the print result in your repl, but is four last invocation is the "put!" fn then you have to find the print result on the nrepl-server terminal.
This behavior and necessary workspace it's introduced on the beginning of Timothy Baldridge video of core.async
And here one screenshot of my 2 frames in emacs, one for the nrepl and the other for the nrepl-server output
Don't wrap put! and take! with go. Inside go you want <! or !>:
user=> (require '[clojure.core.async :refer (chan put! take! <! >!)])
nil
user=> (def h (chan))
#'user/h
user=> (put! h "hello")
true
user=> (take! h println)
hello
nil
user=> (put! h "hello")
true
user=> (take! h println)
hello
nil
user=> (put! h "hello")
true
user=> (take! h println)
hello
nil
user=> (put! h "hello")
true
user=> (take! h println)
hello
nil
user=> (take! h println)
nil
user=> (put! h "hello")
hello
true
Related
I am using [org.clojure/clojure "1.10.1"],[org.clojure/core.async "1.2.603"] and the latest Amazon Corretto 11 JVM if there was anything to do with them.
The following code is a simplified version of the code used in production and it does cause memory leak. I have no idea why that happened but suspect it might due to sub/unsub of channels. Can anyone help point out where my code may go wrong or how I can fix the memory leak?
(ns test-gc.core
(:require [clojure.core.async :as a :refer [chan put! close! <! go >! go-loop timeout]])
(:import [java.util UUID]))
(def global-msg-ch (chan (a/sliding-buffer 200)))
(def global-msg-pub (a/pub global-msg-ch :id))
(defn io-promise []
(let [id (UUID/randomUUID)
ch (chan)]
(a/sub global-msg-pub id ch)
[id (go
(let [x (<! ch)]
(a/unsub global-msg-pub id ch)
(:data x)))]))
(defn -main []
(go-loop []
(<! (timeout 1))
(let [[pid pch] (io-promise)
cmd {:id pid
:data (rand-int 1E5)}]
(>! global-msg-ch cmd)
(println (<! pch)))
(recur))
(while true
(Thread/yield)))
A quick heap dump gives the following statistics for example:
Class by number of instances
java.util.LinkedList 5,157,128 (14.4%)
java.util.concurrent.atomic.AtomicReference 3,698,382 (10.3%)
clojure.lang.Atom 3,094,279 (8.6%)
...
Class by size of instances
java.lang.Object[] 210,061,752 B (13.8%)
java.util.LinkedList 206,285,120 B (13.6%)
clojure.lang.Atom 148,525,392 B (9.8%)
clojure.core.async.impl.channels.ManyToManyChannel 132,022,336 B (8.7%)
...
I finally figured out why. By looking at the source code, we get the following segment:
(defn pub
"Creates and returns a pub(lication) of the supplied channel, ..."
...
(let [mults (atom {}) ;;topic->mult
ensure-mult (fn [topic]
(or (get #mults topic)
(get (swap! mults
#(if (% topic) % (assoc % topic (mult (chan (buf-fn topic))))))
topic)))
p (reify
Mux
(muxch* [_] ch)
Pub
(sub* [p topic ch close?]
(let [m (ensure-mult topic)]
(tap m ch close?)))
(unsub* [p topic ch]
(when-let [m (get #mults topic)]
(untap m ch)))
(unsub-all* [_] (reset! mults {}))
(unsub-all* [_ topic] (swap! mults dissoc topic)))]
...
p)))
We can see mults stores all topic hence shall increase monotonically if we do not clear it. We may add something like (a/unsub-all* global-msg-pub pid) to fix that.
How can I make a words chan out of chars chan?
I have chan c that read all characters from file. And i need words(str) chan from c chan.(
words need to be made from с, not from a file)
(def file-as-str (slurp "src/clojure2/text.txt"))
(def read (str/split file-as-str #""))
(defn ch
[c]
(go
(doseq [o read]
(>! c o)
))
)
(defn word
[c]
(let [k (chan)]
(go-loop []
(let [o (<! c)]
(when (not= " " o)
(>! k o)
(recur))))
k))
(defn -main
[& args]
(let [c (chan)
words (chan)]
;(go-loop []
; (let [o (<! c)]
; (println o))
; (recur))
(ch c)
(word c)
)
)
UPDATE
And so I made a function that returns a chan of characters for 1 word. But I don’t understand how to combine this into 1 word
You can create chan with transducer which converts incoming characters to outgoing words.
(let [ch (async/chan 100 (comp
(partition-by (complement #{\space}))
(map #(apply str %))
(remove #{" "})))]
(doseq [c "Hello, world! "]
(async/>!! ch c))
(async/<!! (async/into [] (async/take 2 ch))))
=> ["Hello," "world!"]
The number of possible executions of a function should be throttled. So after calling a function, any repeated call should be ignored within a time period. If there where calls in the meantime, the last one should be executed after the time period.
Here's my approach with core.async. The problem here is, that additional calls are summing up in the channel c. I'd need a channel with only one position inside, which will be overridden by put! everytime.
(defn throttle [f time]
(let [c (chan 1)]
(go-loop []
(apply f (<! c))
(<! (timeout time))
(recur))
(fn [& args]
(put! c (if args args [])))))
usage:
(def throttled (throttle #(print %) 4000))
(doseq [x (range 10)]
(throttled x))
; 0
;... after 4 seconds
; 9
Does anyone have an idea how to fix this?
Solution
(defn throttle [f time]
(let [c (chan (sliding-buffer 1))]
(go-loop []
(apply f (<! c))
(<! (timeout time))
(recur))
(fn [& args]
(put! c (or args [])))))
To solve your channel question you can use a chan with a sliding buffer:
user> (require '[clojure.core.async :as async])
nil
user> (def c (async/chan (async/sliding-buffer 1)))
#'user/c
user> (async/>!! c 1)
true
user> (async/>!! c 2)
true
user> (async/>!! c 3)
true
user> (async/<!! c)
3
that way only the last value put into the channel will be computed at the next interval.
You can use a debounce function.
I'll copy it out here:
(defn debounce [in ms]
(let [out (chan)]
(go-loop [last-val nil]
(let [val (if (nil? last-val) (<! in) last-val)
timer (timeout ms)
[new-val ch] (alts! [in timer])]
(condp = ch
timer (do (>! out val) (recur nil))
in (recur new-val))))
out))
Here only when in has not emitted a message for ms is the last value it emitted forwarded onto the out channel. While in continues to emit without a long enough pause between emits then all-but-the-last-message are continuously discarded.
I've tested this function. It waits 4 seconds and then prints out 9, which is nearly what you asked for - some tweaking required!
(defn my-sender [to-chan values]
(go-loop [[x & xs] values]
(>! to-chan x)
(when (seq xs) (recur xs))))
(defn my-receiver [from-chan f]
(go-loop []
(let [res (<! from-chan)]
(f res)
(recur))))
(defn setup-and-go []
(let [in (chan)
ch (debounce in 4000)
sender (my-sender in (range 10))
receiver (my-receiver ch #(log %))]))
And this is the version of debounce that will output as required by the question, which is 0 immediately, then wait four seconds, then 9:
(defn debounce [in ms]
(let [out (chan)]
(go-loop [last-val nil
first-time true]
(let [val (if (nil? last-val) (<! in) last-val)
timer (timeout (if first-time 0 ms))
[new-val ch] (alts! [in timer])]
(condp = ch
timer (do (>! out val) (recur nil false))
in (recur new-val false))))
out))
I've used log rather than print as you did. You can't rely on ordinary println/print functions with core.async. See here for an explanation.
This is taken from David Nolens blog's source code:
(defn throttle*
([in msecs]
(throttle* in msecs (chan)))
([in msecs out]
(throttle* in msecs out (chan)))
([in msecs out control]
(go
(loop [state ::init last nil cs [in control]]
(let [[_ _ sync] cs]
(let [[v sc] (alts! cs)]
(condp = sc
in (condp = state
::init (do (>! out v)
(>! out [::throttle v])
(recur ::throttling last
(conj cs (timeout msecs))))
::throttling (do (>! out v)
(recur state v cs)))
sync (if last
(do (>! out [::throttle last])
(recur state nil
(conj (pop cs) (timeout msecs))))
(recur ::init last (pop cs)))
control (recur ::init nil
(if (= (count cs) 3)
(pop cs)
cs)))))))
out))
(defn throttle-msg? [x]
(and (vector? x)
(= (first x) ::throttle)))
(defn throttle
([in msecs] (throttle in msecs (chan)))
([in msecs out]
(->> (throttle* in msecs out)
(filter #(and (vector? %) (= (first %) ::throttle)))
(map second))))
You probably also want to add a dedupe transducer to the channel.
I needed to pass a function to capture the args because I was using it for an input event and it was passing a mutable object. 🤷
(defn throttle-for-mutable-args [time f arg-capture-fn]
(let [c (async/chan (async/sliding-buffer 1))]
(async-m/go-loop []
(f (async/<! c))
(async/<! (async/timeout time))
(recur))
(fn [& args]
(async/put! c (apply arg-capture-fn (or args []))))))
And I use like
[:input
{:onChange (util/throttle-for-mutable-args
500
#(really-use-arg %)
#(-> % .-target .-value))}]
For example given a channel with operations and another channel with data, how to write a go block that will apply the operation on whatever was the last value on the data channel?
(go-loop []
(let [op (<! op-ch)
data (<! data-ch)]
(put! result-ch (op data))))
Obviously that doesn't work because it would require both channels to have the same frequency.
(see http://rxmarbles.com/#withLatestFrom)
Using alts! you could accomplish what you want.
The with-latest-from shown below implements the same behavior found in the withLatestFrom from RxJS (I think :P).
(require '[clojure.core.async :as async])
(def op-ch (async/chan))
(def data-ch (async/chan))
(defn with-latest-from [chs f]
(let [result-ch (async/chan)
latest (vec (repeat (count chs) nil))
index (into {} (map vector chs (range)))]
(async/go-loop [latest latest]
(let [[value ch] (async/alts! chs)
latest (assoc latest (index ch) value)]
(when-not (some nil? latest)
(async/put! result-ch (apply f latest)))
(when value (recur latest))))
result-ch))
(def result-ch (with-latest-from [op-ch data-ch] str))
(async/go-loop []
(prn (async/<! result-ch))
(recur))
(async/put! op-ch :+)
;= true
(async/put! data-ch 1)
;= true
; ":+1"
(async/put! data-ch 2)
;= true
; ":+2"
(async/put! op-ch :-)
;= true
; ":-2"
There's an :priority true option for the alts!.
An expression which always returns the latest seen value in some channel would look something like this:
(def in-chan (chan))
(def mem (chan))
(go (let [[ch value] (alts! [in-chan mem] :priority true)]
(take! mem) ;; clear mem (take! is non-blocking)
(>! mem value) ;; put the new (or old) value in the mem
value ;; return a chan with the value in
It's untested, it's probably not efficient (a volatile variable is probably better). The go-block returns a channel with only the value, but the idea could be expanded to some "memoized" channel.
I have a namespace like this:
(ns foo.core)
(def ^:dynamic *debug-fn*
"A function taking arguments [bar baz]"
nil)
(defn bar-info
[bar _]
(println bar))
(defn baz-info
[_ baz]
(println baz))
(defn do-stuff
[bar baz]
(when *debug-fn* (*debug-fn* bar baz)))
(defn -main
[& {:keys [debug-fn]}]
(binding [*debug-fn* (symbol debug-fn)] ;; THIS WON'T WORK!
(do-stuff 27 42)))
What I would like to do is allow a debug function to be specified from the command line like this: lein run bar-info or lein run baz-info.
I'm not sure how to take the string specified as a command-line argument and turn it into the namespace-qualified function to bind. Do I need a macro to do this?
Use ns-resolve, you will need to specify namespace where your function is defined though.
user=> (defn f [n] (* n n n))
#'user/f
user=> ((ns-resolve *ns* (symbol "f")) 10)
1000
Use alter-var-root:
user=> (doc alter-var-root)
-------------------------
clojure.core/alter-var-root
([v f & args])
Atomically alters the root binding of var v by applying f to its
current value plus any args
nil
user=> (alter-var-root #'*debug-fn* (fn [v] (fn [x] (println x) x)))
#<user$eval171$fn__172$fn__173 user$eval171$fn__172$fn__173#7c93d88e>
user=> (*debug-fn* 1)
1
1
Though I've accepted Guillermo's answer above, I figured that it might also be useful to add the solution I ended up going with:
(def debug-fns
{:bar-info (fn [bar _] (println bar))
:baz-info (fn [_ baz] (println baz))
(def active-debug-fns (atom []))
(defn activate-debug-fn!
[fn-key]
(let [f (debug-fns fn-key)]
(if f
(swap! active-debug-fns conj f)
(warn (str "Debug function " fn-key " not found! Available functions are: "
(join " " (map name (keys debug-fns))))))))
(defn debug-fn-keys
[args]
(if (args "--debug")
(split (or (args "--debug") "") #",")
[]))
(defn do-stuff
[bar baz]
(doseq [f #active-debug-fns]
(f bar baz)))
(defn -main
[& args]
(let [args (apply hash-map args)]
(doseq [f (debug-fn-keys args)]
(activate-debug-fn! (keyword k)))
(do-stuff 27 42)))
So now you can say something like lein run --debug bar-info to get info on bars, or lein run --debug bar,baz to get info on both bars and bazes.
Any suggestions to make this more idiomatic will be happily accepted and edited in. :)