I have a problem with ClojureScript Core-Async . This only happens in ClojureScript and not in Clojure.
I have the following code:
(defn cc [x]
(go
(println "cc: " x)
x))
(defn foo [x]
(go
(when (and (= :ok (<! (cc x)))
(= :ok (<! (cc :ok))))
(print "after and"))))
(foo 1)
(foo :ok)
When calling (foo :ok), the result is as expected - the function cc is called twice, and the console shows cc: :ok cc: :ok after and.
But, when running (foo 1), the function cc is also run twice and the console shows cc: 1 cc: :ok. So, even though the first condition isn't fullfilled, the second one is still checked!
This is apparently a bug with the ClojureScript core.async library. I opened a Jira issue for this.
In the meantime I was able to work around this by using:
(go
(when (= :ok (<! (cc x))
(when (= :ok (<! (cc :ok)))
(print "after and")))))
Related
I'm experimenting with core.async on Clojure and ClojureScript, to try and understand how merge works. In particular, whether merge makes any values put on input channels available to take immediately on the merged channel.
I have the following code:
(ns async-merge-example.core
(:require
#?(:clj [clojure.core.async :as async] :cljs [cljs.core.async :as async])
[async-merge-example.exec :as exec]))
(defn async-fn-timeout
[v]
(async/go
(async/<! (async/timeout (rand-int 5000)))
v))
(defn async-fn-exec
[v]
(exec/exec "sh" "-c" (str "sleep " (rand-int 5) "; echo " v ";")))
(defn merge-and-print-results
[seq async-fn]
(let [chans (async/merge (map async-fn seq))]
(async/go
(while (when-let [v (async/<! chans)]
(prn v)
v)))))
When I try async-fn-timeout with a large-ish seq:
(merge-and-print-results (range 20) async-fn-timeout)
For both Clojure and ClojureScript I get the result I expect, as in, results start getting printed pretty much immediately, with the expected delays.
However, when I try async-fn-exec with the same seq:
(merge-and-print-results (range 20) async-fn-exec)
For ClojureScript, I get the result I expect, as in results start getting printed pretty much immediately, with the expected delays. However for Clojure even though the sh processes are executed concurrently (subject to the size of the core.async thread pool), the results appear to be initially delayed, then mostly printed all at once! I can make this difference more obvious by increasing the size of the seq e.g. (range 40)
Since the results for async-fn-timeout are as expected on both Clojure and ClojureScript, the finger is pointed at the differences between the Clojure and ClojureScript implementation for exec..
But I don't know why this difference would cause this issue?
Notes:
These observations were made in WSL on Windows 10
The source code for async-merge-example.exec is below
In exec, the implementation differs for Clojure and ClojureScript due to differences between Clojure/Java and ClojureScript/NodeJS.
(ns async-merge-example.exec
(:require
#?(:clj [clojure.core.async :as async] :cljs [cljs.core.async :as async])))
; cljs implementation based on https://gist.github.com/frankhenderson/d60471e64faec9e2158c
; clj implementation based on https://stackoverflow.com/questions/45292625/how-to-perform-non-blocking-reading-stdout-from-a-subprocess-in-clojure
#?(:cljs (def spawn (.-spawn (js/require "child_process"))))
#?(:cljs
(defn exec-chan
"spawns a child process for cmd with args. routes stdout, stderr, and
the exit code to a channel. returns the channel immediately."
[cmd args]
(let [c (async/chan), p (spawn cmd (if args (clj->js args) (clj->js [])))]
(.on (.-stdout p) "data" #(async/put! c [:out (str %)]))
(.on (.-stderr p) "data" #(async/put! c [:err (str %)]))
(.on p "close" #(async/put! c [:exit (str %)]))
c)))
#?(:clj
(defn exec-chan
"spawns a child process for cmd with args. routes stdout, stderr, and
the exit code to a channel. returns the channel immediately."
[cmd args]
(let [c (async/chan)]
(async/go
(let [builder (ProcessBuilder. (into-array String (cons cmd (map str args))))
process (.start builder)]
(with-open [reader (clojure.java.io/reader (.getInputStream process))
err-reader (clojure.java.io/reader (.getErrorStream process))]
(loop []
(let [line (.readLine ^java.io.BufferedReader reader)
err (.readLine ^java.io.BufferedReader err-reader)]
(if (or line err)
(do (when line (async/>! c [:out line]))
(when err (async/>! c [:err err]))
(recur))
(do
(.waitFor process)
(async/>! c [:exit (.exitValue process)]))))))))
c)))
(defn exec
"executes cmd with args. returns a channel immediately which
will eventually receive a result map of
{:out [stdout-lines] :err [stderr-lines] :exit [exit-code]}"
[cmd & args]
(let [c (exec-chan cmd args)]
(async/go (loop [output (async/<! c) result {}]
(if (= :exit (first output))
(assoc result :exit (second output))
(recur (async/<! c) (update result (first output) #(conj (or % []) (second output)))))))))
Your Clojure implementation uses blocking IO in a single thread. You are first reading from stdout and then stderr in a loop. Both do a blocking readLine so they will only return once they actually finished reading a line. So unless your process creates the same amount of output to stdout and stderr one stream will end up blocking the other one.
Once the process is finished the readLine will no longer block and just return nil once the buffer is empty. So the loop just finishes reading the buffered output and then finally completes explaining the "all at once" messages.
You'll probably want to start a second thread that deals reading from stderr.
node does not do blocking IO so everything happens async by default and one stream doesn't block the other.
I'm writing a Clojure wrapper for an object-oriented API that heavily involves resource handling. For instance, for the Foo object, I've written three basic functions: foo?, which returns true iff something is a Foo; create-foo, which attempts to obtain the resources to create a Foo, then returns a map containing a return code and (if the construction succeeded) the newly created Foo; and destroy-foo, which takes a Foo and releases its resources. Here are some stubs for those three functions:
(def foo? (comp boolean #{:placeholder}))
(defn create-foo []
(let [result (rand-nth [::success ::bar-too-full ::baz-not-available])]
(merge {::result result}
(when (= ::success result)
{::foo :placeholder}))))
(defn destroy-foo [foo] {:pre [(foo? foo)]} nil)
Obviously, every time create-foo is called and succeeds, destroy-foo must be called with the returned Foo. Here's a simple example that doesn't use any custom macros:
(let [{:keys [::result ::foo]} (create-foo)]
(if (= ::success result)
(try
(println "Got a Foo:")
(prn foo)
(finally
(destroy-foo foo)))
(do
(println "Got an error:")
(prn result))))
There's a lot of boilerplate here: the try-finally-destroy-foo construct must be present to ensure that all Foo resources are released, and the (= ::success result) test must be present to ensure that nothing gets run assuming a Foo when there is no Foo.
Some of that boilerplate can be eliminated by a with-foo macro, similar to the with-open macro in clojure.core:
(defmacro with-foo [bindings & body]
{:pre [(vector? bindings)
(= 2 (count bindings))
(symbol? (bindings 0))]}
`(let ~bindings
(try
~#body
(finally
(destroy-foo ~(bindings 0))))))
While this does help somewhat, it doesn't do anything about the (= ::success result) boilerplate, and now two separate binding forms are required to achieve the desired result:
(let [{:keys [::result] :as m} (create-foo)]
(if (= ::success result)
(with-foo [foo (::foo m)]
(println "Got a Foo:")
(prn foo))
(do
(println "Got an error:")
(prn result))))
I simply can't figure out a good way to handle this. I mean, I could complect the behaviors of if-let and with-foo into some sort of if-with-foo macro:
(defmacro if-with-foo [bindings then else]
{:pre [(vector? bindings)
(= 2 (count bindings))]}
`(let [{result# ::result foo# ::foo :as m#} ~(bindings 1)
~(bindings 0) m#]
(if (= ::success result#)
(try
~then
(finally
(destroy-foo foo#)))
~else)))
This does eliminate even more boilerplate:
(if-with-foo [{:keys [::result ::foo]} (create-foo)]
(do
(println "Got a Foo:")
(prn foo))
(do
(println "Got a result:")
(prn result)))
However, I don't like this if-with-foo macro for several reasons:
it's very tightly coupled to the specific structure of the map returned by create-foo
unlike if-let, it causes all bindings to be in scope in both branches
its ugly name reflects its ugly complexity
Are these macros the best I can do here? Or is there a more elegant way to handle resource handling with possible resource obtainment failure? Perhaps this is a job for monads; I don't have enough experience with monads to know whether they would be useful tool here.
I'd add an error-handler to with-foo. This way the macro has a focus on what should be done. However, this simplifies the code only when all error-cases are treated by a handful of error handlers. If you have to define a custom error-handler every time you call with-foo this solution makes readability worse than an if-else construct.
I added copy-to-map. copy-to-map should copy all relevant information from the object to a map. This way the user of the macro doesn't by accident return the foo-object, since it gets destroyed inside the macro
(defn foo? [foo]
(= ::success (:result foo)))
(defn create-foo [param-one param-two]
(rand-nth (map #(merge {:obj :foo-obj :result %} {:params [param-one param-two]})
[::success ::bar-too-full ::baz-not-available])))
(defn destroy-foo [foo]
nil)
(defn err-handler [foo]
[:error foo])
(defn copy-to-map [foo]
;; pseudo code here
(into {} foo))
(defmacro with-foo [[f-sym foo-params & {:keys [on-error]}] & body]
`(let [foo# (apply ~create-foo [~#foo-params])
~f-sym (copy-to-map foo#)]
(if (foo? foo#)
(try ~#body
(finally (destroy-foo foo#)))
(when ~on-error
(apply ~on-error [~f-sym])))))
Now you call it
(with-foo [f [:param-one :param-two] :on-error err-handler]
[:success (str "i made it: " f)])
Building from #murphy's excellent idea to put the error handler into with-foo's bindings to keep the focus on the normal case, I've ended up with a solution that I like quite a lot:
(defmacro with-foo [bindings & body]
{:pre [(vector? bindings)
(even? (count bindings))]}
(if-let [[sym init temp error] (not-empty bindings)]
(let [error? (= :error temp)]
`(let [{result# ::result foo# ::foo :as m#} ~init]
(if (contains? m# ::foo)
(try
(let [~sym foo#]
(with-foo ~(subvec bindings (if error? 4 2))
~#body))
(finally
(destroy-foo foo#)))
(let [f# ~(if error? error `(constantly nil))]
(f# result#)))))
`(do
~#body)))
like my if-with-foo macro in the question, this with-foo macro is still tied to the structure returned by create-foo; unlike my if-with-foo macro and #murphy's with-foo macro, it eliminates the need for the user to manually take apart that structure
all names are properly scoped; the user's sym is only bound in the main body, not in the :error handler, and conversely, the ::result is only bound in the :error handler, not in the main body
like #murphy's solution, this macro has a nice, fitting name, instead of something ugly like if-with-foo
unlike #murphy's with-foo macro, this with-foo macro allows the user to provide any init value, rather than forcing a call to create-foo, and doesn't transform the returned value
The most basic use case simply binds a symbol to a Foo returned by create-foo in some body, returning nil if the construction fails:
(with-foo [foo (create-foo)]
["Got a Foo!" foo])
To handle the exceptional case, an :error handler can be added to the binding:
(with-foo [foo (create-foo)
:error (partial vector "Got an error!")]
["Got a Foo!" foo])
Any number of Foo bindings can be used:
(with-foo [foo1 (create-foo)
foo2 (create-foo)]
["Got some Foos!" foo1 foo2])
Each binding can have its own :error handler; any missing error handlers are replaced with (constantly nil):
(with-foo [foo1 (create-foo)
:error (partial vector "Got an error!")
foo2 (create-foo)]
["Got some Foos!" foo1 foo2])
I'm trying to write unit tests when using core.async go macros. Writing the test naively, as follows, it appears that the code inside the go blocks don't get executed.
(ns app.core-test
(:require [clojure.test :refer :all]
[clojure.core.async :as async]))
(deftest test1 []
(let [chan (async/chan)]
(async/go
(is (= (async/<! chan) "Hello")))
(async/go
(async/>! chan "Hello"))))
I've managed to get the following working, but it's extremely hacky.
(deftest test1 []
(let [result (async/chan)
chan (async/chan)]
(async/go
(is (= (async/<! chan) "Hello"))
(async/>! result true))
(async/go
(async/>! chan "Hello"))
(async/alts!! [result (async/timeout 10000)])))
Any suggestions as to how I can do this properly?
Tests are executed synchronously, so if you go async the test-runner won't. In Clojure you need to block the test runner via <!!, in ClojureScript you have to return an async test object. This is a generic helper function I use in all my async CLJC tests:
(defn test-async
"Asynchronous test awaiting ch to produce a value or close."
[ch]
#?(:clj
(<!! ch)
:cljs
(async done
(take! ch (fn [_] (done))))))
Your test using it, CLJC compatible and looking way less "hacky":
(deftest test1
(let [ch (chan)]
(go (>! ch "Hello"))
(test-async
(go (is (= "Hello" (<! ch)))))))
It is good practice to assert that the test unblocks, especially during test driven development where you want to avoid locking your test runner. Also, locking is a common cause of failure in async programming, so testing against it is highly reasonable.
To do that I wrote a helper similar to your timeout thing:
(defn test-within
"Asserts that ch does not close or produce a value within ms. Returns a
channel from which the value can be taken."
[ms ch]
(go (let [t (timeout ms)
[v ch] (alts! [ch t])]
(is (not= ch t)
(str "Test should have finished within " ms "ms."))
v)))
You can use it to write your test like:
(deftest test1
(let [ch (chan)]
(go (>! ch "Hello"))
(test-async
(test-within 1000
(go (is (= "Hello" (<! ch)))))))
your test is finishing, and then failing. This happens more reliably if I put a sleep in and then make it fail:
user> (deftest test1 []
(async/<!!
(let [chan (async/chan)]
(async/go
(async/go
(async/<! (async/timeout 1000))
(is (= (async/<! chan) "WRONG")))
(async/go
(async/>! chan "Hello"))))))
#'user/test1
user> (clojure.test/run-tests)
Testing user
Ran 1 tests containing 0 assertions.
0 failures, 0 errors.
{:test 1, :pass 0, :fail 0, :error 0, :type :summary}
user>
FAIL in (test1) (form-init8563497779572341831.clj:5)
expected: (= (async/<! chan) "WRONG")
actual: (not (= "Hello" "WRONG"))
here we can see it report that nothing fails, then it prints the failure message. We can fix this by explicitly coordinating the end of the test and that action finishing by, like most solutions in core.async, adding one more chan.
user> (deftest test1 []
(async/<!!
(let [all-done-chan (async/chan)
chan (async/chan)]
(async/go
(async/go
(async/<! (async/timeout 1000))
(is (= (async/<! chan) "WRONG"))
(async/close! all-done-chan ))
(async/go
(async/>! chan "Hello"))
(async/<! all-done-chan)))))
#'user/test1
user> (clojure.test/run-tests)
Testing user
FAIL in (test1) (form-init8563497779572341831.clj:6)
expected: (= (async/<! chan) "WRONG")
actual: (not (= "Hello" "WRONG"))
Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
{:test 1, :pass 0, :fail 1, :error 0, :type :summary}
Which is equivalent to your solution using alts. I don't think your solution is hackey. With asynchronous code it's always required to pay attention to when things finish, even if you conciously decide to ignore the result.
I'm using an approach similar to Leon's, but with no extra go blocks:
(defn <!!?
"Reads from chan synchronously, waiting for a given maximum of milliseconds.
If the value does not come in during that period, returns :timed-out. If
milliseconds is not given, a default of 1000 is used."
([chan]
(<!!? chan 1000))
([chan milliseconds]
(let [timeout (async/timeout milliseconds)
[value port] (async/alts!! [chan timeout])]
(if (= chan port)
value
:timed-out))))
You can use it simply as:
(is (= 42 (<!!? result-chan)))
Most of the time I just want to read the value from the channel without any extra hassle.
I'd like to use memoize for a function that uses core.async and <! e.g
(defn foo [x]
(go
(<! (timeout 2000))
(* 2 x)))
(In the real-life, it could be useful in order to cache the results of server calls)
I was able to achieve that by writing a core.async version of memoize (almost the same code as memoize):
(defn memoize-async [f]
(let [mem (atom {})]
(fn [& args]
(go
(if-let [e (find #mem args)]
(val e)
(let [ret (<! (apply f args))]; this line differs from memoize [ret (apply f args)]
(swap! mem assoc args ret)
ret))))))
Example of usage:
(def foo-memo (memoize-async foo))
(go (println (<! (foo-memo 3)))); delay because of (<! (timeout 2000))
(go (println (<! (foo-memo 3)))); subsequent calls are memoized => no delay
I am wondering if there are simpler ways to achieve the same result.
**Remark: I need a solution that works with <!. For <!!, see this question: How to memoize a function that uses core.async and blocking channel read? **
You can use the built in memoize function for this. Start by defining a method that reads from a channel and returns the value:
(defn wait-for [ch]
(<!! ch))
Note that we'll use <!! and not <! because we want this function block until there is data on the channel in all cases. <! only exhibits this behavior when used in a form inside of a go block.
You can then construct your memoized function by composing this function with foo, like such:
(def foo-memo (memoize (comp wait-for foo)))
foo returns a channel, so wait-for will block until that channel has a value (i.e. until the operation inside foo finished).
foo-memo can be used similar to your example above, except you do not need the call to <! because wait-for will block for you:
(go (println (foo-memo 3))
You can also call this outside of a go block, and it will behave like you expect (i.e. block the calling thread until foo returns).
This was a little trickier than I expected. Your solution isn't correct, because when you call your memoized function again with the same arguments, sooner than the first run finishes running its go block, you will trigger it again and get a miss. This is often the case when you process lists with core.async.
The one below uses core.async's pub/sub to solve this (tested in CLJS only):
(def lookup-sentinel #?(:clj ::not-found :cljs (js-obj))
(def pending-sentinel #?(:clj ::pending :cljs (js-obj))
(defn memoize-async
[f]
(let [>in (chan)
pending (pub >in :args)
mem (atom {})]
(letfn
[(memoized [& args]
(go
(let [v (get #mem args lookup-sentinel)]
(condp identical? v
lookup-sentinel
(do
(swap! mem assoc args pending-sentinel)
(go
(let [ret (<! (apply f args))]
(swap! mem assoc args ret)
(put! >in {:args args :ret ret})))
(<! (apply memoized args)))
pending-sentinel
(let [<out (chan 1)]
(sub pending args <out)
(:ret (<! <out)))
v))))]
memoized)))
NOTE: it probably leaks memory, subscriptions and <out channels are not closed
I have used this function in one of my projects to cache HTTP calls. The function caches results for a given amount of time and uses a barrier to prevent executing the function multiple times when the cache is "cold" (due to the context switch inside the go block).
(defn memoize-af-until
[af ms clock]
(let [barrier (async/chan 1)
last-return (volatile! nil)
last-return-ms (volatile! nil)]
(fn [& args]
(async/go
(>! barrier :token)
(let [now-ms (.now clock)]
(when (or (not #last-return-ms) (< #last-return-ms (- now-ms ms)))
(vreset! last-return (<! (apply af args)))
(vreset! last-return-ms now-ms))
(<! barrier)
#last-return)))))
You can test that it works properly by setting the cache time to 0 and observe that the two function calls take approximately 10 seconds. Without the barrier the two calls would finish at the same time:
(def memo (memoize-af-until #(async/timeout 5000) 0 js/Date))
(async/take! (memo) #(println "[:a] Finished"))
(async/take! (memo) #(println "[:b] Finished"))
I don't know how to implement this piece of Python code in Clojure
for i in range(3):
try:
......
except e:
if i == 2:
raise e
else:
continue
else:
break
I wonder why something so simple in Python is so hard in Clojure. I think the difficulty is because Clojure is a functional programming language and thus is not suitable for such an imperative task. This is my attempt:
(first
(remove #(instance? Exception %)
(for [i (range 3)]
(try (......)
(catch Exception e
(if (== i 2)
(throw e)
e)))))))
It is very ugly, and worse, it doesn't work as expected. The for loop is actually evaluated fully instead of lazily (I realized this when I put a println inside).
If anyone has a better idea to implement that, please enlighten me.
Similar to Marcyk's answer, but no macro trickery:
(defn retry
[retries f & args]
(let [res (try {:value (apply f args)}
(catch Exception e
(if (zero? retries)
(throw e)
{:exception e})))]
(if (:exception res)
(recur (dec retries) f args)
(:value res))))
Slightly complicated because you can't recur inside a catch clause. Note that this takes a function:
(retry 3 (fn []
(println "foo")
(if (zero? (rand-int 2))
(throw (Exception. "foo"))
2)))
=>
foo ;; one or two or three of these
foo
2
Here's one approach:
(defmacro retry
"Evaluates expr up to cnt + 1 times, retrying if an exception
is thrown. If an exception is thrown on the final attempt, it
is allowed to bubble up."
[cnt expr]
(letfn [(go [cnt]
(if (zero? cnt)
expr
`(try ~expr
(catch Exception e#
(retry ~(dec cnt) ~expr)))))]
(go cnt)))
Example from the REPL:
user> (retry 2 (do (println :foo) (throw (RuntimeException. "foo"))))
:foo
:foo
:foo
; Evaluation aborted.
(Passing 2 to retry asks expr to be retried twice it if fails the first time round, for a total of three attempts. Three :foos are printed, because the println occurs before the throw in the do form passed to retry. The final ; Evaluation aborted. means an exception was thrown.)
Also, about the for loop from your snippet:
If you try looping over a longer range (replace (range 3) with (range 10), say), the output will end after i reaches 3. Also, if you put in a println before the form which throws the exception, it will of course print out whatever you pass to it; if the println occurs after the exception-throwing form, there will be no printout. In any case, at most three calls to println will be executed (assuming an exception is thrown on every iteration).
(cond (every? nil? (for [x (range (inc retry)) :while (not #tmp-doc)]
...do sth) )
;all failed
:else
;at least one success
You can do like this:
(defn retry
"Tries at most n times, return first try satisfying pred or nil"
[times pred? lazy-seq]
(let [successful-trial (drop-while (complement pred?) (take times lazy-seq))]
(if (empty? successful-trial)
nil
(first successful-trial))))
Then you could use the function as such:
(when-not (retry 3 pos? (repeatedly #(rand-nth [-1 -2 -3 2 1]))
(throw (Exception. "my exception message"))
This would try at most three times to take a positive number at random from the vector and if it doesn't succeed, throws an exception.
Retry with sleep and another implementation based on this question's answers:
(defn retry [retries sleep-ms func url options]
(let [{:keys [status headers body error] :as resp} #(func url options)]
(if error
(do
(log/error (str "Retry " retries " returned error: " e))
(Thread/sleep sleep-ms)
(if (= retries 1)
resp
(retry (dec retries) sleep-ms func url options)))
resp)))