I have this code:
(def x 10)
(def io (java.io.StringWriter.))
(with-redefs [x 11]
(with-redefs [io (java.io.StringWriter.)]
(.write io "one"))
(with-redefs [io (java.io.StringWriter.)]
(.write io "two")
(.toString io)))
(with-redefs [x 11]
(with-redefs [*out* (java.io.StringWriter.)]
(print "one"))
(with-redefs [*out* (java.io.StringWriter.)]
(print "two")
(.toString *out*)))
The first with-redefs block return "two" as expected. The second one returns "onetwo". It is as if it were reutilizing the same mocked **out* for some reason. Is this the expected behaviour?
The problem is that the *out* you care about is a thread-local, not the root value. Try:
(with-redefs [x 11]
(binding [*out* (java.io.StringWriter.)]
(print "one"))
(binding [*out* (java.io.StringWriter.)]
(print "two")
(.toString *out*)))
The with-redefs macro only alters the root binding; any thread-local bindings are not affected. When code is executed by the REPL server, it is done so in a context that re-binds *out* to a client-specific writer, so that output can be captured and sent back to the client for display (otherwise it would go to the server's stdout). The original code would probably work if run from the command-line instead of the REPL.
Related
I'm experimenting with core.async on Clojure and ClojureScript, to try and understand how merge works. In particular, whether merge makes any values put on input channels available to take immediately on the merged channel.
I have the following code:
(ns async-merge-example.core
(:require
#?(:clj [clojure.core.async :as async] :cljs [cljs.core.async :as async])
[async-merge-example.exec :as exec]))
(defn async-fn-timeout
[v]
(async/go
(async/<! (async/timeout (rand-int 5000)))
v))
(defn async-fn-exec
[v]
(exec/exec "sh" "-c" (str "sleep " (rand-int 5) "; echo " v ";")))
(defn merge-and-print-results
[seq async-fn]
(let [chans (async/merge (map async-fn seq))]
(async/go
(while (when-let [v (async/<! chans)]
(prn v)
v)))))
When I try async-fn-timeout with a large-ish seq:
(merge-and-print-results (range 20) async-fn-timeout)
For both Clojure and ClojureScript I get the result I expect, as in, results start getting printed pretty much immediately, with the expected delays.
However, when I try async-fn-exec with the same seq:
(merge-and-print-results (range 20) async-fn-exec)
For ClojureScript, I get the result I expect, as in results start getting printed pretty much immediately, with the expected delays. However for Clojure even though the sh processes are executed concurrently (subject to the size of the core.async thread pool), the results appear to be initially delayed, then mostly printed all at once! I can make this difference more obvious by increasing the size of the seq e.g. (range 40)
Since the results for async-fn-timeout are as expected on both Clojure and ClojureScript, the finger is pointed at the differences between the Clojure and ClojureScript implementation for exec..
But I don't know why this difference would cause this issue?
Notes:
These observations were made in WSL on Windows 10
The source code for async-merge-example.exec is below
In exec, the implementation differs for Clojure and ClojureScript due to differences between Clojure/Java and ClojureScript/NodeJS.
(ns async-merge-example.exec
(:require
#?(:clj [clojure.core.async :as async] :cljs [cljs.core.async :as async])))
; cljs implementation based on https://gist.github.com/frankhenderson/d60471e64faec9e2158c
; clj implementation based on https://stackoverflow.com/questions/45292625/how-to-perform-non-blocking-reading-stdout-from-a-subprocess-in-clojure
#?(:cljs (def spawn (.-spawn (js/require "child_process"))))
#?(:cljs
(defn exec-chan
"spawns a child process for cmd with args. routes stdout, stderr, and
the exit code to a channel. returns the channel immediately."
[cmd args]
(let [c (async/chan), p (spawn cmd (if args (clj->js args) (clj->js [])))]
(.on (.-stdout p) "data" #(async/put! c [:out (str %)]))
(.on (.-stderr p) "data" #(async/put! c [:err (str %)]))
(.on p "close" #(async/put! c [:exit (str %)]))
c)))
#?(:clj
(defn exec-chan
"spawns a child process for cmd with args. routes stdout, stderr, and
the exit code to a channel. returns the channel immediately."
[cmd args]
(let [c (async/chan)]
(async/go
(let [builder (ProcessBuilder. (into-array String (cons cmd (map str args))))
process (.start builder)]
(with-open [reader (clojure.java.io/reader (.getInputStream process))
err-reader (clojure.java.io/reader (.getErrorStream process))]
(loop []
(let [line (.readLine ^java.io.BufferedReader reader)
err (.readLine ^java.io.BufferedReader err-reader)]
(if (or line err)
(do (when line (async/>! c [:out line]))
(when err (async/>! c [:err err]))
(recur))
(do
(.waitFor process)
(async/>! c [:exit (.exitValue process)]))))))))
c)))
(defn exec
"executes cmd with args. returns a channel immediately which
will eventually receive a result map of
{:out [stdout-lines] :err [stderr-lines] :exit [exit-code]}"
[cmd & args]
(let [c (exec-chan cmd args)]
(async/go (loop [output (async/<! c) result {}]
(if (= :exit (first output))
(assoc result :exit (second output))
(recur (async/<! c) (update result (first output) #(conj (or % []) (second output)))))))))
Your Clojure implementation uses blocking IO in a single thread. You are first reading from stdout and then stderr in a loop. Both do a blocking readLine so they will only return once they actually finished reading a line. So unless your process creates the same amount of output to stdout and stderr one stream will end up blocking the other one.
Once the process is finished the readLine will no longer block and just return nil once the buffer is empty. So the loop just finishes reading the buffered output and then finally completes explaining the "all at once" messages.
You'll probably want to start a second thread that deals reading from stderr.
node does not do blocking IO so everything happens async by default and one stream doesn't block the other.
I'm trying to write unit tests when using core.async go macros. Writing the test naively, as follows, it appears that the code inside the go blocks don't get executed.
(ns app.core-test
(:require [clojure.test :refer :all]
[clojure.core.async :as async]))
(deftest test1 []
(let [chan (async/chan)]
(async/go
(is (= (async/<! chan) "Hello")))
(async/go
(async/>! chan "Hello"))))
I've managed to get the following working, but it's extremely hacky.
(deftest test1 []
(let [result (async/chan)
chan (async/chan)]
(async/go
(is (= (async/<! chan) "Hello"))
(async/>! result true))
(async/go
(async/>! chan "Hello"))
(async/alts!! [result (async/timeout 10000)])))
Any suggestions as to how I can do this properly?
Tests are executed synchronously, so if you go async the test-runner won't. In Clojure you need to block the test runner via <!!, in ClojureScript you have to return an async test object. This is a generic helper function I use in all my async CLJC tests:
(defn test-async
"Asynchronous test awaiting ch to produce a value or close."
[ch]
#?(:clj
(<!! ch)
:cljs
(async done
(take! ch (fn [_] (done))))))
Your test using it, CLJC compatible and looking way less "hacky":
(deftest test1
(let [ch (chan)]
(go (>! ch "Hello"))
(test-async
(go (is (= "Hello" (<! ch)))))))
It is good practice to assert that the test unblocks, especially during test driven development where you want to avoid locking your test runner. Also, locking is a common cause of failure in async programming, so testing against it is highly reasonable.
To do that I wrote a helper similar to your timeout thing:
(defn test-within
"Asserts that ch does not close or produce a value within ms. Returns a
channel from which the value can be taken."
[ms ch]
(go (let [t (timeout ms)
[v ch] (alts! [ch t])]
(is (not= ch t)
(str "Test should have finished within " ms "ms."))
v)))
You can use it to write your test like:
(deftest test1
(let [ch (chan)]
(go (>! ch "Hello"))
(test-async
(test-within 1000
(go (is (= "Hello" (<! ch)))))))
your test is finishing, and then failing. This happens more reliably if I put a sleep in and then make it fail:
user> (deftest test1 []
(async/<!!
(let [chan (async/chan)]
(async/go
(async/go
(async/<! (async/timeout 1000))
(is (= (async/<! chan) "WRONG")))
(async/go
(async/>! chan "Hello"))))))
#'user/test1
user> (clojure.test/run-tests)
Testing user
Ran 1 tests containing 0 assertions.
0 failures, 0 errors.
{:test 1, :pass 0, :fail 0, :error 0, :type :summary}
user>
FAIL in (test1) (form-init8563497779572341831.clj:5)
expected: (= (async/<! chan) "WRONG")
actual: (not (= "Hello" "WRONG"))
here we can see it report that nothing fails, then it prints the failure message. We can fix this by explicitly coordinating the end of the test and that action finishing by, like most solutions in core.async, adding one more chan.
user> (deftest test1 []
(async/<!!
(let [all-done-chan (async/chan)
chan (async/chan)]
(async/go
(async/go
(async/<! (async/timeout 1000))
(is (= (async/<! chan) "WRONG"))
(async/close! all-done-chan ))
(async/go
(async/>! chan "Hello"))
(async/<! all-done-chan)))))
#'user/test1
user> (clojure.test/run-tests)
Testing user
FAIL in (test1) (form-init8563497779572341831.clj:6)
expected: (= (async/<! chan) "WRONG")
actual: (not (= "Hello" "WRONG"))
Ran 1 tests containing 1 assertions.
1 failures, 0 errors.
{:test 1, :pass 0, :fail 1, :error 0, :type :summary}
Which is equivalent to your solution using alts. I don't think your solution is hackey. With asynchronous code it's always required to pay attention to when things finish, even if you conciously decide to ignore the result.
I'm using an approach similar to Leon's, but with no extra go blocks:
(defn <!!?
"Reads from chan synchronously, waiting for a given maximum of milliseconds.
If the value does not come in during that period, returns :timed-out. If
milliseconds is not given, a default of 1000 is used."
([chan]
(<!!? chan 1000))
([chan milliseconds]
(let [timeout (async/timeout milliseconds)
[value port] (async/alts!! [chan timeout])]
(if (= chan port)
value
:timed-out))))
You can use it simply as:
(is (= 42 (<!!? result-chan)))
Most of the time I just want to read the value from the channel without any extra hassle.
This piece of code runs on the server and it detects the changes to a file and sends it to the client. This is working for the first time and after that the file length is not getting updated even the I changed the file and saved it. I guess the clojure immutability is the reason here. How can I make this work?
(def clients (atom {}))
(def rfiles (atom {}))
(def file-pointers (atom {}))
(defn get-rfile [filename]
(let [rdr ((keyword filename) #rfiles)]
(if rdr
rdr
(let [rfile (RandomAccessFile. filename "rw")]
(swap! rfiles assoc (keyword filename) rfile)
rfile))))
(defn send-changes [changes]
(go (while true
(let [[op filename] (<! changes)
rfile (get-rfile filename)
ignore (println (.. rfile getChannel size))
prev ((keyword filename) #file-pointers)
start (if prev prev 0)
end (.length rfile) // file length is not getting updated even if I changed the file externally
array (byte-array (- end start))]
(do
(println (str "str" start " end" end))
(.seek rfile start)
(.readFully rfile array)
(swap! file-pointers assoc (keyword filename) end)
(doseq [client #clients]
(send! (key client) (json/write-str
{:changes (apply str (map char array))
:fileName filename}))
false))))))
There is no immutability here. In the rfiles atom, you store standard Java objects that are mutable.
This code works well only if data are appended to the end of the file, and the size is always increasing.
If there is an update/addition (of length +N) in the file other than at the end, the pointers start and end won't point to the modified data, but just to the last N characters and you will send dummy stuff to the clients.
If there is a delete or any change that decrease the length,
array (byte-array (- end start))
will throw a NegativeArraySizeException you don't see (eaten by the go bloc?). You can add some (try (...) catch (...)) or test that (- end start) is alway positive or null, to manage it and do the appropriate behaviour: resetting the pointers?,...
Are you sure the files you scan for changes are only changed by appending data? If not, you need to handle this case by resetting or updating the pointers accordingly.
I hope it will help.
EDIT test environment.
I defined the following. There is no change to the code you provided.
;; define the changes channel
(def notif-chan (chan))
;; define some clients
(def clients (atom {:foo "foo" :bar "bar"}))
;; helper function to post a notif of change in the channel
(defn notify-a-change [op filename]
(go (>! notif-chan [op filename])))
;; mock of the send! function used in send-changes
(defn send! [client message]
(println client message))
;; main loop
(defn -main [& args]
(send-changes notif-chan))
in a repl, I ran:
repl> (-main)
in a shell (I tested with an editor too):
sh> echo 'hello there' >> ./foo.txt
in the repl:
repl> (notify-a-change "x" "./foo.txt")
str0 end12
:bar {"changes":"hello there\n","fileName":".\/foo.txt"}
:foo {"changes":"hello there\n","fileName":".\/foo.txt"}
repl> (notify-a-change "x" "./foo.txt")
str12 end12
:bar {"changes":"","fileName":".\/foo.txt"}
:foo {"changes":"","fileName":".\/foo.txt"}
in a shell:
sh> echo 'bye bye' >> ./foo.txt
in a repl:
repl> (notify-a-change "x" "./foo.txt")
str12 end20
:bar {"changes":"bye bye\n","fileName":".\/foo.txt"}
:foo {"changes":"bye bye\n","fileName":".\/foo.txt"}
I have a Ring handler which uses several functions to build the response. If any of these functions throw an exception, it should be caught so a custom response body can be written before returning a 500.
I'm writing the unit test for the handler, and I want to ensure that an exception thrown by any of these functions will be handled as described above. My instinct was to use with-redefs inside a doseq:
(doseq [f [ns1/fn1 ns1/fn2 ns2/fn1]]
(with-redefs [f (fn [& args] (throw (RuntimeException. "fail!"))]
(let [resp (app (request :get "/foo")))))]
(is (= (:status resp) 500))
(is (= (:body resp) "Something went wrong")))))
Of course, given that with-redefs wants to change the root bindings of vars, it is treating f as a symbol. Is there any way to get it to rebind the var referred to by f? I suspect that I'm going to need a macro to accomplish this, but I'm hoping that someone can come up with a clever solution.
with-redefs is just sugar over repeated calls to alter-var-root, so you can just write the desugared form yourself, something like:
(doseq [v [#'ns1/fn1 #'ns1/fn2 #'ns2/fn1]]
(let [old #v]
(try
(alter-var-root v (constantly (fn [& args]
(throw (RuntimeException. "fail!")))))
(let [resp (app (request :get "/foo")))
(is (= (:status resp) 500))
(is (= (:body resp) "Something went wrong")))
(finally (alter-var-root v (constantly old))))))
Following on from amalloy's great answer, here's a utility function that I wrote to use in my tests:
(defn with-mocks
"Iterates through a list of functions to-mock, rebinding each to mock-fn, and calls
test-fn with the optional test-args.
Example:
(defn foobar [a b]
(try (+ (foo a) (bar b))
(catch Exception _ 1)))
(deftest test-foobar
(testing \"Exceptions are handled\"
(with-mocks
[#'foo #'bar]
(fn [& _] (throw (RuntimeException. \"\")))
(fn [a b] (is (= 1 (foobar a b)))) 1 2)))"
[to-mock mock-fn test-fn & test-args]
(doseq [f to-mock]
(let [real-fn #f]
(try
(alter-var-root f (constantly mock-fn))
(apply test-fn test-args)
(finally
(alter-var-root f (constantly real-fn)))))))
I'm new to Clojure, and am trying to redirect output to a file by rebinding *out*. In a simple case, it works nicely:
(binding [*out* (new java.io.FileWriter "test.txt")]
(println "Hi"))
This does what I expect, printing "Hi" to the file test.txt. However, if I introduce a for loop, things go awry:
(binding [*out* (new java.io.FileWriter "test.txt")]
(for [x [1 2]]
(println "Hi" x)))
This time, all the output goes to stdout, and the file is empty. What's going on here?
I'm using Leiningen, if that makes any difference:
Leiningen 2.0.0 on Java 1.7.0_13 Java HotSpot(TM) 64-Bit Server VM
you have been bitten by the lazy bug.
put a doall or dorun around the for and within the binding
(binding [*out* (new java.io.FileWriter "test.txt")]
(doall (for [x [1 2]]
(println "Hi" x))))
In your example the printing is happening then the result is printed by the repl after it is returned from the binding. So at the time of printing the binding is no longer in place.
Nothing is printed because result is a lazy sequence that will later be evaluated when used
user> (def result (binding [*out* (new java.io.FileWriter "test.txt")]
(for [x [1 2]]
(println "Hi" x))))
#'user/result
When the repl prints the resuls the printlns are evaluated:
user> result
(Hi 1
Hi 2
nil nil)
If we force the evaluation of the lazy sequence returned by for within the binding
nothing is printed to the repl,
user> (def result (binding [*out* (new java.io.FileWriter "test.txt")]
(doall (for [x [1 2]]
(println "Hi" x)))))
#'user/result
user> result
(nil nil)
and instead the output ends up in the file:
arthur#a:~/hello$ cat test.txt
Hi 1
Hi 2