I am trying to wrap a go routine inside a function (as in Method 1 below). Method 2 works perfectly fine, but Method 1 doesn't. The only difference is that in Method 1, I am passing a channel as a parameter to a function and the put is inside the function. What are the exact rules concerning go routines and functions?
(defn doit [ch i]
(print "g\n")
(async/>! ch i)
(print "f\n"))
;Method 1
(let [c1 (async/chan)]
(async/go (while true
(let [[v ch] (async/alts! [c1])]
(println "Read" v "from" ch))))
(dotimes [i 10]
(async/go (doit c1 i))))
;Method 2
(let [ch (async/chan)]
(async/go (while true
(let [[v ch] (async/alts! [ch])]
(println "Read" v "from" ch))))
(dotimes [i 10]
(async/go
(print "g\n")
(async/>! ch i)
(print "f\n"))))
I also noticed that if I remove the go in Method 1 and move it to the do it function as shown below, the function prints "g" but not "f" but otherwise works fine. Why?
(defn doit [ch i]
(async/go
(print "g\n")
(async/>! ch i)
(print "f\n")))
>! and <! are not really functions that get called (see here and here). The go macro identifies these two symbols and auto-magically generates code according to the semantics of those two operators. So this macro has no way of knowing that a function uses the >! or <! operator internally, since all it gets is the forms that calls that function.
Method 1 is actually throwing an exception on each call to doit since the actual code for >! and <! is just an assertion that always fails. Evaluating the code for this method in a REPLy session started with lein repl shows the exception Exception in thread "async-dispatch-46" java.lang.AssertionError: Assert failed: >! used not in (go ...) block a bunch of times (10 to be exact). If you are using a REPL through an nREPL client you might not be seeing this because the exception is thrown asynchronously in the server and the client is not taking this into account.
Additionally instead of using (print "something\n") you could just use (println "something"), not really related to your question but thought I mentioned this.
Related
I want to use constantly in a test to model a scenario that throws an Exception. Using the off-the-shelf constantly the body is evaluated when the code is read, not executed. I.e. I can't do this:
(def x (constantly (throw (Exception. "X"))))
(x 1 2 3)
Instead, the throw happens immediately.
This works:
(defn x [&] (throw (Exception. "X")))
But constantly is so handy and idiomatic, I wonder if there's a built-in equivalent that does this, maybe using a macro?
One alternative
#(throw (Exception. (str %&)))
constantly is a function not a macro like fn so you need to use (fn [& args]) to achieve this kind of operation.
constantly eagerly evaluates its parameters that's why it fails immediately.
It isn't built in, but it's easy to define. Let's call it defer:
(defmacro defer [exp]
(list 'fn ['& '_] exp))
Your example becomes
(def x (defer (throw (Exception. "X"))))
=> #'user/x
(x 1 2 3)
=> Exception X user/x (form-init7339591407440568822.clj:10)
This has no practical advantage over using the # reader form directly, as tap does, but it is what you asked for.
I changed the generated function to accept arguments, as the question called for. So it is no longer a thunk.
There are several ways to delay a computation in Clojure
The most obvious is delay:
(def x (delay (throw (ex-info "myException" {}))))
#x ;; exception is thrown
You could also use a lambda, similar to what would have to be done in other languages, or use laziness.
Given your code sample in the question, it looks like you are looking for something like this:
(defn x [& args]
(throw (ex-info "myException" {:args args})))
(try
(x 1 2 3)
(catch Exception e
(println "Exception! data is " (ex-data e))))
Note the use of ex-info and ex-data which could be useful to pass information.
I'm learning core.async and have written a simple producer consumer code:
(ns webcrawler.parallel
(:require [clojure.core.async :as async
:refer [>! <! >!! <!! go chan buffer close! thread alts! alts!! timeout]]))
(defn consumer
[in out f]
(go (loop [request (<! in)]
(if (nil? request)
(close! out)
(do (print f)
(let [result (f request)]
(>! out result))
(recur (<! in)))))))
(defn make-consumer [in f]
(let [out (chan)]
(consumer in out f)
out))
(defn process
[f s no-of-consumers]
(let [in (chan (count s))
consumers (repeatedly no-of-consumers #(make-consumer in f))
out (async/merge consumers)]
(map #(>!! in %1) s)
(close! in)
(loop [result (<!! out)
results '()]
(if (nil? result)
results
(recur (<!! out)
(conj results result))))))
This code works fine when I step in through the process function in debugger supplied with Emacs' cider.
(process (partial + 1) '(1 2 3 4) 1)
(5 4 3 2)
However, if I run it by itself (or hit continue in the debugger) I get an empty result.
(process (partial + 1) '(1 2 3 4) 1)
()
My guess is that in the second case for some reason producer doesn't wait for consumers before exiting, but I'm not sure why. Thanks for help!
The problem is that your call to map is lazy, and will not run until something asks for the results. Nothing does this in your code.
There are 2 solutions:
(1) Use the eager function mapv:
(mapv #(>!! in %1) items)
(2) Use the doseq, which is intended for side-effecting operations (like putting values on a channel):
(doseq [item items]
(>!! in item))
Both will work and produce output:
(process (partial + 1) [1 2 3 4] 1) => (5 4 3 2)
P.S. You have a debug statement in (defn consumer ...)
(print f)
that produces a lot of noise in the output:
<#clojure.core$partial$fn__5561 #object[clojure.core$partial$fn__5561 0x31cced7
"clojure.core$partial$fn__5561#31cced7"]>
That is repeated 5 times back to back. You probably want to avoid that, as printing function "refs" is pretty useless to a human reader.
Also, debug printouts in general should normally use println so you can see where each one begins and ends.
I'm going to take a safe stab that this is being caused by the lazy behavior of map, and this line that's carrying out side effects:
(map #(>!! in %1) s)
Because you never explicitly use the results, it never runs. Change it to use mapv, which is strict, or more correctly, use doseq. Never use map to run side effects. It's meant to lazily transform a list, and abuse of it leads to behaviour like this.
So why is it working while debugging? I'm going to guess because the debugger forces evaluation as part of its operation, which is masking the problem.
As you can read from docstring map returns a lazy sequence. And I think the best way is to use dorun. Here is an example from clojuredocs:
;;map a function which makes database calls over a vector of values
user=> (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"])
JdbcSQLException The object is already closed [90007-170] org.h2.message.DbE
xception.getJdbcSQLException (DbException.java:329)
;;database connection was closed before we got a chance to do our transactions
;;lets wrap it in dorun
user=> (dorun (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"]))
DEBUG :db insert into person values name = 'Fred'
DEBUG :db insert into person values name = 'Ethel'
DEBUG :db insert into person values name = 'Lucy'
DEBUG :db insert into person values name = 'Ricardo'
nil
When I evaluate the following core.async clojurescript code I get an error: "Uncaught Error: <! used not in (go ...) block"
(let [chans [(chan)]]
(go
(doall (for [c chans]
(let [x (<! c)]
x)))))
What am I doing wrong here? It definitely looks like the <! is in the go block.
because go blocks can't cross function boundaries I tend to fall back on loop/recur for a lot of these cases. the (go (loop pattern is so common that it has a short-hand form in core.async that is useful in cases like this:
user> (require '[clojure.core.async :as async])
user> (async/<!! (let [chans [(async/chan) (async/chan) (async/chan)]]
(doseq [c chans]
(async/go (async/>! c 42)))
(async/go-loop [[f & r] chans result []]
(if f
(recur r (conj result (async/<! f)))
result))))
[42 42 42]
Why dont you use alts! from Core.Async?
This function lets you listen on multiple channels and know which channel you read from on each data.
For example:
(let [chans [(chan)]]
(go
(let [[data ch] (alts! chans)]
data)))))
You can ask of the channel origin too:
...
(let [slow-chan (chan)
fast-chan (chan)
[data ch] (alts! [slow-chan fast-chan])]
(when (= ch slow-chan)
...))
From the Docs:
Completes at most one of several channel operations. Must be called
inside a (go ...) block. ports is a vector of channel endpoints,
which can be either a channel to take from or a vector of
[channel-to-put-to val-to-put], in any combination. Takes will be
made as if by !. Unless
the :priority option is true, if more than one port operation is
ready a non-deterministic choice will be made. If no operation is
ready and a :default value is supplied, [default-val :default] will
be returned, otherwise alts! will park until the first operation to
become ready completes. Returns [val port] of the completed
operation, where val is the value taken for takes, and a
boolean (true unless already closed, as per put!) for put
Doumentation ref
I'd like to use memoize for a function that uses core.async and <!! e.g
(defn foo [x]
(go
(<!! (timeout 2000))
(* 2 x)))
(In the real-life, it could be useful in order to cache the results of server calls)
I was able to achieve that by writing a core.async version of memoize (almost the same code as memoize):
(defn memoize-async [f]
(let [mem (atom {})]
(fn [& args]
(go
(if-let [e (find #mem args)]
(val e)
(let [ret (<! (apply f args))]; this line differs from memoize [ret (apply f args)]
(swap! mem assoc args ret)
ret))))))
Example of usage:
(def foo-memo (memoize-async foo))
(go (println (<!! (foo-memo 3)))); delay because of (<!! (timeout 2000))
(go (println (<!! (foo-memo 3)))); subsequent calls are memoized => no delay
I am wondering if there are simpler ways to achieve the same result.
Remark: I need a solution that works with <!!. For <!, see this question: How to memoize a function that uses core.async and non-blocking channel read?
You can use the built in memoize function for this. Start by defining a method that reads from a channel and returns the value:
(defn wait-for [ch]
(<!! ch))
Note that we'll use <!! and not <! because we want this function block until there is data on the channel in all cases. <! only exhibits this behavior when used in a form inside of a go block.
You can then construct your memoized function by composing this function with foo, like such:
(def foo-memo (memoize (comp wait-for foo)))
foo returns a channel, so wait-for will block until that channel has a value (i.e. until the operation inside foo finished).
foo-memo can be used similar to your example above, except you do not need the call to <!! because wait-for will block for you:
(go (println (foo-memo 3))
You can also call this outside of a go block, and it will behave like you expect (i.e. block the calling thread until foo returns).
What would be an idiomatic way of executing a function within a time limit? Something like,
(with-timeout 5000
(do-somthing))
Unless do-something returns within 5000 throw an exception or return nil.
EDIT: before someone points it out there is,
clojure (with-timeout ... macro)
but with that the future keeps executing that does not work in my case.
I think you can do this reasonably reliably by using the timeout capability within futures:
(defmacro with-timeout [millis & body]
`(let [future# (future ~#body)]
(try
(.get future# ~millis java.util.concurrent.TimeUnit/MILLISECONDS)
(catch java.util.concurrent.TimeoutException x#
(do
(future-cancel future#)
nil)))))
A bit of experimenting verified that you need to do a future-cancel to stop the future thread from continuing to execute....
What about?
(defn timeout [timeout-ms callback]
(let [fut (future (callback))
ret (deref fut timeout-ms ::timed-out)]
(when (= ret ::timed-out)
(future-cancel fut))
ret))
(timeout 100 #(Thread/sleep 1000))
;=> :user/timed-out
This isn't something you can do 100% reliably on the JVM. The only way to stop something after a while is to give it a new thread, and then send that thread an exception when you want it to stop. But their code can catch the exception, or they can spin up another thread that you don't control, or...
But most of the time, and especially if you control the code that's being timed out, you can do something like we do in clojail:
If you wanted to make that prettier you could define a macro like
(defmacro with-timeout [time & body]
`(thunk-timeout (fn [] ~#body) ~time))
It's a quite a breeze using clojure's channel facilities
https://github.com/clojure/core.async
require respective namespace
(:require [clojure.core.async :refer [>! alts!! timeout chan go]])
the function wait takes a timeout [ms], a function [f] and optional parameters [args]
(defn wait [ms f & args]
(let [c (chan)]
(go (>! c (apply f args)))
(first (alts!! [c (timeout ms)]))))
third line pops off the call to f to another thread. fourth line consumes the result of the function call or (if faster) the timeout.
consider the following example calls
(wait 1000 (fn [] (do (Thread/sleep 100) 2)))
=> 2
but
(wait 50 (fn [] (do (Thread/sleep 100) 2)))
=> nil
You can probably use an agent, and then await-for it.
Adding a possible (macro-less) alternative to the mix (though the macro isn't required in the accepted answer of course)
(defn with-timeout [f ms]
(let [p (promise)
h (future
(deliver p (f)))
t (future
(Thread/sleep ms)
(future-cancel h)
(deliver p nil))]
#p))
Requires two threads, but just an idea.