Suppose I use the following code to ensure my code times out:
(defmacro with-timeout [millis & body]
`(let [future# (future ~#body)]
(try
(.get future# ~millis java.util.concurrent.TimeUnit/MILLISECONDS)
(catch java.util.concurrent.TimeoutException x#
(do
(future-cancel future#)
nil)))))
Now I want to make it retry 3 times (if it times out) after a 5 second interval. Do I do it on the TimeoutException - or do I compose it some other way?
My question is: How to execute a function with retries on timeout?
deref has a variant that accepts a timeout. Using this we can implement with-timeout without Java interop.
(defmacro with-timeout [millis & body]
`(loop [tries# 3]
(if (pos? tries#)
(let [future# (future ~#body)
result# (deref future# ~millis ::timeout)]
(if (= result# ::timeout)
(do (future-cancel future#)
(recur (dec tries#)))
result#)))))
(This is not a 100% 'hygienic' because the ::timeout sentinel value could theoretically clash with the value returned by the body forms.)
Here is one solution using core.async. It is quite naive as it does not deal with error handling. I am also not sure if it is exiting the go call cleanly on timeout.
(ns async
(:require [clojure.core.async :as a]))
(defmacro with-timeout [millis & body]
`(loop [tries# 0]
(if (> tries# 3)
(throw (Exception. "Timed out. (3)"))
(let [result# (a/alt!!
(a/go ~#body) ([v# ch] v#)
(a/timeout ~millis) :async/timed-out)]
(if (= result# :async/timed-out)
(recur (inc tries#))
result#)))))
Related
If any step in my test setup fails, I want to report this as a failure, and cease any subsequent tests in the current deftest block (or current namespace). One way to do this now:
(if some-condition-is-ok
(do
... do tests)
(is (= 1 0) "Failure, condition not met")
The above:
Reports failure if some-condition-is-ok is not met
Does not run any tests, since the setup condition was not met
Except that it does not flow well, and does not work well for multiple conditions. I'd like something like:
(let [;; setup here...]
(assert-or-stop-tests some-condition-is-ok)
... continue with tests here
Any ideas on a clean way to do this?
You could use Mark Engelberg's better-cond for this:
(require '[better-cond.core :as b]
'[clojure.test :refer [is]])
(def some-condition-is-ok true)
(def some-other-condition-is-ok false)
(deftest a-test
(b/cond
:let [#_"setup here..."]
:when (is some-condition-is-ok)
:let [_ (is (= 0 1))]
:when (is some-other-condition-is-ok)
:let [_ (is (= 1 2))]))
Or if you want to avoid the :let [_ ,,,], you could define your own macro:
(defmacro ceasing [& exprs]
(when-let [[left & [right & less :as more]] (seq exprs)]
(if (= :assert left)
`(when (is ~right)
(ceasing ~#less))
`(do
~left
(ceasing ~#more)))))
(deftest b-test
(let [#_"setup here..."]
(ceasing
:assert some-condition-is-ok
(is (= 0 1))
:assert some-other-condition-is-ok
(is (= 1 2)))))
When I evaluate the following core.async clojurescript code I get an error: "Uncaught Error: <! used not in (go ...) block"
(let [chans [(chan)]]
(go
(doall (for [c chans]
(let [x (<! c)]
x)))))
What am I doing wrong here? It definitely looks like the <! is in the go block.
because go blocks can't cross function boundaries I tend to fall back on loop/recur for a lot of these cases. the (go (loop pattern is so common that it has a short-hand form in core.async that is useful in cases like this:
user> (require '[clojure.core.async :as async])
user> (async/<!! (let [chans [(async/chan) (async/chan) (async/chan)]]
(doseq [c chans]
(async/go (async/>! c 42)))
(async/go-loop [[f & r] chans result []]
(if f
(recur r (conj result (async/<! f)))
result))))
[42 42 42]
Why dont you use alts! from Core.Async?
This function lets you listen on multiple channels and know which channel you read from on each data.
For example:
(let [chans [(chan)]]
(go
(let [[data ch] (alts! chans)]
data)))))
You can ask of the channel origin too:
...
(let [slow-chan (chan)
fast-chan (chan)
[data ch] (alts! [slow-chan fast-chan])]
(when (= ch slow-chan)
...))
From the Docs:
Completes at most one of several channel operations. Must be called
inside a (go ...) block. ports is a vector of channel endpoints,
which can be either a channel to take from or a vector of
[channel-to-put-to val-to-put], in any combination. Takes will be
made as if by !. Unless
the :priority option is true, if more than one port operation is
ready a non-deterministic choice will be made. If no operation is
ready and a :default value is supplied, [default-val :default] will
be returned, otherwise alts! will park until the first operation to
become ready completes. Returns [val port] of the completed
operation, where val is the value taken for takes, and a
boolean (true unless already closed, as per put!) for put
Doumentation ref
I'd like to use memoize for a function that uses core.async and <!! e.g
(defn foo [x]
(go
(<!! (timeout 2000))
(* 2 x)))
(In the real-life, it could be useful in order to cache the results of server calls)
I was able to achieve that by writing a core.async version of memoize (almost the same code as memoize):
(defn memoize-async [f]
(let [mem (atom {})]
(fn [& args]
(go
(if-let [e (find #mem args)]
(val e)
(let [ret (<! (apply f args))]; this line differs from memoize [ret (apply f args)]
(swap! mem assoc args ret)
ret))))))
Example of usage:
(def foo-memo (memoize-async foo))
(go (println (<!! (foo-memo 3)))); delay because of (<!! (timeout 2000))
(go (println (<!! (foo-memo 3)))); subsequent calls are memoized => no delay
I am wondering if there are simpler ways to achieve the same result.
Remark: I need a solution that works with <!!. For <!, see this question: How to memoize a function that uses core.async and non-blocking channel read?
You can use the built in memoize function for this. Start by defining a method that reads from a channel and returns the value:
(defn wait-for [ch]
(<!! ch))
Note that we'll use <!! and not <! because we want this function block until there is data on the channel in all cases. <! only exhibits this behavior when used in a form inside of a go block.
You can then construct your memoized function by composing this function with foo, like such:
(def foo-memo (memoize (comp wait-for foo)))
foo returns a channel, so wait-for will block until that channel has a value (i.e. until the operation inside foo finished).
foo-memo can be used similar to your example above, except you do not need the call to <!! because wait-for will block for you:
(go (println (foo-memo 3))
You can also call this outside of a go block, and it will behave like you expect (i.e. block the calling thread until foo returns).
I'd like to use memoize for a function that uses core.async and <! e.g
(defn foo [x]
(go
(<! (timeout 2000))
(* 2 x)))
(In the real-life, it could be useful in order to cache the results of server calls)
I was able to achieve that by writing a core.async version of memoize (almost the same code as memoize):
(defn memoize-async [f]
(let [mem (atom {})]
(fn [& args]
(go
(if-let [e (find #mem args)]
(val e)
(let [ret (<! (apply f args))]; this line differs from memoize [ret (apply f args)]
(swap! mem assoc args ret)
ret))))))
Example of usage:
(def foo-memo (memoize-async foo))
(go (println (<! (foo-memo 3)))); delay because of (<! (timeout 2000))
(go (println (<! (foo-memo 3)))); subsequent calls are memoized => no delay
I am wondering if there are simpler ways to achieve the same result.
**Remark: I need a solution that works with <!. For <!!, see this question: How to memoize a function that uses core.async and blocking channel read? **
You can use the built in memoize function for this. Start by defining a method that reads from a channel and returns the value:
(defn wait-for [ch]
(<!! ch))
Note that we'll use <!! and not <! because we want this function block until there is data on the channel in all cases. <! only exhibits this behavior when used in a form inside of a go block.
You can then construct your memoized function by composing this function with foo, like such:
(def foo-memo (memoize (comp wait-for foo)))
foo returns a channel, so wait-for will block until that channel has a value (i.e. until the operation inside foo finished).
foo-memo can be used similar to your example above, except you do not need the call to <! because wait-for will block for you:
(go (println (foo-memo 3))
You can also call this outside of a go block, and it will behave like you expect (i.e. block the calling thread until foo returns).
This was a little trickier than I expected. Your solution isn't correct, because when you call your memoized function again with the same arguments, sooner than the first run finishes running its go block, you will trigger it again and get a miss. This is often the case when you process lists with core.async.
The one below uses core.async's pub/sub to solve this (tested in CLJS only):
(def lookup-sentinel #?(:clj ::not-found :cljs (js-obj))
(def pending-sentinel #?(:clj ::pending :cljs (js-obj))
(defn memoize-async
[f]
(let [>in (chan)
pending (pub >in :args)
mem (atom {})]
(letfn
[(memoized [& args]
(go
(let [v (get #mem args lookup-sentinel)]
(condp identical? v
lookup-sentinel
(do
(swap! mem assoc args pending-sentinel)
(go
(let [ret (<! (apply f args))]
(swap! mem assoc args ret)
(put! >in {:args args :ret ret})))
(<! (apply memoized args)))
pending-sentinel
(let [<out (chan 1)]
(sub pending args <out)
(:ret (<! <out)))
v))))]
memoized)))
NOTE: it probably leaks memory, subscriptions and <out channels are not closed
I have used this function in one of my projects to cache HTTP calls. The function caches results for a given amount of time and uses a barrier to prevent executing the function multiple times when the cache is "cold" (due to the context switch inside the go block).
(defn memoize-af-until
[af ms clock]
(let [barrier (async/chan 1)
last-return (volatile! nil)
last-return-ms (volatile! nil)]
(fn [& args]
(async/go
(>! barrier :token)
(let [now-ms (.now clock)]
(when (or (not #last-return-ms) (< #last-return-ms (- now-ms ms)))
(vreset! last-return (<! (apply af args)))
(vreset! last-return-ms now-ms))
(<! barrier)
#last-return)))))
You can test that it works properly by setting the cache time to 0 and observe that the two function calls take approximately 10 seconds. Without the barrier the two calls would finish at the same time:
(def memo (memoize-af-until #(async/timeout 5000) 0 js/Date))
(async/take! (memo) #(println "[:a] Finished"))
(async/take! (memo) #(println "[:b] Finished"))
What would be an idiomatic way of executing a function within a time limit? Something like,
(with-timeout 5000
(do-somthing))
Unless do-something returns within 5000 throw an exception or return nil.
EDIT: before someone points it out there is,
clojure (with-timeout ... macro)
but with that the future keeps executing that does not work in my case.
I think you can do this reasonably reliably by using the timeout capability within futures:
(defmacro with-timeout [millis & body]
`(let [future# (future ~#body)]
(try
(.get future# ~millis java.util.concurrent.TimeUnit/MILLISECONDS)
(catch java.util.concurrent.TimeoutException x#
(do
(future-cancel future#)
nil)))))
A bit of experimenting verified that you need to do a future-cancel to stop the future thread from continuing to execute....
What about?
(defn timeout [timeout-ms callback]
(let [fut (future (callback))
ret (deref fut timeout-ms ::timed-out)]
(when (= ret ::timed-out)
(future-cancel fut))
ret))
(timeout 100 #(Thread/sleep 1000))
;=> :user/timed-out
This isn't something you can do 100% reliably on the JVM. The only way to stop something after a while is to give it a new thread, and then send that thread an exception when you want it to stop. But their code can catch the exception, or they can spin up another thread that you don't control, or...
But most of the time, and especially if you control the code that's being timed out, you can do something like we do in clojail:
If you wanted to make that prettier you could define a macro like
(defmacro with-timeout [time & body]
`(thunk-timeout (fn [] ~#body) ~time))
It's a quite a breeze using clojure's channel facilities
https://github.com/clojure/core.async
require respective namespace
(:require [clojure.core.async :refer [>! alts!! timeout chan go]])
the function wait takes a timeout [ms], a function [f] and optional parameters [args]
(defn wait [ms f & args]
(let [c (chan)]
(go (>! c (apply f args)))
(first (alts!! [c (timeout ms)]))))
third line pops off the call to f to another thread. fourth line consumes the result of the function call or (if faster) the timeout.
consider the following example calls
(wait 1000 (fn [] (do (Thread/sleep 100) 2)))
=> 2
but
(wait 50 (fn [] (do (Thread/sleep 100) 2)))
=> nil
You can probably use an agent, and then await-for it.
Adding a possible (macro-less) alternative to the mix (though the macro isn't required in the accepted answer of course)
(defn with-timeout [f ms]
(let [p (promise)
h (future
(deliver p (f)))
t (future
(Thread/sleep ms)
(future-cancel h)
(deliver p nil))]
#p))
Requires two threads, but just an idea.