What would be an idiomatic way of executing a function within a time limit? Something like,
(with-timeout 5000
(do-somthing))
Unless do-something returns within 5000 throw an exception or return nil.
EDIT: before someone points it out there is,
clojure (with-timeout ... macro)
but with that the future keeps executing that does not work in my case.
I think you can do this reasonably reliably by using the timeout capability within futures:
(defmacro with-timeout [millis & body]
`(let [future# (future ~#body)]
(try
(.get future# ~millis java.util.concurrent.TimeUnit/MILLISECONDS)
(catch java.util.concurrent.TimeoutException x#
(do
(future-cancel future#)
nil)))))
A bit of experimenting verified that you need to do a future-cancel to stop the future thread from continuing to execute....
What about?
(defn timeout [timeout-ms callback]
(let [fut (future (callback))
ret (deref fut timeout-ms ::timed-out)]
(when (= ret ::timed-out)
(future-cancel fut))
ret))
(timeout 100 #(Thread/sleep 1000))
;=> :user/timed-out
This isn't something you can do 100% reliably on the JVM. The only way to stop something after a while is to give it a new thread, and then send that thread an exception when you want it to stop. But their code can catch the exception, or they can spin up another thread that you don't control, or...
But most of the time, and especially if you control the code that's being timed out, you can do something like we do in clojail:
If you wanted to make that prettier you could define a macro like
(defmacro with-timeout [time & body]
`(thunk-timeout (fn [] ~#body) ~time))
It's a quite a breeze using clojure's channel facilities
https://github.com/clojure/core.async
require respective namespace
(:require [clojure.core.async :refer [>! alts!! timeout chan go]])
the function wait takes a timeout [ms], a function [f] and optional parameters [args]
(defn wait [ms f & args]
(let [c (chan)]
(go (>! c (apply f args)))
(first (alts!! [c (timeout ms)]))))
third line pops off the call to f to another thread. fourth line consumes the result of the function call or (if faster) the timeout.
consider the following example calls
(wait 1000 (fn [] (do (Thread/sleep 100) 2)))
=> 2
but
(wait 50 (fn [] (do (Thread/sleep 100) 2)))
=> nil
You can probably use an agent, and then await-for it.
Adding a possible (macro-less) alternative to the mix (though the macro isn't required in the accepted answer of course)
(defn with-timeout [f ms]
(let [p (promise)
h (future
(deliver p (f)))
t (future
(Thread/sleep ms)
(future-cancel h)
(deliver p nil))]
#p))
Requires two threads, but just an idea.
Related
I have a requirement for a function that when called with particular input args executes a supplied function g, but only after another supplied function f has finished executing with the same input args. There is also a requirement that when the function is called multiple times with the same input args, f is only executed once on the first call, and the other calls wait for this to complete, then execute g directly.
Edit: The solution should work when run in parallel on different threads, and should also use threads efficiently. E.g. blocking should be on a per input basis rather than the whole function.
My first attempt at the function is as follows:
(defn dependent-func
([f g]
(let [mem (atom {})]
(fn [& args]
(->> (get (locking mem
(swap! mem (fn [latch-map args]
(if (contains? latch-map args)
latch-map
(let [new-latch (CountDownLatch. 1)
new-latch-map (assoc latch-map args new-latch)]
(->> (Thread. #(do (apply f args)
(.countDown new-latch)))
(.start))
new-latch-map))) args)) args)
(.await))
(apply g args)))))
This appears to meet my requirements, and awaits on f are on a per input basis, so I'm relatively happy with that. Initially I had hoped to just use swap! to do the mem updating but unfortunately swap! explicitly states that the function in the swap! could be called multiple times (I have seen this in testing). As a result of this I ended up having to lock on mem when updating which is really ugly.
I am sure there must be a cleaner way of doing this that leverages Closure's concurrency mechanisms better than I have, but so far I've been unable to find it.
Any advice would be greatly appreciated.
Thanks,
Matt.
Clojure's combination of future, promise, and deliver is well suited to starting a process and have several threads wait for it to finish.
Future is used to start a thread in the background (it can do more, though in this example I didn't need it to)
Promise is used to immediately return an object that will contain the answer once it is ready.
Deliver is used to supply the promised answer once it is ready.
I'll also split the waiting part into it's own function to make the code easier to follow, and so I can use the built in memoize function:
This question is a very good example of when to use promise and deliver rather than simply a future.
Because we are going to use memoize where it's not safe to run the function twice,
we need to be careful that the two calls don't enter memoize at exactly the same
time. so we are going to lock only the moment we enter memoize, not the duration
of the memoized function.
hello.core> (def lock [])
#'hello.core/lock
this function will always return the same future Object for every time f is called
with a given set of arguments, except we need to make memoize safe by wrapping this
in a function that does the locking (you could also use an agent for this)
hello.core> (def wait-for-function-helper
(memoize (fn [f args]
(let [answer (promise)]
(println "waiting for function " f " with args" args)
(future (deliver answer (apply f args)))
answer))))
#'hello.core/wait-for-function-helper
hello.core> (defn wait-for-function [& args]
(locking lock
(apply wait-for-function-helper args)))
#'hello.core/wait-for-function
and now we write the actual dependent-func function that uses the safely memoized,
future producing, wait-for-function function.
hello.core> (defn dependent-func [f g & args]
#(wait-for-function f args)
(apply g args))
#'hello.core/dependent-func
and define a slow opperation to see it in action:
hello.core> (defn slow-f-1 [x]
(println "starting slow-f-1")
(Thread/sleep 10000)
(println "finishing slow-f-1")
(dec x))
#'hello.core/slow-f-1
and to test it we want to start two of the same function at exactly the same time.
hello.core> (do (future
(println "first" (dependent-func slow-f-1 inc 4)))
(future
(println "second" (dependent-func slow-f-1 inc 4))))
waiting for function
#object[clojure.core$future_call$reify__6736 0x40534083 {:status :pending, :val nil}] with args (4)
#object[hello.core$slow_f_1 0x4f9b3396 hello.core$slow_f_1#4f9b3396]
starting slow-f-1
finishing slow-f-1
second
first
5
5
and if we call it again we see that slow-f-1 only ever ran once:
hello.core> (do (future
(println "first" (dependent-func slow-f-1 inc 4)))
(future
(println "second" (dependent-func slow-f-1 inc 4))))
#object[clojure.core$future_call$reify__6736 0x3935ea29 {:status :pending, :val nil}]
first 5
second 5
Something like this is a much simpler answer to your question:
(defn waiter
[f g & args]
(let [f-result (f args)
g-result (g args) ]
(println (format "waiter: f-result=%d g-result=%d" f-result g-result))))
(defn my-f
[args]
(let [result (apply + args)]
(println "my-f running:" result)
result))
; change your orig prob a bit, and define/use my-f-memo instead of the original my-f
(def my-f-memo (memoize my-f))
(defn my-g
[args]
(let [result (apply * args)]
(println "my-g running:" result)
result))
(waiter my-f-memo my-g 2 3 4)
(waiter my-f-memo my-g 2 3 4)
> lein run
my-f running: 9
my-g running: 24
waiter: f-result=9 g-result=24
my-g running: 24
waiter: f-result=9 g-result=24
main - enter
If you change the problem statement a bit and pass in a memoized version of your first function f, the solution is much easier.
Just calling the functions in sequence in a (let [...]...) form enforces the completion of the first before the execution of the 2nd function.
Also, you could force the waiter function to do the memoization of f for you, but it would be a bit more work to manually simulate what memoize already does.
Update: The original problem didn't explicitly imply it needed to work in a concurrent environment. If multiple threads are an issue, just change the definition of waiter to be:
(defn waiter
[f g & args]
(let [f-result (locking f (f args))
g-result (g args) ]
(println (format "waiter: f-result=%d g-result=%d" f-result g-result))))
There is little purpose to starting up a thread to run f on, if the very next thing you will do is wait for that thread to complete. You might as well just run f on the current thread. In that case, your problem decomposes nicely into two subproblems:
How to memoize the call to f without risking concurrent execution like the standard memoize does.
Returning a lambda that uses that memoized function and then calls g.
Let's solve these in reverse order, by first assuming (my-memoize f) works as you need it to, and then later writing it:
(defn dependent-func [f g]
(let [f' (my-memoize f)]
(fn [& args]
(apply f' args)
(apply g args))))
Very simple with a competent memoize, right? Now, to implement memoize there are a few things you can do. You could use locking, as you did, and I think that's pretty reasonable, since you explicitly want to prevent concurrent execution; once you throw out the thread-launching business it is very easy as well:
(defn my-memoize [f]
(let [memo (atom {})]
(fn [& args]
(locking memo
(if (contains? #memo args)
(get #memo args)
(get (swap! memo assoc args (apply f args))))))))
Or you can reinvent locking yourself, by storing a delay in the atom and then having each call dereference it instead:
(defn my-memoize [f]
(let [memo (atom {})]
(fn [& args]
(-> memo
(swap! update-in [args]
(fn [v]
(or v (delay (apply f args)))))
(get args)
(deref)))))
It's readable and "clever", because it does everything in a swap!, and I felt quite smug back when I figured this out the first time, but later I realized that this is just hijacking the mutex in Delay.deref() to accomplish locking, so honestly I think you might as well just use locking to make it clear there is a lock.
I'd like to use memoize for a function that uses core.async and <!! e.g
(defn foo [x]
(go
(<!! (timeout 2000))
(* 2 x)))
(In the real-life, it could be useful in order to cache the results of server calls)
I was able to achieve that by writing a core.async version of memoize (almost the same code as memoize):
(defn memoize-async [f]
(let [mem (atom {})]
(fn [& args]
(go
(if-let [e (find #mem args)]
(val e)
(let [ret (<! (apply f args))]; this line differs from memoize [ret (apply f args)]
(swap! mem assoc args ret)
ret))))))
Example of usage:
(def foo-memo (memoize-async foo))
(go (println (<!! (foo-memo 3)))); delay because of (<!! (timeout 2000))
(go (println (<!! (foo-memo 3)))); subsequent calls are memoized => no delay
I am wondering if there are simpler ways to achieve the same result.
Remark: I need a solution that works with <!!. For <!, see this question: How to memoize a function that uses core.async and non-blocking channel read?
You can use the built in memoize function for this. Start by defining a method that reads from a channel and returns the value:
(defn wait-for [ch]
(<!! ch))
Note that we'll use <!! and not <! because we want this function block until there is data on the channel in all cases. <! only exhibits this behavior when used in a form inside of a go block.
You can then construct your memoized function by composing this function with foo, like such:
(def foo-memo (memoize (comp wait-for foo)))
foo returns a channel, so wait-for will block until that channel has a value (i.e. until the operation inside foo finished).
foo-memo can be used similar to your example above, except you do not need the call to <!! because wait-for will block for you:
(go (println (foo-memo 3))
You can also call this outside of a go block, and it will behave like you expect (i.e. block the calling thread until foo returns).
I'd like to use memoize for a function that uses core.async and <! e.g
(defn foo [x]
(go
(<! (timeout 2000))
(* 2 x)))
(In the real-life, it could be useful in order to cache the results of server calls)
I was able to achieve that by writing a core.async version of memoize (almost the same code as memoize):
(defn memoize-async [f]
(let [mem (atom {})]
(fn [& args]
(go
(if-let [e (find #mem args)]
(val e)
(let [ret (<! (apply f args))]; this line differs from memoize [ret (apply f args)]
(swap! mem assoc args ret)
ret))))))
Example of usage:
(def foo-memo (memoize-async foo))
(go (println (<! (foo-memo 3)))); delay because of (<! (timeout 2000))
(go (println (<! (foo-memo 3)))); subsequent calls are memoized => no delay
I am wondering if there are simpler ways to achieve the same result.
**Remark: I need a solution that works with <!. For <!!, see this question: How to memoize a function that uses core.async and blocking channel read? **
You can use the built in memoize function for this. Start by defining a method that reads from a channel and returns the value:
(defn wait-for [ch]
(<!! ch))
Note that we'll use <!! and not <! because we want this function block until there is data on the channel in all cases. <! only exhibits this behavior when used in a form inside of a go block.
You can then construct your memoized function by composing this function with foo, like such:
(def foo-memo (memoize (comp wait-for foo)))
foo returns a channel, so wait-for will block until that channel has a value (i.e. until the operation inside foo finished).
foo-memo can be used similar to your example above, except you do not need the call to <! because wait-for will block for you:
(go (println (foo-memo 3))
You can also call this outside of a go block, and it will behave like you expect (i.e. block the calling thread until foo returns).
This was a little trickier than I expected. Your solution isn't correct, because when you call your memoized function again with the same arguments, sooner than the first run finishes running its go block, you will trigger it again and get a miss. This is often the case when you process lists with core.async.
The one below uses core.async's pub/sub to solve this (tested in CLJS only):
(def lookup-sentinel #?(:clj ::not-found :cljs (js-obj))
(def pending-sentinel #?(:clj ::pending :cljs (js-obj))
(defn memoize-async
[f]
(let [>in (chan)
pending (pub >in :args)
mem (atom {})]
(letfn
[(memoized [& args]
(go
(let [v (get #mem args lookup-sentinel)]
(condp identical? v
lookup-sentinel
(do
(swap! mem assoc args pending-sentinel)
(go
(let [ret (<! (apply f args))]
(swap! mem assoc args ret)
(put! >in {:args args :ret ret})))
(<! (apply memoized args)))
pending-sentinel
(let [<out (chan 1)]
(sub pending args <out)
(:ret (<! <out)))
v))))]
memoized)))
NOTE: it probably leaks memory, subscriptions and <out channels are not closed
I have used this function in one of my projects to cache HTTP calls. The function caches results for a given amount of time and uses a barrier to prevent executing the function multiple times when the cache is "cold" (due to the context switch inside the go block).
(defn memoize-af-until
[af ms clock]
(let [barrier (async/chan 1)
last-return (volatile! nil)
last-return-ms (volatile! nil)]
(fn [& args]
(async/go
(>! barrier :token)
(let [now-ms (.now clock)]
(when (or (not #last-return-ms) (< #last-return-ms (- now-ms ms)))
(vreset! last-return (<! (apply af args)))
(vreset! last-return-ms now-ms))
(<! barrier)
#last-return)))))
You can test that it works properly by setting the cache time to 0 and observe that the two function calls take approximately 10 seconds. Without the barrier the two calls would finish at the same time:
(def memo (memoize-af-until #(async/timeout 5000) 0 js/Date))
(async/take! (memo) #(println "[:a] Finished"))
(async/take! (memo) #(println "[:b] Finished"))
I'm looking for a very simple way to call a function periodically in Clojure.
JavaScript's setInterval has the kind of API I'd like. If I reimagined it in Clojure, it'd look something like this:
(def job (set-interval my-callback 1000))
; some time later...
(clear-interval job)
For my purposes I don't mind if this creates a new thread, runs in a thread pool or something else. It's not critical that the timing is exact either. In fact, the period provided (in milliseconds) can just be a delay between the end of one call completing and the commencement of the next.
If you want very simple
(defn set-interval [callback ms]
(future (while true (do (Thread/sleep ms) (callback)))))
(def job (set-interval #(println "hello") 1000))
=>hello
hello
...
(future-cancel job)
=>true
Good-bye.
There's also quite a few scheduling libraries for Clojure:
(from simple to very advanced)
at-at
chime (core.async integration)
monotony
quartzite
Straight from the examples of the github homepage of at-at:
(use 'overtone.at-at)
(def my-pool (mk-pool))
(let [schedule (every 1000 #(println "I am cool!") my-pool)]
(do stuff while schedule runs)
(stop schedule))
Use (every 1000 #(println "I am cool!") my-pool :fixed-delay true) if you want a delay of a second between end of task and start of next, instead of between two starts.
This is how I would do the core.async version with stop channel.
(defn set-interval
[f time-in-ms]
(let [stop (chan)]
(go-loop []
(alt!
(timeout time-in-ms) (do (<! (thread (f)))
(recur))
stop :stop))
stop))
And the usage
(def job (set-interval #(println "Howdy") 2000))
; Howdy
; Howdy
(close! job)
The simplest approach would be to just have a loop in a separate thread.
(defn periodically
[f interval]
(doto (Thread.
#(try
(while (not (.isInterrupted (Thread/currentThread)))
(Thread/sleep interval)
(f))
(catch InterruptedException _)))
(.start)))
You can cancel execution using Thread.interrupt():
(def t (periodically #(println "Hello!") 1000))
;; prints "Hello!" every second
(.interrupt t)
You could even just use future to wrap the loop and future-cancel to stop it.
I took a stab at coding this up, with a slightly modified interface than specified in the original question. Here's what I came up with.
(defn periodically [fn millis]
"Calls fn every millis. Returns a function that stops the loop."
(let [p (promise)]
(future
(while
(= (deref p millis "timeout") "timeout")
(fn)))
#(deliver p "cancel")))
Feedback welcomed.
Another option would be to use java.util.Timer's scheduleAtFixedRate method
edit - multiplex tasks on a single timer, and stop a single task rather than the entire timer
(defn ->timer [] (java.util.Timer.))
(defn fixed-rate
([f per] (fixed-rate f (->timer) 0 per))
([f timer per] (fixed-rate f timer 0 per))
([f timer dlay per]
(let [tt (proxy [java.util.TimerTask] [] (run [] (f)))]
(.scheduleAtFixedRate timer tt dlay per)
#(.cancel tt))))
;; Example
(let [t (->timer)
job1 (fixed-rate #(println "A") t 1000)
job2 (fixed-rate #(println "B") t 2000)
job3 (fixed-rate #(println "C") t 3000)]
(Thread/sleep 10000)
(job3) ;; stop printing C
(Thread/sleep 10000)
(job2) ;; stop printing B
(Thread/sleep 10000)
(job1))
Using core.async
(ns your-namespace
(:require [clojure.core.async :as async :refer [<! timeout chan go]])
)
(def milisecs-to-wait 1000)
(defn what-you-want-to-do []
(println "working"))
(def the-condition (atom true))
(defn evaluate-condition []
#the-condition)
(defn stop-periodic-function []
(reset! the-condition false )
)
(go
(while (evaluate-condition)
(<! (timeout milisecs-to-wait))
(what-you-want-to-do)))
I have a number of (unevaluated) expressions held in a vector; [ expr1 expr2 expr3 ... ]
What I wish to do is hand each expression to a separate thread and wait until one returns a value. At that point I'm not interested in the results from the other threads and would like to cancel them to save CPU resource.
( I realise that this could cause non-determinism in that different runs of the program might cause different expressions to be evaluated first. I have this in hand. )
Is there a standard / idiomatic way of achieving the above?
Here's my take on it.
Basically you have to resolve a global promise inside each of your futures, then return a vector containing future list and the resolved value and then cancel all the futures in the list:
(defn run-and-cancel [& expr]
(let [p (promise)
run-futures (fn [& expr] [(doall (map #(future (deliver p (eval %1))) expr)) #p])
[fs res] (apply run-futures expr)]
(map future-cancel fs)
res))
It's not reached an official release yet, but core.async looks like it might be an interesting way of solving your problem - and other asynchronous problems, very neatly.
The leiningen incantation for core.async is (currently) as follows:
[org.clojure/core.async "0.1.0-SNAPSHOT"]
And here's some code to make a function that will take a number of time-consuming functions, and block until one of them returns.
(require '[clojure.core.async :refer [>!! chan alts!! thread]]))
(defn return-first [& ops]
(let [v (map vector ops (repeatedly chan))]
(doseq [[op c] v]
(thread (>!! c (op))))
(let [[value channel] (alts!! (map second v))]
value)))
;; Make sure the function returns what we expect with a simple Thread/sleep
(assert (= (return-first (fn [] (Thread/sleep 3000) 3000)
(fn [] (Thread/sleep 2000) 2000)
(fn [] (Thread/sleep 5000) 5000))
2000))
In the sample above:
chan creates an asynchronous channel
>!! puts a value onto a channel
thread executes the body in another thread
alts!! takes a vector of channels, and returns when a value appears on any of them
There's way more to it than this, and I'm still getting my head round it, but there's a walkthrough here: https://github.com/clojure/core.async/blob/master/examples/walkthrough.clj
And David Nolen's blog has some great, if mind-boggling, posts on it (http://swannodette.github.io/)
Edit
Just seen that Michał Marczyk has answered a very similar question, but better, here, and it allows you to cancel/short-circuit.
with Clojure threading long running processes and comparing their returns
What you want is Java's CompletionService. I don't know of any wrapper around this in clojure, but it wouldn't be hard to do with interop. The example below is loosely based around the example on the JavaDoc page for the ExecutorCompletionService.
(defn f [col]
(let [cs (ExecutorCompletionService. (Executors/newCachedThreadPool))
futures (map #(.submit cs %) col)
result (.get (.take cs))]
(map #(.cancel % true) futures)
result))
You could use future-call to get a list of all futures, storing them in an Atom. then, compose each running future with a "shoot the other ones in the head" function so the first one will terminate all the remaining ones. Here is an example:
(defn first-out [& fns]
(let [fs (atom [])
terminate (fn [] (println "cancling..") (doall (map future-cancel #fs)))]
(reset! fs (doall (map (fn [x] (future-call #((x) (terminate)))) fns)))))
(defn wait-for [n s]
(fn [] (print "start...") (flush) (Thread/sleep n) (print s) (flush)))
(first-out (wait-for 1000 "long") (wait-for 500 "short"))
Edit
Just noticed that the previous code does not return the first results, so it is mainly useful for side-effects. here is another version that returns the first result using a promise:
(defn first-out [& fns]
(let [fs (atom [])
ret (promise)
terminate (fn [x] (println "cancling.." )
(doall (map future-cancel #fs))
(deliver ret x))]
(reset! fs (doall (map (fn [x] (future-call #(terminate (x)))) fns)))
#ret))
(defn wait-for [n s]
"this time, return the value"
(fn [] (print "start...") (flush) (Thread/sleep n) (print s) (flush) s))
(first-out (wait-for 1000 "long") (wait-for 500 "short"))
While I don't know if there is an idiomatic way to achieve your goal but Clojure Future looks like a good fit.
Takes a body of expressions and yields a future object that will
invoke the body in another thread, and will cache the result and
return it on all subsequent calls to deref/#. If the computation has
not yet finished, calls to deref/# will block, unless the variant of
deref with timeout is used.