I want to use Clojure core.async to write an application that sits in a loop polling a service, but my attempts so far terminate unexpectedly.
When I run this program it prints the message then exits:
(defn -main
[]
(println "Running forever...?")
(async/go-loop [n 0]
(prn n)
(async/<!! (async/timeout 1000))
(recur (inc n))))
I would like the program to run forever (until the JVM process is killed).
What is the accepted way to achieve this?
The main thread is what will keep the JVM process running (it won't care about the threads in the go pool).
Keep it running by blocking on the main thread. e.g.
(defn -main
[]
(println "Running forever...?")
(async/<!! (async/go-loop [n 0]
(prn n)
(async/<! (async/timeout 1000))
(recur (inc n)))))
Note: you shouldn't use <!! inside of a go block. You should instead use <!.
How many ways are there to block the main thread......?
(defn -main []
(println "Running forever...?")
; go-loop runs forever in another (daemon) thread
(async/go-loop [n 0]
(prn n)
(async/<!! (async/timeout 1000))
(recur (inc n))
; hang the main thread
(Thread/sleep 111222333444)) ; long enough....
(or you could use Long/MAX_VALUE).
Related
How do you print your output into REPL whilst the script is still running? I've notice that it stores everything to buffer, and then prints once it completes the code.
(defn -main
[x]
(when (pos? x)
(println x)
(Thread/sleep 10000)
(recur (dec x))))
(-main 10)
Java (& thus Clojure) use buffered output. If you are trying to print in the middle of a tight loop, you need the flush:
(println x)
(flush)
This piece of code returns immediately:
user=> (dorun (pmap + [1 2] [3 4]))
nil
However, when I run the same piece of code in main method using lein:
(ns practice.core)
(defn -main [& args]
(dorun (pmap + [1 2] [3 4])))
why does it never return?
Interestingly, if I replace pmap by map, it returns normally.
You need to call shutdown-agents at the end of your -main method.
(defn -main [& args]
(dorun (pmap + [1 2] [3 4]))
(shutdown-agents))
This is mentioned on http://clojure.org/agents:
Note that use of Agents starts a pool of non-daemon background threads
that will prevent shutdown of the JVM. Use shutdown-agents to
terminate these threads and allow shutdown.
pmap uses futures which run on the agent thread pool.
As I tested, a separate thread is used for each new agent, when I create them.
Could several agents be run in one thread?
My idea is to create 10K+ light-weight agents (like actors in erlang), so is it a challenge for Clojure?
Thanks
This is incorrect. Agents use a thread pool which is the number of core + 2 in size. So on a quad core machine even 10k+ agents will only use 6 worker threads.
With send, that is. With send-off new threads will be started.
Consider using a j.u.c.DelayQueue
Here's a sketch of how it would work,
the (delayed-function is a bit cumbersome here, but it basically constructs an instance of j.u.c.Delayed for submission to the queue.)
(import [java.util.concurrent Delayed DelayQueue TimeUnit])
(defn delayed-function [f]
(let [execute-time (+ 5000 (System/currentTimeMillis))
remaining-delay (fn [t] (.convert t
(- execute-time
(System/currentTimeMillis))
TimeUnit/MILLISECONDS))]
(reify
Delayed (getDelay [_ t] (remaining-delay t))
Runnable (run [_] (f))
Comparable (compareTo [_ _] 0))))
;;; Use java's DelayQueue from clojure.
;;; See http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/DelayQueue.html
(def q (DelayQueue.))
(defn delayed
"put the function f on the queue, it will only execute after the delay
expires"
[f]
(.offer q (delayed-function f)))
(defn start-processing
"starts a thread that endlessly reads from the delay queue and
executes the function found in the queue"
[]
(.start
(Thread.
#(while true
(.run (.take q))))))
user> (start-processing)
user> (delayed #(println "Hello"))
; 5 seconds passes
Hello
the at function of the at-at library that was developed to support the (in my opinion fantastic) Overtone music synthesizer provides a nice clean interfase for running functions at a specific point in time.
(use 'overtone.at-at)
(def my-pool (mk-pool))
(after 1000 #(println "hello from the past!") my-pool)
I've defined a simple factorial function in the REPL:
(defn factorial [n]
(loop [current n fact 1]
(if
(= current 1)
fact
(recur (dec current) (* current fact)))))
The function works fine. But when I try to call the function multiple times with a dotimes loop the REPL seems to stop working. I don't get any results back anymore for whatever expression I type and have to restart the REPL.
I loop with:
(dotimes [x 10]
(println "Factorial of " x " is " (factorial x)))
I'm using IntelliJ with the La Clojure plugin (Clojure version 1.3.0).
I bet it takes an awfully long time to compute (factorial 0) with that function definition...
I'm experimenting with filtering through elements in parallel. For each element, I need to perform a distance calculation to see if it is close enough to a target point. Never mind that data structures already exist for doing this, I'm just doing initial experiments for now.
Anyway, I wanted to run some very basic experiments where I generate random vectors and filter them. Here's my implementation that does all of this
(defn pfilter [pred coll]
(map second
(filter first
(pmap (fn [item] [(pred item) item]) coll))))
(defn random-n-vector [n]
(take n (repeatedly rand)))
(defn distance [u v]
(Math/sqrt (reduce + (map #(Math/pow (- %1 %2) 2) u v))))
(defn -main [& args]
(let [[n-str vectors-str threshold-str] args
n (Integer/parseInt n-str)
vectors (Integer/parseInt vectors-str)
threshold (Double/parseDouble threshold-str)
random-vector (partial random-n-vector n)
u (random-vector)]
(time (println n vectors
(count
(pfilter
(fn [v] (< (distance u v) threshold))
(take vectors (repeatedly random-vector))))))))
The code executes and returns what I expect, that is the parameter n (length of vectors), vectors (the number of vectors) and the number of vectors that are closer than a threshold to the target vector. What I don't understand is why the programs hangs for an additional minute before terminating.
Here is the output of a run which demonstrates the error
$ time lein run 10 100000 1.0
[null] 10 100000 12283
[null] "Elapsed time: 3300.856 msecs"
real 1m6.336s
user 0m7.204s
sys 0m1.495s
Any comments on how to filter in parallel in general are also more than welcome, as I haven't yet confirmed that pfilter actually works.
You need to call shutdown-agents to kill the threads backing the threadpool used by pmap.
About pfilter, it should work but run slower than filter, since your predicate is simple. Parallelization isn't free so you have to give each thread moderately intensive tasks to offset the multithreading overhead. Batch your items before filtering them.