I am just getting curious about lisp programming and wanted to
know how to use concurrent lisp by making threads.
-I also wanted to be clear with the pcall() function in lisp.
the comments to your question give you the right key to solve the problem. I hope this is still helpful.
If you want to use concurrency with common lisp, One of the ways is Eager Future2 http://common-lisp.net/project/eager-future/, but you can choose other models here http://www.cliki.net/concurrency.
Eager Future is based, like the name suggested, in Futures
"Eager Future2 is a Common Lisp library that provides composable
concurrency primitives that unify parallel and lazy evaluation, are
integrated with CL's condition system, and have automatic resource
management.
The basic unit of concurrency in Eager Future2 is the future, which is
a data type that acts as a proxy for the values of a no-argument
function (thunk) that is computed concurrently."
Let's see how works pcall, from documentation of Eager Furture2 we find:
"
function pcall (thunk &optional (future-type default-future-type)) => future
Given a function of no arguments, returns an object (called a
future) that can later be used to retrieve the values computed by the
function.
future-type (by default the value of default-future-type) can either
be :eager, :speculative, or :lazy. See the documentation of
default-future-type for an explanation of the different future
types.
The function is called in an unspecified dynamic environment."
sample lisp code
;; use quicklisp to add Eager-Future2 to your common lisp
;; implementation in my case I use GVIM, SLIMV and SBCL
(quicklisp:quickload "EAGER-FUTURE2")
(defparameter *future* (eager-future2:pcall #'compute))
*future*
;; Returns non nil if the future values have been computed, nil otherwise
(eager-future2:ready-to-yield? *future*)
;;Returns the computed values, see delayed future, computes the value in current thread,
;;and speculative future computes the future in a current thread if not is evalatuated
;;in another thread.
(eager-future2:yield *future*)
;; function for the long computation, from http://www.cliki.net/fibonacci
(defun fib (n)
"Tail-recursive computation of the nth element of the Fibonacci sequence"
(check-type n (integer 0 *))
(labels ((fib-aux (n f1 f2)
(if (zerop n)
f1 (fib-aux (1- n) f2 (+ f1 f2)))))
(fib-aux n 0 1)))
;; function with no arguments for pcall that computes fibonacci 10000th number
(defun compute ()
(fib 10000))
Use in REPL
SBCL 1.2.4-1.fc21 Port: 4005 Pid: 1188
; SWANK 2014-10-10
CL-USER> (quicklisp:quickload "EAGER-FUTURE2")
To load "eager-future2":
Load 1 ASDF system:
alexandria
Install 3 Quicklisp releases:
bordeaux-threads eager-future2 trivial-garbage
; Fetching #<URL "http://beta.quicklisp.org/archive/trivial-garbage/2013-03-12/trivial-garbage-20130312-git.tgz">
; 8.00KB
==================================================
8,197 bytes in 0.01 seconds (1143.55KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/bordeaux-threads/2013-06-15/bordeaux-threads-0.8.3.tgz">
; 18.31KB
==================================================
18,754 bytes in 0.06 seconds (327.04KB/sec)
; Fetching #<URL "http://beta.quicklisp.org/archive/eager-future2/2011-03-20/eager-future2-0.2.tgz">
; 17.34KB
==================================================
17,758 bytes in 0.07 seconds (262.75KB/sec)
; Loading "eager-future2"
[package bordeaux-threads]........................
[package trivial-garbage].........................
[package eager-future2]..
("EAGER-FUTURE2")
CL-USER> (defun fib (n)
"Tail-recursive computation of the nth element of the Fibonacci sequence"
(check-type n (integer 0 *))
(labels ((fib-aux (n f1 f2) ...))))
FIB
CL-USER> (fib 1000)
43466557686937456435688527675040625802564660517371780402481729089536555417949051890403879840079255169295922593080322634775209689623239873322471161642996440906533187938298969649928516003704476137795166849228875
CL-USER> (fib 10000)
33644764876431783266621612005107543310302148460680063906564769974680081442166662368155595513633734025582065332680836159373734790483865268263040892463056431887354544369559827491606602099884183933864652731300088830269235673613135117579297437854413752130520504347701602264758318906527890855154366159582987279682987510631200575428783453215515103870818298969791613127856265033195487140214287532698187962046936097879900350962302291026368131493195275630227837628441540360584402572114334961180023091208287046088923962328835461505776583271252546093591128203925285393434620904245248929403901706233888991085841065183173360437470737908552631764325733993712871937587746897479926305837065742830161637408969178426378624212835258112820516370298089332099905707920064367426202389783111470054074998459250360633560933883831923386783056136435351892133279732908133732642652633989763922723407882928177953580570993691049175470808931841056146322338217465637321248226383092103297701648054726243842374862411453093812206564914032751086643394517512161526545361333111314042436854805106765843493523836959653428071768775328348234345557366719731392746273629108210679280784718035329131176778924659089938635459327894523777674406192240337638674004021330343297496902028328145933418826817683893072003634795623117103101291953169794607632737589253530772552375943788434504067715555779056450443016640119462580972216729758615026968443146952034614932291105970676243268515992834709891284706740862008587135016260312071903172086094081298321581077282076353186624611278245537208532365305775956430072517744315051539600905168603220349163222640885248852433158051534849622434848299380905070483482449327453732624567755879089187190803662058009594743150052402532709746995318770724376825907419939632265984147498193609285223945039707165443156421328157688908058783183404917434556270520223564846495196112460268313970975069382648706613264507665074611512677522748621598642530711298441182622661057163515069260029861704945425047491378115154139941550671256271197133252763631939606902895650288268608362241082050562430701794976171121233066073310059947366875
CL-USER> (defun compute ()
(fib 10000))
COMPUTE
CL-USER> (defparameter *future* (eager-future2:pcall #'compute))
*FUTURE*
CL-USER> *future*
#<EAGER-FUTURE2:FUTURE {1004F20383}>
CL-USER> (eager-future2:ready-to-yield? *future*)
T
CL-USER> (eager-future2:yield *future*)
33644764876431783266621612005107543310302148460680063906564769974680081442166662368155595513633734025582065332680836159373734790483865268263040892463056431887354544369559827491606602099884183933864652731300088830269235673613135117579297437854413752130520504347701602264758318906527890855154366159582987279682987510631200575428783453215515103870818298969791613127856265033195487140214287532698187962046936097879900350962302291026368131493195275630227837628441540360584402572114334961180023091208287046088923962328835461505776583271252546093591128203925285393434620904245248929403901706233888991085841065183173360437470737908552631764325733993712871937587746897479926305837065742830161637408969178426378624212835258112820516370298089332099905707920064367426202389783111470054074998459250360633560933883831923386783056136435351892133279732908133732642652633989763922723407882928177953580570993691049175470808931841056146322338217465637321248226383092103297701648054726243842374862411453093812206564914032751086643394517512161526545361333111314042436854805106765843493523836959653428071768775328348234345557366719731392746273629108210679280784718035329131176778924659089938635459327894523777674406192240337638674004021330343297496902028328145933418826817683893072003634795623117103101291953169794607632737589253530772552375943788434504067715555779056450443016640119462580972216729758615026968443146952034614932291105970676243268515992834709891284706740862008587135016260312071903172086094081298321581077282076353186624611278245537208532365305775956430072517744315051539600905168603220349163222640885248852433158051534849622434848299380905070483482449327453732624567755879089187190803662058009594743150052402532709746995318770724376825907419939632265984147498193609285223945039707165443156421328157688908058783183404917434556270520223564846495196112460268313970975069382648706613264507665074611512677522748621598642530711298441182622661057163515069260029861704945425047491378115154139941550671256271197133252763631939606902895650288268608362241082050562430701794976171121233066073310059947366875
CL-USER>
I hope that this will be usefull
Related
I am newbie to Clojure. I am invoking Clojure function using java and I want to record the time a particular line of clojure code execution takes:
Suppose if my clojure function is:
(defn sampleFunction [sampleInput]
(fun1 (fun2 sampleInput))
Above function I am invoking from java which returns some String value and I want to record the time it takes for executing fun2.
I have a another function say logTime which will write the parameter passed to it in to some database:
(defn logTime [time]
.....
)
My Question is: How can I modify my sampleFunction(..) to invoke logTime for recording time it took to execute fun2.
Thank you in advance.
I'm not entirely sure how the different pieces of your code fit together and interoperate with Java, but here's something that could work with the way you described it.
To get the execution time of a piece of code, there's a core function called time. However, this function doesn't return the execution time, it just prints it... So given that you want to log that time into a database, we need to write a macro to capture both the return value of fun2 as well the time it took to execute:
(defmacro time-execution
[& body]
`(let [s# (new java.io.StringWriter)]
(binding [*out* s#]
(hash-map :return (time ~#body)
:time (.replaceAll (str s#) "[^0-9\\.]" "")))))
What this macro does is bind standard output to a Java StringWriter, so that we can use it to store whatever the time function prints. To return both the result of fun2 and the time it took to execute, we package the two values in a hash-map (could be some other collection too - we'll end up destructuring it later). Notice that the code whose execution we're timing is wrapped in a call to time, so that we trigger the printing side effect and capture it in s#. Finally, the .replaceAll is just to ensure that we're only extracting the actual numeric value (in miliseconds), since time prints something of the form "Elapsed time: 0.014617 msecs".
Incorporating this into your code, we need to rewrite sampleFunction like so:
(defn sampleFunction [sampleInput]
(let [{:keys [return time]} (time-execution (fun2 sampleInput))]
(logTime time)
(fun1 return)))
We're simply destructuring the hash-map to access both the return value of fun2 and the time it took to execute, then we log the execution time using logTime, and finally we finish by calling fun1 on the return value of fun2.
The library tupelo.prof gives you many options if you want to capture execution time for one or more functions and accumulate it over multiple calls. An example:
(ns tst.demo.core
(:use tupelo.core tupelo.test)
(:require
[tupelo.profile :as prof]))
(defn add2 [x y] (+ x y))
(prof/defnp fast [] (reduce add2 0 (range 10000)))
(prof/defnp slow [] (reduce add2 0 (range 10000000)))
(dotest
(prof/timer-stats-reset)
(dotimes [i 10000] (fast))
(dotimes [i 10] (slow))
(prof/print-profile-stats)
)
with result:
--------------------------------------
Clojure 1.10.2-alpha1 Java 14
--------------------------------------
Testing tst.demo.core
---------------------------------------------------------------------------------------------------
Profile Stats:
Samples TOTAL MEAN SIGMA ID
10000 0.955 0.000096 0.000045 :tst.demo.core/fast
10 0.905 0.090500 0.000965 :tst.demo.core/slow
---------------------------------------------------------------------------------------------------
If you want detailed timing for a single method, the Criterium library is what you need. Start off with the quick-bench function.
Since the accepted answer has some shortcomings around eating up logs etc,
A simpler solution compared to the accepted answer perhaps
(defmacro time-execution [body]
`(let [st# (System/currentTimeMillis)
return# ~body
se# (System/currentTimeMillis)]
{:return return#
:time (double (/ (- se# st#) 1000))}))
I have been trying some simple tests using Clojure and its Java interop capabilities, and am running into some issues. I am interfacing with the Parallel Colt Java library, trying a LU factorization - a feature offered by Parallel Colt. I see errors of the form
"NoSuchMethodError edu.emory.mathcs.utils.ConcurrencyUtils.getThreadsBeginN_1D" ...
Here's some simple code I am using:
(ns colt-test.core
(:import [cern.colt.matrix.tdouble DoubleMatrix1D DoubleMatrix2D]
[cern.colt.matrix.tdouble.impl DenseDoubleMatrix2D DenseDoubleMatrix1D]
[cern.colt.matrix.tdouble.algo.decomposition DenseDoubleLUDecomposition]
[cern.colt.list.tdouble DoubleArrayList])
;; make a 1D double array, size N of random values up to val
(defn make-1D-rand-array [N val]
(let [v (repeatedly N #(rand val))]
(double-array v)))
;; make a 2D double array, size NxN of random values up to val
(defn make-2D-rand-array [N val]
(let [v (repeatedly N #(make-1D-rand-array N val))]
(into-array v)))
;; Test LU factorization
;; matrix A size 10 x 10, random values
(def A (DenseDoubleMatrix2D. (make-2D-rand-array 5 10.0)))
;; vector b of size 10, random values
(def b (DenseDoubleMatrix1D. (make-1D-rand-array 5 10.0)))
I can call some functions on "b", the DenseDoubleMatrix1D, such as size(), and elements(). Also, I can call
set(int index, double value)
using the clojure interop syntax as
(.set b 0 10.5)
and it will update the first element of "b" to 10.5.
However, when I call some other methods on "b", such as
assign(double value)
as
(.assign b 10.0)
to set all the elements of b to 10.0, I get a NoSuchMethodError. Essentially, I seem to have correct calls for some interop methods, but not others, and I am not sure what I am doing wrong.
I see the same error when I try
(DenseDoubleLUDecomposition. A)
For reference, here is the parallel colt javadoc (http://incanter.org/docs/parallelcolt/api/)
Note, I am using a slightly newer version of parallel colt.
I have figured out how to get this to work. I was using parallel-colt 0.10.0, and when I upgraded to version 0.10.1, everything works as expected.
Hope this helps someone.
Say I have a function, (get-events "feed"), that returns a vector of events in chronological order, taken from an external source.
Now, at any given moment, that function returns a list of events up to that point in time. Called a few seconds later, it will return a few more events, etc, as the feed continually grows.
If I want to create a lazy-seq that forever pulls new events from the feed, making sure it doesn't repeat those that have already been seen, how would I write this? I'm running into a stack overflow error when I don't use recur, but I can't use recur, because it doesn't appear in a tail position.
(def continually-list-events
([feed] (continually-list-events feed (hash-set)))
([feed seen]
(let [events-now (get-events feed)]
(into (remove seen events-now)
(lazy-seq
(continually-list-events feed
(into seen events-now))))))
You can see I'm trying to use an accumulator to track events already seen (in a set), and I'm making sure to always filter out the ones I've seen.
If each step keeps track of how many events have been received so far then that iteration can return a sequence of new events by dropping the old ones.
user> (->> (iterate (fn [[events-so-far contents]]
(let [events (get-events)
new-events (drop events-so-far events)]
[(count events) new-events])))
(mapcat second))
Then you can drop the counts from the sequence and flatten the chunks of events into a sequence of single events.
In your example the stackoverflow is because there is no call to cons after the call to lazy-seq so it's calculating the whole list as the first item in the sequence.
user> (defn example [x] (lazy-seq (cons x (example (inc x)))))
#'user/example
user> (take 5 (example 4))
(4 5 6 7 8)
user> (defn example [x] (lazy-seq (example (inc x))))
#'user/example
user> (take 5 (example 4))
... long pause then out of memory ...
PS: using lazy-seq directly is somewhat uncommon, though it's important to know how it works.
I've defined a simple factorial function in the REPL:
(defn factorial [n]
(loop [current n fact 1]
(if
(= current 1)
fact
(recur (dec current) (* current fact)))))
The function works fine. But when I try to call the function multiple times with a dotimes loop the REPL seems to stop working. I don't get any results back anymore for whatever expression I type and have to restart the REPL.
I loop with:
(dotimes [x 10]
(println "Factorial of " x " is " (factorial x)))
I'm using IntelliJ with the La Clojure plugin (Clojure version 1.3.0).
I bet it takes an awfully long time to compute (factorial 0) with that function definition...
I'm experimenting with filtering through elements in parallel. For each element, I need to perform a distance calculation to see if it is close enough to a target point. Never mind that data structures already exist for doing this, I'm just doing initial experiments for now.
Anyway, I wanted to run some very basic experiments where I generate random vectors and filter them. Here's my implementation that does all of this
(defn pfilter [pred coll]
(map second
(filter first
(pmap (fn [item] [(pred item) item]) coll))))
(defn random-n-vector [n]
(take n (repeatedly rand)))
(defn distance [u v]
(Math/sqrt (reduce + (map #(Math/pow (- %1 %2) 2) u v))))
(defn -main [& args]
(let [[n-str vectors-str threshold-str] args
n (Integer/parseInt n-str)
vectors (Integer/parseInt vectors-str)
threshold (Double/parseDouble threshold-str)
random-vector (partial random-n-vector n)
u (random-vector)]
(time (println n vectors
(count
(pfilter
(fn [v] (< (distance u v) threshold))
(take vectors (repeatedly random-vector))))))))
The code executes and returns what I expect, that is the parameter n (length of vectors), vectors (the number of vectors) and the number of vectors that are closer than a threshold to the target vector. What I don't understand is why the programs hangs for an additional minute before terminating.
Here is the output of a run which demonstrates the error
$ time lein run 10 100000 1.0
[null] 10 100000 12283
[null] "Elapsed time: 3300.856 msecs"
real 1m6.336s
user 0m7.204s
sys 0m1.495s
Any comments on how to filter in parallel in general are also more than welcome, as I haven't yet confirmed that pfilter actually works.
You need to call shutdown-agents to kill the threads backing the threadpool used by pmap.
About pfilter, it should work but run slower than filter, since your predicate is simple. Parallelization isn't free so you have to give each thread moderately intensive tasks to offset the multithreading overhead. Batch your items before filtering them.