I'm using agents to manipulate a structure, but I don't have all my side effects.
All the messages are sent(I've printed and counted them), but there are times when I don't have all my side-effects. As if not all of my functions were applied on the agent's state, or if the last send is applied on a previous state..
I experimented with doall, dorun but haven't find a solution, appreciate any help.
;; aux function for adding an element to a hashmap
(defn extend-regs [reg s o]
(let [os (get reg s)]
(if (nil? os)
(assoc reg s [o])
(assoc reg s (conj os o)))))
;; the agent's altering function - adding an element to the :regs field(a hashmap)
(defn add-reg! [d s o]
(send d (fn [a] (assoc a :regs (extend-regs (:regs a) s o)))))
;; Creating the agents, dct/init returns an agent
;; pds: data for fields
(defn pdcts->init-dcts! [pds]
(doall (map dct/init (map :nam pds) (repeat nil))))
;; Altering one agent's state, dct/add-reg sends an assoc message to the agent
;; d: agent, pd: data for fields
(defn dct->add-regs! [d pd]
(dorun (map (fn [s r] (dct/add-reg! d s r))
(:syms pd)
(:regs pd)))
d)
;; Going through all agents
;; ds: agents, pds: datas
(defn dcts->add-regs! [ds pds]
(dorun (map (fn [d pd] (dct->add-regs! d pd))
ds
pds))
ds)
EDIT: =====================================================
Okay it turned out I just haven't wait enough to my threads to finish their tasks. Now the question is how can I monitor my agents. How can I know that there are unfinished threads in the queue? I've only found swank.core/active-threads and similar ones but they are not a solution.
I do not have a solution for your problem, but I can't resist suggesting some improvement on the first two functions:
(defn extend-regs [reg s o]
(let [os (get reg s)]
(if (nil? os)
(assoc reg s [o])
(assoc reg s (conj os o)))))
;; => place the 'if inside the assoc:
(defn extend-regs [reg s o]
(let [os (get reg s)]
(assoc reg s (if (nil? os) [o] (conj os o)))))
;; => this (if (nil? x) ...) is the pattern of function fnil, so ...
(defn extend-regs [reg s o]
(let [os (get reg s)]
(assoc reg s ((fnil conj []) os o))))
;; with update-in, this will be even clearer, and we can remove the let altogether:
(defn extend-regs [reg s o]
(update-in reg [s] (fnil conj []) o))
As for the second:
(defn add-reg! [d s o]
(send d (fn [a] (assoc a :regs (extend-regs (:regs a) s o)))))
;; => We can, again, use update-in instead of assoc:
(defn add-reg! [d s o]
(send d (fn [a] (update-in a [:regs] extend-regs s o))))
;; or, if you can get rid of extend-regs:
(defn add-reg! [d s o]
(send d (fn [a] (update-in a [:regs s] (fnil conj []) o)))
Finally, as a matter of style, I would place add-reg in a separate function, and directly use the idiom of sending to the agent in the client code (or have a simplified add-reg! function):
(defn add-reg [v s o] (update-in v [:regs s] (fnil conj []) o))
(defn add-reg! [d s o] (send d add-reg))
I know this doesn't answer your initial question, but it was fun to write this step by step refactoring
Use await or await-for to wait until an agent finished it's current work queue:
(await agent1 agent2 agent3)
or
(apply await list-of-agents)
A minor improvement of add-reg:
(defn extend-regs [reg s o]
(update-in reg [s] conj o))
This works because of
(conj nil :b) ; => [:b]
thus
(update-in {} [:a] conj :b) ; => {:a [:b]}
finally we have.
(defn add-reg! [d s o]
(send d update-in s [:regs] conj o)
Related
How would I create a transducer from the following ordinary code, where combo is the alias for clojure.math.combinatorics:
(defn row->evenly-divided [xs]
(->> (combo/combinations (sort-by - xs) 2)
(some (fn [[big small]]
(assert (>= big small))
(let [res (/ big small)]
(when (int? res)
res))))))
As noted in a comment transducers are only applicable for processing each item. With this is mind I've made the code a little more transducer friendly by shifting the sorting so that it is now being done for each item. I don't think there's anything that can be done about the combinations part however!
(defn row->evenly-divided [xs]
(->> (combo/combinations xs 2)
(some (fn [xy]
(let [res (apply / (sort-by - xy))]
(when (int? res)
res))))))
This is the same function but with an introduced transducer:
(def x-row->evenly-divided (comp
(map (partial sort-by -))
(map (partial apply /))
(filter int?)))
(defn row->evenly-divided-2 [xs]
(->> (combo/combinations xs 2)
(sequence x-row->evenly-divided)
first))
I have some similar reagent components, that can render a given number in certain ways:
(defn plain-number [n]
[:h1 n])
(defn pie-chart [n]
(render-fancy-chart n))
And there is some (simplified) state:
(def state (r/atom {:a 5 :b 10 :c 7}))
And I know how to write a component, that can access that state and use one of the components to render the state:
(def fetch-and-render-pie [k]
(let [v (get #state k)]
[pie-chart v]))
[fetch-and-render-pie :a] ; renders pie with 5
So far, so good. But that's coupled and repetitive.
The Goal:
A decorator would be nice, that can fetch some state and pass it to the children. The usage would look something like this:
[fetch :a
[pie-chart]]
Possible Solution:
(defn fetch [k wrapped]
(let [v (get #state k)]
(conj wrapped v)))
This worked, but it messes with the vector of the component definition and it assumes a lot of the wrapped component's arguments. And it failed for chained decorators.
There must be a clever and robust solution out there. Any ideas?
You can't rid of assumptions about arguments of decorated component just because you pass one to it and should know how to do it the same way as with function call. But you have no need to pass vector to your decorator, just a component itself should work:
(defn fetch [k component]
(let [v (get #state k)]
[component v]))
[fetch :test pie-chart] ; ~ [pie-chart (get #state :test)]
For chaining you'd want to support extra args for component:
(defn fetch [k component & args]
(let [v (get #state k)]
(into [component v] args)))
(defn prepare [s component & args]
(let [v (keyword s)]
(into [component v] args)))
[prepare "test" fetch pie-chart {:colourful true}]
; ~ [pie-chart (get #state (keyword "test")) {:colourful true}]
This resembles HOFs and threading macros a little bit.
with reagent only, we could use cursor
(defn com-a [state]
(fn []
[:h1 #state]))
(defn fetch [db]
(let [state (reagent/cursor db :k)] ; assume {:k "dd"}
(fn []
[com-a state])))
with re-frame
(re-frame/reg-sub
:chart-data
(fn [db [_ query]]
(get-in db query)))
(defn com-a [state]
(fn []
[:h1 #state]))
(defn fetch [db]
(let [state (re-frame/subscribe [:chart-data [:k]])]
(fn []
[com-a state])))
I have a situation where I am creating and destroying objects in one clojure namespace, and want another namespace to co-ordinate. However I do not want the first namespace to have to call the second explicitly on object destruction.
In Java, I could use a listener. Unfortunately the underlying java libraries do not signal events on object destruction. If I were in Emacs-Lisp, then I'd use hooks which do the trick.
Now, in clojure I am not so sure. I have found the Robert Hooke library https://github.com/technomancy/robert-hooke. But this is more like defadvice in elisp terms -- I am composing functions. More over the documentation says:
"Hooks are meant to extend functions you don't control; if you own the target function there are obviously better ways to change its behaviour."
Sadly, I am not finding it so obvious.
Another possibility would be to use add-watch, but this is marked as alpha.
Am I missing another obvious solution?
Example Added:
So First namespace....
(ns scratch-clj.first
(:require [scratch-clj.another]))
(def listf (ref ()))
(defn add-object []
(dosync
(ref-set listf (conj
#listf (Object.))))
(println listf))
(defn remove-object []
(scratch-clj.another/do-something-useful (first #listf))
(dosync
(ref-set listf (rest #listf)))
(println listf))
(add-object)
(remove-object)
Second namespace
(ns scratch-clj.another)
(defn do-something-useful [object]
(println "object removed is:" object))
The problem here is that scratch-clj.first has to require another and explicitly push removal events across. This is a bit clunky, but also doesn't work if I had "yet-another" namespace, which also wanted to listen.
Hence I thought of hooking the first function.
Is this solution suitable to your requirements?
scratch-clj.first:
(ns scratch-clj.first)
(def listf (atom []))
(def destroy-listeners (atom []))
(def add-listeners (atom []))
(defn add-destroy-listener [f]
(swap! destroy-listeners conj f))
(defn add-add-listener [f]
(swap! add-listeners conj f))
(defn add-object []
(let [o (Object.)]
(doseq [f #add-listeners] (f o))
(swap! listf conj o)
(println #listf)))
(defn remove-object []
(doseq [f #destroy-listeners] (f (first #listf)))
(swap! listf rest)
(println #listf))
Some listeners:
(ns scratch-clj.another
(:require [scratch-clj.first :as fst]))
(defn do-something-useful-on-remove [object]
(println "object removed is:" object))
(defn do-something-useful-on-add [object]
(println "object added is:" object))
Init binds:
(ns scratch-clj.testit
(require [scratch-clj.another :as another]
[scratch-clj.first :as fst]))
(defn add-listeners []
(fst/add-destroy-listener another/do-something-useful-on-remove)
(fst/add-add-listener another/do-something-useful-on-add))
(defn test-it []
(add-listeners)
(fst/add-object)
(fst/remove-object))
test:
(test-it)
=> object added is: #<Object java.lang.Object#c7aaef>
[#<Object java.lang.Object#c7aaef>]
object removed is: #<Object java.lang.Object#c7aaef>
()
It sounds a lot like what you're describing is callbacks.
Something like:
(defn make-object
[destructor-fn]
{:destructor destructor-fn :other-data "data"})
(defn destroy-object
[obj]
((:destructor obj) obj))
; somewhere at the calling code...
user> (defn my-callback [o] (pr [:destroying o]))
#'user/my-callback
user> (destroy-object (make-object my-callback))
[:destroying {:destructor #<user$my_callback user$my_callback#73b8cdd5>, :other-data "data"}]
nil
user>
So, here is my final solution following mobytes suggestion. A bit more work, but
I suspect that I will want this in future.
Thanks for all the help
;; hook system
(defn make-hook []
(atom []))
(defn add-hook [hook func]
(do
(when-not
(some #{func} #hook)
(swap! hook conj func))
#hook))
(defn remove-hook [hook func]
(swap! hook
(partial
remove #{func})))
(defn clear-hook [hook]
(reset! hook []))
(defn run-hook
([hook]
(doseq [func #hook] (func)))
([hook & rest]
(doseq [func #hook] (apply func rest))))
(defn phils-hook []
(println "Phils hook"))
(defn phils-hook2 []
(println "Phils hook2"))
(def test-hook (make-hook))
(add-hook test-hook phils-hook)
(add-hook test-hook phils-hook2)
(run-hook test-hook)
(remove-hook test-hook phils-hook)
(run-hook test-hook)
I'm trying to compute folder size in parallel.
Maybe it's naive approach.
What I do, is that I give computation of every branch node (directory) to an agent.
All leaf nodes have their file sizes added to my-size.
Well it doesn't work. :)
'scan' works ok, serially.
'pscan' prints only files from first level.
(def agents (atom []))
(def my-size (atom 0))
(def root-dir (clojure.java.io/file "/"))
(defn scan [listing]
(doseq [f listing]
(if (.isDirectory f)
(scan (.listFiles f))
(swap! my-size #(+ % (.length f))))))
(defn pscan [listing]
(doseq [f listing]
(if (.isDirectory f)
(let [a (agent (.listFiles f))]
(do (swap! agents #(conj % a))
(send-off a pscan)
(println (.getName f))))
(swap! my-size #(+ % (.length f))))))
Do you have any idea, what have i done wrong?
Thanks.
No need to keep state using atoms. Pure functional:
(defn psize [f]
(if (.isDirectory f)
(apply + (pmap psize (.listFiles f)))
(.length f)))
So counting filesizes in parallel should be so easy?
It's not :)
I tried to solve this issue better. I realized that i'm doing blocking I/O operations so pmap doesn't do the job.
I was thinking maybe giving chunks of directories (branches) to agents to process it independently would make sense. Looks it does :)
Well I haven't benchmarked it yet.
It works, but, there might be some problems with symbolic links on UNIX-like systems.
(def user-dir (clojure.java.io/file "/home/janko/projects/"))
(def root-dir (clojure.java.io/file "/"))
(def run? (atom true))
(def *max-queue-length* 1024)
(def *max-wait-time* 1000) ;; wait max 1 second then process anything left
(def *chunk-size* 64)
(def queue (java.util.concurrent.LinkedBlockingQueue. *max-queue-length* ))
(def agents (atom []))
(def size-total (atom 0))
(def a (agent []))
(defn branch-producer [node]
(if #run?
(doseq [f node]
(when (.isDirectory f)
(do (.put queue f)
(branch-producer (.listFiles f)))))))
(defn producer [node]
(future
(branch-producer node)))
(defn node-consumer [node]
(if (.isFile node)
(.length node)
0))
(defn chunk-length []
(min (.size queue) *chunk-size*))
(defn compute-sizes [a]
(doseq [i (map (fn [f] (.listFiles f)) a)]
(swap! size-total #(+ % (apply + (map node-consumer i))))))
(defn consumer []
(future
(while #run?
(when-let [size (if (zero? (chunk-length))
false
(chunk-length))] ;appropriate size of work
(binding [a (agent [])]
(dotimes [_ size] ;give us all directories to process
(when-let [item (.poll queue)]
(set! a (agent (conj #a item)))))
(swap! agents #(conj % a))
(send-off a compute-sizes))
(Thread/sleep *max-wait-time*)))))
You can start it by typing
(producer (list user-dir))
(consumer)
For result type
#size-total
You can stop it by (there are running futures - correct me if I'm wrong)
(swap! run? not)
If you find any errors/mistakes, you're welcome to share your ideas!
I have just start reading Let over lambda and I thought I would try and write a clojure version of the block-scanner in the closures chapter.
I have the following so far:
(defn block-scanner [trigger-string]
(let [curr (ref trigger-string) trig trigger-string]
(fn [data]
(doseq [c data]
(if (not (empty? #curr))
(dosync(ref-set curr
(if (= (first #curr) c)
(rest #curr)
trig)))))
(empty? #curr))))
(def sc (block-scanner "jihad"))
This works I think, but I would like know what I did right and what I could do better.
I would not use ref-set but alter because you don't reset the state to a completely new value, but update it to a new value which is obtained from the old one.
(defn block-scanner
[trigger-string]
(let [curr (ref trigger-string)
trig trigger-string]
(fn [data]
(doseq [c data]
(when (seq #curr)
(dosync
(alter curr
#(if (-> % first (= c))
(rest %)
trig)))))
(empty? #curr))))
Then it is not necessary to use refs since you don't have to coordinate changes. Here an atom is a better fit, since it can be changed without all the STM ceremony.
(defn block-scanner
[trigger-string]
(let [curr (atom trigger-string)
trig trigger-string]
(fn [data]
(doseq [c data]
(when (seq #curr)
(swap! curr
#(if (-> % first (= c))
(rest %)
trig))))
(empty? #curr))))
Next I would get rid of the imperative style.
it does more than it should: it traverses all data - even if we found a match already. We should stop early.
it is not thread-safe, since we access the atom multiple times - it might change in between. So we must touch the atom only once. (Although this is probably not interesting in this case, but it's good to make it a habit.)
it's ugly. We can do all the work functionally and just save the state, when we come to a result.
(defn block-scanner
[trigger-string]
(let [state (atom trigger-string)
advance (fn [trigger d]
(when trigger
(condp = d
(first trigger) (next trigger)
; This is maybe a bug in the book. The book code
; matches "foojihad", but not "jijihad".
(first trigger-string) (next trigger-string)
trigger-string)))
update (fn [trigger data]
(if-let [data (seq data)]
(when-let [trigger (advance trigger (first data))]
(recur trigger (rest data)))
trigger))]
(fn [data]
(nil? (swap! state update data)))))