I'm trying to handle duplicate events (MIDI note-on and note-off signals coming in triplicate). It seems like using agents and locking works, unless the events happen (nearly) simultaneously.
Here's an example:
(def test-val (agent 0))
(def allow-update (agent true))
(defn increase-val []
(locking allow-update
(when #allow-update
(send allow-update (fn [_ x] x) false)
(send test-val + 1)))
(print #test-val))
(defn run-futures [delay-time]
(send allow-update (fn [_ x] x) true)
(send test-val * 0)
(dotimes [_ 20]
(Thread/sleep delay-time)
(future (increase-val))))
If I test it with a slight delay between calls to increase-val:
(run-futures 2)
;outputs 0111111111111111111 every time, as expected
But if I let all the calls to increase-val happen all at once:
(run-futures 0)
;001222222222222222
(run-futures 0)
;000000145555555555
(run-futures 0)
;000000013677777777
It seems like the lock doesn't have time to get turned on, so the agent gets increased by multiple futures.
I'm hoping I'm missing something here that will allow me to make sure I don't act on duplicate simultaneous events.
Thanks!
Because Agents are asynchronous and uncoordinated, there can be a delay between when you send them a message and the actual running of the action that was sent. During this delay
it's possible for most, or all, of the calls to (send test-val + 1) to make it through the locked section before the first one actually runs and sets #allow-update to false. You can see this in the leading 0s before the state of test-val gets changed.
atoms may be better suited to interacting with locks, though refs may be the right tool for coordinated access to multiple identities.
Related
I have a number of threads that need to access a collection of values, some of these values also need to be persisted to a database when any changes are made to them so that I don't lose the state in case of a server reboot etc.
Currently I'm using an atom to store these values, I have a set of functions which I call when something in the atom needs to change.
Inside these functions I'm also persisting data to a database before calling swap!, I chose this approach because I need to frequently read the values inside the atom, and it doesn't seem performant to open/close db connections every time I'm interested in one of the values.
the question:
Is this approach viable? I'm interested to know if anyone has had success implementing a similar solution or are there pitfalls I should be aware of?
Atoms are fine.
An alternative approach would be to use https://github.com/clojure/core.memoize or core.cached directly, like suggested by Stefan Kamphausen.
Approach:
Cache query results on the function level. This way you are sure that what you get back is exactly how the database would return it and not the way you would think it would serialize/deserialize.
Invalidate key/args after you have inserted/changed something in the database.
One benefit of this approach is that you can tweak the caching behavior: TTL, LRU, FIFO, etc.
Demo:
(require '[clojure.core.memoize :as memo])
;; suppose this is a real DB
(def db (atom {}))
(defn my-get [k]
;; expensive database call
(Thread/sleep 5000)
(get #db k))
(def my-get-cached
(memo/memo my-get))
(defn my-put
[k val]
(swap! db assoc k val)
(memo/memo-clear! my-get-cached [k]))
(comment
(my-put :foo "the value")
(my-get-cached :foo) ;; wait 5 seconds, "the value"
(my-get-cached :foo) ;; "the value", instantly
(my-put :bar "other-value")
(my-get-cached :foo) ;; "the value", still instantly
(my-get-cached :bar) ;; wait 5 seconds, "other value"
(my-put :foo "changed")
(my-get-cached :foo) ;; wait 5 seconds, "changed"
)
I am working on a Clojure / Jetty web service. I have a special url that I want to only be serviced one request at a time. If the url was requested, and before it returns, the url is requested again, I want to immediately return. So in more core.clj, where I defined my routes, I have something like this:
(def work-in-progress (ref false))
Then sometime later
(compojure.core/GET "/myapp/internal/do-work" []
(if #work-in-progress
"Work in Progress please try again later"
(do
(dosync
(ref-set work-in-progress true))
(do-the-work)
(dosync
(ref-set rebuild-in-progress false))
"Job completed Successfully")))
I have tried this on local Jetty server but I seem to be able to hit the url twice and double the work. What is a good pattern / way to implement this in Clojure in a threaded web server environment?
Imagine a following race condition for the solution proposed in the question.
Thread A starts to execute handler's body. #work-in-progress is false, so it enters the do expression. However, before it managed to set the value of work-in-progress to true...
Thread B starts to execute handler's body. #work-in-progress is false, so it enters the do expression.
Now two threads are executing (do-the-work) concurrently. That's not what we want.
To prevent this problem check and set the value of the ref in a dosync transaction.
(compojure.core/GET "/myapp/internal/do-work" []
(if (dosync
(when-not #work-in-progress
(ref-set work-in-progress true)))
(try
(do-the-work)
"Job completed Successfully"
(finally
(dosync
(ref-set work-in-progress false))))
"Work in Progress please try again later"))
Another abstraction which you might find useful in this scenario is an atom and compare-and-set!.
(def work-in-progress (atom false))
(compojure.core/GET "/myapp/internal/do-work" []
(if (compare-and-set! work-in-progress false true)
(try
(do-the-work)
"Job completed Successfully"
(finally
(reset! work-in-progress false)))
"Work in Progress please try again later"))
Actually this is the natural use case for a lock; in particular, a java.util.concurrent.locks.ReentrantLock.
The same pattern came up in my answer to an earlier SO question, Canonical Way to Ensure Only One Instance of a Service Is Running / Starting / Stopping in Clojure?; I'll repeat the relevant piece of code here:
(import java.util.concurrent.locks.ReentrantLock)
(def lock (ReentrantLock.))
(defn start []
(if (.tryLock lock)
(try
(do-stuff)
(finally (.unlock lock)))
(do-other-stuff)))
The tryLock method attempts to acquire the lock, returning true if it succeeds in doing so and false otherwise, not blocking in either case.
Consider queueing the access to the resource as well - in addition to getting an equivalent functionality to that of locks/flags, queues let you observe the resource contention, among other advantages.
I am learning clojure and try out its concurrency and effectiveness via a producer consumer example.
Did that and it felt pretty akward with having to use ref and deref and also watch and unwatch.
I tried to check other code snippet; but is there a better way of re factoring this other than using Java Condition await() and signal() methods along with Java Lock. I did not want to use anything in Java.
here is the code; i guess I would have made many mistakes here with my usage...
;a simple producer class
(ns my.clojure.producer
(:use my.clojure.consumer)
(:gen-class)
)
(def tasklist( ref (list) )) ;this is declared as a global variable; to make this
;mutable we need to use the fn ref
(defn gettasklist[]
(deref tasklist) ;we need to use deref fn to return the task list
)
(def testagent (agent 0)); create an agent
(defn emptytasklist[akey aref old-val new-val]
(doseq [item (gettasklist)]
(println(str "item is") item)
(send testagent consume item)
(send testagent increment item)
)
(. java.lang.Thread sleep 1000)
(dosync ; adding a transaction for this is needed to reset
(remove-watch tasklist "key123"); removing the watch on the tasklist so that it does not
; go to a recursive call
(ref-set tasklist (list ) ) ; we need to make it as a ref to reassign
(println (str "The number of tasks now remaining is=") (count (gettasklist)))
)
(add-watch tasklist "key123" emptytasklist)
)
(add-watch tasklist "key123" emptytasklist)
(defn addtask [task]
(dosync ; adding a transaction for this is needed to refset
;(println (str "The number of tasks before") (count (gettasklist)))
(println (str "Adding a task") task)
(ref-set tasklist (conj (gettasklist) task )) ; we need to make it as a ref to reassign
;(println (str "The number of tasks after") (count (gettasklist)))
)
)
Here is the consumer code
(ns my.clojure.consumer
)
(defn consume[c item]
(println "In the consume method:Item is " c item )
item
)
(defn increment [c n]
(println "parmeters are" c n)
(+ c n)
)
And here is the test code ( I have used maven to run clojure code and used NetBeans to edit as this is more familiar to me coming from Java - folder structure and pom at - https://github.com/alexcpn/clojure-evolve
(ns my.clojure.Testproducer
(:use my.clojure.producer)
(:use clojure.test)
(:gen-class)
)
(deftest test-addandcheck
(addtask 1)
(addtask 2)
(is(= 0 (count (gettasklist))))
(println (str "The number of tasks are") (count (gettasklist)))
)
If anybody can refactor this lightly so that I can read and understand the code then it will be great; Else I guess I will have to learn more
Edit -1
I guess using a global task list and making it available to other functions by de-referencing it (deref) and again making it mutable by ref is not the way in clojure;
So changing the addTask method to directly send the incoming tasks to an agent
(defn addtask [task]
(dosync ; adding a transaction for this is needed to refset
(println (str "Adding a task") task)
;(ref-set tasklist (conj (gettasklist) task )) ; we need to make it as a ref to reassign
(def testagent (agent 0)); create an agent
(send testagent consume task)
(send testagent increment task)
)
However when I tested it
(deftest test-addandcheck
(loop [task 0]
(when ( < task 100)
(addtask task)
(recur (inc task))))
(is(= 0 (count (gettasklist))))
(println (str "The number of tasks are") (count (gettasklist)))
)
after sometime the I am getting Java rejected execution exception -- This is fine if you do Java threads, because you take full control. But from clojure this looks odd, especially since you are not selecting the ThreadPool stratergy yourself
Adding a task 85
Exception in thread "pool-1-thread-4" java.util.concurrent.RejectedExecutionExce
ption
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution
(ThreadPoolExecutor.java:1759)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.jav
a:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.ja
va:658)
at clojure.lang.Agent$Action.execute(Agent.java:56)
at clojure.lang.Agent$Action.doRun(Agent.java:95)
at clojure.lang.Agent$Action.run(Agent.java:106)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:885)Adding a task 86
Adding a task 87
I think modeling producer & consumer in clojure would be done the easiest (and most efficiently) using lamina channels.
I have made similar clojure program with producer/consumer pattern in Computing Folder Sizes Asynchronously.
I don't think that you really need to use Refs.
You have single tasklist that is mutable state that you want to alter synchronously. Changes are relatively quick and do not depend on any other external state, only the variable that comes in to consumer.
As far with atoms there is swap! function which can help you do the changes the way I understood you need.
You can look at my snippet Computing folder size i think it can at least show you proper use of atoms & agents. I played with it a lot, so it should be correct.
Hope it helps!
Regards,
Jan
I was looking at the Clojure examples for Twitter storm. He just used LinkedBlockingQueue. It's easy to use, concurrent and performs well. Sure, it lacks the sex-appeal of an immutable Clojure solution, but it will work well.
I've come across several use cases where I need the ability to:
strictly control the number of worker threads on both the producer and consumer side
control the maximum size of the "work queue" in order to limit memory consumption
detect when all work has been completed so that I can shut down the workers
I've found that the clojure built-in concurrency features (while amazingly simple and useful in their own right) make the first two bullet points difficult. lamina looks great, but I didn't see a way that it would address my particular use cases w/o the same sort of extra plumbing that I'd need to do around an implementation based on BlockingQueue.
So, I ended up hacking together a simple clojure library to try to solve my problems. It is basically just a wrapper around BlockingQueue that attempts to conceal some of the Java constructs and provide a higher-level producer-consumer API. I'm not entirely satisfied with the API yet; it'll likely evolve a bit further... but it's operational:
https://github.com/cprice-puppet/freemarket
Example usage:
(def myproducer (producer producer-work-fn num-workers max-work))
(def myconsumer (consumer myproducer consumer-work-fn num-workers max-results))
(doseq [result (work-queue->seq (:result-queue myconsumer))]
(println result))
Feedback / suggestions / contributions would be welcomed!
How can I create a constantly running background process in Clojure? Is using "future" with a loop that never ends the right way?
You could just start a Thread with a function that runs forever.
(defn forever []
;; do stuff in a loop forever
)
(.start (Thread. forever))
If you don't want the background thread to block process exit, make sure to make it a daemon thread:
(doto
(Thread. forever)
(.setDaemon true)
(.start))
If you want some more finesse you can use the java.util.concurrent.Executors factory to create an ExecutorService. This makes it easy to create pools of threads, use custom thread factories, custom incoming queues, etc.
The claypoole lib wraps some of the work execution stuff up into a more clojure-friendly api if that's what you're angling towards.
My simple higher-order infinite loop function (using futures):
(def counter (atom 1))
(defn infinite-loop [function]
(function)
(future (infinite-loop function))
nil)
;; note the nil above is necessary to avoid overflowing the stack with futures...
(infinite-loop
#(do
(Thread/sleep 1000)
(swap! counter inc)))
;; wait half a minute....
#counter
=> 31
I strongly recommend using an atom or one of Clojures other reference types to store results (as per the counter in the example above).
With a bit of tweaking you could also use this approach to start/stop/pause the process in a thread-safe manner (e.g. test a flag to see if (function) should be executed in each iteration of the loop).
Maybe, or perhaps Lein-daemon? https://github.com/arohner/lein-daemon
After a few weekends exploring Clojure I came up with this program. It allows you to move a little rectangle in a window. Here's the code:
(import java.awt.Color)
(import java.awt.Dimension)
(import java.awt.event.KeyListener)
(import javax.swing.JFrame)
(import javax.swing.JPanel)
(def x (ref 0))
(def y (ref 0))
(def panel
(proxy [JPanel KeyListener] []
(getPreferredSize [] (Dimension. 100 100))
(keyPressed [e]
(let [keyCode (.getKeyCode e)]
(if (== 37 keyCode) (dosync (alter x dec))
(if (== 38 keyCode) (dosync (alter y dec))
(if (== 39 keyCode) (dosync (alter x inc))
(if (== 40 keyCode) (dosync (alter y inc))
(println keyCode)))))))
(keyReleased [e])
(keyTyped [e])))
(doto panel
(.setFocusable true)
(.addKeyListener panel))
(def frame (JFrame. "Test"))
(doto frame
(.add panel)
(.pack)
(.setDefaultCloseOperation JFrame/EXIT_ON_CLOSE)
(.setVisible true))
(defn drawRectangle [p]
(doto (.getGraphics p)
(.setColor (java.awt.Color/WHITE))
(.fillRect 0 0 100 100)
(.setColor (java.awt.Color/BLUE))
(.fillRect (* 10 (deref x)) (* 10 (deref y)) 10 10)))
(loop []
(drawRectangle panel)
(Thread/sleep 10)
(recur))
Despite being an experienced C++ programmer I found it very challenging to write even a simple application in a language that uses a radically different style than what I'm used to.
On top of that, this code probably sucks. I suspect the globalness of the various values is a bad thing. It's also not clear to me if it's appropriate to use references here for the x and y values.
Any hints for improving this code are welcome.
Those ifs in keyPressed can be replaced with a single case. Also, the dosync can be moved outside to wrap the case. In fact, alter can be moved out too, so that if you e.g. decide to change it to commute, there's just the one place to make the change. The result:
(def panel
(proxy [JPanel KeyListener] []
(getPreferredSize [] (Dimension. 100 100))
(keyPressed [e]
(let [keyCode (.getKeyCode e)]
(dosync
(apply alter
(case keyCode
37 [x dec]
38 [y dec]
39 [x inc]
40 [y inc])))
(println keyCode)))
(keyReleased [e])
(keyTyped [e])))
You could also rewrite the imports more concisely:
(import [java.awt Color Dimension event.ActionListener])
(import [javax.swing JFrame JPanel])
-- whether you'd want to is a matter of style.
I would rename drawRectangle to draw-rectangle (that's the idiomatic style for function names in Clojure) and, more to the point, rewrite it as a pure function accepting the coordinates as explicit arguments. Then you could write a small wrapper around that to use your Refs, if indeed your design would benefit from the use of Refs. (Hard to say without knowing how you might want to use & modify them etc.)
Prefer while to (loop [] ... (recur)) (see (doc while) and (clojure.contrib.repl-utils/source while)).
Also -- and this is important -- don't put anything except definitions at top level. That's because top-level forms are actually executed when the code is compiled (try loading a library with a (println :foo) at top level). That infinite loop should be wrapped inside a function; the standard name for a "main" function in Clojure is -main; same goes for panel and frame. This doesn't apply when playing around at the REPL, of course, but it's an important gotcha to know about.
Incidentally, (doto foo ...) returns foo, so you can just write (doto (proxy ...) (.setFocusable true) ...).
Otherwise, I'd say the code is fine. Normally you'd want to put it in a namespace; then all the imports would go in the ns form.
HTH
Edit, while writing the below post in a hurry I forgot to say you must not put parens around constants eg java.awt.Color/WHITE.
In addition to Michał's comments:
when you use an infinite loop, condition it with an atom (eg #switch) thus you'll be able to stop it (unless the loops runs on the repl thread -- so lauch it in a separat thread with "future")
you use refs, so it means the values of x and y are intended to be coordinated (in your code they are not: each update is independent); hence in drawRectangle you should wrap the reads in a dosync to be sure that you get a consistent view -- again in your actual code it doesn't matter because x and y are always updated independently.
hth,
(shameless plug: http://conj-labs.eu/)
Although not exactly Clojure advice - consider using KeyAdapter instead of KeyListener. That way you won't have to provide empty implementations for keyReleased and keyTyped. It's generally a good rule to use adapter classes when you're not needing all of the methods of a Listener interface.