I have a a web service endpoint that uses a mutable resource from a Java library. That web service endpoint can receive multiple queries at the same time. (the endpoint is implemented using Ring/Compojure). Creating these resources is costly, so re-creating them for every web service call is really inefficient.
What I want to do is to create a pool of that resource that I populate when the web service starts. Then each time the endpoint is called, it takes a resource from the pool, use it for its processing, and then push it back into the pool and wait for the next call to happen.
I am wondering what would be the best way to do that in Clojure? Is there a "pool" Clojure library that could help me with that?
I naively tried to implement that using an vector in an atom where each item of the vector is that resource. However, it quickly learned that it could not really work that way.
This is based on Timothy Pratley's idea of using refs:
(def pool (ref ['a 'b 'c]))
(defn take' [pool]
(dosync
(let [[h & t] #pool]
(ref-set pool (vec t))
h)))
(defn put [pool x]
(dosync
(alter pool conj x)
nil))
(take' pool) ;; => 'a
(put pool 'a) ;; => nil
(take' pool) ;; => 'a
(take' pool) ;; => 'b
(take' pool) ;; => 'c
Maybe not the best way to attack this. But I like the simplicity of it.
To implement a pool you need to address 2 concerns:
Concurrency. Use locking https://clojuredocs.org/clojure.core/locking or a ref instead of an atom. Requests can be simultaneous, so you need to be careful that it is impossible for two consumers to receive the same resource.
Releasing resources. Consider using a pattern like (with-open ...) i.e. a macro that expands to a try-finally where the resource is released back to the open pool when you leave the block scope.
You might want to designate an 'error' status as well as 'available' or 'in use', where the resource may need to be released and recreated.
Have a look to this kul/pool. It uses Apache Commons Pool. I hope it's useful.
You can also use an atom:
(def pool (atom ['c 'b 'a]))
(defn take'
[pool]
(loop []
(let [p #pool]
(if (compare-and-set! pool p (pop p))
(peek p)
(recur)))))
(defn put
[pool x]
(swap! pool conj x)
nil)
(take' pool) ;; => 'a
(put pool 'a) ;; => nil
(take' pool) ;; => 'a
(take' pool) ;; => 'b
(take' pool) ;; => 'c
Related
I'd like to test whether a collection is transient. I tried the following function:
(defn transient? [coll]
(extends? clojure.lang.ITransientCollection (type coll)))
But it doesn't work:
user=> (transient? {})
NullPointerException clojure.core/implements? (core_deftype.clj:512)
user=> (transient? (transient {}))
NullPointerException clojure.core/implements? (core_deftype.clj:512)
The documentation of extends? doesn't mention that it would work with Java interfaces anyway. I was just hoping they'd work. Is there any other way to determine if a collection is transient?
I'm using Clojure 1.5.1.
You need to use instance? not extends?.
extends? asks whether a type extends a protocol. ITransientCollection is a Java interface, not a protocol, so instance? is the tool for that.
user=> (instance? clojure.lang.ITransientCollection (transient {}))
true
user=> (instance? clojure.lang.ITransientCollection {})
false
I'm trying to handle duplicate events (MIDI note-on and note-off signals coming in triplicate). It seems like using agents and locking works, unless the events happen (nearly) simultaneously.
Here's an example:
(def test-val (agent 0))
(def allow-update (agent true))
(defn increase-val []
(locking allow-update
(when #allow-update
(send allow-update (fn [_ x] x) false)
(send test-val + 1)))
(print #test-val))
(defn run-futures [delay-time]
(send allow-update (fn [_ x] x) true)
(send test-val * 0)
(dotimes [_ 20]
(Thread/sleep delay-time)
(future (increase-val))))
If I test it with a slight delay between calls to increase-val:
(run-futures 2)
;outputs 0111111111111111111 every time, as expected
But if I let all the calls to increase-val happen all at once:
(run-futures 0)
;001222222222222222
(run-futures 0)
;000000145555555555
(run-futures 0)
;000000013677777777
It seems like the lock doesn't have time to get turned on, so the agent gets increased by multiple futures.
I'm hoping I'm missing something here that will allow me to make sure I don't act on duplicate simultaneous events.
Thanks!
Because Agents are asynchronous and uncoordinated, there can be a delay between when you send them a message and the actual running of the action that was sent. During this delay
it's possible for most, or all, of the calls to (send test-val + 1) to make it through the locked section before the first one actually runs and sets #allow-update to false. You can see this in the leading 0s before the state of test-val gets changed.
atoms may be better suited to interacting with locks, though refs may be the right tool for coordinated access to multiple identities.
I have a piece of code which I run in parallel using a future -
(def services [s1 s2])
(defn get-data []
(->>
(map #(future
(->> (p/fetch %)
(p/parse %)))
services)
(map deref)
(apply concat)))
(get-data)
The requirement for the above piece of code is to run two services in parallel. So I want to create a test that determines if the function (get-data) runs code for both the services in parallel or not. How do write such a test?
Use an agent to record the start and stop events into a sequence, then read the sequence to ensure that it contains at least two start events before the first stop event.
(map #(future
(send watcher conj :start)
(let [result (->> (p/fetch %)
(p/parse %))]
(send watcher conj :finish)
result)))
or you can use with-redefs to wrap your fetch and parse functions to avoid mixing test and production code.
I am learning clojure and try out its concurrency and effectiveness via a producer consumer example.
Did that and it felt pretty akward with having to use ref and deref and also watch and unwatch.
I tried to check other code snippet; but is there a better way of re factoring this other than using Java Condition await() and signal() methods along with Java Lock. I did not want to use anything in Java.
here is the code; i guess I would have made many mistakes here with my usage...
;a simple producer class
(ns my.clojure.producer
(:use my.clojure.consumer)
(:gen-class)
)
(def tasklist( ref (list) )) ;this is declared as a global variable; to make this
;mutable we need to use the fn ref
(defn gettasklist[]
(deref tasklist) ;we need to use deref fn to return the task list
)
(def testagent (agent 0)); create an agent
(defn emptytasklist[akey aref old-val new-val]
(doseq [item (gettasklist)]
(println(str "item is") item)
(send testagent consume item)
(send testagent increment item)
)
(. java.lang.Thread sleep 1000)
(dosync ; adding a transaction for this is needed to reset
(remove-watch tasklist "key123"); removing the watch on the tasklist so that it does not
; go to a recursive call
(ref-set tasklist (list ) ) ; we need to make it as a ref to reassign
(println (str "The number of tasks now remaining is=") (count (gettasklist)))
)
(add-watch tasklist "key123" emptytasklist)
)
(add-watch tasklist "key123" emptytasklist)
(defn addtask [task]
(dosync ; adding a transaction for this is needed to refset
;(println (str "The number of tasks before") (count (gettasklist)))
(println (str "Adding a task") task)
(ref-set tasklist (conj (gettasklist) task )) ; we need to make it as a ref to reassign
;(println (str "The number of tasks after") (count (gettasklist)))
)
)
Here is the consumer code
(ns my.clojure.consumer
)
(defn consume[c item]
(println "In the consume method:Item is " c item )
item
)
(defn increment [c n]
(println "parmeters are" c n)
(+ c n)
)
And here is the test code ( I have used maven to run clojure code and used NetBeans to edit as this is more familiar to me coming from Java - folder structure and pom at - https://github.com/alexcpn/clojure-evolve
(ns my.clojure.Testproducer
(:use my.clojure.producer)
(:use clojure.test)
(:gen-class)
)
(deftest test-addandcheck
(addtask 1)
(addtask 2)
(is(= 0 (count (gettasklist))))
(println (str "The number of tasks are") (count (gettasklist)))
)
If anybody can refactor this lightly so that I can read and understand the code then it will be great; Else I guess I will have to learn more
Edit -1
I guess using a global task list and making it available to other functions by de-referencing it (deref) and again making it mutable by ref is not the way in clojure;
So changing the addTask method to directly send the incoming tasks to an agent
(defn addtask [task]
(dosync ; adding a transaction for this is needed to refset
(println (str "Adding a task") task)
;(ref-set tasklist (conj (gettasklist) task )) ; we need to make it as a ref to reassign
(def testagent (agent 0)); create an agent
(send testagent consume task)
(send testagent increment task)
)
However when I tested it
(deftest test-addandcheck
(loop [task 0]
(when ( < task 100)
(addtask task)
(recur (inc task))))
(is(= 0 (count (gettasklist))))
(println (str "The number of tasks are") (count (gettasklist)))
)
after sometime the I am getting Java rejected execution exception -- This is fine if you do Java threads, because you take full control. But from clojure this looks odd, especially since you are not selecting the ThreadPool stratergy yourself
Adding a task 85
Exception in thread "pool-1-thread-4" java.util.concurrent.RejectedExecutionExce
ption
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution
(ThreadPoolExecutor.java:1759)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.jav
a:767)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.ja
va:658)
at clojure.lang.Agent$Action.execute(Agent.java:56)
at clojure.lang.Agent$Action.doRun(Agent.java:95)
at clojure.lang.Agent$Action.run(Agent.java:106)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
utor.java:885)Adding a task 86
Adding a task 87
I think modeling producer & consumer in clojure would be done the easiest (and most efficiently) using lamina channels.
I have made similar clojure program with producer/consumer pattern in Computing Folder Sizes Asynchronously.
I don't think that you really need to use Refs.
You have single tasklist that is mutable state that you want to alter synchronously. Changes are relatively quick and do not depend on any other external state, only the variable that comes in to consumer.
As far with atoms there is swap! function which can help you do the changes the way I understood you need.
You can look at my snippet Computing folder size i think it can at least show you proper use of atoms & agents. I played with it a lot, so it should be correct.
Hope it helps!
Regards,
Jan
I was looking at the Clojure examples for Twitter storm. He just used LinkedBlockingQueue. It's easy to use, concurrent and performs well. Sure, it lacks the sex-appeal of an immutable Clojure solution, but it will work well.
I've come across several use cases where I need the ability to:
strictly control the number of worker threads on both the producer and consumer side
control the maximum size of the "work queue" in order to limit memory consumption
detect when all work has been completed so that I can shut down the workers
I've found that the clojure built-in concurrency features (while amazingly simple and useful in their own right) make the first two bullet points difficult. lamina looks great, but I didn't see a way that it would address my particular use cases w/o the same sort of extra plumbing that I'd need to do around an implementation based on BlockingQueue.
So, I ended up hacking together a simple clojure library to try to solve my problems. It is basically just a wrapper around BlockingQueue that attempts to conceal some of the Java constructs and provide a higher-level producer-consumer API. I'm not entirely satisfied with the API yet; it'll likely evolve a bit further... but it's operational:
https://github.com/cprice-puppet/freemarket
Example usage:
(def myproducer (producer producer-work-fn num-workers max-work))
(def myconsumer (consumer myproducer consumer-work-fn num-workers max-results))
(doseq [result (work-queue->seq (:result-queue myconsumer))]
(println result))
Feedback / suggestions / contributions would be welcomed!
How to achieve Aspect-Oriented Programming in Clojure? Do we need AOP in Clojure?
Let's say we want plain vanilla Clojure solution (no AspectJ).
Aspect-Oriented Programming is typically used to add cross-cutting functionality to code that would otherwise get hopelessly intertwined with business logic. A great example is logging - you don't really want logging code scattered everywhere in your code base.
You don't really need AOP in Clojure because it's easy to achieve this with other techniques in Clojure.
For example, you can use higher-order functions to "wrap" other functions with cross cutting functionality:
; a simple function - the "business logic"
(defn my-calculation [a b]
(+ a b))
; higher order function that adds logging to any other function
(defn wrap-with-logging [func]
(fn [& args]
(let [result (apply func args)]
(println "Log result: " result)
result)))
; create a wrapped version of the original function with logging added
(def my-logged-calculation (wrap-with-logging my-calculation))
(my-logged-calculation 7 9)
=> Log result: 16
=> 16
AOP IMHO is just an artifact of certain kinds of static programming languages. AFAIKS it's usually just a bunch of non-standard compiler extensions. I've not yet seen any application of AOP that can't be solved better & natively in more dynamic languages. Clojure is certainly dynamic enough, and that's without even considering macros.
I may be wrong, but if so, I'd need to see an actual AOP use case that can't be implemented just as well in pure clojure.
Edit: just to be clear: I refuse to see things like elisp's advice as aspect oriented. In dynamic languages, those are just techniques to be used whenever you need them, with no need for language support other than rebinding of function definitions - which all lisps support anyway.
There's no need to treat them as special - you can easily define your own defadvice-like function in clojure. See for example, compojure's wrap! macro, which is actually deprecated since you generally don't even need it.
Aspect oriented programming is a great way to achieve seperation of concernes in Java. Clojure's composable abstractions achieve this very well. See this question also. The topic is covered really well in The Joy Of Clojure.
as for an example of Aspect Oriented Clojure by another name check out the Ring web framework
Well you could be more AOP w/ Clojure easily. Just use metadata in functions to informe when you want logs:
(defn ^:log my-calculation
[a b]
(+ a b))
Then you can redefine all functions, wrapping them w/ logging automatically. Part of this code (together w/ unwrap functions bellow):
(defn logfn
[f topic severity error-severity]
(fn [& args]
(try
(if severity
(let [r (apply f args)]
(log* topic {:args args, :ret r} severity)
r)
(apply f args))
(catch Exception e
(if error-severity
(let [data {:args args, :error (treat-error e), :severity error-severity}]
(log* topic data error-severity)
(throw e))
(throw e))))))
(defn logfn-ns
"Wrap function calls for logging on call or on error.
By default, do nothing. When any :log or :log-error, enables logging. If ^:log,
only log on error (default severity error).
Can customize log severity w/ e.g. ^{:log info} or on error log severity likewise."
[ns alias]
(doseq [s (keys (ns-interns ns))
:let [v (ns-resolve ns s)
f #v
log (-> v meta :log)
log-error (-> v meta :log-error)]
:when (and (ifn? f)
(-> v meta :macro not)
(-> v meta :logged not) ;; make it idempotent
(or log log-error))]
(let [log (if (= log true) nil log)
log-error (or log-error "error")
f-with-log (logfn f
(str alias "/" s)
log
log-error)]
(alter-meta! (intern ns s f-with-log)
(fn [x]
(-> x
(assoc :logged true)
(assoc :unlogged #v)))))))
(defn unlogfn-ns
"Reverts logfn-ns."
[ns]
(doseq [s (keys (ns-interns ns))
:let [v (ns-resolve ns s)]
:when (-> v meta :logged)]
(let [f-without-log (-> v meta :unlogged)]
(alter-meta! (intern ns s f-without-log)
(fn [x]
(-> x
(dissoc :logged)
(dissoc :unlogged)))))))
You just call (log/logfn-ns 'my.namespace "some alias") and all is wrapped w/ logging (and some).
PS: My custom logger above have a topic which is "some alias/function name"
PS2: Also wrapped w/ try/catch.
PS3: Didn't like this so much. Reverted to have explicit logging.