The thing i need - is simple queue, something, where i can put tasks, and retrieve them one-by-one from workers(without maintaining order of tasks).
I wrote something like this:
;; Definition
(def q (ref []))
;; Put
(defn put-in-queue [i]
(dosync (alter q conj i)))
;; Get
(defn get-from-queue []
(dosync
(let [v (peek q)]
(alter q pop)
v)))
Is it correct solution? (maybe there are also better solutions for this task)
You should use a clojure.lang.PersistentQueue, right tool for the job and all that :).
Sample below "borrowed" from Hidden features of Clojure:
user> (-> (clojure.lang.PersistentQueue/EMPTY)
(conj 1 2 3)
pop)
(2 3)
See the remainder of that answer for specifics.
The correct solution is probably to use a java.util.concurrent queue, most likely java.util.concurrent.LinkedBlockingQueue. j.u.c queues are robust, perfectly suited to the task and work in Clojure just fine.
See my answer to the Producer consumer with qualifications SO question for a scenario with 5 producers, 2 consumers and a limited-size queue implemented in two ways for comparison: first with c.l.PersistentQueue, second with j.u.c.LinkedBlockingQueue.
as dsm points out, when you need a queue, use a queue, and for times where you really need a vector and want to add things to the end and remove them from the front, subvec and conj are both O(1) functions on vectors.
user> (subvec [1 2 3] 1)
[2 3]
user> (conj [1 2 3] 4)
[1 2 3 4]
user> (-> [] (conj 1 2) (conj 3) (subvec 1))
Related
I am currently working on a tree search algorithm to walk the optimal, cost efficient path through a tree structure.
(def tree
'([1]
[3 5]
[5 6 4]
[2 3 4 5]))
The function initiates multiple "workers" through a recursive function call, each time a new node is reached.
(defn treewalk [tree]
(let [worker_history {}]
(letfn [(get-neighbours [level current_position]
(;; get adjacent nodes one level below)
(worker [current_level current_position cost individual_history]
(if (;; current level above bottom)
(let [neighbours (get-neighbours current_level current_position)]
;; recursive worker call for next 2 branches
(worker (+ level1 1) (first_neighbour) (+ cost (current_cost))
(;; updated individual_history))
(worker (+ level1 1) (second neighbour) (+ cost (current_cost))
(;; updated individual_history)))
;; else: worker at bottom -> insert cost and individual history into worker_history
(assoc worker_history cost individual_history))
)))))
The worker_history map is supposed to store the costs and individual paths and is updated with these values when each individual worker reaches bottom of the tree structure. I am aware that working with these sides effects is not the most elegant way to approach that problem in Clojure!
Currently, I am running into the problem that worker_history only returns ine entry of the very last worker that has finished and does not behave as a static variable within the function scope, so there is no concurrent access of that ofject. How could I still modify my approach in order to achieve this level of "concurrency"?
Your structure isn't tree. In functional languages, trees are usually written as nested lists (or vectors), like this: '(3 (4) (5)) or this '(3 (4 (2) (8)) (5)). If I stick to your instructions recursive tree search, return costs and paths for all workers, this should work:
(defn tree-costs [tree]
(let [worker-history (atom [])]
(letfn [(treewalk [tree cost history]
(if (= (count tree) 1) (swap! worker-history
conj [(+ (first tree) cost)
(conj history (first tree))])
(doseq [node (rest tree)]
(treewalk node
(+ cost (first tree))
(conj history (first tree))))))]
(treewalk tree 0 [])
#worker-history
)))
(tree-costs '(3 (4) (5))) => [[7 [3 4]] [8 [3 5]]]
(tree-costs '(3 (4 (2) (8)) (5))) => [[9 [3 4 2]] [15 [3 4 8]] [8 [3 5]]]
Also check clojure.core.async for thread concurrency.
In order to update worker_hisoty concurently, you can try atom
...
(let [worker_history (atom {})]
...
then update it with swap!
...
(swap! worker_history assoc cost individual_history)
...
Simplistically you think about atom as a global mutable variable with atomic updates
I would caution against using stateful techniques as you miss out on the benefits of pure functions. Without knowing the full context of the problem you are trying to solve I cannot say if using atom will serve you well.
Rather than update an atom for each tree path, a map can be constructed using reduce:
(defn paths [tree]
(when-let [root (ffirst tree)]
(if-let [r (next tree)]
(let [left (paths r)
right (paths (map rest r))]
(map #(cons root %) (concat left right)))
[[root]])))
(defn tree-costs [tree]
(reduce (fn [m path] (update m (reduce + path) (fnil conj #{}) path))
{}
(paths tree)))
I'm currently learning Clojure, and I'm trying to learn how to do things the best way. Today I'm looking at the basic concept of doing things on a sequence, I know the basics of map, filter and reduce. Now I want to try to do a thing to pairs of elements in a sequence, and I found two ways of doing it. The function I apply is println. The output is simply 12 34 56 7
(def xs [1 2 3 4 5 6 7])
(defn work_on_pairs [xs]
(loop [data xs]
(if (empty? data)
data
(do
(println (str (first data) (second data)))
(recur (drop 2 data))))))
(work_on_pairs xs)
I mean, I could do like this
(map println (zipmap (take-nth 2 xs) (take-nth 2 (drop 1 xs))))
;; prints [1 2] [3 4] [5 6], and we loose the last element because zip.
But it is not really nice.. My background is in Python, where I could just say zip(xs[::2], xs[1::2]) But I guess this is not the Clojure way to do it.
So I'm looking for suggestions on how to do this same thing, in the best Clojure way.
I realize I'm so new to Clojure I don't even know what this kind of operation is called.
Thanks for any input
This can be done with partition-all:
(def xs [1 2 3 4 5 6 7])
(->> xs
(partition-all 2) ; Gives ((1 2) (3 4) (5 6) (7))
(map (partial apply str)) ; or use (map #(apply str %))
(apply println))
12 34 56 7
The map line is just to join the pairs so the "()" don't end up in the output.
If you want each pair printed on its own line, change (apply println) to (run! println). Your expected output seems to disagree with your code, so that's unclear.
If you want to dip into transducers, you can do something similar to the threading (->>) form of the accepted answer, but in a single pass over the data.
Assuming
(def xs [1 2 3 4 5 6 7])
has been evaluated already,
(transduce
(comp
(partition-all 2)
(map #(apply str %)))
conj
[]
xs)
should give you the same output if you wrap it in
(apply println ...)
We supply conj (reducing fn) and [] (initial data structure) to specify how the reduce process inside transduce should build up the result.
I wouldn't use a transducer for a list that small, or a process that simple, but it's good to know what's possible!
I wanted a Clojure vector in an atom to model a stateful FIFO (push to the end, pop from the beginning). tried
(def stack (atom []))
then push as follows:
(swap! stack #(conj % 1))
(swap! stack #(conj % 2))
expecting
[1 2]
but getting
(2 1)
Not a big deal, it just means that I must reverse (O(n)) the value of the atom (persistent list) to get the items in the order than I pushed them (e.g., a stream of imperative commands to a virtual machine in order). Still, it was a surprise.
Is there a clojure.core FIFO I can pack into an atom? I thought of priority-map, but it seems overkill. The examples for swap! on clojuredocs.org use lists or maps, not quite what I wanted. I found lots of samples by googling "FIFO Clojure," but some are a bit rich, e.g., clojure.core.cache (map-like and not vector-like); amalloy's ring-buffer (external dependency). Looking for something really small and straightforward. I didn't see an answer in the automatic suggestions from StackOverflow.
Something is not quite right in the wider context of your code; I'm assuming that you have much more going on than what's in the example you posted?
Here's the result in my REPL and what I'd expect:
user=> (def stack (atom []))
#'user/stack
user=> (swap! stack #(conj % 1))
[1]
user=> (swap! stack #(conj % 2))
[1 2]
user=> #stack
[1 2]
When using Vectors, items are added to the end of the collection. The result that you're seeing looks like the behaviour you'd get if stack was a list instead, e.g. adding to the front:
user=> (def stack (atom '()))
#'user/stack
user=> (swap! stack #(conj % 1))
(1)
user=> (swap! stack #(conj % 2))
(2 1)
user=> #stack
(2 1)
So I'm wondering if somewhere in your code, you've somehow ended up defining a list, rather than a vector.
Im trying to understand how to make stateful transducers in core.async.
For example how would I make a transducer that counts the number of elements that have come throgh a channel? For example I want to input to be transfomed into a count that depends on the number of objects that have come before it.
From what I have read the way to go is to use volatile! to hold the state inside the transducer but im still not sure how to put all things together.
You need a stateful transducer returning a reducing function closed over a volatile! tracking the count.
(defn count-xf [rf]
(let [ctr (volatile! 0)]
(fn
([] (rf))
([result] (rf result))
([result _] ; we ignore the input as
(rf result (vswap! ctr inc)))))) ; we just pass on the count
This can be simplified using the core function completing
(defn count-xf [rf]
(let [ctr (volatile! 0)]
(completing
(fn [result _]
(rf result (vswap! ctr inc))))))
E. g. use it so
(let [ch (chan 1 count-xf)]
(onto-chan ch (repeat 10 true))
(<!! (clojure.core.async/into [] ch)))
;-> [1 2 3 4 5 6 7 8 9 10]
Alternatively, you could just use the map-indexed transducer but this would likely help you less to understand how transducers work. Also it requires a bit additional per-step overhead for this particular usecase.
(def count-xf (map-indexed (fn [i _] (inc i))))
Observe that its implementation diverges little from the implementation above.
Further reference: http://clojure.org/reference/transducers
I have an infinite list like that:
((1 1)(3 9)(5 17)...)
I would like to make a hash map out of it:
{:1 1 :3 9 :5 17 ...)
Basically 1st element of the 'inner' list would be a keyword, while second element a value. I am not sure if it would not be easier to do it at creation time, to create the list I use:
(iterate (fn [[a b]] [(computation for a) (computation for b)]) [1 1])
Computation of (b) requires (a) so I believe at this point (a) could not be a keyword... The whole point of that is so one can easily access a value (b) given (a).
Any ideas would be greatly appreciated...
--EDIT--
Ok so I figured it out:
(def my-map (into {} (map #(hash-map (keyword (str (first %))) (first (rest %))) my-list)))
The problem is: it does not seem to be lazy... it just goes forever even though I haven't consumed it. Is there a way to force it to be lazy?
The problem is that hash-maps can be neither infinite nor lazy. They designed for fast key-value access. So, if you have a hash-map you'll be able to perform fast key look-up. Key-value access is the core idea of hash-maps, but it makes creation of lazy infinite hash-map impossible.
Suppose, we have an infinite 2d list, then you can just use into to create hash-map:
(into {} (vec (map vec my-list)))
But there is no way to make this hash-map infinite. So, the only solution for you is to create your own hash-map, like Chouser suggested. In this case you'll have an infinite 2d sequence and a function to perform lazy key lookup in it.
Actually, his solution can be slightly improved:
(def my-map (atom {}))
(def my-seq (atom (partition 2 (range))))
(defn build-map [stop]
(when-let [[k v] (first #my-seq)]
(swap! my-seq rest)
(swap! my-map #(assoc % k v))
(if (= k stop)
v
(recur stop))))
(defn get-val [k]
(if-let [v (#my-map k)]
v
(build-map k)))
my-map in my example stores the current hash-map and my-seq stores the sequence of not yet processed elements. get-val function performs a lazy look-up, using already processed elements in my-map to improve its performance:
(get-val 4)
=> 5
#my-map
=> {4 5, 2 3, 0 1}
And a speed-up:
(time (get-val 1000))
=> Elapsed time: 7.592444 msecs
(time (get-val 1000))
=> Elapsed time: 0.048192 msecs
In order to be lazy, the computer will have to do a linear scan of the input sequence each time a key is requested, at the very least if the key is beyond what has been scanned so far. A naive solution is just to scan the sequence every time, like this:
(defn get-val [coll k]
(some (fn [[a b]] (when (= k a) b)) coll))
(get-val '((1 1)(3 9)(5 17))
3)
;=> 9
A slightly less naive solution would be to use memoize to cache the results of get-val, though this would still scan the input sequence more than strictly necessary. A more aggressively caching solution would be to use an atom (as memoize does internally) to cache each pair as it is seen, thereby only consuming more of the input sequence when a lookup requires something not yet seen.
Regardless, I would not recommend wrapping this up in a hash-map API, as that would imply efficient immutable "updates" that would likely not be needed and yet would be difficult to implement. I would also generally not recommend keywordizing the keys.
If you flatten it down to a list of (k v k v k v k v) with flatten then you can use apply to call hash-map with that list as it's arguments which will git you the list you seek.
user> (apply hash-map (flatten '((1 1)(3 9)(5 17))))
{1 1, 3 9, 5 17}
though it does not keywordize the first argument.
At least in clojure the last value associated with a key is said to be the value for that key. If this is not the case then you can't produce a new map with a different value for a key that is already in the map, because the first (and now shadowed key) would be returned by the lookup function. If the lookup function searches to the end then it is not lazy. You can solve this by writing your own map implementation that uses association lists, though it would lack the performance guarantees of Clojure's trei based maps because it would devolve to linear time in the worst case.
Im not sure keeping the input sequence lazy will have the desired results.
To make a hashmap from your sequence you could try:
(defn to-map [s] (zipmap (map (comp keyword str first) s) (map second s)))
=> (to-map '((1 1)(3 9)(5 17)))
=> {:5 17, :3 9, :1 1}
You can convert that structure to a hash-map later this way
(def it #(iterate (fn [[a b]] [(+ a 1) (+ b 1)]) [1 1]))
(apply hash-map (apply concat (take 3 (it))))
=> {1 1, 2 2, 3 3}