I am trying to implement a solution for minimum-swaps required to sort an array in clojure.
The code works, but takes about a second to solve for the 7 element vector, which is very poor compared to a similar solution in Java. (edited)
I already tried providing the explicit types, but doesnt seem to make a difference
I tried using transients, but has an open bug for subvec, that I am using in my solution- https://dev.clojure.org/jira/browse/CLJ-787
Any pointers on how I can optimize the solution?
;; Find minimumSwaps required to sort the array. The algorithm, starts by iterating from 0 to n-1. In each iteration, it places the least element in the ith position.
(defn minimumSwaps [input]
(loop [mv input, i (long 0), swap-count (long 0)]
(if (< i (count input))
(let [min-elem (apply min (drop i mv))]
(if (not= min-elem (mv i))
(recur (swap-arr mv i min-elem),
(unchecked-inc i),
(unchecked-inc swap-count))
(recur mv,
(unchecked-inc i),
swap-count)))
swap-count)))
(defn swap-arr [vec x min-elem]
(let [y (long (.indexOf vec min-elem))]
(assoc vec x (vec y) y (vec x))))
(time (println (minimumSwaps [7 6 5 4 3 2 1])))
There are a few things that can be improved in your solution, both algorithmically and efficiency-wise. The main improvement is to remember both the minimal element in the vector and its position when you search for it. This allows you to not search for the minimal element again with .indexOf.
Here's my revised solution that is ~4 times faster:
(defn swap-arr [v x y]
(assoc v x (v y) y (v x)))
(defn find-min-and-position-in-vector [v, ^long start-from]
(let [size (count v)]
(loop [i start-from, min-so-far (long (nth v start-from)), min-pos start-from]
(if (< i size)
(let [x (long (nth v i))]
(if (< x min-so-far)
(recur (inc i) x i)
(recur (inc i) min-so-far min-pos)))
[min-so-far min-pos]))))
(defn minimumSwaps [input]
(loop [mv input, i (long 0), swap-count (long 0)]
(if (< i (count input))
(let [[min-elem min-pos] (find-min-and-position-in-vector mv i)]
(if (not= min-elem (mv i))
(recur (swap-arr mv i min-pos),
(inc i),
(inc swap-count))
(recur mv,
(inc i),
swap-count)))
swap-count)))
To understand where are the performance bottlenecks in your program, it is better to use https://github.com/clojure-goes-fast/clj-async-profiler rather than to guess.
Notice how I dropped unchecked-* stuff from your code. It is not as important here, and it is easy to get it wrong. If you want to use them for performance, make sure to check the resulting bytecode with a decompiler: https://github.com/clojure-goes-fast/clj-java-decompiler
A similar implementation in java, runs almost in half the time.
That's actually fairly good for Clojure, given that you use immutable vectors where in Java you probably use arrays. After rewriting the Clojure solution to arrays, the performance would be almost the same.
Related
I defined a function is-prime? in Clojure that returns if a number is prime or not, and I am trying to define a function prime-seq that returns all the prime numbers between two numbersn and m.
I have created the function in Common Lisp since I am more comfortable with it and I am trying to translate the code to Clojure. However, I cannot find how to replace the collect function in Lisp to Clojure.
This is my prime-seq function in Lisp:
(defun prime-seq (i j)
(loop for x from i to j
when (is-prime x)
collect x
)
)
And this is the try I did in Clojure but it is not working:
(defn prime-seq? [n m]
(def list ())
(loop [k n]
(cond
(< k m) (if (prime? k) (def list (conj list k)))
)
)
(println list)
)
Any ideas?
loop in Clojure is not the same as CL loop. You probably want for:
(defn prime-seq [i j]
(for [x (range i j)
:when (is-prime x)]
x))
Which is basically the same as saying:
(defn prime-seq [i j]
(filter is-prime (range i j)))
which may be written using the ->> macro for readability:
(defn prime-seq [i j]
(->> (range i j)
(filter is-prime)))
However, you might actually want a lazy-sequence of all prime numbers which you could write with something like this:
(defonce prime-seq
(let [c (fn [m numbers] (filter #(-> % (mod m) (not= 0)) numbers))
f (fn g [s]
(when (seq s)
(cons (first s)
(lazy-seq (g (c (first s) (next s)))))))]
(f (iterate inc 2))))
The lazy sequence will cache the results of the previous calculation, and you can use things like take-while and drop-while to filter the sequence.
Also, you probably shouldn't be using def inside a function call like that. def is for defining a var, which is essentially global. Then using def to change that value completely destroys the var and replaces it with another var pointing to the new state. It's something that's allowed to enable iterative REPL based development and shouldn't really be used in that way. Var's are designed to isolate changes locally to a thread, and are used as containers for global things like functions and singletons in your system. If the algorithm you're writing needs a local mutable state you could use a transient or an atom, and define a reference to that using let, but it would be more idiomatic to use the sequence processing lib or maybe a transducer.
Loop works more like a tail recursive function:
(defn prime-seq [i j]
(let [l (transient [])]
(loop [k i]
(when (< k j)
(when (is-prime k)
(conj! l k))
(recur (inc k))))
(persistent! l)))
But that should be considered strictly a performance optimisation. The decision to use transients shouldn't be taken lightly, and it's often best to start with a more functional algorithm, benchmark and optimise accordingly. Here is a way to write the same thing without the mutable state:
(defn prime-seq [i j]
(loop [k i
l []]
(if (< k j)
(recur (inc k)
(if (is-prime k)
(conj l k)
l))
l)))
I'd try to use for:
(for [x (range n m) :when (is-prime? x)] x)
So this is my first time programming in clojure and I am running into some issues with stackoverflow with my program. This program basically tries to find all the possible solutions to the N-queens problem
http://en.wikipedia.org/wiki/Eight_queens_puzzle
When I call sol-count (finds number of solutions for a given N) on 10 or higher I get a stack overflow
(defn qextends?
"Returns true if a queen at rank extends partial-sol."
[partial-sol rank]
(if (>= (count partial-sol) 1)
(and
(not= (first partial-sol) (- rank (count partial-sol)))
(not= (first partial-sol) (+ rank (count partial-sol)))
(not= (first partial-sol) rank)
(qextends? (rest partial-sol) rank))
true)
)
(defn qextend-helper [n x partial-sol partial-sol-list]
(if (<= x n)
(if (qextends? partial-sol x)
(qextend-helper n (inc x) partial-sol (conj partial-sol-list (conj partial-sol x)))
(qextend-helper n (inc x) partial-sol partial-sol-list)
)
partial-sol-list)
)
(defn qextend
"Given a vector *partial-sol-list* of all partial solutions of length k,
returns a vector of all partial solutions of length k + 1.
"
[n partial-sol-list]
(if (>= (count partial-sol-list) 1)
(vec (concat (qextend-helper n 1 (first partial-sol-list) [])
(qextend n (rest partial-sol-list))))
nil))
(defn sol-count-helper
[n x partial-sol-list]
(if (<= x (- n 1))
(sol-count-helper n (+ 1 x) (qextend n partial-sol-list))
(qextend n partial-sol-list))
)
(defn sol-count
"Returns the total number of n-queens solutions on an n x n board."
[n]
(count (sol-count-helper n 1 [[]]))
)
Completing dizzystar's answer:
Most of your recursions can be recurred: You can put recur inside if, and hence under and and or, macros which expand to if forms. For example ...
(defn qextend-helper [n x partial-sol partial-sol-list]
(if (<= x n)
(if (qextends? partial-sol x)
(recur n (inc x) partial-sol (conj partial-sol-list (conj partial-sol x)))
(recur n (inc x) partial-sol partial-sol-list))
partial-sol-list)
)
But the recursive call in qextend:
(vec (concat ( ...)
(qextend n (rest partial-sol-list))))
... can't be dealt with by recur, since it is buried in function calls, to concat and vec.
You can solve this problem by returning a lazy sequence, so making the partial-sol-list argument lazy:
get rid of vec - return the sequence - and
replace concat with lazy-cat.
The resulting function is
(defn qextend [n partial-sol-list]
(if (seq partial-sol-list)
(lazy-cat (qextend-helper n 1 (first partial-sol-list) [])
(qextend n (rest partial-sol-list)))
nil))
You also have to avoid counting (and hence realizing) the lazy sequence: so useseq instead to test whether there is anything in partial-sol-list.
It works!
=> (sol-count 11)
2680
For starters, Clojure does not have TCO like other lisps. To mitigate this, Clojure offers loop and recur. In Clojure and other lisps, you generally do not have explicit returns, instead, the focus is on returning values.
I fixed up the qextend-helper to give you a start. I tested a few of your answers against the snippet that I replaced here, and they all resolve to the same solution. You still can't go past 10 even with this snippet inside of it, but if you continue to remove all the tail call recursions, you should be able to get past the stackoverflow errors:
(defn qextend-helper [n x partial-sol partial-sol-list]
(loop [n n
x x
partail-sol partial-sol
partial-sol-list partial-sol-list]
(when (and (<= x n)
(qextends? partail-sol x))
(recur n (inc x) partail-sol (conj partial-sol-list (conj partial-sol x))))))
I should point out also that the above isn't great Clojure either. There are many other solutions online that are less LOC and more Clojure-y, but since you are starting down this route, I am simply attempting to encourage you to remove the errors that you are running into. In this case, removing the tail-calls is a good place to start. With that said, there is nothing that would stop you from using for or other constructs and I encourage you to look into other options after you solved this.
When using recur, resist the temptation to put recur at the end of a function to simulate TCO:
(defn my-funct [x]
.....
(recur x))
Finally, this indention style and putting parens on their own lines does not improve readability.:
(qextend-helper n (inc x) partial-sol partial-sol-list)
)
partial-sol-list)
)
In clojure, this is valid:
(loop [a 5]
(if (= a 0)
"done"
(recur (dec a))))
However, this is not:
(let [a 5]
(if (= a 0)
"done"
(recur (dec a))))
So I'm wondering: why are loop and let separated, given the fact they both (at least conceptually) introduce lexical bindings? That is, why is loop a recur target while let is not?
EDIT: originally wrote "loop target" which I noticed is incorrect.
Consider the following example:
(defn pascal-step [v n]
(if (pos? n)
(let [l (concat v [0])
r (cons 0 v)]
(recur (map + l r) (dec n)))
v))
This function calculates n+mth line of pascal triangle by given mth line.
Now, imagine, that let is a recur target. In this case I won't be able to recursively call the pascal-step function itself from let binding using recur operator.
Now let's make this example a little bit more complex:
(defn pascal-line [n]
(loop [v [1]
i n]
(if (pos? i)
(let [l (concat v [0])
r (cons 0 v)]
(recur (map + l r) (dec i)))
v)))
Now we're calculating nth line of a pascal triangle. As you can see, I need both loop and let here.
This example is quite simple, so you may suggest removing let binding by using (concat v [0]) and (cons 0 v) directly, but I'm just showing you the concept. There may be a more complex examples where let inside a loop is unavoidable.
I implemented a naive solution for printing a Pascal's Triangle of N depth which I'll include below. My question is, in what ways could this be improved to make it more idiomatic? I feel like there are a number of things that seem overly verbose or awkward, for example, this if block feels unnatural: (if (zero? (+ a b)) 1 (+ a b)). Any feedback is appreciated, thank you!
(defn add-row [cnt acc]
(let [prev (last acc)]
(loop [n 0 row []]
(if (= n cnt)
row
(let [a (nth prev (- n 1) 0)
b (nth prev n 0)]
(recur (inc n) (conj row (if (zero? (+ a b)) 1 (+ a b)))))))))
(defn pascals-triangle [n]
(loop [cnt 1 acc []]
(if (> cnt n)
acc
(recur (inc cnt) (conj acc (add-row cnt acc))))))
(defn pascal []
(iterate (fn [row]
(map +' `(0 ~#row) `(~#row 0)))
[1]))
Or if you're going for maximum concision:
(defn pascal []
(->> [1] (iterate #(map +' `(0 ~#%) `(~#% 0)))))
To expand on this: the higher-order-function perspective is to look at your original definition and realize something like: "I'm actually just computing a function f on an initial value, and then calling f again, and then f again...". That's a common pattern, and so there's a function defined to cover the boring details for you, letting you just specify f and the initial value. And because it returns a lazy sequence, you don't have to specify n now: you can defer that, and work with the full infinite sequence, with whatever terminating condition you want.
For example, perhaps I don't want the first n rows, I just want to find the first row whose sum is a perfect square. Then I can just (first (filter (comp perfect-square? sum) (pascal))), without having to worry about how large an n I'll need to choose up front (assuming the obvious definitions of perfect-square? and sum).
Thanks to fogus for an improvement: I need to use +' rather than just + so that this doesn't overflow when it gets past Long/MAX_VALUE.
(defn next-row [row]
(concat [1] (map +' row (drop 1 row)) [1]))
(defn pascals-triangle [n]
(take n (iterate next-row '(1))))
Not as terse as the others, but here's mine:)
(defn A []
(iterate
(comp (partial map (partial reduce +))
(partial partition-all 2 1) (partial cons 0))
[1]))
I am trying to prove Clojure performance can be on equal footing with Java. An important use case I've found is the Quicksort. I have written an implementation as follows:
(set! *unchecked-math* true)
(defn qsort [^longs a]
(let [qs (fn qs [^long low, ^long high]
(when (< low high)
(let [pivot (aget a low)
[i j]
(loop [i low, j high]
(let [i (loop [i i] (if (< (aget a i) pivot)
(recur (inc i)) i))
j (loop [j j] (if (> (aget a j) pivot)
(recur (dec j)) j))
[i j] (if (<= i j)
(let [tmp (aget a i)]
(aset a i (aget a j)) (aset a j tmp)
[(inc i) (dec j)])
[i j])]
(if (< i j) (recur i j) [i j])))]
(when (< low j) (qs low j))
(when (< i high) (qs i high)))))]
(qs 0 (dec (alength a))))
a)
Also, this helps call the Java quicksort:
(defn jqsort [^longs a] (java.util.Arrays/sort a) a))
Now, for the benchmark.
user> (def xs (let [rnd (java.util.Random.)]
(long-array (repeatedly 100000 #(.nextLong rnd)))))
#'user/xs
user> (def ys (long-array xs))
#'user/ys
user> (time (qsort ys))
"Elapsed time: 163.33 msecs"
#<long[] [J#3ae34094>
user> (def ys (long-array xs))
user> (time (jqsort ys))
"Elapsed time: 13.895 msecs"
#<long[] [J#1b2b2f7f>
Performance is worlds apart (an order of magnitude, and then some).
Is there anything I'm missing, any Clojure feature I may have used? I think the main source of performance degradation is when I need to return several values from a loop and must allocate a vector for that. Can this be avoided?
BTW running Clojure 1.4. Also note that I have run the benchmark multiple times in order to warm up the HotSpot. These are the times when they settle down.
Update
The most terrible weakness in my code is not just the allocation of vectors, but the fact that they force boxing and break the primitive chain. Another weakness is using results of loop because they also break the chain. Yep, performance in Clojure is still a minefield.
This version is based on #mikera's, is just as fast and doesn't require the use of ugly macros. On my machine this takes ~12ms vs ~9ms for java.util.Arrays/sort:
(set! *unchecked-math* true)
(set! *warn-on-reflection* true)
(defn swap [^longs a ^long i ^long j]
(let [t (aget a i)]
(aset a i (aget a j))
(aset a j t)))
(defn ^long apartition [^longs a ^long pivot ^long i ^long j]
(loop [i i j j]
(if (<= i j)
(let [v (aget a i)]
(if (< v pivot)
(recur (inc i) j)
(do
(when (< i j)
(aset a i (aget a j))
(aset a j v))
(recur i (dec j)))))
i)))
(defn qsort
([^longs a]
(qsort a 0 (long (alength a))))
([^longs a ^long lo ^long hi]
(when
(< (inc lo) hi)
(let [pivot (aget a lo)
split (dec (apartition a pivot (inc lo) (dec hi)))]
(when (> split lo)
(swap a lo split))
(qsort a lo split)
(qsort a (inc split) hi)))
a))
(defn ^longs rand-long-array []
(let [rnd (java.util.Random.)]
(long-array (repeatedly 100000 #(.nextLong rnd)))))
(comment
(dotimes [_ 10]
(let [as (rand-long-array)]
(time
(dotimes [_ 1]
(qsort as)))))
)
The need for manual inlining is mostly unnecessary starting with Clojure 1.3. With a few type hints only on the function arguments the JVM will do the inlining for you. There is no need to cast index arguments to int for the the array operations - Clojure does this for you.
One thing to watch out for is that nested loop/recur does present problems for JVM inlining since loop/recur doesn't (at this time) support returning primitives. So you have to break apart your code into separate fns. This is for the best as nested loop/recurs get very ugly in Clojure anyhow.
For a more detailed look on how to consistently achieve Java performance (when you actually need it) please examine and understand test.benchmark.
This is slightly horrific because of the macros, but with this code I think you can match the Java speed (I get around 11ms for the benchmark):
(set! *unchecked-math* true)
(defmacro swap [a i j]
`(let [a# ~a
i# ~i
j# ~j
t# (aget a# i#)]
(aset a# i# (aget a# j#))
(aset a# j# t#)))
(defmacro apartition [a pivot i j]
`(let [pivot# ~pivot]
(loop [i# ~i
j# ~j]
(if (<= i# j#)
(let [v# (aget ~a i#)]
(if (< v# pivot#)
(recur (inc i#) j#)
(do
(when (< i# j#)
(aset ~a i# (aget ~a j#))
(aset ~a j# v#))
(recur i# (dec j#)))))
i#))))
(defn qsort
([^longs a]
(qsort a 0 (alength a)))
([^longs a ^long lo ^long hi]
(let [lo (int lo)
hi (int hi)]
(when
(< (inc lo) hi)
(let [pivot (aget a lo)
split (dec (apartition a pivot (inc lo) (dec hi)))]
(when (> split lo) (swap a lo split))
(qsort a lo split)
(qsort a (inc split) hi)))
a)))
The main tricks are:
Do everything with primitive arithmetic
Use ints for the array indexes (this avoids some unnecessary casts, not a big deal but every little helps....)
Use macros rather than functions to break up the code (avoids function call overhead and parameter boxing)
Use loop/recur for maximum speed in the inner loop (i.e. partitioning the subarray)
Avoid constructing any new objects on the heap (so avoid vectors, sequences, maps etc.)
The Joy of Clojure, Chapter 6.4 describes a lazy quicksort algorithm.The beauty of lazy sorting is that it will only do as much work as necessary to find the first x values. So if x << n this algorithm is O(n).
(ns joy.q)
(defn sort-parts
"Lazy, tail-recursive, incremental quicksort. Works against
and creates partitions based on the pivot, defined as 'work'."
[work]
(lazy-seq
(loop [[part & parts] work]
(if-let [[pivot & xs] (seq part)]
(let [smaller? #(< % pivot)]
(recur (list*
(filter smaller? xs)
pivot
(remove smaller? xs)
parts)))
(when-let [[x & parts] parts]
(cons x (sort-parts parts)))))))
(defn qsort [xs]
(sort-parts (list xs)))
By examining the main points from mikera's answer, you can see that they are mostly focused on eliminating the overhead introduced by using idiomatic (as opposed to tweaked) Clojure, which would probably not exist in an idiomatic Java implementation:
primitive arithmetic - slightly easier and more idiomatic in Java, you are more likely to use ints than Integers
ints for the array indexes - the same
Use macros rather than functions to break up the code (avoids functional call overhead and boxing) - fixes a problem introduced by using the language. Clojure encourages functional style, hence a function call overhead (and boxing).
Use loop/recur for maximum speed in the inner loop - in Java you'd idiomatically use an ordinary loop (which is what loop/recur compiles to anyway, as far as I know)
That being said, there actually is another trivial solution. Write (or find) an efficient Java implementation of Quick Sort, say something with a signature like this:
Sort.quickSort(long[] elems)
And then call it from Clojure:
(Sort/quickSort elems)
Checklist:
as efficient as in Java - yes
idiomatic in Clojure - arguably yes, I'd say that Java-interop is one of Clojure's core features.
reusable - yes, there's a good chance that you can easily find a very efficient Java implementation already written.
I'm not trying to troll, I understand what you are trying to find out with these experiments I'm just adding this answer for the sake of completeness. Let's not overlook the obvious one! :)