I found this code in Clojure to sieve out first n prime numbers:
(defn sieve [n]
(let [n (int n)]
"Returns a list of all primes from 2 to n"
(let [root (int (Math/round (Math/floor (Math/sqrt n))))]
(loop [i (int 3)
a (int-array n)
result (list 2)]
(if (>= i n)
(reverse result)
(recur (+ i (int 2))
(if (< i root)
(loop [arr a
inc (+ i i)
j (* i i)]
(if (>= j n)
arr
(recur (do (aset arr j (int 1)) arr)
inc
(+ j inc))))
a)
(if (zero? (aget a i))
(conj result i)
result)))))))
Then I wrote the equivalent (I think) code in Scheme (I use mit-scheme)
(define (sieve n)
(let ((root (round (sqrt n)))
(a (make-vector n)))
(define (cross-out t to dt)
(cond ((> t to) 0)
(else
(vector-set! a t #t)
(cross-out (+ t dt) to dt)
)))
(define (iter i result)
(cond ((>= i n) (reverse result))
(else
(if (< i root)
(cross-out (* i i) (- n 1) (+ i i)))
(iter (+ i 2) (if (vector-ref a i)
result
(cons i result))))))
(iter 3 (list 2))))
The timing results are:
For Clojure:
(time (reduce + 0 (sieve 5000000)))
"Elapsed time: 168.01169 msecs"
For mit-scheme:
(time (fold + 0 (sieve 5000000)))
"Elapsed time: 3990 msecs"
Can anyone tell me why mit-scheme is more than 20 times slower?
update: "the difference was in iterpreted/compiled mode. After I compiled the mit-scheme code, it was running comparably fast. – abo-abo Apr 30 '12 at 15:43"
Modern incarnations of the Java Virtual Machine have extremely good performance when compared to interpreted languages. A significant amount of engineering resource has gone into the JVM, in particular the hotspot JIT compiler, highly tuned garbage collection and so on.
I suspect the difference you are seeing is primarily down to that. For example if you look Are the Java programs faster? you can see a comparison of java vs ruby which shows that java outperforms by a factor of 220 on one of the benchmarks.
You don't say what JVM options you are running your clojure benchmark with. Try running java with the -Xint flag which runs in pure interpreted mode and see what the difference is.
Also, it's possible that your example is too small to really warm-up the JIT compiler. Using a larger example may yield an even larger performance difference.
To give you an idea of how much Hotspot is helping you. I ran your code on my MBP 2011 (quad core 2.2Ghz), using java 1.6.0_31 with default opts (-server hotspot) and interpreted mode (-Xint) and see a large difference
; with -server hotspot (best of 10 runs)
>(time (reduce + 0 (sieve 5000000)))
"Elapsed time: 282.322 msecs"
838596693108
; in interpreted mode using -Xint cmdline arg
> (time (reduce + 0 (sieve 5000000)))
"Elapsed time: 3268.823 msecs"
838596693108
As to comparing Scheme and Clojure code, there were a few things to simplify at the Clojure end:
don't rebind the mutable array in loops;
remove many of those explicit primitive coercions, no change in performance. As of Clojure 1.3 literals in function calls compile to primitives if such a function signature is available, and generally the difference in performance is so small that it gets quickly drowned by any other operations happening in a loop;
add a primitive long annotation into the fn signature, thus removing the rebinding of n;
call to Math/floor is not needed -- the int coercion has the same semantics.
Code:
(defn sieve [^long n]
(let [root (int (Math/sqrt n))
a (int-array n)]
(loop [i 3, result (list 2)]
(if (>= i n)
(reverse result)
(do
(when (< i root)
(loop [inc (+ i i), j (* i i)]
(when (>= j n) (aset a j 1) (recur inc (+ j inc)))))
(recur (+ i 2) (if (zero? (aget a i))
(conj result i)
result)))))))
Related
I'm starting to learn Clojure and have decided that doing some projects on HackerRank is a good way to do that. What I'm finding is that my Clojure solutions are horribly slow. I'm assuming that's because I'm still thinking imperatively or just don't know enough about how Clojure operates. The latest problem I wrote solutions for was Down To Zero II. Here's my Java code
import java.io.BufferedReader;
import java.io.InputStreamReader;
public class Solution {
private static final int MAX_NUMBER = 1000000;
private static final BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
public static int[] precompute() {
int[] values = new int[MAX_NUMBER];
values[0] = 0;
values[1] = 1;
for (int i = 1; i < MAX_NUMBER; i += 1) {
if ((values[i] == 0) || (values[i] > (values[i - 1] + 1))) {
values[i] = (values[i - 1] + 1);
}
for (int j = 1; j <= i && (i * j) < MAX_NUMBER; j += 1) {
int mult = i * j;
if ((values[mult] == 0) || (values[mult] > (values[i] + 1))) {
values[mult] = values[i] + 1;
}
}
}
return values;
}
public static void main(String[] args) throws Exception {
int numQueries = Integer.parseInt(reader.readLine());
int[] values = Solution.precompute();
for (int loop = 0; loop < numQueries; loop += 1) {
int query = Integer.parseInt(reader.readLine());
System.out.println(values[query]);
}
}
}
My Clojure implementation is
(def MAX-NUMBER 1000000)
(defn set-i [out i]
(cond
(= 0 i) (assoc out i 0)
(= 1 i) (assoc out i 1)
(or (= 0 (out i))
(> (out i) (inc (out (dec i)))))
(assoc out i (inc (out (dec i))))
:else out))
(defn set-j [out i j]
(let [mult (* i j)]
(if (or (= 0 (out mult)) (> (out mult) (inc (out i))))
(assoc out mult (inc (out i)))
out)))
;--------------------------------------------------
; Precompute the values for all possible inputs
;--------------------------------------------------
(defn precompute []
(loop [i 0 out (vec (repeat MAX-NUMBER 0))]
(if (< i MAX-NUMBER)
(recur (inc i) (loop [j 1 new-out (set-i out i)]
(if (and (<= j i) (< (* i j) MAX-NUMBER))
(recur (inc j) (set-j new-out i j))
new-out)))
out)))
;--------------------------------------------------
; Read the number of queries
;--------------------------------------------------
(def num-queries (Integer/parseInt (read-line)))
;--------------------------------------------------
; Precompute the solutions
;--------------------------------------------------
(def values (precompute))
;--------------------------------------------------
; Read and process each query
;--------------------------------------------------
(loop [iter 0]
(if (< iter num-queries)
(do
(println (values (Integer/parseInt (read-line))))
(recur (inc iter)))))
The Java code runs in about 1/10 of a second on my machine, while the Clojure code takes close to 2 seconds. Since it's the same machine, with the same JVM, it means I'm doing something wrong in Clojure.
How do people go about trying to translate this type of code? What are the gotchas that are causing it to be so much slower?
I'm going to do some transformations to your code (which might be slightly outside of what you were originally asking)
and then address your more specific questions.
I know it's almost two years later, but after running across your question and spending way too much time fighting with
HackerRank and its time limits, I thought I would post an answer. Does achieving a solution within HR's environment and
time limits make us better Clojure programmers? I didn't learn the answer to that. But I'll share what I did learn.
I found a slightly slimmer version of your same algorithm. It still has two loops, but the update only happens once in
the inner loop, and many of the conditions are handled in a min function. Here is my adaptation of it:
(defn compute
"Returns a vector of down-to-zero counts for all numbers from 0 to m."
[m]
(loop [i 2 out (vec (range (inc m)))]
(if (<= i m)
(recur (inc i)
(loop [j 1 out out]
(let [ij (* i j)]
(if (and (<= j i) (<= ij m))
(recur (inc j)
(assoc out ij (min (out ij) ;; current value
(inc (out (dec ij))) ;; steps from value just below
(inc (out i))))) ;; steps from a factor
out))))
out)))
Notice we're still using loop/recur (twice), we're still using a vector to hold the output. But some differences:
We initialize out to incrementing integers. This is the worst case number of steps for every value, and once
initialized, we don't have to test that a value equals 0 and we can skip indices 0 and 1 and start the outer loop at
index 2. (We also fix a bug in your original and make sure out contains MAX-NUMBER+1 values.)
All three tests happen inside a min function that encapsulates the original logic: a value will be
updated only if it's a shorter number of steps from the number just below it, or from one of it's factors.
The tests are now simple enough that we don't need to break them out into separate functions.
This code (along with your original) is fast enough to pass some of the test cases in HR, but not all. Here are some
things to speed this up:
Use int-array instead of vec. This means we'll use aset instead of assoc and aget instead of calling out
with an index. It also means that loop/recur isn't the best structure anymore (because we are no longer passing
around new versions of an immutable vector, but actually mutating a java.util.Array); instead we'll use doseq.
Type hints. This alone makes a huge speed difference. When testing your code, include a form at the top (set! *warn-on-reflection* true) and you'll see where Clojure is having to do extra work to figure out what types it is
dealing with.
Use custom I/O functions to read the input. HR's boilerplate I/O code is supposed to let you focus on solving the
challenge and not worry about I/O, but it is basically garbage, and often the culprit behind your program timing out.
Below is a version that incorporates the tips above and runs fast enough to pass all test cases. I've included my custom
I/O approach that I've been using for all my HR challenges. One nice benefit of using doseq is we can include a
:let and a :while clause within the binding form, removing some of the indentation within the body of doseq. Also
notice a few strategically placed type hints that really speed up the program.
(ns down-to-zero-int-array)
(set! *warn-on-reflection* true)
(defn compute
"Returns a vector of down-to-zero counts for all numbers from 0 to m."
^ints [m]
(let [out ^ints (int-array (inc m) (range (inc m)))]
(doseq [i (range 2 (inc m)) j (range 1 (inc i)) :let [ij (* i j)] :while (<= ij m)]
(aset out ij (min (aget out ij)
(inc (aget out (dec ij)))
(inc (aget out i)))))
out))
(let [tokens ^java.io.StreamTokenizer
(doto (java.io.StreamTokenizer. (java.io.BufferedReader. *in*))
(.parseNumbers))]
(defn next-int []
"Read next integer from input. As fast as `read-line` for a single value,
and _much_ faster than `read-line`+`split` for multiple values on same line."
(.nextToken tokens)
(int (.-nval tokens))))
(def MAX 1000000)
(let [q (next-int)
down-to-zero (compute MAX)]
(doseq [n (repeatedly q next-int)]
(println (aget down-to-zero n))))
I am trying to implement a solution for minimum-swaps required to sort an array in clojure.
The code works, but takes about a second to solve for the 7 element vector, which is very poor compared to a similar solution in Java. (edited)
I already tried providing the explicit types, but doesnt seem to make a difference
I tried using transients, but has an open bug for subvec, that I am using in my solution- https://dev.clojure.org/jira/browse/CLJ-787
Any pointers on how I can optimize the solution?
;; Find minimumSwaps required to sort the array. The algorithm, starts by iterating from 0 to n-1. In each iteration, it places the least element in the ith position.
(defn minimumSwaps [input]
(loop [mv input, i (long 0), swap-count (long 0)]
(if (< i (count input))
(let [min-elem (apply min (drop i mv))]
(if (not= min-elem (mv i))
(recur (swap-arr mv i min-elem),
(unchecked-inc i),
(unchecked-inc swap-count))
(recur mv,
(unchecked-inc i),
swap-count)))
swap-count)))
(defn swap-arr [vec x min-elem]
(let [y (long (.indexOf vec min-elem))]
(assoc vec x (vec y) y (vec x))))
(time (println (minimumSwaps [7 6 5 4 3 2 1])))
There are a few things that can be improved in your solution, both algorithmically and efficiency-wise. The main improvement is to remember both the minimal element in the vector and its position when you search for it. This allows you to not search for the minimal element again with .indexOf.
Here's my revised solution that is ~4 times faster:
(defn swap-arr [v x y]
(assoc v x (v y) y (v x)))
(defn find-min-and-position-in-vector [v, ^long start-from]
(let [size (count v)]
(loop [i start-from, min-so-far (long (nth v start-from)), min-pos start-from]
(if (< i size)
(let [x (long (nth v i))]
(if (< x min-so-far)
(recur (inc i) x i)
(recur (inc i) min-so-far min-pos)))
[min-so-far min-pos]))))
(defn minimumSwaps [input]
(loop [mv input, i (long 0), swap-count (long 0)]
(if (< i (count input))
(let [[min-elem min-pos] (find-min-and-position-in-vector mv i)]
(if (not= min-elem (mv i))
(recur (swap-arr mv i min-pos),
(inc i),
(inc swap-count))
(recur mv,
(inc i),
swap-count)))
swap-count)))
To understand where are the performance bottlenecks in your program, it is better to use https://github.com/clojure-goes-fast/clj-async-profiler rather than to guess.
Notice how I dropped unchecked-* stuff from your code. It is not as important here, and it is easy to get it wrong. If you want to use them for performance, make sure to check the resulting bytecode with a decompiler: https://github.com/clojure-goes-fast/clj-java-decompiler
A similar implementation in java, runs almost in half the time.
That's actually fairly good for Clojure, given that you use immutable vectors where in Java you probably use arrays. After rewriting the Clojure solution to arrays, the performance would be almost the same.
I am trying to prove Clojure performance can be on equal footing with Java. An important use case I've found is the Quicksort. I have written an implementation as follows:
(set! *unchecked-math* true)
(defn qsort [^longs a]
(let [qs (fn qs [^long low, ^long high]
(when (< low high)
(let [pivot (aget a low)
[i j]
(loop [i low, j high]
(let [i (loop [i i] (if (< (aget a i) pivot)
(recur (inc i)) i))
j (loop [j j] (if (> (aget a j) pivot)
(recur (dec j)) j))
[i j] (if (<= i j)
(let [tmp (aget a i)]
(aset a i (aget a j)) (aset a j tmp)
[(inc i) (dec j)])
[i j])]
(if (< i j) (recur i j) [i j])))]
(when (< low j) (qs low j))
(when (< i high) (qs i high)))))]
(qs 0 (dec (alength a))))
a)
Also, this helps call the Java quicksort:
(defn jqsort [^longs a] (java.util.Arrays/sort a) a))
Now, for the benchmark.
user> (def xs (let [rnd (java.util.Random.)]
(long-array (repeatedly 100000 #(.nextLong rnd)))))
#'user/xs
user> (def ys (long-array xs))
#'user/ys
user> (time (qsort ys))
"Elapsed time: 163.33 msecs"
#<long[] [J#3ae34094>
user> (def ys (long-array xs))
user> (time (jqsort ys))
"Elapsed time: 13.895 msecs"
#<long[] [J#1b2b2f7f>
Performance is worlds apart (an order of magnitude, and then some).
Is there anything I'm missing, any Clojure feature I may have used? I think the main source of performance degradation is when I need to return several values from a loop and must allocate a vector for that. Can this be avoided?
BTW running Clojure 1.4. Also note that I have run the benchmark multiple times in order to warm up the HotSpot. These are the times when they settle down.
Update
The most terrible weakness in my code is not just the allocation of vectors, but the fact that they force boxing and break the primitive chain. Another weakness is using results of loop because they also break the chain. Yep, performance in Clojure is still a minefield.
This version is based on #mikera's, is just as fast and doesn't require the use of ugly macros. On my machine this takes ~12ms vs ~9ms for java.util.Arrays/sort:
(set! *unchecked-math* true)
(set! *warn-on-reflection* true)
(defn swap [^longs a ^long i ^long j]
(let [t (aget a i)]
(aset a i (aget a j))
(aset a j t)))
(defn ^long apartition [^longs a ^long pivot ^long i ^long j]
(loop [i i j j]
(if (<= i j)
(let [v (aget a i)]
(if (< v pivot)
(recur (inc i) j)
(do
(when (< i j)
(aset a i (aget a j))
(aset a j v))
(recur i (dec j)))))
i)))
(defn qsort
([^longs a]
(qsort a 0 (long (alength a))))
([^longs a ^long lo ^long hi]
(when
(< (inc lo) hi)
(let [pivot (aget a lo)
split (dec (apartition a pivot (inc lo) (dec hi)))]
(when (> split lo)
(swap a lo split))
(qsort a lo split)
(qsort a (inc split) hi)))
a))
(defn ^longs rand-long-array []
(let [rnd (java.util.Random.)]
(long-array (repeatedly 100000 #(.nextLong rnd)))))
(comment
(dotimes [_ 10]
(let [as (rand-long-array)]
(time
(dotimes [_ 1]
(qsort as)))))
)
The need for manual inlining is mostly unnecessary starting with Clojure 1.3. With a few type hints only on the function arguments the JVM will do the inlining for you. There is no need to cast index arguments to int for the the array operations - Clojure does this for you.
One thing to watch out for is that nested loop/recur does present problems for JVM inlining since loop/recur doesn't (at this time) support returning primitives. So you have to break apart your code into separate fns. This is for the best as nested loop/recurs get very ugly in Clojure anyhow.
For a more detailed look on how to consistently achieve Java performance (when you actually need it) please examine and understand test.benchmark.
This is slightly horrific because of the macros, but with this code I think you can match the Java speed (I get around 11ms for the benchmark):
(set! *unchecked-math* true)
(defmacro swap [a i j]
`(let [a# ~a
i# ~i
j# ~j
t# (aget a# i#)]
(aset a# i# (aget a# j#))
(aset a# j# t#)))
(defmacro apartition [a pivot i j]
`(let [pivot# ~pivot]
(loop [i# ~i
j# ~j]
(if (<= i# j#)
(let [v# (aget ~a i#)]
(if (< v# pivot#)
(recur (inc i#) j#)
(do
(when (< i# j#)
(aset ~a i# (aget ~a j#))
(aset ~a j# v#))
(recur i# (dec j#)))))
i#))))
(defn qsort
([^longs a]
(qsort a 0 (alength a)))
([^longs a ^long lo ^long hi]
(let [lo (int lo)
hi (int hi)]
(when
(< (inc lo) hi)
(let [pivot (aget a lo)
split (dec (apartition a pivot (inc lo) (dec hi)))]
(when (> split lo) (swap a lo split))
(qsort a lo split)
(qsort a (inc split) hi)))
a)))
The main tricks are:
Do everything with primitive arithmetic
Use ints for the array indexes (this avoids some unnecessary casts, not a big deal but every little helps....)
Use macros rather than functions to break up the code (avoids function call overhead and parameter boxing)
Use loop/recur for maximum speed in the inner loop (i.e. partitioning the subarray)
Avoid constructing any new objects on the heap (so avoid vectors, sequences, maps etc.)
The Joy of Clojure, Chapter 6.4 describes a lazy quicksort algorithm.The beauty of lazy sorting is that it will only do as much work as necessary to find the first x values. So if x << n this algorithm is O(n).
(ns joy.q)
(defn sort-parts
"Lazy, tail-recursive, incremental quicksort. Works against
and creates partitions based on the pivot, defined as 'work'."
[work]
(lazy-seq
(loop [[part & parts] work]
(if-let [[pivot & xs] (seq part)]
(let [smaller? #(< % pivot)]
(recur (list*
(filter smaller? xs)
pivot
(remove smaller? xs)
parts)))
(when-let [[x & parts] parts]
(cons x (sort-parts parts)))))))
(defn qsort [xs]
(sort-parts (list xs)))
By examining the main points from mikera's answer, you can see that they are mostly focused on eliminating the overhead introduced by using idiomatic (as opposed to tweaked) Clojure, which would probably not exist in an idiomatic Java implementation:
primitive arithmetic - slightly easier and more idiomatic in Java, you are more likely to use ints than Integers
ints for the array indexes - the same
Use macros rather than functions to break up the code (avoids functional call overhead and boxing) - fixes a problem introduced by using the language. Clojure encourages functional style, hence a function call overhead (and boxing).
Use loop/recur for maximum speed in the inner loop - in Java you'd idiomatically use an ordinary loop (which is what loop/recur compiles to anyway, as far as I know)
That being said, there actually is another trivial solution. Write (or find) an efficient Java implementation of Quick Sort, say something with a signature like this:
Sort.quickSort(long[] elems)
And then call it from Clojure:
(Sort/quickSort elems)
Checklist:
as efficient as in Java - yes
idiomatic in Clojure - arguably yes, I'd say that Java-interop is one of Clojure's core features.
reusable - yes, there's a good chance that you can easily find a very efficient Java implementation already written.
I'm not trying to troll, I understand what you are trying to find out with these experiments I'm just adding this answer for the sake of completeness. Let's not overlook the obvious one! :)
I'm learning clojure by going through project euler and am working on problem number 10 (find the sum of all the prime number below two million. I implemented a pretty literal algorithm for the sieve of eratosthenes but it works far too slowly to be useful for up to two million. I tried implementing it with loop-recur to not create any new frames but that didn't have a big impact on performance.
(defn find-primes-sum [last-prime nums]
(loop [p last-prime n nums sum 0]
(println p)
(if (empty? n)
sum
(recur (first n) (doall (remove #(zero? (mod % (first n))) n)) (+ sum (first n))))))
(defn sieve-primes-until [limit]
(find-primes-sum 2 (filter odd? (range 2 (inc limit)))))
(println (sieve-primes-until 2000000))
(set! *unchecked-math* true)
(defmacro iloop [[b t n] & body]
`(loop [~#b]
(when ~t
~#body
(recur ~n))))
(defn count-primes [^long n]
(let [c (inc n)
^booleans prime? (make-array Boolean/TYPE c)]
(iloop [(i 2) (<= i n) (inc i)]
(aset prime? i true))
(iloop [(i 2) (<= (* i i) n) (inc i)]
(if (aget prime? i)
(iloop [(j i) (<= (* i j) n) (inc j)]
(aset prime? (* i j) false))))
(areduce prime? i r 0
(if (aget prime? i)
(inc r)
r))))
This version targets Clojure 1.3.0 alpha. It counts primes up to 1e8 in 2 seconds on my machine. It can be easily altered to collect them. It was originally written to show that you can implement the sieve so that it runs as fast as the comparable Java.
http://dosync.posterous.com/lispers-know-the-value-of-everything-and-the
Personally, I'd structure the code differently. I've got an implementation of this problem that generates a lazy seq of all primes first, then sums the first 2,000,000 items. Takes 16 seconds on clojure 1.2, but it's hardly optimized except for using recur.
You're also doing a lot of unnecessary comparing with your input range; (remove #(zero? (mod % (first n))) n) tests all candidates against all odd numbers, which is nonsense**, especially when you force it with doall. You actually only have to test candidate prime x against all known primes <= sqrt(x) and discard the candidate when you find your first match.
** I just noticed your algorithm is not that stupid, but I'd still recommend you rewrite your algorithm as a lazy seq finds "the next prime" given a seq of previous primes and a candidate number.
I've been working on solving Project Euler problems in Clojure to get better, and I've already run into prime number generation a couple of times. My problem is that it is just taking way too long. I was hoping someone could help me find an efficient way to do this in a Clojure-y way.
When I fist did this, I brute-forced it. That was easy to do. But calculating 10001 prime numbers took 2 minutes this way on a Xeon 2.33GHz, too long for the rules, and too long in general. Here was the algorithm:
(defn next-prime-slow
"Find the next prime number, checking against our already existing list"
([sofar guess]
(if (not-any? #(zero? (mod guess %)) sofar)
guess ; Then we have a prime
(recur sofar (+ guess 2))))) ; Try again
(defn find-primes-slow
"Finds prime numbers, slowly"
([]
(find-primes-slow 10001 [2 3])) ; How many we need, initial prime seeds
([needed sofar]
(if (<= needed (count sofar))
sofar ; Found enough, we're done
(recur needed (concat sofar [(next-prime-slow sofar (last sofar))])))))
By replacing next-prime-slow with a newer routine that took some additional rules into account (like the 6n +/- 1 property) I was able to speed things up to about 70 seconds.
Next I tried making a sieve of Eratosthenes in pure Clojure. I don't think I got all the bugs out, but I gave up because it was simply way too slow (even worse than the above, I think).
(defn clean-sieve
"Clean the sieve of what we know isn't prime based"
[seeds-left sieve]
(if (zero? (count seeds-left))
sieve ; Nothing left to filter the list against
(recur
(rest seeds-left) ; The numbers we haven't checked against
(filter #(> (mod % (first seeds-left)) 0) sieve)))) ; Filter out multiples
(defn self-clean-sieve ; This seems to be REALLY slow
"Remove the stuff in the sieve that isn't prime based on it's self"
([sieve]
(self-clean-sieve (rest sieve) (take 1 sieve)))
([sieve clean]
(if (zero? (count sieve))
clean
(let [cleaned (filter #(> (mod % (last clean)) 0) sieve)]
(recur (rest cleaned) (into clean [(first cleaned)]))))))
(defn find-primes
"Finds prime numbers, hopefully faster"
([]
(find-primes 10001 [2]))
([needed seeds]
(if (>= (count seeds) needed)
seeds ; We have enough
(recur ; Recalculate
needed
(into
seeds ; Stuff we've already found
(let [start (last seeds)
end-range (+ start 150000)] ; NOTE HERE
(reverse
(self-clean-sieve
(clean-sieve seeds (range (inc start) end-range))))))))))
This is bad. It also causes stack overflows if the number 150000 is smaller. This despite the fact I'm using recur. That may be my fault.
Next I tried a sieve, using Java methods on a Java ArrayList. That took quite a bit of time, and memory.
My latest attempt is a sieve using a Clojure hash-map, inserting all the numbers in the sieve then dissoc'ing numbers that aren't prime. At the end, it takes the key list, which are the prime numbers it found. It takes about 10-12 seconds to find 10000 prime numbers. I'm not sure it's fully debugged yet. It's recursive too (using recur and loop), since I'm trying to be Lispy.
So with these kind of problems, problem 10 (sum up all primes under 2000000) is killing me. My fastest code came up with the right answer, but it took 105 seconds to do it, and needed quite a bit of memory (I gave it 512 MB just so I wouldn't have to fuss with it). My other algorithms take so long I always ended up stopping them first.
I could use a sieve to calculate that many primes in Java or C quite fast and without using so much memory. I know I must be missing something in my Clojure/Lisp style that's causing the problem.
Is there something I'm doing really wrong? Is Clojure just kinda slow with large sequences? Reading some of the project Euler discussions people have calculated the first 10000 primes in other Lisps in under 100 miliseconds. I realize the JVM may slow things down and Clojure is relatively young, but I wouldn't expect a 100x difference.
Can someone enlighten me on a fast way to calculate prime numbers in Clojure?
Here's another approach that celebrates Clojure's Java interop. This takes 374ms on a 2.4 Ghz Core 2 Duo (running single-threaded). I let the efficient Miller-Rabin implementation in Java's BigInteger#isProbablePrime deal with the primality check.
(def certainty 5)
(defn prime? [n]
(.isProbablePrime (BigInteger/valueOf n) certainty))
(concat [2] (take 10001
(filter prime?
(take-nth 2
(range 1 Integer/MAX_VALUE)))))
The Miller-Rabin certainty of 5 is probably not very good for numbers much larger than this. That certainty is equal to 96.875% certain it's prime (1 - .5^certainty)
I realize this is a very old question, but I recently ended up looking for the same and the links here weren't what I'm looking for (restricted to functional types as much as possible, lazily generating ~every~ prime I want).
I stumbled upon a nice F# implementation, so all credits are his. I merely ported it to Clojure:
(defn gen-primes "Generates an infinite, lazy sequence of prime numbers"
[]
(letfn [(reinsert [table x prime]
(update-in table [(+ prime x)] conj prime))
(primes-step [table d]
(if-let [factors (get table d)]
(recur (reduce #(reinsert %1 d %2) (dissoc table d) factors)
(inc d))
(lazy-seq (cons d (primes-step (assoc table (* d d) (list d))
(inc d))))))]
(primes-step {} 2)))
Usage is simply
(take 5 (gen-primes))
Very late to the party, but I'll throw in an example, using Java BitSets:
(defn sieve [n]
"Returns a BitSet with bits set for each prime up to n"
(let [bs (new java.util.BitSet n)]
(.flip bs 2 n)
(doseq [i (range 4 n 2)] (.clear bs i))
(doseq [p (range 3 (Math/sqrt n))]
(if (.get bs p)
(doseq [q (range (* p p) n (* 2 p))] (.clear bs q))))
bs))
Running this on a 2014 Macbook Pro (2.3GHz Core i7), I get:
user=> (time (do (sieve 1e6) nil))
"Elapsed time: 64.936 msecs"
See the last example here:
http://clojuredocs.org/clojure_core/clojure.core/lazy-seq
;; An example combining lazy sequences with higher order functions
;; Generate prime numbers using Eratosthenes Sieve
;; See http://en.wikipedia.org/wiki/Sieve_of_Eratosthenes
;; Note that the starting set of sieved numbers should be
;; the set of integers starting with 2 i.e., (iterate inc 2)
(defn sieve [s]
(cons (first s)
(lazy-seq (sieve (filter #(not= 0 (mod % (first s)))
(rest s))))))
user=> (take 20 (sieve (iterate inc 2)))
(2 3 5 7 11 13 17 19 23 29 31 37 41 43 47 53 59 61 67 71)
Here's a nice and simple implementation:
http://clj-me.blogspot.com/2008/06/primes.html
... but it is written for some pre-1.0 version of Clojure. See lazy_seqs in Clojure Contrib for one that works with the current version of the language.
(defn sieve
[[p & rst]]
;; make sure the stack size is sufficiently large!
(lazy-seq (cons p (sieve (remove #(= 0 (mod % p)) rst)))))
(def primes (sieve (iterate inc 2)))
with a 10M stack size, I get the 1001th prime in ~ 33 seconds on a 2.1Gz macbook.
So I've just started with Clojure, and yeah, this comes up a lot on Project Euler doesn't it? I wrote a pretty fast trial division prime algorithm, but it doesn't really scale too far before each run of divisions becomes prohibitively slow.
So I started again, this time using the sieve method:
(defn clense
"Walks through the sieve and nils out multiples of step"
[primes step i]
(if (<= i (count primes))
(recur
(assoc! primes i nil)
step
(+ i step))
primes))
(defn sieve-step
"Only works if i is >= 3"
[primes i]
(if (< i (count primes))
(recur
(if (nil? (primes i)) primes (clense primes (* 2 i) (* i i)))
(+ 2 i))
primes))
(defn prime-sieve
"Returns a lazy list of all primes smaller than x"
[x]
(drop 2
(filter (complement nil?)
(persistent! (sieve-step
(clense (transient (vec (range x))) 2 4) 3)))))
Usage and speed:
user=> (time (do (prime-sieve 1E6) nil))
"Elapsed time: 930.881 msecs
I'm pretty happy with the speed: it's running out of a REPL running on a 2009 MBP. It's mostly fast because I completely eschew idiomatic Clojure and instead loop around like a monkey. It's also 4X faster because I'm using a transient vector to work on the sieve instead of staying completely immutable.
Edit: After a couple of suggestions / bug fixes from Will Ness it now runs a whole lot faster.
Here's a simple sieve in Scheme:
http://telegraphics.com.au/svn/puzzles/trunk/programming-in-scheme/primes-up-to.scm
Here's a run for primes up to 10,000:
#;1> (include "primes-up-to.scm")
; including primes-up-to.scm ...
#;2> ,t (primes-up-to 10000)
0.238s CPU time, 0.062s GC time (major), 180013 mutations, 130/4758 GCs (major/minor)
(2 3 5 7 11 13...
Here is a Clojure solution. i is the current number being considered and p is a list of all prime numbers found so far. If division by some prime numbers has a remainder of zero, the number i is not a prime number and recursion occurs with the next number. Otherwise the prime number is added to p in the next recursion (as well as continuing with the next number).
(defn primes [i p]
(if (some #(zero? (mod i %)) p)
(recur (inc i) p)
(cons i (lazy-seq (primes (inc i) (conj p i))))))
(time (do (doall (take 5001 (primes 2 []))) nil))
; Elapsed time: 2004.75587 msecs
(time (do (doall (take 10001 (primes 2 []))) nil))
; Elapsed time: 7700.675118 msecs
Update:
Here is a much slicker solution based on this answer above.
Basically the list of integers starting with two is filtered lazily. Filtering is performed by only accepting a number i if there is no prime number dividing the number with remainder of zero. All prime numbers are tried where the square of the prime number is less or equal to i.
Note that primes is used recursively but Clojure manages to prevent endless recursion. Also note that the lazy sequence primes caches results (that's why the performance results are a bit counter intuitive at first sight).
(def primes
(lazy-seq
(filter (fn [i] (not-any? #(zero? (rem i %))
(take-while #(<= (* % %) i) primes)))
(drop 2 (range)))))
(time (first (drop 10000 primes)))
; Elapsed time: 542.204211 msecs
(time (first (drop 20000 primes)))
; Elapsed time: 786.667644 msecs
(time (first (drop 40000 primes)))
; Elapsed time: 1780.15807 msecs
(time (first (drop 40000 primes)))
; Elapsed time: 8.415643 msecs
Based on Will's comment, here is my take on postponed-primes:
(defn postponed-primes-recursive
([]
(concat (list 2 3 5 7)
(lazy-seq (postponed-primes-recursive
{}
3
9
(rest (rest (postponed-primes-recursive)))
9))))
([D p q ps c]
(letfn [(add-composites
[D x s]
(loop [a x]
(if (contains? D a)
(recur (+ a s))
(persistent! (assoc! (transient D) a s)))))]
(loop [D D
p p
q q
ps ps
c c]
(if (not (contains? D c))
(if (< c q)
(cons c (lazy-seq (postponed-primes-recursive D p q ps (+ 2 c))))
(recur (add-composites D
(+ c (* 2 p))
(* 2 p))
(first ps)
(* (first ps) (first ps))
(rest ps)
(+ c 2)))
(let [s (get D c)]
(recur (add-composites
(persistent! (dissoc! (transient D) c))
(+ c s)
s)
p
q
ps
(+ c 2))))))))
Initial submission for comparison:
Here is my attempt to port this prime number generator from Python to Clojure. The below returns an infinite lazy sequence.
(defn primes
[]
(letfn [(prime-help
[foo bar]
(loop [D foo
q bar]
(if (nil? (get D q))
(cons q (lazy-seq
(prime-help
(persistent! (assoc! (transient D) (* q q) (list q)))
(inc q))))
(let [factors-of-q (get D q)
key-val (interleave
(map #(+ % q) factors-of-q)
(map #(cons % (get D (+ % q) (list)))
factors-of-q))]
(recur (persistent!
(dissoc!
(apply assoc! (transient D) key-val)
q))
(inc q))))))]
(prime-help {} 2)))
Usage:
user=> (first (primes))
2
user=> (second (primes))
3
user=> (nth (primes) 100)
547
user=> (take 5 (primes))
(2 3 5 7 11)
user=> (time (nth (primes) 10000))
"Elapsed time: 409.052221 msecs"
104743
edit:
Performance comparison, where postponed-primes uses a queue of primes seen so far rather than a recursive call to postponed-primes:
user=> (def counts (list 200000 400000 600000 800000))
#'user/counts
user=> (map #(time (nth (postponed-primes) %)) counts)
("Elapsed time: 1822.882 msecs"
"Elapsed time: 3985.299 msecs"
"Elapsed time: 6916.98 msecs"
"Elapsed time: 8710.791 msecs"
2750161 5800139 8960467 12195263)
user=> (map #(time (nth (postponed-primes-recursive) %)) counts)
("Elapsed time: 1776.843 msecs"
"Elapsed time: 3874.125 msecs"
"Elapsed time: 6092.79 msecs"
"Elapsed time: 8453.017 msecs"
2750161 5800139 8960467 12195263)
Idiomatic, and not too bad
(def primes
(cons 1 (lazy-seq
(filter (fn [i]
(not-any? (fn [p] (zero? (rem i p)))
(take-while #(<= % (Math/sqrt i))
(rest primes))))
(drop 2 (range))))))
=> #'user/primes
(first (time (drop 10000 primes)))
"Elapsed time: 0.023135 msecs"
=> 104729
From: http://steloflute.tistory.com/entry/Clojure-%ED%94%84%EB%A1%9C%EA%B7%B8%EB%9E%A8-%EC%B5%9C%EC%A0%81%ED%99%94
Using Java array
(defmacro loopwhile [init-symbol init whilep step & body]
`(loop [~init-symbol ~init]
(when ~whilep ~#body (recur (+ ~init-symbol ~step)))))
(defn primesUnderb [limit]
(let [p (boolean-array limit true)]
(loopwhile i 2 (< i (Math/sqrt limit)) 1
(when (aget p i)
(loopwhile j (* i 2) (< j limit) i (aset p j false))))
(filter #(aget p %) (range 2 limit))))
Usage and speed:
user=> (time (def p (primesUnderb 1e6)))
"Elapsed time: 104.065891 msecs"
After coming to this thread and searching for a faster alternative to those already here, I am surprised nobody linked to the following article by Christophe Grand :
(defn primes3 [max]
(let [enqueue (fn [sieve n factor]
(let [m (+ n (+ factor factor))]
(if (sieve m)
(recur sieve m factor)
(assoc sieve m factor))))
next-sieve (fn [sieve candidate]
(if-let [factor (sieve candidate)]
(-> sieve
(dissoc candidate)
(enqueue candidate factor))
(enqueue sieve candidate candidate)))]
(cons 2 (vals (reduce next-sieve {} (range 3 max 2))))))
As well as a lazy version :
(defn lazy-primes3 []
(letfn [(enqueue [sieve n step]
(let [m (+ n step)]
(if (sieve m)
(recur sieve m step)
(assoc sieve m step))))
(next-sieve [sieve candidate]
(if-let [step (sieve candidate)]
(-> sieve
(dissoc candidate)
(enqueue candidate step))
(enqueue sieve candidate (+ candidate candidate))))
(next-primes [sieve candidate]
(if (sieve candidate)
(recur (next-sieve sieve candidate) (+ candidate 2))
(cons candidate
(lazy-seq (next-primes (next-sieve sieve candidate)
(+ candidate 2))))))]
(cons 2 (lazy-seq (next-primes {} 3)))))
Plenty of answers already, but I have an alternative solution which generates an infinite sequence of primes. I was also interested on bechmarking a few solutions.
First some Java interop. for reference:
(defn prime-fn-1 [accuracy]
(cons 2
(for [i (range)
:let [prime-candidate (-> i (* 2) (+ 3))]
:when (.isProbablePrime (BigInteger/valueOf prime-candidate) accuracy)]
prime-candidate)))
Benjamin # https://stackoverflow.com/a/7625207/3731823 is primes-fn-2
nha # https://stackoverflow.com/a/36432061/3731823 is primes-fn-3
My implementations is primes-fn-4:
(defn primes-fn-4 []
(let [primes-with-duplicates
(->> (for [i (range)] (-> i (* 2) (+ 5))) ; 5, 7, 9, 11, ...
(reductions
(fn [known-primes candidate]
(if (->> known-primes
(take-while #(<= (* % %) candidate))
(not-any? #(-> candidate (mod %) zero?)))
(conj known-primes candidate)
known-primes))
[3]) ; Our initial list of known odd primes
(cons [2]) ; Put in the non-odd one
(map (comp first rseq)))] ; O(1) lookup of the last element of the vec "known-primes"
; Ugh, ugly de-duplication :(
(->> (map #(when (not= % %2) %) primes-with-duplicates (rest primes-with-duplicates))
(remove nil?))))
Reported numbers (time in milliseconds to count first N primes) are the fastest from the run of 5, no JVM restarts between experiments so your mileage may vary:
1e6 3e6
(primes-fn-1 5) 808 2664
(primes-fn-1 10) 952 3198
(primes-fn-1 20) 1440 4742
(primes-fn-1 30) 1881 6030
(primes-fn-2) 1868 5922
(primes-fn-3) 489 1755 <-- WOW!
(primes-fn-4) 2024 8185
If you don't need a lazy solution and you just want a sequence of primes below a certain limit, the straight forward implementation of the Sieve of Eratosthenes is pretty fast. Here's my version using transients:
(defn classic-sieve
"Returns sequence of primes less than N"
[n]
(loop [nums (transient (vec (range n))) i 2]
(cond
(> (* i i) n) (remove nil? (nnext (persistent! nums)))
(nums i) (recur (loop [nums nums j (* i i)]
(if (< j n)
(recur (assoc! nums j nil) (+ j i))
nums))
(inc i))
:else (recur nums (inc i)))))
I just started using Clojure so I don't know if it's good but here is my solution:
(defn divides? [x i]
(zero? (mod x i)))
(defn factors [x]
(flatten (map #(list % (/ x %))
(filter #(divides? x %)
(range 1 (inc (Math/floor (Math/sqrt x))))))))
(defn prime? [x]
(empty? (filter #(and divides? (not= x %) (not= 1 %))
(factors x))))
(def primes
(filter prime? (range 2 java.lang.Integer/MAX_VALUE)))
(defn sum-of-primes-below [n]
(reduce + (take-while #(< % n) primes)))