Noticed something odd while trying to troubleshoot a complex computation. I had an odd error so I started to build up the computation incrementally and pretty early on discovered that a very very long seq was being unexpectedly realized. I have a combinatoric seq of lists like so:
(0 1 2 3)
(0 1 2 4)
(0 1 2 5)
for n-choose-k for n = 143 and k = 4 somewhat unoriginally named "combos". I plan to manipulate this as a seq to keep the memory consumption reasonable but this failed:
(def combos (combinations 4 143))
(def semantically-also-combos
(filter nil? (map identity combos)))
;; this prints instantly and uses almost no memory, as expected
(println (first combos)) ; prints (0 1 2 3)
;; this takes minutes and runs the JVM out of memory
;; without printing anything
(println (first semantically-also-combos))
According to type they are both clojure.lang.LazySeq but one works as expected and the other crashes the process. Why does the entire seq get realized just to run it through the identity function and check if it's nil?
Complete code to reproduce
(ns my-project.core
(:gen-class))
;;; Copied from rosetta code
(defn combinations
"If m=1, generate a nested list of numbers [0,n)
If m>1, for each x in [0,n), and for each list in the recursion on [x+1,n), cons the two"
[m n]
(letfn [(comb-aux
[m start]
(if (= 1 m)
(for [x (range start n)]
(list x))
(for [x (range start n)
xs (comb-aux (dec m) (inc x))]
(cons x xs))))]
(comb-aux m 0)))
(println "Generating combinations...")
(def combos (combinations 4 143))
(def should-also-be-combos
(filter nil? (map identity combos)))
(defn -main
"Calculates combos"
[& _args]
(println (type combos))
(println (type should-also-be-combos)))
Why don't you check that (filter nil? (map identity combos)) actually returns the result you would expect for much smaller arguments passed to combinations? I did that. This is what combinations returns with arguments 2 and 5:
(combinations 2 5)
;; => ((0 1) (0 2) (0 3) (0 4) (1 2) (1 3) (1 4) (2 3) (2 4) (3 4))
This is what we get with the extra lazy sequence operations of the example:
(filter nil? (map identity (combinations 2 5)))
;; => ()
What (filter nil? ...) does is to keep all elements that satisfy the predicate. And none of the elements in the input sequence is nil?, so it will scan the entire input sequence and will find none. Maybe you wanted to use (remove nil? ...), which will remove elements satisfying a predicate?
(remove nil? (map identity (combinations 2 5)))
;; => ((0 1) (0 2) (0 3) (0 4) (1 2) (1 3) (1 4) (2 3) (2 4) (3 4))
Going back to the original example, this is what I get using remove:
(first (combinations 4 143))
;; => (0 1 2 3)
(first (remove nil? (map identity (combinations 4 143))))
;; => (0 1 2 3)
My general advice is to start testing your function with "smaller" data such as 2 and 5, before going to "bigger" data such as 4 and 143.
I modified the example a bit as follows:
(ns tst.demo.core
(:use tupelo.core tupelo.test))
(def cnt (atom 0))
; Copied from rosetta code
(defn combinations
"If m=1, generate a nested list of numbers [0,n)
If m>1, for each x in [0,n), and for each list in the recursion on [x+1,n), cons the two"
[m n]
(let [comb-aux (fn comb-aux
[m start]
(swap! cnt inc)
(when (zero? (mod #cnt 100))
(print \.)
(flush))
(when (zero? (mod #cnt 10000))
(newline)
(print #cnt " "))
(if (= 1 m)
(for [x (range start n)]
(list x))
(for [x (range start n)
xs (comb-aux (dec m) (inc x))]
(cons x xs))))]
(comb-aux m 0)))
(println "Generating combinations...")
(def combos (combinations 4 143))
(def should-also-be-combos
(filter nil? (map identity combos)))
and the invocation:
(dotest
(newline)
(spyx (type combos))
(spyx (type should-also-be-combos))
(newline)
(println :1)
(println (first combos))
(newline)
(println :2)
(println (first should-also-be-combos))
(newline))
with result:
-------------------------------
Clojure 1.10.2 Java 15
-------------------------------
Testing tst.demo.core
(type combos) => clojure.lang.LazySeq
(type should-also-be-combos) => clojure.lang.LazySeq
:1
(0 1 2 3)
:2
....................................................................................................
10000 ....................................................................................................
20000 ....................................................................................................
30000 ....................................................................................................
40000 ....................................................................................................
50000 ....................................................................................................
60000 ....................................................................................................
70000 ....................................................................................................
80000 ....................................................................................................
90000 ....................................................................................................
100000 ....................................................................................................
110000 ....................................................................................................
120000 ....................................................................................................
130000 ....................................................................................................
140000 ....................................................................................................
150000 ....................................................................................................
160000 ....................................................................................................
170000 ....................................................................................................
180000 ....................................................................................................
190000 ....................................................................................................
200000 ....................................................................................................
210000 ....................................................................................................
220000 ....................................................................................................
230000 ....................................................................................................
240000 ....................................................................................................
250000 ....................................................................................................
260000 ....................................................................................................
270000 ....................................................................................................
280000 ....................................................................................................
290000 ....................................................................................................
300000 ....................................................................................................
310000 ....................................................................................................
320000 ....................................................................................................
330000 ....................................................................................................
340000 ....................................................................................................
350000 ....................................................................................................
360000 ....................................................................................................
370000 ....................................................................................................
380000 ....................................................................................................
390000 ....................................................................................................
400000 ....................................................................................................
410000 ....................................................................................................
420000 ....................................................................................................
430000 ....................................................................................................
440000 ....................................................................................................
450000 ....................................................................................................
460000 ....................................................................................................
470000 ....................................................................................................
480000 ..........................................................................nil
with timing on my desktop computer (5 yrs old, 8 core, 32Gb RAM):
Passed all tests
Finished at 16:37:28.484 (run time: 5.354s)
So you can see the function is being invoked about 1/2 million times, which takes 5.3 seconds on my computer.
I believe you are seeing something similar to that described in Clojure Don'ts: Concat. It's recursive form also reminds me of the famous Ackerman Function, which is an example of how a seemingly innocuous function can "blow up" even for small input values.
Related
How to achieve this using Clojure atom on the Fibonacci series.
(def x 0)
(def y 0)
(def z 1)
(defn problem []
(def a (atom 0))
(while (< #a 10 )
(do
;logic code need to write to achieve the output
; how can I add b,c == c,b???????.)
(swap! a inc)))
(problem)
output should be 0 1 1 2 3 5 8 13 21 34 55
As mentioned in the comments, defining and using an atom in a function is a totally non-clojure way of solving the question which makes it all the more difficult to solve. You should get familiar with the concept of immutability and higher order functions like map, filter, reduce, etc.
The code below gives you what you want.
(def fib1
(fn [n]
(cond
(= n 0) 1
(= n 1) 1
:else (+ (fib1 (dec n)) (fib1 (- n 2))))))
since this gives you just nth Fibonacci number, you should take + map with it if you want to get what you want.
(take 10 (map fib1 (range))) => (1 1 2 3 5 8 13 21 34 55)
take a look at this beautiful solution I found on internet.
(def fib-seq-iterate (map first (iterate (fn [[n m]] [m (+ n m)]) [0 1])))
This returns a lazy sequence, which can be called with
(take 5 fib-seq-iterate) => (0 1 1 2 3)
(def x 0)
(def y 0)
(def z 1)
(defn problem []
(def a (atom 0))
(while (< #a 10 )
(do
(println x)
(def x (+ y z))
(def y z)
(def z x)
(swap! a inc))))
(problem)
Output:
0 1 2 3 5 8 13 21 34 55
We've been given a task to print the first ten multiples of any number for which we have written the below code. It is throwing an error. In simple words, if n is 2 then we need to create a table of 2's till 10.
(defn multiples [n]
(while ( n < 11)
(println( n * n))
(swap! n inc)))
(def n (Integer/parseInt (clojure.string/trim (read-line))))
(multiples n)
With this, we're getting the error:
Exception in thread "main" java.lang.ClassCastException: java.lang.Integer cannot be cast to clojure.lang.
(defn multiples [n]
(map #(* n %) (range 1 (+ 10 1))))
user=> (multiples 1)
;; => (1 2 3 4 5 6 7 8 9 10)
user=> (multiples 2)
;; => (2 4 6 8 10 12 14 16 18 20)
The resulting list you can loop over and println each of the elements.
(for [i (multiples 2)]
(println i))
;; or:
(map println (multiples 2)) ;; though one usually doesn't apply
;; `map` on side effect functions ...
To improve your own construct:
You, coming from an imperative language, try to work with mutations.
That is very un-idiomatic clojure.
However, by declaring a value atom, you can access using the # operator to its place. And mutate the variable's value.
(defn multiples [n]
(let [i (atom 1)] ;; i is an atom
(while (< #i 11) ;; #i is the value saved into i
(println (* #i n))
(swap! i inc)))) ;; and correctly you can increase the value
With this multiples, you can also print the values.
You can't apply swap! to normal variables, only to atoms.
while loops one should apply only if number of elements not known.
In this case, one knows very well, when to stop. So use rather
a for loop.
(defn multiples [n]
(for [i (range 1 11)]
(println (* i n))))
Look at what iterate function does here
(defn multiples-of [n]
(iterate (partial * n) n))
(def ten-multiples-of-ten
(take 10 (multiples-of 10)))
EDIT: I misread the author of the question, I believe he wants to just generate a sequence of squares. Here is one way using transducers, cause why not ;)
(def xf
(comp
(map inc)
(map #(* % %))))
(defn first-n-squares [n]
(into [] xf (take n (range))))
You can use recur in a loop:
(defn multiples [n]
(if (< n 11)
(do ; then
(println (* n n))
(recur (inc n)))
nil)) ; else return nil
Running this by invoking
(multiples 1)
in a REPL will produce
1
4
9
16
25
36
49
64
81
100
nil
(defn multiply-xf
[]
(fn [xf]
(let [product (volatile! 1)]
(fn
([] (xf))
([result]
(xf result #product)
(xf result))
([result input]
(let [new-product (* input #product)]
(vreset! product new-product)
(if (zero? new-product)
(do
(println "reduced")
(reduced ...)) <----- ???
result)))))))
This is a simple transducer which multiples numbers. I am wondering what would be the reduced value to allow early termination?
I've tried (transient []) but that means the transducer only works with vectors.
I'm assuming you want this transducer to produce a running product sequence and terminate early if the product reaches zero. Although in the example the reducing function xf is never called in the 2-arity step function, and it's called twice in the completion arity.
(defn multiply-xf
[]
(fn [rf]
(let [product (volatile! 1)]
(fn
([] (rf))
([result] (rf result))
([result input]
(let [new-product (vswap! product * input)]
(if (zero? new-product)
(reduced result)
(rf result new-product))))))))
Notice for early termination, we don't care what result is. That's the responsibility of the reducing function rf a.k.a xf in your example. I also consolidated vreset!/#product with vswap!.
(sequence (multiply-xf) [2 2 2 2 2])
=> (2 4 8 16 32)
It will terminate if the running product reaches zero:
(sequence (multiply-xf) [2 2 0 2 2])
=> (2 4)
We can use transduce to sum the output. Here the reducing function is +, but your transducer doesn't need to know anything about that:
(transduce (multiply-xf) + [2 2 2 2])
=> 30
I've tried (transient []) but that means the transducer only works with vectors.
This transducer also doesn't need to concern itself the type of sequence/collection it's given.
(eduction (multiply-xf) (range 1 10))
=> (1 2 6 24 120 720 5040 40320 362880)
(sequence (multiply-xf) '(2.0 2.0 0.5 2 1/2 2 0.5))
=> (2.0 4.0 2.0 4.0 2.0 4.0 2.0)
(into #{} (multiply-xf) [2.0 2.0 0.5 2 1/2 2 0.5])
=> #{2.0 4.0}
This can be done without transducers as well:
(take-while (complement zero?) (reductions * [2 2 0 2 2]))
=> (2 4)
I'm trying to get my feet wet with parallel processing by implementing a wheel sieve for primes, the order of operation roughly as follows,
given some upper bound N, construct 8 different spokes of the form
[p, p + 30, p+ 60, ..., p + 30n] for p in {1, 7, 11, 13, 17, 19, 23, 29}.
combine these all together into one list
For each p, sieve all the primes in the list created in step 2 (using pmap)
take these 8 bitsets, parse them, and rebuild some sorted list of all primes less than N
My code is listed below (primes and primes2 are single-threaded implementations of a simple sieve, and wheel sieve respectively, for comparison)
The problem I'm having is that my current attempts to implement step 4 (in functions primes4, primes5, and primes6) are all dominating steps 1-3.
Can anyone give me any pointers on how I should complete primes3 (which implements only steps 1-3)? otherwise, if this is hopeless, can someone explain some other strategy for splitting up the work for separate threads to do in isolation?
(defn spoke-30 [p N]
(map #(+ p (* %1 30)) (range 0 (/ N 30))))
(defn interleaved-spokes
"returns all the spoke30 elements less than or equal to N (but not the first which is 1)"
[N]
(rest (filter #(< % N) (apply interleave (map #(spoke-30 % N) '(1 7 11 13 17 19 23 29))))))
(defn get-orbit
"for a spacing diff, generates the orbit of diff under + in Z_{30}"
[diff]
(map #(mod (* diff %) 30) (range 0 30)))
(defn _all-orbits
"returns a map of maps where each key is an element of (1 7 11 13 17 19 23 29),
and each value is the orbit of that element under + in Z_30)"
[]
(let [primes '(1 7 11 13 17 19 23 29)]
(zipmap primes (map #(zipmap (get-orbit %) (range 30 0 -1)) primes))))
(def all-orbits (memoize _all-orbits))
(defn start
"for a prime N and a spoke on-spoke, determine the optimal starting point"
[N on-spoke]
(let [dist (mod (- N) 30)
lowest (* (((all-orbits) dist) on-spoke) N)
sqrN (* N N)]
; this might be one away from where I need to be, but it is a cheaper
; calculation than the absolute best start.
(cond (>= lowest sqrN) lowest
(= lowest N) (* 31 N)
true (+ lowest (* N 30 (int (/ N 30)))))))
(defn primes2 [bound]
(let [bset (new java.util.BitSet bound)
sqrtBound (Math/sqrt bound)
pList (interleaved-spokes sqrtBound)]
(.flip bset 2 bound)
(doseq [i (range 9 bound 6)] (.clear bset i)) ;clear out the special case 3s
(doseq [i (range 25 bound 10)] (.clear bset i)) ;clear out the special case 5s
(doseq [x '(1 7 11 13 17 19 23 29)
y pList
z (range (start y x) bound (* 30 y))]
(.clear bset z))
(conj (filter (fn [x] (.get bset x)) (range 1 bound 2)) 2)))
(defn scaled-start [N on-spoke]
(let [dist (mod (- N) 30)
k (((all-orbits) dist) on-spoke)
lowest (int (/ (* k N) 30))
remaining (* (int (/ (- N k) 30)) N)
start (+ lowest remaining)]
(if (> start 0) start N)))
;TODO do I even *need* this bitset!?? ...
(defn mark-composites [bound spoke pList]
(let [scaledbound (int (/ bound 30))
bset (new java.util.BitSet scaledbound)]
(if (= spoke 1) (.set bset 0)) ; this won't be marked as composite - but it isn't prime!
(doseq [x pList
y (range (scaled-start x spoke) scaledbound x)]
(.set bset y))
[spoke bset]))
;TODO now need to find a quick way of reconstructing the required list
; need the raw bitsets ... will then loop over [0 scaledbound] and the bitsets, adding to a list in correct order if the element is true and not present already (it shouldn't!)
(defn primes3 [bound]
(let [pList (interleaved-spokes (Math/sqrt bound))]
(pmap #(mark-composites bound % pList) '(1 7 11 13 17 19 23 29))))
(defn primes4 [bound]
(let [pList (interleaved-spokes (Math/sqrt bound))
bits (pmap #(mark-composites bound % pList) '(1 7 11 13 17 19 23 29))
L (new java.util.ArrayList)]
(.addAll L '(2 3 5))
(doseq [z (range 0 (int (/ bound 30))) [x y] bits ]
(if (not (.get y z)) (.add L (+ x (* 30 z))))
(println x y z L)
)))
(defn primes5 [bound]
(let [pList (interleaved-spokes (Math/sqrt bound))
bits (pmap #(mark-composites bound % pList) '(1 7 11 13 17 19 23 29))]
(for [z (range 0 (int (+ 1 (/ bound 30)))) [x y] bits ]
(if (not (.get y z)) (+ x (* 30 z))))
))
(defn primes6 [bound]
(let [pList (interleaved-spokes (Math/sqrt bound))]
(concat '(2 3 5) (filter #(not= % 0) (apply interleave (pmap #(mark-composites2 bound % pList) '(1 7 11 13 17 19 23 29)))))))
(defn primes [n]
"returns a list of prime numbers less than or equal to n"
(let [bs (new java.util.BitSet n)]
(.flip bs 2 n)
;(doseq [i (range 4 n 2)] (.clear bs i)) ;clear out the special case 2s
(doseq [i (range 3 (Math/sqrt n))]
(if (.get bs i) ; it seems faster to check if odd than to range in steps of 2
(doseq [j (range (* i i) n (* 2 i))] (.clear bs j))))
(conj (filter (fn [x] (.get bs x)) (range 1 n 2)) 2)))
some timings are:
user=> (time (count (primes 1000000)))
"Elapsed time: 117.023543 msecs"
78498
user=> (time (count (primes2 1000000)))
"Elapsed time: 77.10944 msecs"
78498
user=> (time (count (primes3 1000000)))
"Elapsed time: 22.447898 msecs"
8
user=> (time (count (primes4 1000000)))
"Elapsed time: 647.586234 msecs"
78506
user=> (time (count (primes5 1000000)))
"Elapsed time: 721.62017 msecs"
266672
user=> (time (count (primes6 1000000)))
"Elapsed time: 306.280182 msecs"
I am trying to show the importance of lazy-sequences or lazy-evaluation to the non-FP programmers. I have written this implementation of prime-generation to show the concept:
(defn primes-gen [sieve]
(if-not (empty? sieve)
(let [prime (first sieve)]
(cons prime
(lazy-seq (primes-gen
(filter (fn [x]
(not= 0 (mod x prime)))
(rest sieve))))))))
;;;;; --------- TO SHOW ABOUT THE LAZY-THINGS
;; (take 400 (primes-gen (iterate inc 2)))
;; (take 400 (primes-gen (range 2 1000000000000N)))
However, i get stack-overflow-exception if i give any bigger value to take.
The stack is :
user> (pst)
StackOverflowError
clojure.core/range/fn--4269 (core.clj:2664)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:60)
clojure.lang.RT.seq (RT.java:484)
clojure.core/seq (core.clj:133)
clojure.core/filter/fn--4226 (core.clj:2523)
clojure.lang.LazySeq.sval (LazySeq.java:42)
clojure.lang.LazySeq.seq (LazySeq.java:60)
clojure.lang.RT.seq (RT.java:484)
clojure.core/seq (core.clj:133)
It seems that filter thunks are getting accumulated.
But if do (doall (filter ... then i would not be able to process the infinite sequences i.e. (take 1000 (primes-gen (iterate inc 2))) would not work anymore.
What is the right way to do it ?
Your analysis is spot on: you are nesting filters too much.
You should modify prime-gen to take two args: the set of known primes and candidates.
See my blog for some other ideas on implementing the Erathostenes' sieve.
Update:
So you stack filters over filters and at some point the stack is too big when you want to fetch the new candidate.
You have to merge all the filters into a single (or a reasonable number of) pass. Here it's easy because filters are very homogeneous. So I replace the filters stack by a collection holding the known primes.
(defn primes-gen
([candidates] (primes-gen candidates []))
([candidates known-primes]
(lazy-seq ; I prefer having the lazy-seq up here
(when-first [prime candidates] ; little known macro
(let [known-primes (conj known-primes prime)]
(cons prime
(primes-gen
(drop-while (fn [n] (some #(zero? (mod n %)) known-primes)) candidates)
known-primes)))))))
One of the possible solutions would be moving generator function inside lazy seq. For example (taken from here):
(def primes
(concat
[2 3 5 7]
(lazy-seq
(let [primes-from
(fn primes-from [n [f & r]]
(if (some #(zero? (rem n %))
(take-while #(<= (* % %) n) primes))
(recur (+ n f) r)
(lazy-seq (cons n (primes-from (+ n f) r)))))
wheel (cycle [2 4 2 4 6 2 6 4 2 4 6 6 2 6 4 2
6 4 6 8 4 2 4 2 4 8 6 4 6 2 4 6
2 6 6 4 2 4 6 2 6 4 2 4 2 10 2 10])]
(primes-from 11 wheel)))))