I have the following Clojure code to calculate a number with a certain "factorable" property. (what exactly the code does is secondary).
(defn factor-9
([]
(let [digits (take 9 (iterate #(inc %) 1))
nums (map (fn [x] ,(Integer. (apply str x))) (permutations digits))]
(some (fn [x] (and (factor-9 x) x)) nums)))
([n]
(or
(= 1 (count (str n)))
(and (divisible-by-length n) (factor-9 (quot n 10))))))
Now, I'm into TCO and realize that Clojure can only provide tail-recursion if explicitly told so using the recur keyword. So I've rewritten the code to do that (replacing factor-9 with recur being the only difference):
(defn factor-9
([]
(let [digits (take 9 (iterate #(inc %) 1))
nums (map (fn [x] ,(Integer. (apply str x))) (permutations digits))]
(some (fn [x] (and (factor-9 x) x)) nums)))
([n]
(or
(= 1 (count (str n)))
(and (divisible-by-length n) (recur (quot n 10))))))
To my knowledge, TCO has a double benefit. The first one is that it does not use the stack as heavily as a non tail-recursive call and thus does not blow it on larger recursions. The second, I think is that consequently it's faster since it can be converted to a loop.
Now, I've made a very rough benchmark and have not seen any difference between the two implementations although. Am I wrong in my second assumption or does this have something to do with running on the JVM (which does not have automatic TCO) and recur using a trick to achieve it?
Thank you.
The use of recur does speed things up, but only by about 3 nanoseconds (really) over a recursive call. When things get that small they can be hidden in the noise of the rest of the test. I wrote four tests (link below) that are able to illustrate the difference in performance.
I'd also suggest using something like criterium when benchmarking. (Stack Overflow won't let me post with more than 1 link since I've got no reputation to speak of, so you'll have to google it, maybe "clojure criterium")
For formatting reasons, I've put the tests and results in this gist.
Briefly, to compare relatively, if the recursive test is 1, then the looping test is about 0.45, and the TCO tests about 0.87 and the absolute difference between the recursive and TCO tests are around 3ns.
Of course, all the caveats about benchmarking apply.
When optimizing any code, it's good to start from potential or actual bottlenecks and optimize that first.
It seems to me that this particular piece of code is eating most of your CPU time:
(map (fn [x] ,(Integer. (apply str x))) (permutations digits))
And that doesn't depend on TCO in any way - it is executed in same way. So, tail call in this particular example will allow you not to use up all the stack, but to achieve better performance, try optimizing this.
just a gentile reminder that clojure has no TCO
After evaluating factor-9 (quot n 10) an and and an or has to be evaluated before the function can return. Thus it is not tail-recursive.
Related
I find myself writing a lot of clojure in this manner:
(defn my-fun [input]
(let [result1 (some-complicated-procedure input)
result2 (some-other-procedure result1)]
(do-something-with-results result1 result2)))
This let statement seems very... imperative. Which I don't like. In principal, I could be writing the same function like this:
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
The problem with this is that it involves recomputation of some-complicated-procedure, which may be arbitrarily expensive. Also you can imagine that some-complicated-procedure is actually a series of nested function calls, and then I either have to write a whole new function, or risk that changes in the first invocation don't get applied to the second:
E.g. this works, but I have to have an extra shallow, top-level function that makes it hard to do a mental stack trace:
(defn some-complicated-procedure [input] (lots (of (nested (operations input)))))
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
E.g. this is dangerous because refactoring is hard:
(defn my-fun [input]
(do-something-with-results (lots (of (nested (operations (mistake input))))) ; oops made a change here that wasn't applied to the other nested calls
(some-other-procedure (lots (of (nested (operations input))))))))
Given these tradeoffs, I feel like I don't have any alternatives to writing long, imperative let statements, but when I do, I cant shake the feeling that I'm not writing idiomatic clojure. Is there a way I can address the computation and code cleanliness problems raised above and write idiomatic clojure? Are imperitive-ish let statements idiomatic?
The kind of let statements you describe might remind you of imperative code, but there is nothing imperative about them. Haskell has similar statements for binding names to values within bodies, too.
If your situation really needs a bigger hammer, there are some bigger hammers that you can either use or take for inspiration. The following two libraries offer some kind of binding form (akin to let) with a localized memoization of results, so as to perform only the necessary steps and reuse their results if needed again: Plumatic Plumbing, specifically the Graph part; and Zach Tellman's Manifold, whose let-flow form furthermore orchestrates asynchronous steps to wait for the necessary inputs to become available, and to run in parallel when possible. Even if you decide to maintain your present course, their docs make good reading, and the code of Manifold itself is educational.
I recently had this same question when I looked at this code I wrote
(let [user-symbols (map :symbol states)
duplicates (for [[id freq] (frequencies user-symbols) :when (> freq 1)] id)]
(do-something-with duplicates))
You'll note that map and for are lazy and will not be executed until do-something-with is executed. It's also possible that not all (or even not any) of the states will be mapped or the frequencies calculated. It depends on what do-something-with actually requests of the sequence returned by for. This is very much functional and idiomatic functional programming.
i guess the simplest approach to keep it functional would be to have a pass-through state to accumulate the intermediate results. something like this:
(defn with-state [res-key f state]
(assoc state res-key (f state)))
user> (with-state :res (comp inc :init) {:init 10})
;;=> {:init 10, :res 11}
so you can move on to something like this:
(->> {:init 100}
(with-state :inc'd (comp inc :init))
(with-state :inc-doubled (comp (partial * 2) :inc'd))
(with-state :inc-doubled-squared (comp #(* % %) :inc-doubled))
(with-state :summarized (fn [st] (apply + (vals st)))))
;;=> {:init 100,
;; :inc'd 101,
;; :inc-doubled 202,
;; :inc-doubled-squared 40804,
;; :summarized 41207}
The let form is a perfectly functional construct and can be seen as syntactic sugar for calls to anonymous functions. We can easily write a recursive macro to implement our own version of let:
(defmacro my-let [bindings body]
(if (empty? bindings)
body
`((fn [~(first bindings)]
(my-let ~(rest (rest bindings)) ~body))
~(second bindings))))
Here is an example of calling it:
(my-let [a 3
b (+ a 1)]
(* a b))
;; => 12
And here is a macroexpand-all called on the above expression, that reveal how we implement my-let using anonymous functions:
(clojure.walk/macroexpand-all '(my-let [a 3
b (+ a 1)]
(* a b)))
;; => ((fn* ([a] ((fn* ([b] (* a b))) (+ a 1)))) 3)
Note that the expansion doesn't rely on let and that the bound symbols become parameter names in the anonymous functions.
As others write, let is actually perfectly functional, but at times it can feel imperative. It's better to become fully comfortable with it.
You might, however, want to kick the tires of my little library tl;dr that lets you write code like for example
(compute
(+ a b c)
where
a (f b)
c (+ 100 b))
I'm writing a lazy implementation of the Recamán's Sequence, and ran into some confusion regarding where calls to lazy-seq should happen.
This first version I came up with this morning was:
(defn lazy-recamans-sequence []
(let [f (fn rec [n seen last-s]
(let [back (- last-s n)
new-s (if (and (pos? back) (not (seen back)))
back
(+ last-s n))]
(lazy-seq ; Here
(cons new-s (rec (inc n) (conj seen new-s) new-s)))))]
(f 0 #{} 0)))
Then I realized that my placement of lazy-seq was kind of arbitrary, and that it could be placed higher to wrap more of the computations:
(defn lazy-recamans-sequence2 []
(let [f (fn rec [n seen last-s]
(lazy-seq ; Here
(let [back (- last-s n)
new-s (if (and (pos? back) (not (seen back)))
back
(+ last-s n))]
(cons new-s (rec (inc n) (conj seen new-s) new-s)))))]
(f 0 #{} 0)))
Then I looked back on a review that someone gave me last night:
(defn recaman []
(letfn [(tail [previous n seen]
(let [nx (if (and (> previous n) (not (seen (- previous n))))
(- previous n)
(+ previous n))]
; Here, inside "cons"
(cons nx (lazy-seq (tail nx (inc n) (conj seen nx))))))]
(tail 0 0 #{})))
And they have theirs inside of the call to cons!
Thinking this over, it seems like it wouldn't make a difference. With a broader scope (like the second version), more code is inside the explicit function that's passed to LazySeq. With a narrower scope however, the function itself may be smaller, but since the passed function involves a recursive call, it will be executing the same code anyways.
They seem to preform nearly identically and give the same answers. Is there any reason to prefer placing lazy-seq in one place over another? Is this simply a stylistic choice, or can this have actual repercussions?
In the first two examples the lazy-seq wraps the cons call. This means that when you generate call the function you return a lazy sequence immediately without calculating the first item of the sequence.
In the first example the let expression is still outside of lazy-seq so the value of the first item is calculated immediately but the returned sequence is still lazy and not realized.
The second example is similar to the first. The lazy-seq wraps the cons cell and also the let block. This means that the function will return immediatetly and the value of the first item is calculated only when the caller starts to consume the lazy sequence.
In the third example the value of the first item in the list is calculated immediately and only the tail of the returned sequence is lazy.
Is there any reason to prefer placing lazy-seq in one place over another?
It depends on what you want to achieve. Do you want to return a sequence immediately without calculating any values? In this case make the scope of lazy-seq as broad as possible. Otherwise try to restrict the scope of lazy-seq to calculate only the tail part of the sequence.
When I was first learning Clojure, I was a bit confused by the many possible choices of lazy-seq constructs, the lack of clarity in terms of which construct to choose, and the somewhat vague explanation for how lazy-seq creates laziness in the first place (it is implemented as a Java class of ~240 lines).
To reduce repetition and keep things as simple as possible, I created the lazy-cons macro. It is used like so:
(defn lazy-countdown [n]
(when (<= 0 n)
(lazy-cons n (lazy-countdown (dec n)))))
(deftest t-all
(is= (lazy-countdown 5) [5 4 3 2 1 0] )
(is= (lazy-countdown 1) [1 0] )
(is= (lazy-countdown 0) [0] )
(is= (lazy-countdown -1) nil ))
This version does realize the initial value n immediately.
I never worry about chunking (typically batches of 32) or trying to precisely control the number of elements realized in a lazy sequence. IMHO, if you need fine-grained control such as this, it is better to use an explicit loop than to make assumptions on the timing of realizations in a lazy sequence.
I'm trying to work through Stuart Halloway's book Programming Clojure. This whole functional stuff is very new to me.
I understand how
(defn fibo[]
(map first (iterate (fn [[a b]] [b (+ a b)]) [0 1])))
generates the Fibonacci sequence lazily. I do not understand why
(last (take 1000000 (fibo)))
works, while
(nth (fibo) 1000000)
throws an OutOfMemoryError. Could someone please explain how these two expressions differ? Is (nth) somehow holding on to the head of the sequence?
Thanks!
I think you are talking about issue that was discussed in google group and Rich Hickey provided patch that solved the problem. And the book, whick was published later, didn't cover this topic.
In clojure 1.3 your nth example works with minor improvements in fibo function. Now, due to changes in 1.3, we should explicitly flag M to use arbitrary precision, or it falls with throwIntOverflow.
(defn fibo[]
(map first (iterate (fn [[a b]] [b (+ a b)]) [0M 1M])))
And with these changes
(nth (fibo) 1000000)
succeed (if you have enough memory)
What Clojure version are you using? Try (clojure-version) on a repl. I get identical results for both expressions in 1.3.0, namely an integer overflow.
For
(defn fibo[]
(map first (iterate (fn [[a b]] [b (+ a b)]) [(bigint 0) 1])))
I get correct results for both expressions (a really big integer...).
I think that you may be hitting a specific memory limit for your machine, and not a real difference in function.
Looking at the source code for nth in https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/RT.java it does not look like either nth or take are retaining the head.
However, nth uses zero-based indexing, rather than a count by item number. Your code with nth selects the 1000001st element of the sequence (the one at index 1000000). You code with take is returning the final element in a 1000000 element sequence. That's the item with the index 999999. Given how fast fib grows, that last item could be the one that broke the camel's back.
Also, I was checking the 1.3.0 source. Perhaps earlier versions had different implementations. To get your fibo to work properly in 1.3.0 you need to use the arithmetic functions that will promote numbers to bignums:
(defn fibo[]
(map first (iterate (fn [[a b]] [b (+' a b)]) [0 1])))
I am trying to write a simple sieve function to calculate prime numbers in clojure. I've seen this question about writing an efficient sieve function, but I am not to that point yet. Right now I am just trying to write a very simple (and slow) sieve. Here is what I have come up with:
(defn sieve [potentials primes]
(if-let [p (first potentials)]
(recur (filter #(not= (mod % p) 0) potentials) (conj primes p))
primes))
For small ranges it works fine, but causes a stack overflow for large ranges:
user=> (sieve (range 2 30) [])
[2 3 5 7 11 13 17 19 23 29]
user=> (sieve (range 2 15000) [])
java.lang.StackOverflowError (NO_SOURCE_FILE:0)
I thought that by using recur this would be a non-stack-consuming looping construct? What am I missing?
You're being hit by filter's laziness. Change (filter ...) to (doall (filter ...)) in your recur form and the problem should go away.
A more in-depth explanation:
The call to filter returns a lazy seq, which materialises actual elements of the filtered seq as required. As written, your code stacks filter upon filter upon filter..., adding one more level of filtering at each iteration; at some point this blows up. The solution is to force the whole result at each iteration so that the next one will do its filtering on a fully realised seq and return a fully realised seq instead of adding an extra layer of lazy seq processing; that's what doall does.
Algorithmically the problem is that you continue filtering when there's no more purpose to it. Stopping as early as possible achieves quadratic reduction in recursion depth (sqrt(n) vs. n):
(defn sieve [potentials primes]
(if-let [p (first potentials)]
(if (> (* p p) (last potentials))
(concat primes potentials)
(recur (filter (fn [n] (not= (mod n p) 0)) potentials)
(conj primes p)))
primes))
Runs OK for 16,000 (performing just 30 iterations instead of 1862), and for 160,000 too, on ideone. Even runs 5% faster without the doall.
everyone, I've started working yesterday on the Euler Project in Clojure and I have a problem with one of my solutions I cannot figure out.
I have this function:
(defn find-max-palindrom-in-range [beg end]
(reduce max
(loop [n beg result []]
(if (>= n end)
result
(recur (inc n)
(concat result
(filter #(is-palindrom? %)
(map #(* n %) (range beg end)))))))))
I try to run it like this:
(find-max-palindrom-in-range 100 1000)
and I get this exception:
java.lang.Integer cannot be cast to clojure.lang.IFn
[Thrown class java.lang.ClassCastException]
which I presume means that at some place I'm trying to evaluate an Integer as a function. I however cannot find this place and what puzzles me more is that everything works if I simply evaluate it like this:
(reduce max
(loop [n 100 result []]
(if (>= n 1000)
result
(recur (inc n)
(concat result
(filter #(is-palindrom? %)
(map #(* n %) (range 100 1000))))))))
(I've just stripped down the function definition and replaced the parameters with constants)
Thanks in advance for your help and sorry that I probably bother you with idiotic mistake on my part. Btw I'm using Clojure 1.1 and the newest SLIME from ELPA.
Edit: Here is the code to is-palindrom?. I've implemented it as a text property of the number, not a numeric one.
(defn is-palindrom? [n]
(loop [num (String/valueOf n)]
(cond (not (= (first num) (last num))) false
(<= (.length num) 1) true
:else (recur (.substring num 1 (dec (.length num)))))))
The code works at my REPL (1.1). I'd suggest that you paste it back at yours and try again -- perhaps you simply mistyped something?
Having said that, you could use this as an opportunity to make the code simpler and more obviously correct. Some low-hanging fruit (don't read if you think it could take away from your Project Euler fun, though with your logic already written down I think it shouldn't):
You don't need to wrap is-palindrome? in an anonymous function to pass it to filter. Just write (filter is-palindrome? ...) instead.
That loop in is-palindrome? is pretty complex. Moreover, it's not particularly efficient (first and last both make a seq out of the string first, then last needs to traverse all of it). It would be simpler and faster to (require '[clojure.contrib.str-utils2 :as str]) and use (= num (str/reverse num)).
Since I mentioned efficiency, using concat in this manner is a tad dangerous -- it creates a lazy seq, which might blow up if you pile up two many levels of laziness (this will not matter in the context of Euler 4, but it's good to keep it in mind). If you really need to extend vectors to the right, prefer into.
To further simplify things, you could consider breaking them apart into a function to filter a given sequence so that only palindromes remain and a separate function to return all products of two three-digit numbers. The latter can be accomplished with e.g.
(for [f (range 100 1000)
s (range 100 1000)
:when (<= f s)] ; avoid duplication of effort
(* f s))