This works:
(defn tri*
([] (tri* 0 1))
([sum n]
(let [new-sum (+ sum n)]
(cons new-sum (lazy-seq (tri* new-sum (+ n 1)))))))
but when I use recur in it, I get a CompilerException:
Mismatched argument count to recur, expected 0 args, got: 2
(defn tri*
([] (tri* 0 1))
([sum n]
(let [new-sum (+ sum n)]
(cons new-sum (lazy-seq (recur new-sum (+ n 1)))))))
lazy-seq is a macro which expands to code involving a zero-arg thunk that can be called later as the sequence is realized. You can see that the recur form is captured as the body of this thunk by doing
(macroexpand '(lazy-seq (recur new-sum (+ n 1))))
to get
(new clojure.lang.LazySeq (fn* [] (recur new-sum (+ n 1))))
which shows that the target of recur is the thunk, not tri*.
Another way to think about all of this is that the original call to tri* has long since completed (and returned a Cons object containing LazySeq rest object) by the time recur would be “evaluated.”
Your first version of tri* is fine in that the recursive call to tri* doesn't grow the stack, but instead simply creates a new set of Cons / LazySeq objects.
Related
problem formulation
Informally speaking, I want to write a function which, taking as input a function that generates binary factorizations and an element (usually neutral), creates an arbitrary length factorization generator. To be more specific, let us first define the function nfoldr in Clojure.
(defn nfoldr [f e]
(fn rec [n]
(fn [s]
(if (zero? n)
(if (empty? s) e)
(if (seq s)
(if-some [x ((rec (dec n)) (rest s))]
(f (list (first s) x))))))))
Here nil is used with the meaning "undefined output, input not in function's domain". Additionally, let us view the inverse relation of a function f as a set-valued function defining inv(f)(y) = {x | f(x) = y}.
I want to define a function nunfoldr such that inv(nfoldr(f , e)(n)) = nunfoldr(inv(f) , e)(n) when for every element y inv(f)(y) is finite, for each binary function f, element e and natural number n.
Moreover, I want the factorizations to be generated as lazily as possible, in a 2-dimensional sense of laziness. My goal is that, when getting some part of a factorization for the first time, there does not happen (much) computation needed for next parts or next factorizations. Similarly, when getting one factorization for the first time, there does not happen computation needed for next ones, whereas all the previous ones get in effect fully realized.
In an alternative formulation we can use the following longer version of nfoldr, which is equivalent to the shorter one when e is a neutral element.
(defn nfoldr [f e]
(fn [n]
(fn [s]
(if (zero? n)
(if (empty? s) e)
((fn rec [n]
(fn [s]
(if (= 1 n)
(if (and (seq s) (empty? (rest s))) (first s))
(if (seq s)
(if-some [x ((rec (dec n)) (rest s))]
(f (list (first s) x)))))))
n)))))
a special case
This problem is a generalization of the problem of generating partitions described in that question. Let us see how the old problem can be reduced to the current one. We have for every natural number n:
npt(n) = inv(nconcat(n)) = inv(nfoldr(concat2 , ())(n)) = nunfoldr(inv(concat2) , ())(n) = nunfoldr(pt2 , ())(n)
where:
npt(n) generates n-ary partitions
nconcat(n) computes n-ary concatenation
concat2 computes binary concatenation
pt2 generates binary partitions
So the following definitions give a solution to that problem.
(defn generate [step start]
(fn [x] (take-while some? (iterate step (start x)))))
(defn pt2-step [[x y]]
(if (seq y) (list (concat x (list (first y))) (rest y))))
(def pt2-start (partial list ()))
(def pt2 (generate pt2-step pt2-start))
(def npt (nunfoldr pt2 ()))
I will summarize my story of solving this problem, using the old one to create example runs, and conclude with some observations and proposals for extension.
solution 0
At first, I refined/generalized the approach I took for solving the old problem. Here I write my own versions of concat and map mainly for a better presentation and, in the case of concat, for some added laziness. Of course we can use Clojure's versions or mapcat instead.
(defn fproduct [f]
(fn [s]
(lazy-seq
(if (and (seq f) (seq s))
(cons
((first f) (first s))
((fproduct (rest f)) (rest s)))))))
(defn concat' [s]
(lazy-seq
(if (seq s)
(if-let [x (seq (first s))]
(cons (first x) (concat' (cons (rest x) (rest s))))
(concat' (rest s))))))
(defn map' [f]
(fn rec [s]
(lazy-seq
(if (seq s)
(cons (f (first s)) (rec (rest s)))))))
(defn nunfoldr [f e]
(fn rec [n]
(fn [x]
(if (zero? n)
(if (= e x) (list ()) ())
((comp
concat'
(map' (comp
(partial apply map)
(fproduct (list
(partial partial cons)
(rec (dec n))))))
f)
x)))))
In an attempt to get inner laziness we could replace (partial partial cons) with something like (comp (partial partial concat) list). Although this way we get inner LazySeqs, we do not gain any effective laziness because, before consing, most of the computation required for fully realizing the rest part takes place, something that seems unavoidable within this general approach. Based on the longer version of nfoldr, we can also define the following faster version.
(defn nunfoldr [f e]
(fn [n]
(fn [x]
(if (zero? n)
(if (= e x) (list ()) ())
(((fn rec [n]
(fn [x] (println \< x \>)
(if (= 1 n)
(list (list x))
((comp
concat'
(map' (comp
(partial apply map)
(fproduct (list
(partial partial cons)
(rec (dec n))))))
f)
x))))
n)
x)))))
Here I added a println call inside the main recursive function to get some visualization of eagerness. So let us demonstrate the outer laziness and inner eagerness.
user=> (first ((npt 5) (range 3)))
< (0 1 2) >
< (0 1 2) >
< (0 1 2) >
< (0 1 2) >
< (0 1 2) >
(() () () () (0 1 2))
user=> (ffirst ((npt 5) (range 3)))
< (0 1 2) >
< (0 1 2) >
< (0 1 2) >
< (0 1 2) >
< (0 1 2) >
()
solution 1
Then I thought of a more promising approach, using the function:
(defn transpose [s]
(lazy-seq
(if (every? seq s)
(cons
(map first s)
(transpose (map rest s))))))
To get the new solution we replace the previous argument in the map' call with:
(comp
(partial map (partial apply cons))
transpose
(fproduct (list
repeat
(rec (dec n)))))
Trying to get inner laziness we could replace (partial apply cons) with #(cons (first %) (lazy-seq (second %))) but this is not enough. The problem lies in the (every? seq s) test inside transpose, where checking a lazy sequence of factorizations for emptiness (as a stopping condition) results in realizing it.
solution 2
A first way to tackle the previous problem that came to my mind was to use some additional knowledge about the number of n-ary factorizations of an element. This way we can repeat a certain number of times and use only this sequence for the stopping condition of transpose. So we will replace the test inside transpose with (seq (first s)), add an input count to nunfoldr and replace the argument in the map' call with:
(comp
(partial map #(cons (first %) (lazy-seq (second %))))
transpose
(fproduct (list
(partial apply repeat)
(rec (dec n))))
(fn [[x y]] (list (list ((count (dec n)) y) x) y)))
Let us turn to the problem of partitions and define:
(defn npt-count [n]
(comp
(partial apply *)
#(map % (range 1 n))
(partial comp inc)
(partial partial /)
count))
(def npt (nunfoldr pt2 () npt-count))
Now we can demonstrate outer and inner laziness.
user=> (first ((npt 5) (range 3)))
< (0 1 2) >
(< (0 1 2) >
() < (0 1 2) >
() < (0 1 2) >
() < (0 1 2) >
() (0 1 2))
user=> (ffirst ((npt 5) (range 3)))
< (0 1 2) >
()
However, the dependence on additional knowledge and the extra computational cost make this solution unacceptable.
solution 3
Finally, I thought that in some crucial places I should use a kind of lazy sequences "with a non-lazy end", in order to be able to check for emptiness without realizing. An empty such sequence is just a non-lazy empty list and overall they behave somewhat like the lazy-conss of the early days of Clojure. Using the definitions given below we can reach an acceptable solution, which works under the assumption that always at least one of the concat'ed sequences (when there is one) is non-empty, something that holds in particular when every element has at least one binary factorization and we are using the longer version of nunfoldr.
(def lazy? (partial instance? clojure.lang.IPending))
(defn empty-eager? [x] (and (not (lazy? x)) (empty? x)))
(defn transpose [s]
(lazy-seq
(if-not (some empty-eager? s)
(cons
(map first s)
(transpose (map rest s))))))
(defn concat' [s]
(if-not (empty-eager? s)
(lazy-seq
(if-let [x (seq (first s))]
(cons (first x) (concat' (cons (rest x) (rest s))))
(concat' (rest s))))
()))
(defn map' [f]
(fn rec [s]
(if-not (empty-eager? s)
(lazy-seq (cons (f (first s)) (rec (rest s))))
())))
Note that in this approach the input function f should produce lazy sequences of the new kind and the resulting n-ary factorizer will also produce such sequences. To take care of the new input requirement, for the problem of partitions we define:
(defn pt2 [s]
(lazy-seq
(let [start (list () s)]
(cons
start
((fn rec [[x y]]
(if (seq y)
(lazy-seq
(let [step (list (concat x (list (first y))) (rest y))]
(cons step (rec step))))
()))
start)))))
Once again, let us demonstrate outer and inner laziness.
user=> (first ((npt 5) (range 3)))
< (0 1 2) >
< (0 1 2) >
(< (0 1 2) >
() < (0 1 2) >
() < (0 1 2) >
() () (0 1 2))
user=> (ffirst ((npt 5) (range 3)))
< (0 1 2) >
< (0 1 2) >
()
To make the input and output use standard lazy sequences (sacrificing a bit of laziness), we can add:
(defn lazy-end->eager-end [s]
(if (seq s)
(lazy-seq (cons (first s) (lazy-end->eager-end (rest s))))
()))
(defn eager-end->lazy-end [s]
(lazy-seq
(if-not (empty-eager? s)
(cons (first s) (eager-end->lazy-end (rest s))))))
(def nunfoldr
(comp
(partial comp (partial comp eager-end->lazy-end))
(partial apply nunfoldr)
(fproduct (list
(partial comp lazy-end->eager-end)
identity))
list))
observations and extensions
While creating solution 3, I observed that the old mechanism for lazy sequences in Clojure might not be necessarily inferior to the current one. With the transition, we gained the ability to create lazy sequences without any substantial computation taking place but lost the ability to check for emptiness without doing the computation needed to get one more element. Because both of these abilities can be important in some cases, it would be nice if a new mechanism was introduced, which would combine the advantages of the previous ones. Such a mechanism could use again an outer LazySeq thunk, which when forced would return an empty list or a Cons or another LazySeq or a new LazyCons thunk. This new thunk when forced would return a Cons or perhaps another LazyCons. Now empty? would force only LazySeq thunks while first and rest would also force LazyCons. In this setting map could look like this:
(defn map [f s]
(lazy-seq
(if (empty? s) ()
(lazy-cons
(cons (f (first s)) (map f (rest s)))))))
I have also noticed that the approach taken from solution 1 onwards lends itself to further generalization. If inside the argument in the map' call in the longer nunfoldr we replace cons with concat, transpose with some implementation of Cartesian product and repeat with another recursive call, we can then create versions that "split at different places". For example, using the following as the argument we can define a nunfoldm function that "splits in the middle" and corresponds to an easy-to-imagine nfoldm. Note that all "splitting strategies" are equivalent when f is associative.
(comp
(partial map (partial apply concat))
cproduct
(fproduct (let [n-half (quot n 2)]
(list (rec n-half) (rec (- n n-half))))))
Another natural modification would allow for infinite factorizations. To achieve this, if f generated infinite factorizations, nunfoldr(f , e)(n) should generate the factorizations in an order of type ω, so that each one of them could be produced in finite time.
Other possible extensions include dropping the n parameter, creating relational folds (in correspondence with the relational unfolds we consider here) and generically handling algebraic data structures other than sequences as input/output. This book, which I have just discovered, seems to contain valuable relevant information, given in a categorical/relational language.
Finally, to be able to do this kind of programming more conveniently, we could transfer it into a point-free, algebraic setting. This would require constructing considerable "extra machinery", in fact almost making a new language. This paper demonstrates such a language.
Let's say you have a recursive function defined in a let block:
(let [fib (fn fib [n]
(if (< n 2)
n
(+ (fib (- n 1))
(fib (- n 2)))))]
(fib 42))
This can be mechanically transformed to utilize memoize:
Wrap the fn form in a call to memoize.
Move the function name in as the 1st argument.
Pass the function into itself wherever it is called.
Rebind the function symbol to do the same using partial.
Transforming the above code leads to:
(let [fib (memoize
(fn [fib n]
(if (< n 2)
n
(+ (fib fib (- n 1))
(fib fib (- n 2))))))
fib (partial fib fib)]
(fib 42))
This works, but feels overly complicated. The question is: Is there a simpler way?
I take risks in answering since I am not a scholar but I don't think so. You pretty much did the standard thing which in fine is a partial application of memoization through a fixed point combinator.
You could try to fiddle with macros though (for simple cases it could be easy, syntax-quote would do name resolution for you and you could operate on that). I'll try once I get home.
edit: went back home and tried out stuff, this seems to be ok-ish for simple cases
(defmacro memoize-rec [form]
(let [[fn* fname params & body] form
params-with-fname (vec (cons fname params))]
`(let [f# (memoize (fn ~params-with-fname
(let [~fname (partial ~fname ~fname)] ~#body)))]
(partial f# f#))))
;; (clojure.pprint/pprint (macroexpand '(memoize-rec (fn f [x] (str (f x))))))
((memoize-rec (fn fib [n]
(if (< n 2)
n
(+ (fib (- n 1))
(fib (- n 2)))))) 75) ;; instant
((fn fib [n]
(if (< n 2)
n
(+ (fib (- n 1))
(fib (- n 2))))) 75) ;; slooooooow
simpler than what i thought!
I'm not sure this is "simpler" per se, but I thought I'd share an approach I took to re-implement letfn for a CPS transformer I wrote.
The key is to introduce the variables, but delay assigning them values until they are all in scope. Basically, what you would like to write is:
(let [f nil]
(set! f (memoize (fn []
<body-of-f>)))
(f))
Of course this doesn't work as is, because let bindings are immutable in Clojure. We can get around that, though, by using a reference type — for example, a promise:
(let [f (promise)]
(deliver! f (memoize (fn []
<body-of-f>)))
(#f))
But this still falls short, because we must replace every instance of f in <body-of-f> with (deref f). But we can solve this by introducing another function that invokes the function stored in the promise. So the entire solution might look like this:
(let [f* (promise)]
(letfn [(f []
(#f*))]
(deliver f* (memoize (fn []
<body-of-f>)))
(f)))
If you have a set of mutually-recursive functions:
(let [f* (promise)
g* (promise)]
(letfn [(f []
(#f*))
(g []
(#g*))]
(deliver f* (memoize (fn []
(g))))
(deliver g* (memoize (fn []
(f))))
(f)))
Obviously that's a lot of boiler-plate. But it's pretty trivial to construct a macro that gives you letfn-style syntax.
Yes, there is a simpler way.
It is not a functional transformation, but builds on the impurity allowed in clojure.
(defn fib [n]
(if (< n 2)
n
(+ (#'fib (- n 1))
(#'fib (- n 2)))))
(def fib (memoize fib))
First step defines fib in almost the normal way, but recursive calls are made using whatever the var fib contains. Then fib is redefined, becoming the memoized version of its old self.
I would suppose that clojure idiomatic way will be using recur
(def factorial
(fn [n]
(loop [cnt n acc 1]
(if (zero? cnt)
acc
(recur (dec cnt) (* acc cnt))
;; Memoized recursive function, a mash-up of memoize and fn
(defmacro mrfn
"Returns an anonymous function like `fn` but recursive calls to the given `name` within
`body` use a memoized version of the function, potentially improving performance (see
`memoize`). Only simple argument symbols are supported, not varargs or destructing or
multiple arities. Memoized recursion requires explicit calls to `name` so the `body`
should not use recur to the top level."
[name args & body]
{:pre [(simple-symbol? name) (vector? args) (seq args) (every? simple-symbol? args)]}
(let [akey (if (= (count args) 1) (first args) args)]
;; name becomes extra arg to support recursive memoized calls
`(let [f# (fn [~name ~#args] ~#body)
mem# (atom {})]
(fn mr# [~#args]
(if-let [e# (find #mem# ~akey)]
(val e#)
(let [ret# (f# mr# ~#args)]
(swap! mem# assoc ~akey ret#)
ret#))))))
;; only change is fn to mrfn
(let [fib (mrfn fib [n]
(if (< n 2)
n
(+ (fib (- n 1))
(fib (- n 2)))))]
(fib 42))
Timings on my oldish Mac:
original, Elapsed time: 14089.417441 msecs
mrfn version, Elapsed time: 0.220748 msecs
"Iteration provides an infinite lazy sequence"
(= (range 20) (take 20 (iterate inc 0)))
So my question is why it start from 0 instead of 1 ? How to understand the laziness here ?
The koan asks for the iteration to begin at zero because range starts at 0 by default, and that makes for nice looking statement. It is typical and useful in programming to start counting at 0 rather than 1. But, The koan could have been written to start at 1 (or any other number for that matter)
(= (range 1 21) (take 20 (iterate inc 1)))
Here's how iterate is defined
user=> (source iterate)
(defn iterate
"Returns a lazy sequence of x, (f x), (f (f x)) etc. f must be free of side-effects"
{:added "1.0"
:static true}
[f x] (cons x (lazy-seq (iterate f (f x)))))
So, here's how (iterate inc 0) looks at first
(cons 0, (lazy-seq (iterate inc (inc 0))))
Now when the special lazy-seq element at position 1 (that is, the second position since we count from 0) is first accessed, it is in effect replaced by its expansion.
-- (iterate inc (inc 0))
-> (iterate inc 1)
-> (cons 1, (lazy-seq (iterate inc (inc 1))))
So, the sequence now looks like
-- (cons 0, (lazy-seq (iterate inc (inc 0))))
-> (cons 0, (cons 1 (lazy-seq (iterate inc (inc 1)))))
When the special lazy-seq element at position 2 is first accessed, it is in effect replaced by its expansion.
-- (iterate inc (inc 1))
-> (iterate inc 2)
-> (cons 2, (lazy-seq (iterate inc (inc 2))))
So, the sequence now looks like
-- (cons 0, (cons 1 (lazy-seq (iterate inc (inc 1)))))
-> (cons 0, (cons 1, (cons 2, (lazy-seq (iterate inc (inc 2))))))
Some consequences to be aware, but not typically anything to worry about as a new user
Any side effects in the body are not executed until accessed, and then only once
Because you can return a lazy sequence that is not fully realized, beware dependance on dynamic scope, e.g. returning within a with- block.
If you hold a reference to a long/infinite lazy sequence, e.g. (def numbers (iterate inc 0)) and realize a bunch of them, they will remain in memory. Avoid "holding the head" if this is to be a problem.
clojure.core/iterate takes two arguments:
the fn to apply to the last element in the sequence. When applied, it should produce the next element in the sequence
the initial value of the sequence
(iterate inc 0) has 0 as the initial element of the sequence.
(take 1 (iterate inc 0)) ;; (0)
I'm reading Programming Clojure now and I find out next example
(defn by-pairs [coll]
(let
[take-pair (fn [c] (when (next c) (take 2 c)))]
(lazy-seq
(when-let [pair (seq (take-pair coll))] ;seq calls here
(cons pair (by-pairs (rest coll)))))))
it breaks list into pairs, like
(println (by-pairs [1 2 1])) ((1 2) (2 1))
(println (by-pairs [1 2 1 3])) ((1 2) (2 1) (1 3))
(println (by-pairs [])) ()
(println (by-pairs [1])) ()
What I can not get is why we should invoke seq on take-pair result? So why we can not just write
(defn by-pairs [coll]
(let
[take-pair (fn [c] (when (next c) (take 2 c)))]
(lazy-seq
(when-let [pair (take-pair coll)]
(cons pair (by-pairs (rest coll)))))))
In witch cases there are will be different results or are there are any performance reasons?
Both the code are same and there will be no difference because next and take functions that are being applied to coll in take-pair function, do call seq on the passed parameter i.e next c will first call seq on c or try to check if it is an object which implements ISeq and same in being doing by the take function. So basically in this case if you don't call seq yourself, the next and take will call seq on it.
I can't figure out why this definition of a lazy primes sequence would cause non-termination. The stack-trace I get isn't very helpful (my one complaint about clojure is obtuse stack-traces).
(declare naturals is-prime? primes)
(defn naturals
([] (naturals 1))
([n] (lazy-seq (cons n (naturals (inc n))))))
(defn is-prime? [n]
(not-any? #(zero? (rem n %))
(take-while #(> n (* % %)) (primes))))
(defn primes
([] (lazy-seq (cons 2 (primes 3))))
([n] (let [m (first (filter is-prime? (naturals n)))]
(lazy-seq (cons m (primes (+ 2 m)))))))
(take 10 (primes)) ; this results in a stack overflow error
Let's start executing primes, and we'll magically realise one seq just to be clear. I'll ignore naturals because it's correctly lazy:
> (magically-realise-seq (primes))
=> (magically-realise-seq (lazy-seq (cons 2 (primes 3))))
=> (cons 2 (primes 3))
=> (cons 2 (let [m (first (filter is-prime? (naturals 3)))]
(lazy-seq (cons m (primes (+ 2 3))))))
=> (cons 2 (let [m (first (filter
(fn [n]
(not-any? #(zero? (rem n %))
(take-while #(> n (* % %)) (primes)))))
(naturals 3)))]
(lazy-seq (cons m (primes (+ 2 3))))))
I've substituted is-prime? in as a fn at the end there—you can see that primes will get called again, and realised at least once as take-while pulls out elements. This will then cause the loop.
The issue is that to know to calculate the "primes" function you are using the "is-prime?" function, and then to calculate the "is-prime?" function you are using "(primes)", hence the stack over flow.
So to calculate the "(primes 3)", you are need calculate the "(first (filter is-prime? (naturals 3)))", which is going to call "(is-prime? 1)", which is calling "(primes)", which in turns calls "(primes 3)". In other words you are doing:
user=> (declare a b)
#'user/b
user=> (defn a [] (b))
#'user/a
user=> (defn b [] (a))
#'user/b
user=> (a)
StackOverflowError user/b (NO_SOURCE_FILE:1)
To see how to generate prime numbers: Fast Prime Number Generation in Clojure
I think the problem is, that you're trying to use (primes) before it's already constructed.
Changing is-prime? like that fixes the problem:
(defn is-prime? [n]
(not-any? #(zero? (rem n %))
(take-while #(>= n (* % %)) (next (naturals)))))
(Note, that I've changed > with >=, otherwise it gives that 4 is prime. It still says that 1 is prime, which isn't true and may cause problems if you use is-prime? elsewhere.