I'm trying to solve a 4Clojure problem (sequence reductions), and I've hit a wall. The problem is to reimplement the reductions function.
It seems to me like this function should return a lazy sequence, but it doesn't - evaluating (take 5 (redux + (range))) results in an infinite loop.
Here's my code:
(defn redux
([f coll]
(redux f (first coll) (rest coll)))
([f val coll]
((fn red [val coll s]
(if (empty? coll)
s
(lazy-seq
(let [val (f val (first coll))]
(red val
(rest coll)
(conj s val))))))
val coll [val])))
Why is this function not returning a lazy sequence?
There are a few misconceptions in the code. noisesmith pointed out on #clojurians chat (and Josh's comment stated as well) the following points:
There is no step in the above function where you can evaluate the head and not the tail of a list.
It does an immediate self recursion, and to get the n+1 element, you need to do the recursive call.
lazy-seq should always have a call to cons or some similar function that lets you return the next item of the list without recurring.
conj is never lazy, and vectors are never lazy.
You cannot append to a list without realizing the entire thing.
I modified the code to the following:
(fn redux
([f coll]
(redux f (first coll) (rest coll)))
([f val coll]
(cons val
((fn red [val coll]
(lazy-seq
(when-not (empty? coll)
(let [val (f val (first coll))]
(cons val (red val (rest coll)))))))
val coll))))
Note the use of cons instead of conj.
Related
I need to make filter method, but i can't return method name as argument, not as result.
In my case, i need to input odd? method as argument and call recursion.
I can use only this construction:
(defn my-filter [p xn])
My code:
(defn my-filter [p xs]
(if (p (first xs))
(cons (first xs) (recur p (next xs)))
(recur p (next xs) )))
(my-filter odd? '(1 2 3 4 5))
Error: IllegalArgumentException Argument must be an integer: clojure.core/even? (core.clj:1372)
As i can see, where recursion is called, arguments are calculating result, instead of call recursion with odd? and (next xs) arguments
Two issues need attention. Or maybe only one issue, if you don't need to handle very long lists. 1) The function does not notice when the inputs are exhausted. Open a REPL and try (odd? nil) and you will see what happens! 2) If you try the function on a really long list, you might get a StackOverflow. The clojure.org guide for recursion has an example of how to avoid that problem - actually it illustrates solutions to both problems: https://clojure.org/guides/learn/flow#_recursion
Your code does not compile:
Syntax error (UnsupportedOperationException) compiling recur at ... .
Can only recur from tail position
You have to replace the recurs with explicit recursive calls to my-filter:
(defn my-filter [p xs]
(if (p (first xs))
(cons (first xs) (my-filter p (next xs)))
(my-filter p (next xs))))
Now it compiles, but ...
(my-filter odd? [])
Execution error (IllegalArgumentException) at ...
Argument must be an integer:
You need to check that the sequence argument xs is not empty before doing anything else with it:
(defn my-filter [p xs]
(when (seq xs)
(if (p (first xs))
(cons (first xs) (my-filter p (rest xs)))
(my-filter p (rest xs)))))
The when evaluates to nil if the condition fails. The nil, called on to be a sequence, behaves as an empty one. So ...
(my-filter odd? [])
=> nil
(my-filter odd? (range 10))
=> (1 3 5 7 9)
It works. However, it evaluates (first xs) twice, and mentions (my-filter p (rest xs)) twice. Factoring these out, we get
(defn my-filter [p xs]
(when (seq xs)
(let [head (first xs)
tail (my-filter p (rest xs))]
(if (p head) (cons head tail) tail))))
This uses direct recursion. So it runs out of stack on a long sequence:
(count (my-filter odd? (range 10000)))
Execution error (StackOverflowError) at ...
Wrapping the recursion in lazy-seq flattens the evaluation, devolving it to whatever explores the sequence:
(defn my-filter [p xs]
(lazy-seq
(when (seq xs)
(let [head (first xs)
tail (my-filter p (rest xs))]
(if (p head) (cons head tail) tail)))))
Now ...
(count (my-filter odd? (range 10000)))
=> 5000
If you want an eager version, you had better build the returned sequence as a vector:
(defn eager-filter [p coll]
(loop [answer [], coll (seq coll)]
(if-let [[x & xs] coll]
(recur
(if (p x) (conj answer x) answer)
xs)
(sequence answer))))
This won't run out of stack:
(count (eager-filter odd? (range 10000)))
=> 5000
But it can't handle an endless sequence:
(first (eager-filter odd? (range)))
Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
I had to kill the process.
I really like this solution, therefore I share with you. Very simple. (I know, that is not exactly what was the question but different approach.)
(def data (vec (range 10)))
Map implementation
(defn -map [f coll]
(reduce
(fn [acc v]
(conj acc (f v)))
[]
coll))
Filter implementation
(defn -filter [f coll]
(reduce
(fn [acc v]
(if (f v)
(conj acc v))
[]
coll))
Example usage
(->> data
(-map inc)
(-filter odd?))
I am playing around and trying to create my own reductions implementation, so far I have this which works with this test data:
((fn [func & args]
(reduce (fn [acc item]
(conj acc (func (last acc) item))
)[(first args)] (first (rest args)))) * 2 [3 4 5]
What I don't like is how I am separating the args.
(first args) is what I would expect, i.e. 2 but (rest args) is ([3 4 5]) and so I am getting the remainder like this (first (rest args)) which I do not like.
Am I missing some trick that makes it easier to work with variadic arguments?
Variadic arguments are just about getting an unspecified number of arguments in a list, so all list/destructuring operations can be applied here.
For example:
(let [[fst & rst] a-list]
; fst is the first element
; rst is the rest
)
This is more readable than:
(let [fst (first a-list)
rst (rest a-list)]
; ...
)
You can go further to get the first and second elements of a list (assuming it has >1 elements) in one line:
(let [fst snd & rst]
; ...
)
I originally misread your question and thought you were trying to reimplement the reduce function. Here is a sample implementation I wrote for this answer which does’t use first or rest:
(defn myreduce
;; here we accept the form with no initial value
;; like in (myreduce * [2 3 4 5]), which is equivalent
;; to (myreduce * 2 [3 4 5]). Notice how we use destructuring
;; to get the first/rest of the list passed as a second
;; argument
([op [fst & rst]] (myreduce op fst rst))
;; we take an operator (function), accumulator and list of elements
([op acc els]
;; no elements? give the accumulator back
(if (empty? els)
acc
;; all the function's logic is in here
;; we're destructuring els to get its first (el) and rest (els)
(let [[el & els] els]
;; then apply again the function on the same operator,
;; using (op acc el) as the new accumulator, and the
;; rest of the previous elements list as the new
;; elements list
(recur op (op acc el) els)))))
I hope it helps you see how to work with list destructuring, which is probably what you want in your function. Here is a relevant blog post on this subject.
Tidying up your function.
As #bfontaine commented, you can use (second args) instead of (first (rest args)):
(defn reductions [func & args]
(reduce
(fn [acc item] (conj acc (func (last acc) item)))
[(first args)]
(second args)))
This uses
func
(first args)
(second args)
... but ignores the rest of args.
So we can use destructuring to name the first and second elements of args - init and coll seem suitable - giving
(defn reductions [func & [init coll & _]]
(reduce
(fn [acc item] (conj acc (func (last acc) item)))
[init]
coll))
... where _ is the conventional name for the ignored argument, in this case a sequence.
We can get rid of it, simplifying to
(defn reductions [func & [init coll]] ... )
... and then to
(defn reductions [func init coll] ... )
... - a straightforward function of three arguments.
Dealing with the underlying problems.
Your function has two problems:
slowness
lack of laziness.
Slowness
The flashing red light in this function is the use of last in
(fn [acc item] (conj acc (func (last acc) item)))
This scans the whole of acc every time it is called, even if acc is a vector. So this reductions takes time proportional to the square of the length of coll: hopelessly slow for long sequences.
A simple fix is to replace (last acc) by (acc (dec (count acc))), which takes effectively constant time.
Lack of laziness
We still can't lazily use what the function produces. For example, it would be nice to encapsulate the sequence of factorials like this:
(def factorials (reductions * 1N (next (range)))))
With your reductions, this definition never returns.
You have to entirely recast your function to make it lazy. Let's modify the standard -lazy -reductions to employ destructuring:
(defn reductions [f init coll]
(cons
init
(lazy-seq
(when-let [[x & xs] (seq coll)]
(reductions f (f init x) xs)))))
Now we can define
(def factorials (reductions * 1N (next (range))))
Then, for example,
(take 10 factorials)
;(1N 1N 2N 6N 24N 120N 720N 5040N 40320N 362880N)
Another approach is to derive the sequence from itself, like a railway locomotive laying the track it travels on:
(defn reductions [f init coll]
(let [answer (lazy-seq (reductions f init coll))]
(cons init (map f answer coll))))
But this contains a hidden recursion (hidden from me, at least):
(nth (reductions * 1N (next (range))) 10000)
;StackOverflowError ...
I'm working on 4clojure problems and a similar issue keeps coming up. I'll write a solution that works for all but one of the test cases. It's usually the one that is checking for lazy evaluation. The solution below works for all but the last test case. I've tried all kinds of solutions and can't seem to get it to stop evaluating until integer overflow. I read the chapter on lazy sequences in Joy of Clojure, but I'm having a hard time implementing them. Is there a rule of thumb I'm forgetting, like don't use loop or something like that?
; This version is non working at the moment, will try to edit a version that works
(defn i-between [p k coll]
(loop [v [] coll coll]
(let [i (first coll) coll (rest coll) n (first coll)]
(cond (and i n)
(let [ret (if (p i n) (cons k (cons i v)) (cons i v))]
(recur ret coll))
i
(cons i v )
:else v))))
Problem 132
Ultimate solution for those curious:
(fn i-between [p k coll]
(letfn [(looper [coll]
(if (empty? coll) coll
(let [[h s & xs] coll
c (cond (and h s (p h s))
(list h k )
(and h s)
(list h )
:else (list h))]
(lazy-cat c (looper (rest coll))))
))] (looper coll)))
When I think about lazy sequences, what usually works is thinking about incremental cons'ing
That is, each recursion step only adds a single element to the list, and of course you never use loop.
So what you have is something like this:
(cons (generate first) (recur rest))
When wrapped on lazy-seq, only the needed elements from the sequence are realized, for instance.
(take 5 (some-lazy-fn))
Would only do 5 recursion calls to realize the needed elements.
A tentative, far from perfect solution to the 4clojure problem, that demonstrates the idea:
(fn intercalate
[pred value col]
(letfn [(looper [s head]
(lazy-seq
(if-let [sec (first s)]
(if (pred head sec)
(cons head (cons value (looper (rest s) sec)))
(cons head (looper (rest s) sec)))
(if head [head] []))))]
(looper (rest col) (first col))))
There, the local recursive function is looper, for each element tests if the predicate is true, in that case realizes two elements(adds the interleaved one), otherwise realize just one.
Also, you can avoid recursion using higher order functions
(fn [p v xs]
(mapcat
#(if (p %1 %2) [%1 v] [%1])
xs
(lazy-cat (rest xs) (take 1 xs))))
But as #noisesmith said in the comment, you're just calling a function that calls lazy-seq.
What's the idiomatic way in Clojure to implement take-while-and-n-more below:
=> (take-while-and-n-more #(<= % 3) 1 (range 10))
(0 1 2 3 4)
My try is:
(defn take-while-and-n-more [pred n coll]
(let
[take-while-result (take-while pred coll)
n0 (count take-while-result)]
(concat
take-while-result
(into [] (take n (drop n0 coll))))))
I would use split-with, which is equivalent of getting results of both take-while and drop-while for the same parameters:
(defn take-while-and-n-more [pred n coll]
(let [[head tail] (split-with pred coll)]
(concat head (take n tail))))
Yet another way:
(defn take-while-and-n-more [pred n coll]
(let [[a b] (split-with pred coll)]
(concat a (take n b))))
The following code is a modified version of Clojures take-while. Where Clojures take-while returns nil as a default case (when the predicate does not match), this one invokes take to take the the additional items after the predicate fails.
Note that unlike versions using split-with, this version traverses the sequence only once.
(defn take-while-and-n-more
[pred n coll]
(lazy-seq
(when-let [s (seq coll)]
(if (pred (first s))
(cons (first s) (take-while-and-n-more pred n (rest s)))
(take n s)))))
I'd like to know if there's a function in Clojure equivalent to this:
(defn reduce-1 [f val coll]
(loop [[head & tail] coll
out val]
(if head
(recur tail (f out head tail))
out)))
Note that this differs from the usual 'reduce in that 'tail is passed to 'f.
I'm using this concept (recursively collect interactions of first and rest of list) so much, that I've started to wonder if there's a standard function for that.
You could use iterate.
(defn reduce-with-tail
[f initial coll]
(->> (seq coll)
(iterate next)
(take-while identity)
(reduce (fn [initial [head & tail]] (f initial head tail)) initial)))