Create a lazy sequence by concatenating collections.
Consider the following function:
(defn req []
(Thread/sleep 1000)
(repeat 4 (rand-int 10)))
The sleep is added since the function will finally be a http request, thus it should emulate a delay.
Sample outputs:
(req)
;; (8 8 8 8)
(req)
;; (4 4 4 4)
I'm thinking of a function now, that creates a lazy sequence build by the concatenation of subsequent req results.
(take 10 (f req))
;; (3 3 3 3 2 2 2 2 9 9)
Here is one implementation:
(defn f [g]
(lazy-seq (concat (g) (f g))))
Is this the way to go? I'm somehow guessing that there might be already an abstraction for this available.. I tried lazy-cat, but this macro seems to work only for a fixed number of given sequences.
It turns out that this is a working abstraction:
(take 10 (apply concat (repeatedly req)))
However it looks like chunking of lazy sequences causes req to be called more often than needed here, which would not be acceptable if it's an http request.
The "unneeded" realizations of elements of lazy sequence is happening because apply needs to know the number of arguments that passed function is applied to.
Having a quick look at Clojure core lib, it seems, that it doesn't provide a function that concatenates a sequence of sequences and, at the same time, handles laziness in a way you want (doesn't redundantly realize the elements of passed lazy sequence), so, you'll need to implement it yourself.
Here's possible solution:
(defn apply-concat [xs]
(lazy-seq
(when-let [s (seq xs)]
(concat (first s) (apply-concat (rest s))))))
And then:
user> (defn req []
(println "--> making request")
[1 2 3 4])
#'user/req
user> (dorun (take 4 (apply-concat (repeatedly req))))
--> making request
nil
user> (dorun (take 5 (apply-concat (repeatedly req))))
--> making request
--> making request
nil
user> (dorun (take 8 (apply-concat (repeatedly req))))
--> making request
--> making request
nil
user> (dorun (take 9 (apply-concat (repeatedly req))))
--> making request
--> making request
--> making request
nil
The only concern with this approach is danger of blowing the stack, since apply-concat is potentially infinitely recursive.
Update:
To be precise apply realizes (arity of passed function + 1) elements of passed lazy sequence:
user> (dorun (take 1 (apply (fn [& xs] xs) (repeatedly req))))
--> making request
--> making request
nil
user> (dorun (take 1 (apply (fn [x & xs] xs) (repeatedly req))))
--> making request
--> making request
--> making request
nil
user> (dorun (take 1 (apply (fn [x y & xs] xs) (repeatedly req))))
--> making request
--> making request
--> making request
--> making request
nil
how about
(take 14
(mapcat identity (repeatedly req)))
explanation:
(defn req []
(print ".")
(repeat 4 (rand-int 10)))
(def x
(take 80 (mapcat identity (repeatedly req))))
; prints .... = 4x ; this is probably some repl eagerness
; to take 80 items, 20 realizatons (of 4 items) are requrend
(def y
(doall
(take 80 (mapcat identity (repeatedly req)))))
; prints ..................... = 21x
EDIT: about those 4 early realizations:
I think this is due apply, which us used by mapcat.
It realizes up to 4 args [^clojure.lang.IFn f a b c d & args] given multiple ones.
Related
I often have to run my data through a function if the data fulfill certain criteria. Typically, both the function f and the criteria checker pred are parameterized to the data. For this reason, I find myself wishing for a higher-order if-then-else which knows neither f nor pred.
For example, assume I want to add 10 to all even integers in (range 5). Instead of
(map #(if (even? %) (+ % 10) %) (range 5))
I would prefer to have a helper –let's call it fork– and do this:
(map (fork even? #(+ % 10)) (range 5))
I could go ahead and implement fork as function. It would look like this:
(defn fork
([pred thenf elsef]
#(if (pred %) (thenf %) (elsef %)))
([pred thenf]
(fork pred thenf identity)))
Can this be done by elegantly combining core functions? Some nice chain of juxt / apply / some maybe?
Alternatively, do you know any Clojure library which implements the above (or similar)?
As Alan Thompson mentions, cond-> is a fairly standard way of implicitly getting the "else" part to be "return the value unchanged" these days. It doesn't really address your hope of being higher-order, though. I have another reason to dislike cond->: I think (and argued when cond-> was being invented) that it's a mistake for it to thread through each matching test, instead of just the first. It makes it impossible to use cond-> as an analogue to cond.
If you agree with me, you might try flatland.useful.fn/fix, or one of the other tools in that family, which we wrote years before cond->1.
to-fix is exactly your fork, except that it can handle multiple clauses and accepts constants as well as functions (for example, maybe you want to add 10 to other even numbers but replace 0 with 20):
(map (to-fix zero? 20, even? #(+ % 10)) xs)
It's easy to replicate the behavior of cond-> using fix, but not the other way around, which is why I argue that fix was the better design choice.
1 Apparently we're just a couple weeks away from the 10-year anniversary of the final version of fix. How time flies.
I agree that it could be very useful to have some kind of higher-order functional construct for this but I am not aware of any such construct. It is true that you could implement a higher order fork function, but its usefulness would be quite limited and can easily be achieved using if or the cond-> macro, as suggested in the other answers.
What comes to mind, however, are transducers. You could fairly easily implement a forking transducer that can be composed with other transducers to build powerful and concise sequence processing algorithms.
The implementation could look like this:
(defn forking [pred true-transducer false-transducer]
(fn [step]
(let [true-step (true-transducer step)
false-step (false-transducer step)]
(fn
([] (step))
([dst x] ((if (pred x) true-step false-step) dst x))
([dst] dst))))) ;; flushing not performed.
And this is how you would use it in your example:
(eduction (forking even?
(map #(+ 10 %))
identity)
(range 20))
;; => (10 1 12 3 14 5 16 7 18 9 20 11 22 13 24 15 26 17 28 19)
But it can also be composed with other transducers to build more complex sequence processing algorithms:
(into []
(comp (forking even?
(comp (drop 4)
(map #(+ 10 %)))
(comp (filter #(< 10 %))
(map #(vector % % %))
cat))
(partition-all 3))
(range 20))
;; => [[18 20 11] [11 11 22] [13 13 13] [24 15 15] [15 26 17] [17 17 28] [19 19 19]]
Another way to define fork (with three inputs) could be:
(defn fork [pred then else]
(comp
(partial apply apply)
(juxt (comp {true then, false else} pred) list)))
Notice that in this version the inputs and output can receive zero or more arguments. But let's take a more structured approach, defining some other useful combinators. Let's start by defining pick which corresponds to the categorical coproduct (sum) of morphisms:
(defn pick [actions]
(fn [[tag val]]
((actions tag) val)))
;alternatively
(defn pick [actions]
(comp
(partial apply apply)
(juxt (comp actions first) rest)))
E.g. (mapv (pick [inc dec]) [[0 1] [1 1]]) gives [2 0]. Using pick we can define switch which works like case:
(defn switch [test actions]
(comp
(pick actions)
(juxt test identity)))
E.g. (mapv (switch #(mod % 3) [inc dec -]) [3 4 5]) gives [4 3 -5]. Using switch we can easily define fork:
(defn fork [pred then else]
(switch pred {true then, false else}))
E.g. (mapv (fork even? inc dec) [0 1]) gives [1 0]. Finally, using fork let's also define fork* which receives zero or more predicate and action pairs and works like cond:
(defn fork* [& args]
(->> args
(partition 2)
reverse
(reduce
(fn [else [pred then]]
(fork pred then else))
identity)))
;equivalently
(defn fork* [& args]
(->> args
(partition 2)
(map (partial apply (partial partial fork)))
(apply comp)
(#(% identity))))
E.g. (mapv (fork* neg? -, even? inc) [-1 0 1]) gives [1 1 1].
Depending on the details, it is often easiest to accomplish this goal using the cond-> macro and friends:
(let [myfn (fn [val]
(cond-> val
(even? val) (+ val 10))) ]
with result
(mapv myfn (range 5)) => [10 1 14 3 18]
There is a variant in the Tupelo library that is sometimes helpful:
(mapv #(cond-it-> %
(even? it) (+ it 10))
(range 5))
that allows you to use the special symbol it as you thread the value through multiple stages.
As the examples show, you have the option to define and name the transformer function (my favorite), or use the function literal syntax #(...)
I'm currently learning Clojure, and I'm trying to learn how to do things the best way. Today I'm looking at the basic concept of doing things on a sequence, I know the basics of map, filter and reduce. Now I want to try to do a thing to pairs of elements in a sequence, and I found two ways of doing it. The function I apply is println. The output is simply 12 34 56 7
(def xs [1 2 3 4 5 6 7])
(defn work_on_pairs [xs]
(loop [data xs]
(if (empty? data)
data
(do
(println (str (first data) (second data)))
(recur (drop 2 data))))))
(work_on_pairs xs)
I mean, I could do like this
(map println (zipmap (take-nth 2 xs) (take-nth 2 (drop 1 xs))))
;; prints [1 2] [3 4] [5 6], and we loose the last element because zip.
But it is not really nice.. My background is in Python, where I could just say zip(xs[::2], xs[1::2]) But I guess this is not the Clojure way to do it.
So I'm looking for suggestions on how to do this same thing, in the best Clojure way.
I realize I'm so new to Clojure I don't even know what this kind of operation is called.
Thanks for any input
This can be done with partition-all:
(def xs [1 2 3 4 5 6 7])
(->> xs
(partition-all 2) ; Gives ((1 2) (3 4) (5 6) (7))
(map (partial apply str)) ; or use (map #(apply str %))
(apply println))
12 34 56 7
The map line is just to join the pairs so the "()" don't end up in the output.
If you want each pair printed on its own line, change (apply println) to (run! println). Your expected output seems to disagree with your code, so that's unclear.
If you want to dip into transducers, you can do something similar to the threading (->>) form of the accepted answer, but in a single pass over the data.
Assuming
(def xs [1 2 3 4 5 6 7])
has been evaluated already,
(transduce
(comp
(partition-all 2)
(map #(apply str %)))
conj
[]
xs)
should give you the same output if you wrap it in
(apply println ...)
We supply conj (reducing fn) and [] (initial data structure) to specify how the reduce process inside transduce should build up the result.
I wouldn't use a transducer for a list that small, or a process that simple, but it's good to know what's possible!
I'm learning Clojure and actually I'm doing some exercises to practice but I'm stuck in a problem:
I need to make a sum-consecutives function which sums consecutive elements in a array, resulting in a new one, as example:
[1,4,4,4,0,4,3,3,1] ; should return [1,12,0,4,6,1]
I made this function which should work just fine:
(defn sum-consecutives [a]
(reduce #(into %1 (apply + %2)) [] (partition-by identity a)))
But it throws an error:
IllegalArgumentException Don't know how to create ISeq from:
java.lang.Long clojure.lang.RT.seqFrom (RT.java:542)
Can anyone help me see what is wrong with my func? I've already search this error in web but I find no helpful solutions.
You'll likely want to use conj instead of into, as into is expecting its second argument to be a seq:
(defn sum-consecutives [a]
(reduce
#(conj %1 (apply + %2))
[]
(partition-by identity a)))
(sum-consecutives [1,4,4,4,0,4,3,3,1]) ;; [1 12 0 4 6 1]
Alternatively, if you really wanted to use into, you could wrap your call to apply + in a vector literal like so:
(defn sum-consecutives [a]
(reduce
#(into %1 [(apply + %2)])
[]
(partition-by identity a)))
Your approach is sound in starting with partition-by. But let's
walk through the steps to sum each subsequence that it produces.
(let [xs [1 4 4 4 0 4 3 3 1]]
(partition-by identity xs)) ;=> ((1) (4 4 4) (0) (4) (3 3) (1))
To get a sum, you can use reduce (though a simple apply
instead would also work
here); e.g.:
(reduce + [4 4 4]) ;=> 12
Now put it all together to reduce each subsequence from above with map:
(let [xs [1 4 4 4 0 4 3 3 1]]
(map #(reduce + %) (partition-by identity xs))) ;=> (1 12 0 4 6 1)
A few notes...
I'm using xs to represent your vector (as suggested by the
Clojure Style Guide).
The let is sometimes a convenient form for experimenting with some
data building up to eventual functions.
Commas are not needed and are usually distracting, except occasionally
with hash-maps.
So your final function based on all this could look something like:
(defn sum-consecutives [coll]
(map #(reduce + %) (partition-by identity coll)))
Im trying to understand how to make stateful transducers in core.async.
For example how would I make a transducer that counts the number of elements that have come throgh a channel? For example I want to input to be transfomed into a count that depends on the number of objects that have come before it.
From what I have read the way to go is to use volatile! to hold the state inside the transducer but im still not sure how to put all things together.
You need a stateful transducer returning a reducing function closed over a volatile! tracking the count.
(defn count-xf [rf]
(let [ctr (volatile! 0)]
(fn
([] (rf))
([result] (rf result))
([result _] ; we ignore the input as
(rf result (vswap! ctr inc)))))) ; we just pass on the count
This can be simplified using the core function completing
(defn count-xf [rf]
(let [ctr (volatile! 0)]
(completing
(fn [result _]
(rf result (vswap! ctr inc))))))
E. g. use it so
(let [ch (chan 1 count-xf)]
(onto-chan ch (repeat 10 true))
(<!! (clojure.core.async/into [] ch)))
;-> [1 2 3 4 5 6 7 8 9 10]
Alternatively, you could just use the map-indexed transducer but this would likely help you less to understand how transducers work. Also it requires a bit additional per-step overhead for this particular usecase.
(def count-xf (map-indexed (fn [i _] (inc i))))
Observe that its implementation diverges little from the implementation above.
Further reference: http://clojure.org/reference/transducers
I have the following code that works:
(def *primes*
(let [l "2 3 5 7 11 13 17 19 23 29 31"
f (fn [lst] (filter #(< 0 (count (str/trim %))) lst))
m (fn [lst] (map #(Integer/parseInt %) lst))]
(-> l
(str/partition #"[0-9]+")
f
m)))
If I change it to inline the filter (f) and map (m) functions to this:
(def *primes*
(let [l "2 3 5 7 11 13 17 19 23 29 31"]
(-> l
(str/partition #"[0-9]+")
(fn [lst] (filter #(< 0 (count (str/trim %))) lst))
(fn [lst] (map #(Integer/parseInt %) lst)))))
it doesn't compile anymore. The error is:
#<CompilerException java.lang.RuntimeException: java.lang.IllegalArgumentException: Don't know how to create ISeq from: clojure.lang.Symbol (NO_SOURCE_FILE:227)>
Can anyone explain this to me?
The problem that I'm trying to solve is that map and filter takes the collection as the last parameter, yet str/partition takes the collection as the first, so I'm trying to mix the two using -> but currying map and filter into functions that only take one (the first) parameter for the collection to go into.
You can mix -> and ->> to a certain degree.
(-> l
(str/partition #"[0-9]+")
(->> (filter #(< 0 (count (str/trim %)))))
(->> (map #(Integer/parseInt %))))
But usually having problems like this is a sign that you try to do too much in one form. This simple example could be easily fixed.
(->> (str/partition l #"[0-9]+")
(filter #(< 0 (count (str/trim %))))
(map #(Integer/parseInt %)))
You're using function declarations as function calls. the immediate (ugly) way to fix it is to replace (fn [..] ..) with ((fn [..] ...))
-> is a macro. It manipulates the code you give it, and then executes that code. What happens when you try to use anonymous functions inline like that, is the previous expressions get threaded in as the first argument to fn. That is not what you want. You want them threaded in as the first argument to the actual function.
To use ->, you'd have to declare the functions beforehand, as you did in your first example.