I've read other people's questions about having stack overflow problems in Clojure, and the problem tend to be a lazy sequence being built up somewhere. That appears to be the problem here, but for the life of me I can't figure out where.
Here is the code and after the code is a bit of explanation:
(defn pare-all []
"writes to disk, return new counts map"
(loop [counts (counted-origlabels)
songindex 0]
(let [[o g] (orig-gen-pair songindex)]
(if (< songindex *song-count*) ;if we are not done processing list
(if-not (seq o) ;if there are no original labels
(do
(write-newlabels songindex g);then use the generated ones
(recur counts (inc songindex)))
(let [{labels :labels new-counts :countmap} (pare-keywords o g counts)] ;else pare the pairs
(write-newlabels songindex labels)
(recur new-counts (inc songindex))))
counts))))
There is a map stored in "counts" originally retrieved from the function "counted-origlabels". The map have string keys and integer values. It is 600 or so items long and the values are updated during the iteration but the length stays the same, I've verified this.
The "orig-gen-pair" function reads from a file and returns a short pair of sequences, 10 or so items each.
The "write-newlabels" function just rite the passed sequence to the disk and doesn't have any other side effect nor does it return a value.
"Pare-keywords" returns a short sequence and an updated version of the "counts" map.
I just don't see what lazy sequence could be causing the problem here!
Any tips would be very much appreciated!
----EDIT----
Hello all, I've updated my function to be (hopefully) a little more idiomatic Clojure. But my original problem still remains. First, here is the new code:
(defn process-song [counts songindex]
(let [[o g] (orig-gen-pair songindex)]
(if-not (seq o) ;;if no original labels
(do
(write-newlabels songindex g);then use the generated ones
counts)
(let [{labels :labels new-counts :countmap} (pare-keywords o g counts)] ;else pare the pairs
(write-newlabels songindex labels)
new-counts))))
(defn pare-all []
(reduce process-song (counted-origlabels) (range *song-count*)))
This still ends with java.lang.StackOverflowError (repl-1:331). The stack trace doesn't mean much to me other than it sure seems to indicate lazy sequence mayhem going on. Any more tips? Do I need to post the code to the functions that process-song calls? Thanks!
I cannot quite grasp what you are trying to do without a little more concrete sample data, but it's very evident you're trying to iterate over your data using recursion. You're making things way more painful on yourself than you need to.
If you can generate a function, let's call it do-the-thing, that operates correctly with a single entry in your map, then you can call (map do-the-thing (counted-origlabels)), and it will apply (do-the-thing) to each map entry in (counted-origlabels), passing a single map entry to do-the-thing as it's sole argument and returning a seq of the return values from do-the-thing.
You also look like you need indexes, this is easily solved as well. You can splice in the lazy sequence (range) as the second argument to do-the-thing, and then you'll have a series of indexes generated with each map entry; however maps in clojure are not sorted by default, so unless you are using a sorted map, this index value is relatively meaningless.
Trying to abstract away what you've writen so far, try something like:
(defn do-the-thing [entry index counts]
(let [[o g] (orig-gen-pair index)]
(if-not (seq o)
(write-newlabels index g)
(let [{labels :labels new-counts :countmap} (pare-keywords o g counts)]
(write-newlabels index labels)))))
(map do-the-thing (counted-origlabels) (range) (constantly (counted-origlabels)))
Related
First off, I am a student in week 5 of 12 at The Iron Yard studying Java backend engineering. The course is composed of roughly 60% Java, 25% JavaScript and 15% Clojure.
I have been given the following problem (outlined in the comment):
;; Given an ArrayList of words, return a HashMap> containing a keys for every
;; word's first letter. The value for the key will be an ArrayList of all
;; words in the list that start with that letter. An empty string has no first
;; letter so don't add a key for it.
(defn index-words [word-list]
(loop [word (first word-list)
index {}]
(if (contains? index (subs word 0 1))
(assoc index (subs word 0 1) (let [words (index (subs word 0 1))
word word]
(conj words word)))
(assoc index (subs word 0 1) (conj nil word)))
(if (empty? word-list)
index
(recur (rest word-list) index))))
I was able to get a similar problem working using zipmap but I am positive that I am missing something with this one. The code compiles but fails to run.
Specifically, I am failing to update my hashmap index in the false clause of the 'if'.
I have tested all of the components of this function in the REPL, and they work in isolation. but I am struggling to put them all together.
For your reference, here is the code that calls word-list.
(let [word-list ["aardvark" "apple" "zamboni" "phone"]]
(printf "index-words(%s) -> %s\n" word-list (index-words word-list)))
Rather than getting a working solution from the community, my hope is for a few pointers to get my brain moving in the right direction.
The function assoc does not modify index. You need to work with the new value that assoc returns. Same is true for conj: it does not modify the map you pass it.
I hope, this answer is of the nature you expected to get: just a pointer where your problem is.
BTW: If you can do with a PersistentList this becomes a one-liner when using reduce instead of loop and recur. An interesting function for you could be update-in.
Have fun with Clojure.
The group-by function does what you require.
You can use first as its discriminating function argument. It
returns the first character of a string, or nil if there isn't one:
(first word) is simpler than (subs word 0 1).
Use dissoc to remove the entry for key nil.
You seldom need to use explicit loops in clojure. Most common control patterns have been captured in functions like group-by. Such functions have function and possibly collection arguments. The commonest examples are map and reduce. The Clojure cheat sheet is a most useful guide to them.
I'd like to have a function/macro for checking a list to have truthy value eventually, and I hope the evaluation would be lazy. Here is my illustrative implementation without lazy evaluation:
(defn eventual [cols]
(or (first cols) (if-let [rs (rest cols)]
(eventual rs))
false))
Here is a trivial example to illustrate:
(if (eventual [false (+ 1 2) (* 10000 10000)])
true
false)
I feel that there must be an implication with lazy evaluation. Maybe I'm just blinded at the moment. Please help to help. Thanks
You can check if a sequence contains at least one truthy element with some function:
(some identity col)
If you pass it a lazy sequence as col it will evaluate its contents up to the first truthy element and won't realise the rest:
(let [col (take
10
(iterate
#(do (println "Generating new value from " %) (inc %))
0))]
(some #(> % 5) col))
produces:
Generating new value from 0
Generating new value from 1
Generating new value from 2
Generating new value from 3
Generating new value from 4
Generating new value from 5
true
As you can see, values 6..9 are not produces at all.
You also should double check that the col you pass to some is really lazy and not already realised, because it might confuse you.
Your eventual function is as lazy as it can be. It searches eagerly for the first truthy item then stops. But it has problems:
It fails to terminate on an empty collection. (rest ()) is (),
which is truthy. Use next instead of rest. (next ()) is nil,
which is falsy.
It is truly recursive. It will blow the stack on a long enough
search. Try (eventual (repeat false)). Since the recursion is
tail-recursion, you can fix this by using recur in its place.
While we are at it, it is idiomatic to return nil, not false,
upon running out of a collection. So drop the final false.
We end up with
(defn eventual [cols]
(or (first cols) (if-let [rs (next cols)]
(recur rs))))
I'm a little queasy about what happens if cols is empty. Code based upon the source for some is clearer:
(defn eventual [coll]
(when (seq coll)
(or (first coll) (recur next coll))))
But using (some identity col), as Piotrek suggests, is probably best.
Sorry for the bad title 'cause I don't know how to describe in 10 words. Here's the detail:
I'd like to loop a file in format like:
a:1 b:2...
I want to loop each line, collect all 'k:v' into a hash-map.
{ a 1, b 2...}
I initialize a hash-map in a 'let' form, then loop all lines with 'for' inside let form.
In each loop step, I use 'assoc' to update the original hash-map.
(let [myhash {}]
(for [line #{"A:1 B:2" "C:3 D:4"}
:let [pairs (clojure.string/split line #"\s")]]
(for [[k v] (map #(clojure.string/split %1 #":") pairs)]
(assoc myhash k (Float. v)))))
But in the end I got a lazy-seq of hash-map, like this:
{ {a 1, b 2...} {x 98 y 99 z 100 ...} }
I know how to 'merge' the result now, but still don't understand why 'for' inside 'let' return
a list of result.
What I'm confused is: does the 'myhash' in the inner 'for' refers to the 'myhash' declared in the 'let' form every time? If I do want a list of hash-map like the output, is this the idiomatic way in Clojure ?
Clojure "for" is a list comprehension, so it creates list. It is NOT a for loop.
Also, you seem to be trying to modify the myhash, but Clojure's datastructures are immutable.
The way I would approach the problem is to try to create a list of pair like (["a" 1] ["b" 2] ..) and the use the (into {} the-list-of-pairs)
If the file format is really as simple as you're describing, then something much more simple should suffice:
(apply hash-map (re-seq #"\w+" (slurp "your-file.txt")))
I think it's more readable if you use the ->> threading macro:
(->> "your-file.txt" slurp (re-seq #"\w+") (apply hash-map))
The slurp function reads an entire file into a string. The re-seq function will just return a sequence of all the words in your file (basically the same as splitting on spaces and colons in this case). Now you have a sequence of alternating key-value pairs, which is exactly what hash-map expects...
I know this doesn't really answer your question, but you did ask about more idiomatic solutions.
I think #dAni is right, and you're confused about some fundamental concepts of Clojure (e.g. the immutable collections). I'd recommend working through some of the exercises on 4Clojure as a fun way to get more familiar with the language. Each time you solve a problem, you can compare your own solution to others' solutions and see other (possibly more idomatic) ways to solve the problem.
Sorry, I didn't read your code very thorougly last night when I was posting my answer. I just realized you actually convert the values to Floats. Here are a few options.
1) partition the sequence of inputs into key/val pairs so that you can map over it. Since you now how a sequence of pairs, you can use into to add them all to a map.
(->> "kvs.txt" slurp (re-seq #"\w") (partition 2)
(map (fn [[k v]] [k (Float. v)])) (into {}))
2) Declare an auxiliary map-values function for maps and use that on the result:
(defn map-values [m f]
(into {} (for [[k v] m] [k (f v)])))
(->> "your-file.txt" slurp (re-seq #"\w+")
(apply hash-map) (map-values #(Float. %)))
3) If you don't mind having symbol keys instead of strings, you can safely use the Clojure reader to convert all your keys and values.
(->> "your-file.txt" slurp (re-seq #"\w+")
(map read-string) (apply hash-map))
Note that this is a safe use of read-string because our call to re-seq would filter out any hazardous input. However, this will give you longs instead of floats since numbers like 1 are long integers in Clojure
Does the myhash in the inner for refer to the myhash declared in the let form every time?
Yes.
The let binds myhash to {}, and it is never rebound. myhash is always {}.
assoc returns a modified map, but does not alter myhash.
So the code can be reduced to
(for [line ["A:1 B:2" "C:3 D:4"]
:let [pairs (clojure.string/split line #"\s")]]
(for [[k v] (map #(clojure.string/split %1 #":") pairs)]
(assoc {} k (Float. v))))
... which produces the same result:
(({"A" 1.0} {"B" 2.0}) ({"C" 3.0} {"D" 4.0}))
If I do want a list of hash-map like the output, is this the idiomatic way in Clojure?
No.
See #DaoWen's answer.
I see some examples that show we can get a nice head/tail destructuring of a sequence in clojure as follows:
(if-let [[x & xs] (seq coll)]
However I assume this won't work as desired for lazy sequences because this puts the values into a vector, which aren't lazy. I tried changing the vector form to a list form, and it gave me binding errors, quoted or not.
Without having binding like this, it seems that if I've got a lazy sequence where each element was a computationally-intensive equation of the previous element, I'd have to do that computation twice to get the head and tail as separate statements, right?
(let [head (first my-lazy-seq) ;; has to calculate the value of head.
tail (rest my-lazy-seq)] ;; also has to calculate the value of head to prepare the rest of the sequence.
Is there any way around this, or am I making an incorrect assumption somewhere?
user=> (let [[x & xs] (range)] [x (take 10 xs)])
[0 (1 2 3 4 5 6 7 8 9 10)]
xs is still a lazy seq, so you can use the destructuring with no problems. This will force the first element of xs, though. (Destructuring uses vector notation, but it doesn't necessarily use vectors under the covers.)
With respect to your second question: lazy seqs cache their results, so your second option would also work without extra recalculation. The head will only be calculated once.
The binding vector [x & xs] isn't actually constructing a vector at runtime. It's just the notation used for destructuring into head & tail.
So it works fine on infinite sequences:
(if-let [[x & xs] (range)]
(apply str x (take 9 xs)))
=> "0123456789"
The destructuring form is actually producing a lazy sequence in this case, which you can observe as follows:
(if-let [[x & xs :as my-seq] (range)]
(class my-seq))
=> clojure.lang.LazySeq
I'm trying to use Clojure's update-in function but I can't seem to understand why I need to pass in a function?
update-in takes a function, so you can update a value at a given position depending on the old value more concisely. For example instead of:
(assoc-in m [list of keys] (inc (get-in m [list of keys])))
you can write:
(update-in m [list of keys] inc)
Of course if the new value does not depend on the old value, assoc-in is sufficient and you don't need to use update-in.
This isn't a direct answer to your question, but one reason why a function like update-in could exist would be for efficiency—not just convenience—if it were able to update the value in the map "in-place". That is, rather than
seeking the key in the map,
finding the corresponding key-value tuple,
extracting the value,
computing a new value based on the current value,
seeking the key in the map,
finding the corresponding key-value tuple,
and overwriting the value in the tuple or replacing the tuple with a new one
one can instead imagine an algorithm that would omit the second search for the key:
seek the key in the map,
find the corresponding key-value tuple,
extract the value,
compute a new value based on the current value,
and overwrite the value in the tuple
Unfortunately, the current implementation of update-in does not do this "in-place" update. It uses get for the extraction and assoc for the replacement. Unless assoc is using some caching of the last looked up key and the corresponding key-value tuple, the call to assoc winds up having to seek the key again.
I think the short answer is that the function passed to update-in lets you update values in a single step, rather than 3 (lookup, calculate new value, set).
Coincidentally, just today I ran across this use of update-in in a Clojure presentation by Howard Lewis Ship:
(def in-str "this is this")
(reduce
(fn [m k] (update-in m [k] #(inc (or % 0))))
{}
(seq in-str))
==> {\space 2, \s 3, \i 3, \h 2, \t 2}
Each call to update-in takes a letter as a key, looks it up in the map, and if it's found there increments the letter count (else sets it to 1). The reduce drives the process by starting with an empty map {} and repeatedly applies the update-in with successive characters from the input string. The result is a map of letter frequencies. Slick.
Note 1: clojure.core/frequencies is similar but uses assoc! rather than update-in.
Note 2: You can replace #(inc (or % 0)) with (fnil inc 0). From here: fnil
A practical example you see here.
Type this snippet (in your REPL):
(def my-map {:useless-key "key"})
;;{:useless-key "key"}
(def my-map (update-in my-map [:yourkey] #(cons 1 %)))
;;{:yourkey (1), :useless-key "key"}
Note that :yourkey is new. So the value - of :yourkey - passed to the lambda is null. cons will put 1 as the single element of your list. Now do the following:
(def my-map (update-in my-map [:yourkey] #(cons 25 %)))
;;{:yourkey (25 1), :useless-key "key"}
And that is it, in the second part, the anonymous function takes the list - the value for :yourkey - as argument and just cons 25 to it.
Since our my-map is immutable, update-in will always return a new version of your map letting you do something with the old value of the given key.
Hope it helped!