I have a large text file I want to process in Clojure.
I need to process it 2 lines at a time.
I settled on using a for loop so I could pull 2 lines for each pass with the following binding (rdr is my reader):
[[line-a line-b] (partition 2 (line-seq rdr))]
(I would be interested in knowing other ways to get 2 lines for each loop iteration but that is not the point of my question).
When trying to get the loop to work (using a simpler binding for these tests), I am seeing the following behavior that I can't explain:
Why does
(with-open [rdr (reader "path/to/file")]
(for [line (line-seq rdr)]
line))
trigger a Stream closed exception
while
(with-open [rdr (reader "path/to/file")]
(doseq [line (line-seq rdr)]
(println line)))
works?
for is lazy and just returns the head of the sequence that will eventually read the data from the file. The file is already closed when the for's contents are printed by your repl. you can fix this pu wrapping the for in a doall
(with-open [rdr (reader "path/to/file")]
(doall (for [line (line-seq rdr)]
line)))
Though this unlazys the sequence.
here is a sample of a function out of my misc.clj that lazily closes the file at it's end:
(defn byte-seq [rdr]
"create a lazy seq of bytes in a file and close the file at the end"
(let [result (. rdr read)]
(if (= result -1)
(do (. rdr close) nil)
(lazy-seq (cons result (byte-seq rdr))))))
Related
I have an input csv file and need to generate an output file that has one line for each input line. Each input line could be of a specific type (say "old" or "new") that can be determined only by processing the input line.
In addition to generating the output file, we also want to print the summary of how many lines of each type were in the input file. My actual task involves generating different SQLs based on the input line type, but to keep the example code focussed, I have kept the processing in the function proc-line simple. The function func determines what type an input line is -- again, I have kept it simple by randomly generating a type. The actual logic is more involved.
I have the following code and it does the job. However, to retain a functional style for the task of generating the summary, I chose to return a keyword to signify the type of each line and created a lazy sequence of these for generating the final summary. In an imperative style, we would simply increment a count for each line type. Generating a potentially large collection just for summarizing seems inefficient. Another consequence of the way I have coded it is the repetition of the (.write writer ...) portion. Ideally, I would code that just once.
Any suggestions for eliminating the two problems I have identified (and others)?
(ns file-proc.core
(:gen-class)
(:require [clojure.data.csv :as csv]
[clojure.java.io :as io]))
(defn func [x]
(rand-nth [true false]))
(defn proc-line [line writer]
(if (func line)
(do (.write writer (str line "\n")) :new)
(do (.write writer (str (reverse line) "\n")) :old)))
(defn generate-report [from to]
(with-open
[reader (io/reader from)
writer (io/writer to)]
(->> (csv/read-csv reader)
(rest)
(map #(proc-line % writer))
(frequencies)
(doall))))
I'd try to separate data processing from side-effects like reading/writing files. Hopefully this would allow the IO operations to stay at opposite boundaries of the pipeline, and the "middle" processing logic is agnostic of where the input comes from and where the output is going.
(defn rand-bool [] (rand-nth [true false]))
(defn proc-line [line]
(if (rand-bool)
[line :new]
[(reverse line) :old]))
proc-line no longer takes a writer, it only cares about the line and it returns a vector/2-tuple of the processed line along with a keyword. It doesn't concern itself with string formatting either—we should let csv/write-csv do that. Now you could do something like this:
(defn process-lines [reader]
(->> (csv/read-csv reader)
(rest)
(map proc-line)))
(defn generate-report [from to]
(with-open [reader (io/reader from)
writer (io/writer to)]
(let [lines (process-lines reader)]
(csv/write-csv writer (map first lines))
(frequencies (map second lines)))))
This will work but it's going to realize/keep the entire input sequence in memory, which you don't want for large files. We need a way to keep this pipeline lazy/efficient, but we also have to produce two "streams" from one and in a single pass: the processed lines only to be sent to write-csv, and each line's metadata for calculating frequencies. One "easy" way to do this is to introduce some mutability to track the metadata frequencies as the lazy sequence is consumed by write-csv:
(defn generate-report [from to]
(with-open [reader (io/reader from)
writer (io/writer to)]
(let [freqs (atom {})]
(->> (csv/read-csv reader)
;; processing starts
(rest)
(map (fn [line]
(let [[row tag] (proc-line line)]
(swap! freqs update tag (fnil inc 0))
row)))
;; processing ends
(csv/write-csv writer))
#freqs)))
I removed the process-lines call to make the full pipeline more apparent. By the time write-csv has fully (and lazily) consumed its payload, freqs will be a map like {:old 23, :new 31} which will be the return value of generate-report. There's room for improvement/generalization, but I think this is a start.
As others have mentioned, separating writing and processing work would be ideal. Here's how I usually do this:
(defn product-type [p]
(rand-nth [:new :old]))
(defn row->product [row]
(let [p (zipmap [:id :name :price] row)]
(assoc p :type (product-type p))))
(defmulti to-csv :type)
(defmethod to-csv :new [product] ...)
(defmethod to-csv :old [product] ...)
(defn generate-report [from to]
(with-open [rdr (io/reader from)
wrtr (io/writer to)]
(->> (rest (csv/read-csv rdr))
(map row->product)
(map #(do (.write wrtr (to-csv %)) %))
(map :type)
(frequencies)
(doall))))
(The code might not work—didn't run it, sorry.)
Constructing a hash-map and using multimethods is optional, of course, but it's better to assign a product its type first. This way its data dictates what pipeline is doing, not proc-line.
To refactor the code we need the safety net of at least one characterization test for generate-report. Since that function does file I/O (we will make the code independent from I/O later), we will use this sample CSV file, f1.csv:
Year,Code
1997,A
2000,B
2010,C
1996,D
2001,E
We cannot yet write a test because function func uses a RNG, so we rewrite it to be deterministic by actually looking at the input. While there, we rename it to new?, which is more representative of the problem:
(defn new? [row]
(>= (Integer/parseInt (first row)) 2000))
where, for the sake of the exercise, we assume that a row is "new" if the Year column is >= 2000.
We can now write the test and see it pass (here for brevity we focus only on the frequency calculation, not on the output transformation):
(deftest characterization-as-posted
(is (= {:old 2, :new 3}
(generate-report "f1.csv" "f1.tmp"))))
And now to the refactoring. The main idea is to realize that we need an accumulator, replacing map with reduce and getting rid of frequencies and of doall. Also, we rename "line" with "row", since this is how a line is called in the CSV format:
(defn generate-report [from to] ; 1
(let [[old new _] ; 2
(with-open [reader (io/reader from) ; 3
writer (io/writer to)] ; 4
(->> (csv/read-csv reader) ; 5
(rest) ; 6
(reduce process-row [0 0 writer])))] ; 7
{:old old :new new})) ; 8
The new process-row (originally process-line) becomes:
(defn process-row [[old new writer] row]
(if (new? row)
(do (.write writer (str row "\n")) [old (inc new) writer])
(do (.write writer (str (reverse row) "\n")) [(inc old) new writer])))
Function process-row, as any function to be passed to reduce, has two arguments: first argument [old new writer] is a vector of two accumulators and of the I/O writer (the vector is destructured); second argument row is one element of the collection that is being reduced. It returns the new vector of accumulators, that at the end of the collection is destructured in line 2 of generate-report and used at line 8 to create a hashmap equivalent to the one previously returned by frequencies.
We can do one last refactoring: separate the file I/O from the business logic, so that we can write tests without the scaffolding of preparated input files, as follows.
Function process-row becomes:
(defn process-row [[old-cnt new-cnt writer] row]
(let [[out-row old new] (process-row-pure old-cnt new-cnt row)]
(do (.write writer out-row)
[old new writer])))
and the business logic can be done by the pure (and so easily testable) function:
(defn process-row-pure [old new row]
(if (new? row)
[(str row "\n") old (inc new)]
[(str (reverse row) "\n") (inc old) new]))
All this without mutating anything.
IMHO, I would separate the two different aspects: counting the frequencies and writing to a file:
(defn count-lines
([lines] (count-lines lines 0 0))
([lines count-old count-new]
(if-let [line (first lines)]
(if (func line)
(recur count-old (inc count-new) (rest lines))
(recur (inc count-old) count-new (rest lines)))
{:new count-new :old count-old})))
(defn generate-report [from to]
(with-open [reader (io/reader from)
writer (io/writer to)]
(let [lines (rest (csv/read-csv reader))
frequencies (count-lines lines)]
(doseq [line lines]
(.write writer (str line "\n"))))))
I'm trying to figure out what is wrong with my code here. Basically the idea behind it is that I am reading a very large file and at the end of each line in the file is a number. I want to count the number of lines that have the number at the end greater than 500.
What I have is this and on paper it should work, but something is going wrong and I keep returning nil.
(defn countlines [] (with-open [rdr (clojure.java.io/reader "myfile.txt")]
(doseq [line (line-seq rdr)]
(count (re-find #"(?!500)[56789]\d{2,}|\d{4,}$" line)))))
the reason is that you use doseq:
clojure.core/doseq
[seq-exprs & body]
Macro
Added in 1.0
Repeatedly executes body (presumably for side-effects) with
bindings and filtering as provided by "for". Does not retain
the head of the sequence. Returns nil.
you should probably rewrite it to something like (doall (for [line (line-seq rdr)] ...
but to fulfill your task you need to rewrite it (because your function would return a seq of counts of chars in matches:
user> (count (re-find #"\d+" "123k456"))
3
which is obviously not what you want
what you need to do is:
(count (filter #(re-find #"(?!500)[56789]\d{2,}|\d{4,}$" %)
(line-seq rdr)))
If I understand the question correctly, you should be doing something like this:
(defn countlines [] (with-open [rdr (clojure.java.io/reader "myfile.txt")]
(-> (line-seq rdr)
(filter #(re-find #"(?!500)[56789]\d{2,}|\d{4,}$" %))
(count))))
About Martin Lechner's answer, I think should use Thread last(->>) rather than Thread first(->). So it should be
(defn countlines [] (with-open [rdr (clojure.java.io/reader "myfile.txt")] (->> (line-seq rdr)
(filter #(re-find #"(?!500)[56789]\d{2,}|\d{4,}$" %))
(count))))
I want to read first n lines from a file using clojure. Here is my code:
(defn read-nth-line [file]
(with-open [rdr (reader file)]
(loop [line-number 0]
(when (< line-number 20)
(nth (line-seq rdr) line-number)
(recur (inc line-number))))))
but when I run
user=> (read-nth-line "test.txt")
IndexOutOfBoundsException clojure.lang.RT.nthFrom (RT.java:871)
I have no idea why I got such an error.
Your code produces an out-of-bounds error because you call line-seq multiple times on the same reader. If you want to get a number of lines from a reader, you should call line-seq only once, then take the desired number of lines from that sequence:
(require '[clojure.java.io :as io])
(defn lines [n filename]
(with-open [rdr (io/reader filename)]
(doall (take n (line-seq rdr)))))
Example:
(run! println (lines 20 "test.txt"))
If test.txt contains fewer than 20 lines, this will simply print all the lines in the file.
This would normally be trivial in other language, but I've found no such example in Clojure.
I can println an entire file using:
(with-open [rdr (io/reader "file")]
(doseq [line (line-seq rdr) :while (< count(line) 10)]
(println line)))
But how do I get it to stop at line 5?
Thanks.
You can try this:
(println
(with-open [rdr (clojure.java.io/reader "file")]
(let [ls (line-seq rdr)]
(doall (take 5 ls)))))
This will print first 5 lines of the specified file.
If you need skip some lines that does not satisfy the condition, you can add filter. The following code will print first five lines that the length is less than 10.
(println
(with-open [rdr (clojure.java.io/reader "file")]
(let [ls (line-seq rdr)]
(->> ls
(filter #(< (count %) 10))
(take 5)
(doall)))))
Since filter and take returns lazy sequence, it should be realized within the with-open form. Outside the with-open form, the sequence couldn't be realized and cause exception.
println function also make the sequence realized, you can modify the code like this:
(with-open [rdr (clojure.java.io/reader "data/base_exp.txt")]
(let [ls (line-seq rdr)]
(->> ls
(filter #(> (count %) 10))
(take 5)
(println))))
Simply use take to limit the amount of lines:
Replace
(doseq [line (line-seq rdr) ;; ...
with
(doseq [line (take 5 (line-seq rdr)) ;; ...
I'm have a following code:
(use 'clojure.java.io)
(defrecord Member [id name salary role])
(defrecord Role [id name])
(def member-records (ref ()))
(defn add-member [member]
(dosync (alter member-records conj member)))
;;Test-data -->
(def dev-r(->Role 1 "Developer"))
(def test-member1(->Member 1 "Kirill" 70000.00 dev-r))
;;Test-data <--
(defn save-data-2-file []
(with-open [wrtr (writer "C:/Platform/Work/test.cdf")]
(print-dup #member-records wrtr)))
(defn process-line [line]
(println line))
;;Test line content
;;#BTC.pcost.Member{:id 1, :name "Kirill", :salary 70000.0, :role #BTC.pcost.Role{:id 1, :name "Developer"}})
(defn load-data-from-file []
(with-open [rdr (reader "C:/Platform/Work/test.cdf")]
(doseq [line (line-seq rdr)]
(process-line line))))
I'm want to recreate records after reading file, but i can not understand how i can make it. Yes, i'm know that i can parse text and fill my structure by the elements of parsed line, but it's will be difficult, cause i'm have alot structs like "Member" and "Role". Can anyone to suggest me a way, that i can do?
You can use read-string, and slurp, to pull the records out of the file. read-string is limited to reading the first form of a string, but, from your sample, you are only storing a single form, as a list of records.
(defn load-data-from-file [file]
(read-string (slurp file)))
Lazy Reading
If you need more than the first form, or cannot read the entire stream into memory, you can use read directly, to make a lazy reader.
(defn lazy-read
([rdr] (let [eof (Object.)] (lazy-read rdr (read rdr false eof) eof)))
([rdr data eof]
(if (not= eof data)
(cons data (lazy-seq (lazy-read rdr (read rdr false eof) eof))))))
(defn load-all-data [file]
(with-open [rdr (java.io.PushbackReader. (reader file))]
(doall (lazy-read rdr))))
(load-all-data "C:/Platform/Work/test.cdf")
Security
Also, it is good to mention security when loading code with read-string or read. You should only use them with trusted sources, because, using #= or a Java constructor, the source can execute arbitrary code inside your application. For a longer explanation, take a look at the documentation for read.
Setting *read-eval* to false would prevent the issue, but it would also prevent the reconstruction of the records in your sample. To avoid the issue all together, you can use the clojure.edn/read and clojure.edn/read-string functions, with a whitelist of readers.
(defn edn-read [eof rdr]
(clojure.edn/read {:eof eof :readers {'BTC.pcost.Role map->Role
'BTC.pcost.Member map->Member}}
rdr))
(defn lazy-edn-read
([rdr] (let [eof (Object.)] (lazy-edn-read rdr (edn-read eof rdr) eof)))
([rdr data eof]
(if (not= eof data)
(cons data (lazy-seq (lazy-edn-read rdr (edn-read eof rdr) eof))))))
(defn load-all-data [file]
(with-open [rdr (java.io.PushbackReader. (reader file))]
(doall (take-while (complement nil?) (lazy-edn-read rdr)))))
(load-all-data "C:/Platform/Work/test.cdf")
You can use read.
This function will read one object from a file:
(defn load-data-from-file [filename]
(with-open [rdr (java.io.PushbackReader. (reader filename))]
(read rdr)))
Or this will read all objects from the file:
(defn load-all-data-from-file [filename]
(let [eof (Object.)]
(with-open [rdr (java.io.PushbackReader. (reader filename))]
(doall
(take-while #(not= % eof)
(repeatedly #(read rdr nil eof)))))))
Here's the API documentation for read.
This is a small variation that will read all objects from a string:
(defn load-all-data-from-string [string]
(let [eof (Object.)]
(with-open [rdr (-> string java.io.StringReader. java.io.PushbackReader.)]
(doall
(take-while #(not= % eof)
(repeatedly #(read rdr nil eof)))))))
This is, as far as I know, not possible to do using read-string. Instead we use read with a java.io.StringReader.