Suppose I have a very simple .clj file on disk with the following content:
(def a 2)
(def b 3)
(defn add-two [x y] (+ x y))
(println (add-two a b))
From the context of separate program, I would like to read the above program as a list of S-Expressions, '((def a 2) (def b 3) ... (add-two a b))).
I imagine that one way of doing this involves 1. Using slurp on (io/file file-name.clj) to produce a string containing the file's contents, 2. passing that string to a parser for Clojure code, and 3. injecting the sequence produced by the parser to a list (i.e., (into '() parsed-code)).
However, this approach seems sort of clumsy and error prone. Does anyone know of a more elegant and/or idiomatic way to read a Clojure file as a list of S-Expressions?
Update: Following up on feedback from the comments section, I've decided to try the approach I mentioned on an actual source file using aphyr's clj-antlr as follows:
=> (def file-as-string (slurp (clojure.java.io/file "src/tcl/core.clj")))
=> tcl.core=> (pprint (antlr/parser "src/grammars/Clojure.g4" file-as-string))
{:parser
{:local
#object[java.lang.ThreadLocal 0x5bfcab6 "java.lang.ThreadLocal#5bfcab6"],
:grammar
#object[org.antlr.v4.tool.Grammar 0x5b8cfcb9 "org.antlr.v4.tool.Grammar#5b8cfcb9"]},
:opts
"(ns tcl.core\n (:gen-class)\n (:require [clj-antlr.core :as antlr]))\n\n(def foo 42)\n\n(defn parse-program\n \"uses antlr grammar to \"\n [program]\n ((antlr/parser \"src/grammars/Clojure.g4\") program))\n\n\n(defn -main\n \"I don't do a whole lot ... yet.\"\n [& args]\n (println \"tlc is tcl\"))\n"}
nil
Does anyone know how to transform this output to a list of S-Expressions as originally intended? That is, how might one go about squeezing valid Clojure code/data from the result of parsing with clj-antlr?
(import '[java.io PushbackReader])
(require '[clojure.java.io :as io])
(require '[clojure.edn :as edn])
;; adapted from: http://stackoverflow.com/a/24922859/6264
(defn read-forms [file]
(let [rdr (-> file io/file io/reader PushbackReader.)
sentinel (Object.)]
(loop [forms []]
(let [form (edn/read {:eof sentinel} rdr)]
(if (= sentinel form)
forms
(recur (conj forms form)))))))
(comment
(spit "/tmp/example.clj"
"(def a 2)
(def b 3)
(defn add-two [x y] (+ x y))
(println (add-two a b))")
(read-forms "/tmp/example.clj")
;;=> [(def a 2) (def b 3) (defn add-two [x y] (+ x y)) (println (add-two a b))]
)
Do you need something like this?
(let [exprs (slurp "to_read.clj")]
;; adding braces to form a proper list
(-> (str "(" (str exprs")"))
;; read-string is potentially harmful, since it evals the string
;; there exist non-evaluating readers for clojure but I don't know
;; which one are good
(read-string)
(prn)))
Related
Consider a dataset like this:
(def data [{:url "http://www.url1.com" :type :a}
{:url "http://www.url2.com" :type :a}
{:url "http://www.url3.com" :type :a}
{:url "http://www.url4.com" :type :b}])
The contents of those URL's should be requested in parallel. Depending on the item's :type value those contents should be parsed by corresponding functions. The parsing functions return collections, which should be concatenated, once all the responses have arrived.
So let's assume that there are functions parse-a and parse-b, which both return a collection of strings when they are passed a string containing HTML content.
It looks like core.async could be a good tool for this. One could either have separate channels for each item ore one single channel. I'm not sure which way would be preferable here. With several channels one could use transducers for the postprocessing/parsing. There is also a special promise-chan which might be proper here.
Here is a code-sketch, I'm using a callback based HTTP kit function. Unfortunately, I could not find a generic solution inside the go block.
(defn f [data]
(let [chans (map (fn [{:keys [url type]}]
(let [c (promise-chan (map ({:a parse-a :b parse-b} type)))]
(http/get url {} #(put! c %))
c))
data)
result-c (promise-chan)]
(go (put! result-c (concat (<! (nth chans 0))
(<! (nth chans 1))
(<! (nth chans 2))
(<! (nth chans 3)))))
result-c))
The result can be read like so:
(go (prn (<! (f data))))
I'd say that promise-chan does more harm than good here. The problem is that most of core.async API (a/merge, a/reduce etc.) relies on fact that channels will close at some point, promise-chans in turn never close.
So, if sticking with core.async is crucial for you, the better solution will be not to use promise-chan, but ordinary channel instead, which will be closed after first put!:
...
(let [c (chan 1 (map ({:a parse-a :b parse-b} type)))]
(http/get url {} #(do (put! c %) (close! c)))
c)
...
At this point, you're working with closed channels and things become a bit simpler. To collect all values you could do something like this:
;; (go (put! result-c (concat (<! (nth chans 0))
;; (<! (nth chans 1))
;; (<! (nth chans 2))
;; (<! (nth chans 3)))))
;; instead of above, now you can do this:
(->> chans
async/merge
(async/reduce into []))
UPD (below are my personal opinions):
Seems, that using core.async channels as promises (either in form of promise-chan or channel that closes after single put!) is not the best approach. When things grow, it turns out that core.async API overall is (you may have noticed that) not that pleasant as it could be. Also there are several unsupported constructs, that may force you to write less idiomatic code than it could be. In addition, there is no built-in error handling (if error occurs within go-block, go-block will silently return nil) and to address this you'll need to come up with something of your own (reinvent the wheel). Therefore, if you need promises, I'd recommend to use specific library for that, for example manifold or promesa.
I wanted this functionality as well because I really like core.async but I also wanted to use it in certain places like traditional JavaScript promises. I came up with a solution using macros. In the code below, <? is the same thing as <! but it throws if there's an error. It behaves like Promise.all() in that it returns a vector of all the returned values from the channels if they all are successful; otherwise it will return the first error (since <? will cause it to throw that value).
(defmacro <<? [chans]
`(let [res# (atom [])]
(doseq [c# ~chans]
(swap! res# conj (serverless.core.async/<? c#)))
#res#))
If you'd like to see the full context of the function it's located on GitHub. It's heavily inspired from David Nolen's blog post.
Use pipeline-async in async.core to launch asynchronous operations like http/get concurrently while delivering the result in the same order as the input:
(let [result (chan)]
(pipeline-async
20 result
(fn [{:keys [url type]} ch]
(let [parse ({:a parse-a :b parse-b} type)
callback #(put! ch (parse %)(partial close! ch))]
(http/get url {} callback)))
(to-chan data))
result)
if anyone is still looking at this, adding on to the answer by #OlegTheCat:
You can use a separate channel for errors.
(:require [cljs.core.async :as async]
[cljs-http.client :as http])
(:require-macros [cljs.core.async.macros :refer [go]])
(go (as-> [(http/post <url1> <params1>)
(http/post <url2> <params2>)
...]
chans
(async/merge chans (count chans))
(async/reduce conj [] chans)
(async/<! chans)
(<callback> chans)))
Where in the clojurescript library can I access a function to compile snippets of clojure into js?
I need this to run in the clojure (not clojurescript) repl:
(->js '(fn [x y] (+ x y)))
=> "function(x,y){return x+y}"
Snippet compilation from Clojure REPL
(require '[cljs.analyzer.api :refer [analyze empty-env]])
(require '[cljs.compiler.api :refer [emit]])
(let [ast (analyze (empty-env) '(defn plus [a b] (+ a b)))]
(emit ast))
;; result
"cljs.user.plus = (function cljs$user$plus(a,b){\nreturn (a + b);\n});\n"
Snippet compilation from ClojureScript REPL:
(require '[cljs.js :refer [empty-state compile-str]])
(compile-str (empty-state) "(defn add [x y] (+ x y))" #(println (:value %)))
;; Output (manually formatted for easier reading)
cljs.user.add = (function cljs$user$add(x,y){
return (x + y);
});
compile-str takes a callback as the last argument. It will be called with a map either with a key :value containing result JS as a string or :error with the compilation error.
In both cases org.clojure/tools.reader is needed on your classpath.
there is a lightweight alternative: https://github.com/kriyative/clojurejs which creates the right output asked by the question.
Examples can be seen here: https://github.com/kriyative/clojurejs/wiki/Examples
I want to read file entries in a zip file into a sequence of strings if possible. Currently I'm doing something like this to print out directory names for example:
(defn entries [zipfile]
(lazy-seq
(if-let [entry (.getNextEntry zipfile)]
(cons entry (entries zipfile)))))
(defn with-each-entry [fileName f]
(with-open [z (ZipInputStream. (FileInputStream. fileName))]
(doseq [e (entries z)]
; (println (.getName e))
(f e)
(.closeEntry z))))
(with-each-entry "tmp/my.zip"
(fn [e] (if (.isDirectory e)
(println (.getName e)))))
However this will iterate through the entire zip file. How could I change this so I could take the first few entries say something like:
(take 10 (zip-entries "tmp/my.zip"
(fn [e] (if (.isDirectory e)
(println (.getName e)))))
This seems like a pretty natural fit for the new transducers in CLJ 1.7.
You just build up the transformations you want as a transducer using comp and the usual seq-transforming fns with no seq/collection argument. In your example cases,
(comp (map #(.getName %)) (take 10)) and
(comp (filter #(.isDirectory %)) (map #(-> % .getName println))).
This returns a function of multiple arities which you can use in a lot of ways. In this case you want to eagerly reduce it over the entries sequence (to ensure realization of the entries happens inside with-open), so you use transduce (example zip data made by zipping one of my clojure project folders):
(with-open [z (-> "training-day.zip" FileInputStream. ZipInputStream.)]
(let[transform (comp (map #(.getName %)) (take 10))]
(transduce transform conj (entries z))))
;;return value: [".gitignore" ".lein-failures" ".midje-grading-config.clj" ".nrepl-port" ".travis.yml" "project.clj" "README.md" "target/" "target/classes/" "target/repl-port"]
Here I'm transducing with base function conj which makes a vector of the names. If you instead want your transducer to perform side-effects and not return a value, you can do that with a base function like (constantly nil):
(with-open [z (-> "training-day.zip" FileInputStream. ZipInputStream.)]
(let[transform (comp (filter #(.isDirectory %)) (map #(-> % .getName println)))]
(transduce transform (constantly nil) (entries z))))
which gives output:
target/
target/classes/
target/stale/
test/
A potential downside with this is that you'll probably have to manually incorporate .closeEntry calls into each transducer you use here to prevent holding those resources, because you can't in the general case know when each transducer is done reading the entry.
I have written a function which takes a directory as input and returns a list of files.
(ns musicdb.filesystem)
(import '(java.io.File) '(java.net.url) '(java.io))
(use 'clojure.java.browse)
(require '[clojure.string :as str])
(defn getFiles
"get a list of all files"
[searchPath]
(def directory (clojure.java.io/file searchPath))
(def files (file-seq directory))
(def fonly (filter (fn [x]
(. x isFile)) files))
(def names [])
(doseq [x fonly]
(conj names (. x toString)) ;doesn't seem to work
(println (. x toString))) ;but this DOES print the file path
names)
The only thing that doesn't work here, is the conj call.
Here is my test
(ns musicdb.core-test
(:require [clojure.test :refer :all]
[musicdb.core :refer :all]
[musicdb.filesystem :refer :all]))
(deftest test_0
(testing "getFiles returns valid result"
(is (> (count (getFiles "/home/ls/books/books")) 1))
(doseq [i (take 5 (getFiles "/home/ls/books/books"))] (searchBook i))))
This test fails and shows that the return value of getFiles is empty.
names is an immutable vector. (conj names (. x toString)) creates a new vector but doesn't do anything with it. There are other problems with your code:
you don't want to use doseq. It's for side effects, such as printing things out. If you're creating a collection you usually don't need to iterate in clojure, or if you do you can use an immutable accumulator, loop and recur.
You don't want to use nested defs. You're defining globals, and what you want are function locals. Use let instead.
The clojure naming style is to use dashes instead of camel case (minor, just a convention).
You don't seem to be using your java.io importa in this code.
use in general is not a good idea, unless you restrict it to a few explicitly named functions with :only. This is to avoid confusion when looking at an unqualified name in your code, because you wouldn't know where it came from.
You want something like this:
(defn get-files [search-path]
(let [directory (clojure.java.io/file search-path)
files (file-seq directory)
fonly (filter #(.isFile %) files)]
(map #(.toString %) fonly)))
I want to send var-args of a function to a macro, still as var-args.
Here is my code:
(defmacro test-macro
[& args]
`(println (str "count=" ~(count args) "; args=" ~#args)))
(defn test-fn-calling-macro
[& args]
(test-macro args))
The output of (test-macro "a" "b" "c") is what I want: count=3; args=abc
The output of (test-fn-calling-macro "a" "b" "c") is : count=1; args=("a" "b" "c") because args is sent as a single argument to the macro. How can I expand this args in my function in order to call the macro with the 3 arguments?
I guess I'm just missing a simple core function but I'm not able to find it. Thanks
EDIT 2 - My "real" code, shown in EDIT section below is not a valid situation to use this technique.
As pointed out by #Brian, the macro xml-to-cass can be replaced with a function like this:
(defn xml-to-cass
[zipper table key attr & path]
(doseq [v (apply zf/xml-> zipper path)] (cass/set-attr! table key attr v)))
EDIT - the following section goes beyond my original question but any insight is welcome
The code above is just the most simple I could come with to pinpoint my problem. My real code deals with clj-cassandra and zip-filter. It may also look over-engineering but it's just a toy project and I'm trying to learn the language at the same time.
I want to parse some XML found on mlb.com and insert values found into a cassandra database. Here is my code and the thinking behind it.
Step 1 - Function which works fine but contains code duplication
(ns stats.importer
(:require
[clojure.xml :as xml]
[clojure.zip :as zip]
[clojure.contrib.zip-filter.xml :as zf]
[cassandra.client :as cass]))
(def root-url "http://gd2.mlb.com/components/game/mlb/year_2010/month_05/day_01/")
(def games-table (cass/mk-cf-spec "localhost" 9160 "mlb-stats" "games"))
(defn import-game-xml-1
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))]
(doseq [v (zf/xml-> zipper (zf/attr :type))] (cass/set-attr! games-table game-id :type v))
(doseq [v (zf/xml-> zipper (zf/attr :local_game_time))] (cass/set-attr! games-table game-id :local_game_time v))
(doseq [v (zf/xml-> zipper :team [(zf/attr= :type "home")] (zf/attr :name_full))] (cass/set-attr! games-table game-id :home_team v))))
The parameter to import-game-xml-1 can be for example "gid_2010_05_01_colmlb_sfnmlb_1/". I remove the "gid_" and the trailing slash to make it the key of the ColumnFamily games in my database.
I found that the 3 doseq were a lot of duplication (and there should be more than 3 in the final version). So code templating using a macro seemed appropriate here (correct me if I'm wrong).
Step 2 - Introducing a macro for code templating (still works)
(defmacro xml-to-cass
[zipper table key attr & path]
`(doseq [v# (zf/xml-> ~zipper ~#path)] (cass/set-attr! ~table ~key ~attr v#)))
(defn import-game-xml-2
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))]
(xml-to-cass zipper games-table game-id :type (zf/attr :type))
(xml-to-cass zipper games-table game-id :local_game_time (zf/attr :local_game_time))
(xml-to-cass zipper games-table game-id :home_team :team [(zf/attr= :type "home")] (zf/attr :name_full))))
I believe that's an improvement but I still see some duplication in always reusing the same 3 parameters in my calls to xml-to-cass. That's were I introduced an intermediate function to take care of those.
Step 3 - Adding a function to call the macro (the problem is here)
(defn import-game-xml-3
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))
save-game-attr (fn[key path] (xml-to-cass zipper games-table game-id key path))]
(save-game-attr :type (zf/attr :type)) ; works well because path has only one element
(save-game-attr :local_game_time (zf/attr :local_game_time))
(save-game-attr :home :team [(zf/attr= :type "home"] (zf/attr :name_full))))) ; FIXME this final line doesn't work
Here's a some simple code which may be illuminating.
Macros are about code generation. If you want that to happen at runtime, for some reason, then you have to build and evaluate the code at runtime. This can be a powerful technique.
(defmacro test-macro
[& args]
`(println (str "count=" ~(count args) "; args=" ~#args)))
(defn test-fn-calling-macro
[& args]
(test-macro args))
(defn test-fn-expanding-macro-at-runtime
[& args]
(eval (cons `test-macro args)))
(defmacro test-macro-expanding-macro-at-compile-time
[& args]
(cons `test-macro args))
;; using the splicing notation
(defmacro test-macro-expanding-macro-at-compile-time-2
[& args]
`(test-macro ~#args))
(defn test-fn-expanding-macro-at-runtime-2
[& args]
(eval `(test-macro ~#args)))
(test-macro "a" "b" "c") ;; count=3; args=abc nil
(test-fn-calling-macro "a" "b" "c") ;; count=1; args=("a" "b" "c") nil
(test-fn-expanding-macro-at-runtime "a" "b" "c") ; count=3; args=abc nil
(test-macro-expanding-macro-at-compile-time "a" "b" "c") ; count=3; args=abc nil
(test-macro-expanding-macro-at-compile-time-2 "a" "b" "c") ; count=3; args=abc nil
(test-fn-expanding-macro-at-runtime "a" "b" "c") ; count=3; args=abc nil
If contemplation of the above doesn't prove enlightening, might I suggest a couple of my own blog articles?
In this one I go through macros from scratch, and how clojure's work in particular:
http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-i-getting.html
And in this one I show why run-time code generation might be useful:
http://www.learningclojure.com/2010/09/clojure-faster-than-machine-code.html
The typical way to use a collection as individual arguments to a function is to use (apply function my-list-o-args)
(defn test-not-a-macro [& args]
(print args))
(defn calls-the-not-a-macro [& args]
(apply test-not-a-macro args))
though you wont be able to use apply because test-macro is a macro. to solve this problem you will need to wrap test macro in a function call so you can apply on it.
(defmacro test-macro [& args]
`(println ~#args))
(defn calls-test-macro [& args]
(eval (concat '(test-macro) (args)))) ;you almost never need eval.
(defn calls-calls-test-macro [& args]
(calls-test-macro args))
This is actually a really good example of one of the ways macros are hard to compose. (some would say they cant be composed cleanly, though i think thats an exageration)
Macros are not magic. They are a mechanism to convert code at compile-time to equivalent code; they are not used at run-time. The pain you are feeling is because you are trying to do something you should not be trying to do.
I don't know the library in question, but if cass/set-attr! is a function, I see no reason why the macro you defined has to be a macro; it could be a function instead. You can do what you want to do if you can rewrite your macro as a function instead.
Your requirements aren't clear. I don't see why a macro is necessary here for test-macro, unless you're trying to print the unevaluated forms supplied to your macro.
These functions provide your expected results, but that's because your sample data was self-evaluating.
(defn test-args
[& args]
(println (format "count=%d; args=%s"
(count args)
(apply str args))))
or
(defn test-args
[& args]
(print (format "count=%d; args=" (count args)))
(doseq [a args]
(pr a))
(newline))
You can imagine other variations to get to the same result.
Try calling that function with something that doesn't evaluate to itself, and note the result:
(test-args (+ 1 2) (+ 3 4))
Were you looking to see the arguments printed as "37" or "(+ 1 2)(+ 3 4)"?
If you were instead trying to learn about macros and their expansion in general, as opposed to solving this particular problem, please tune your question to probe further.