I'd like to analyze a file of foreign clojure code. I'm currently using clojure.tools.reader to read all the forms:
(require '[clojure.tools.reader :as reader])
(defn read-all-forms [f]
(let [rdr (indexing-push-back-reader (slurp f))
EOF (Object.)
opts {:eof EOF}]
(loop [ret []]
(let [form (reader/read opts rdr)]
(if (= EOF form)
ret
(recur (conj ret form)))))))
This generally works, except when it encounters a double-colon keyword that refers to an aliased ns. Example:
(ns foo
(:require [foo.bar :as bar]))
::bar/baz
Fails with:
ExceptionInfo Invalid token: ::bar/baz
Is there a way to use clojure.tools.reader to read the file and resolve keywords like this? Am I supposed to somehow keep track of the *alias-map* myself?
tools.reader uses clojure.tools.reader/*alias-map* if it's bound, otherwise it uses (ns-aliases *ns*) to resolve aliases. So if you have auto-resolved keywords in your file, you will need to use one of those approaches to allow auto-resolved aliases to be resolved.
Related
In clojure, can one idiomatically obtain a function's name inside of its body, hopefully accomplishing so without introducing a new wrapper for the function's definition? can one also access the function's name inside of the body of the function's :test attribute as well?
For motivation, this can be helpful for certain logging situations, as well as for keeping the body of :test oblivious to changes to the name of the function which it is supplied for.
A short elucidation of the closest that meta gets follows; there's no this notion to supply to meta, as far as I know, in clojure.
(defn a [] (:name (meta (var a))))
Obviously it is easy to accomplish with a wrapper macro.
Edit: luckily no one so far mentioned lambda combinators.
There are 2 ways to approach your question. However, I suspect that to fully automate what you want to do, you would need to define your own custom defn replacement/wrapper.
The first thing to realize is that all functions are anonymous. When we type:
(defn hello [] (println "hi"))
we are really typing:
(def hello (fn [] (println "hi"))
we are creating a symbol hello that points to an anonymous var which in turn points to an anonymous function. However, we can give the function an "internal name" like so:
(def hello (fn fn-hello [] (println "hi")))
So now we can access the function from the outside via hello or from the inside using either hello of fn-hello symbols (please don't ever use hello in both locations or you create a lot of confusion...even though it is legal).
I frequently use the fn-hello method in (otherwise) anonymous functions since any exceptions thrown will include the fn-hello symbol which makes tracking down the source of the problem much easier (the line number of the error is often missing from the stack trace). For example when using Instaparse we need a map of anonymous transform functions like:
{
:identifier fn-identifier
:string fn-string
:integer (fn fn-integer [arg] [:integer (java.lang.Integer. arg)])
:boolean (fn fn-boolean [arg] [:boolean (java.lang.Boolean. arg)])
:namespace (fn fn-namespace [arg] [:namespace arg])
:prefix (fn fn-prefix [arg] [:prefix arg])
:organization (fn fn-organization [arg] [:organization arg])
:contact (fn fn-contact [arg] [:contact arg])
:description (fn fn-description [arg] [:description arg])
:presence (fn fn-presence [arg] [:presence arg])
:revision (fn fn-revision [& args] (prepend :revision args))
:iso-date (fn fn-iso-date [& args] [:iso-date (str/join args)])
:reference (fn fn-reference [arg] [:reference arg])
:identity (fn fn-identity [& args] (prepend :identity args))
:typedef (fn fn-typedef [& args] (prepend :typedef args))
:container (fn fn-container [& args] (prepend :container args))
:rpc (fn fn-rpc [& args] (prepend :rpc args))
:input (fn fn-input [& args] (prepend :input args))
...<snip>...
}
and giving each function the "internal name" makes debugging much, much easier. Perhaps this would be unnecessary if Clojure had better error messages, but that is a longstanding (& so far unfullfilled) wish.
You can find more details here: https://clojure.org/reference/special_forms#fn
If you read closely, it claims that (defn foo [x] ...) expands into
(def foo (fn foo [x] ...))
although you may need to experiment to see if this has already solved the use-case you are seeking. It works either way as seen in this example where we explicitly avoid the inner fn-fact name:
(def fact (fn [x] ; fn-fact omitted here
(if (zero? x)
1
(* x (fact (dec x))))))
(fact 4) => 24
This version also works:
(def fact (fn fn-fact [x]
(if (zero? x)
1
(* x (fn-fact (dec x))))))
(fact 4) => 24
(fn-fact 4) => Unable to resolve symbol: fn-fact
So we see that the "internal name" fn-fact is hidden inside the function and is invisible from the outside.
A 2nd approach, if using a macro, is to use the &form global data to access the line number from the source code. In the Tupelo library this technique is used to improve error messages for the
(defmacro dotest [& body] ; #todo README & tests
(let [test-name-sym (symbol (str "test-line-" (:line (meta &form))))]
`(clojure.test/deftest ~test-name-sym ~#body)))
This convenience macro allows the use of unit tests like:
(dotest
(is (= 3 (inc 2))))
which evalutes to
(deftest test-line-123 ; assuming this is on line 123 in source file
(is (= 3 (inc 2))))
instead of manually typing
(deftest t-addition
(is (= 3 (inc 2))))
You can access (:line (meta &form)) and other information in any macro which can make your error messages and/or Exceptions much more informative to the poor reader trying to debug a problem.
Besides the above macro wrapper example, another (more involved) example of the same technique can be seen in the Plumatic Schema library, where they wrap clojure.core/defn with an extended version.
You may also wish to view this question for clarification on how Clojure uses the "anonymous" var as an intermediary between a symbol and a function: When to use a Var instead of a function?
Suppose I have a very simple .clj file on disk with the following content:
(def a 2)
(def b 3)
(defn add-two [x y] (+ x y))
(println (add-two a b))
From the context of separate program, I would like to read the above program as a list of S-Expressions, '((def a 2) (def b 3) ... (add-two a b))).
I imagine that one way of doing this involves 1. Using slurp on (io/file file-name.clj) to produce a string containing the file's contents, 2. passing that string to a parser for Clojure code, and 3. injecting the sequence produced by the parser to a list (i.e., (into '() parsed-code)).
However, this approach seems sort of clumsy and error prone. Does anyone know of a more elegant and/or idiomatic way to read a Clojure file as a list of S-Expressions?
Update: Following up on feedback from the comments section, I've decided to try the approach I mentioned on an actual source file using aphyr's clj-antlr as follows:
=> (def file-as-string (slurp (clojure.java.io/file "src/tcl/core.clj")))
=> tcl.core=> (pprint (antlr/parser "src/grammars/Clojure.g4" file-as-string))
{:parser
{:local
#object[java.lang.ThreadLocal 0x5bfcab6 "java.lang.ThreadLocal#5bfcab6"],
:grammar
#object[org.antlr.v4.tool.Grammar 0x5b8cfcb9 "org.antlr.v4.tool.Grammar#5b8cfcb9"]},
:opts
"(ns tcl.core\n (:gen-class)\n (:require [clj-antlr.core :as antlr]))\n\n(def foo 42)\n\n(defn parse-program\n \"uses antlr grammar to \"\n [program]\n ((antlr/parser \"src/grammars/Clojure.g4\") program))\n\n\n(defn -main\n \"I don't do a whole lot ... yet.\"\n [& args]\n (println \"tlc is tcl\"))\n"}
nil
Does anyone know how to transform this output to a list of S-Expressions as originally intended? That is, how might one go about squeezing valid Clojure code/data from the result of parsing with clj-antlr?
(import '[java.io PushbackReader])
(require '[clojure.java.io :as io])
(require '[clojure.edn :as edn])
;; adapted from: http://stackoverflow.com/a/24922859/6264
(defn read-forms [file]
(let [rdr (-> file io/file io/reader PushbackReader.)
sentinel (Object.)]
(loop [forms []]
(let [form (edn/read {:eof sentinel} rdr)]
(if (= sentinel form)
forms
(recur (conj forms form)))))))
(comment
(spit "/tmp/example.clj"
"(def a 2)
(def b 3)
(defn add-two [x y] (+ x y))
(println (add-two a b))")
(read-forms "/tmp/example.clj")
;;=> [(def a 2) (def b 3) (defn add-two [x y] (+ x y)) (println (add-two a b))]
)
Do you need something like this?
(let [exprs (slurp "to_read.clj")]
;; adding braces to form a proper list
(-> (str "(" (str exprs")"))
;; read-string is potentially harmful, since it evals the string
;; there exist non-evaluating readers for clojure but I don't know
;; which one are good
(read-string)
(prn)))
I have written a function which takes a directory as input and returns a list of files.
(ns musicdb.filesystem)
(import '(java.io.File) '(java.net.url) '(java.io))
(use 'clojure.java.browse)
(require '[clojure.string :as str])
(defn getFiles
"get a list of all files"
[searchPath]
(def directory (clojure.java.io/file searchPath))
(def files (file-seq directory))
(def fonly (filter (fn [x]
(. x isFile)) files))
(def names [])
(doseq [x fonly]
(conj names (. x toString)) ;doesn't seem to work
(println (. x toString))) ;but this DOES print the file path
names)
The only thing that doesn't work here, is the conj call.
Here is my test
(ns musicdb.core-test
(:require [clojure.test :refer :all]
[musicdb.core :refer :all]
[musicdb.filesystem :refer :all]))
(deftest test_0
(testing "getFiles returns valid result"
(is (> (count (getFiles "/home/ls/books/books")) 1))
(doseq [i (take 5 (getFiles "/home/ls/books/books"))] (searchBook i))))
This test fails and shows that the return value of getFiles is empty.
names is an immutable vector. (conj names (. x toString)) creates a new vector but doesn't do anything with it. There are other problems with your code:
you don't want to use doseq. It's for side effects, such as printing things out. If you're creating a collection you usually don't need to iterate in clojure, or if you do you can use an immutable accumulator, loop and recur.
You don't want to use nested defs. You're defining globals, and what you want are function locals. Use let instead.
The clojure naming style is to use dashes instead of camel case (minor, just a convention).
You don't seem to be using your java.io importa in this code.
use in general is not a good idea, unless you restrict it to a few explicitly named functions with :only. This is to avoid confusion when looking at an unqualified name in your code, because you wouldn't know where it came from.
You want something like this:
(defn get-files [search-path]
(let [directory (clojure.java.io/file search-path)
files (file-seq directory)
fonly (filter #(.isFile %) files)]
(map #(.toString %) fonly)))
EDIT: Turned out I was using require instead of :require in the namespace declaration. With :require, tools.namespace refreshes the logging namespace, and the problem goes away. I still find it curious, however, that the expression (eval `(var ~(symbol "A/func"))) does not work in the situation described below (that is, if B below is not refreshed).
Summary: I'm using tools.namespace. If I have namespaces A and B, and in B do (eval `(var ~(symbol "A/func"))), and (tools.namespace/refresh) and run the code, that works. But if I make a change to A, do (tools.namespace/refresh), so that only A refreshes, then running that expression gives the error: Cannot resolve var: A/func in this context, even though A/func exists. Why?
Longer version:
In my project, I have a logging module/namespace that uses robert-hooke (see below). I'm using tools.namespace to reload my code when I make changes.
The problem is the following: When I want to log (my logging currently just prints) something, I list the functions that I want to log in my logging namespace and do (t.n/refresh). That works. But if I make changes to the the namespaces that contain the functions that I want to log, and do (t.n/refresh) without making changes to the logging namespace, the logging no longer works (for the functions that have been refreshed). As soon as I make a change to logging, so that it too is refreshed by tools.namespace, it starts working again.
So, it's like the vars in namespaces that have been refreshed don't properly get their logging hooks. But I don't understand why.
Below is my logging namespace. I call add-logging-wrappers each time I run my program.
If I add (eval `(var ~(symbol "sv/register-damage"))) inside add-logging-wrappers, that's fine when logging has just been refreshed and the logging works. But those times the logging does not work, that expression causes the error Cannot resolve var: sv/register-damage in this context.
(ns game.logging
(require [robert.hooke :as rh]
[clojure.pprint :as pp]
[game.server.core :as sv]
[game.client.core :as cl]
[game.math :as math]
(game.common [core-functions :as ccfns]
[graphics :as gfx])
(game.server [pathfinding :as pf]))
(:use [game.utils]))
(defn log-println [name type object]
(println (str (current-thread-name) " // " name " " type ":\n"
(with-out-str
(pp/pprint object)))))
(defn print-output [name f & args]
(let [result (apply f args)]
(log-println name "output" result)
result))
(defn print-input [name f & args]
(log-println name "input" args)
(apply f args))
(defn print-call [name f & args]
(println (str (current-thread-name) "//" name))
(apply f args))
(defmacro make-name-var-list [fn-list]
`[~#(for [fn fn-list]
[(str fn) `(var ~fn)])])
(defmacro defloglist [name & fns]
`(def ~name (make-name-var-list [~#fns])))
(defn add-hooks [name-vars & wrappers]
(when (seq wrappers)
(doseq [[name var] name-vars]
(rh/add-hook var (partial (first wrappers) name)))
(recur name-vars (next wrappers))))
(defn get-ns-name-vars [ns-sym]
(-> (the-ns ns-sym) (#(.name %)) ns-interns))
(defn add-hooks-to-ns [ns-sym & wrappers]
(apply add-hooks (get-ns-name-vars ns-sym) wrappers))
(defloglist log-both
sv/distribute-exp ;; <--- things to log
sv/register-damage
sv/give-exp)
(defloglist log-input)
(defloglist log-output)
(defn add-logging-wrappers []
(dorun (->> (all-ns) (map #(.name %)) (mapcat ns-interns) (map second)
(map rh/clear-hooks)))
(add-hooks log-both print-output print-input)
(add-hooks log-input print-input)
(add-hooks log-output print-output))
I want to write a function that returns the names and docstrings of the public functions in my namespace, like so:
(ns familiar.core
(:require [clojure.repl :refer [doc]]
...))
;; various functions with docstrings here
(defn help
"Halp!"
[]
(let [fns (keys (ns-publics 'familiar.core))]
(for [f fns]
[f (with-out-str (doc f))])))
When I call (help) in the REPL, the docstrings don't come with the functions:
familiar.core=> (help)
([save-experiment ""] [load-experiment ""] [add-data ""] [help ""] ... )
But calling (with-out-str (doc add-data)) in the REPL works as I'd expect:
familiar.core=> (with-out-str (doc add-data))
"-------------------------\nfamiliar.core/add-data\n([& coll])\n
Adds instances of variables with values at active date.\n Examp
le:\n (add-data \"mice\" 6 \"cats\" 2 \"dogs\" 0)\n"
What's going on here?
doc is a macro, so it cannot evaluate the local f in the calling context. Instead, you are simply calling (doc f) ovr and over.
The easiest approach to solve this is to go around the doc macro, and look directly at the data that it uses to produce documentation: metadata on vars.
(defn help
"Halp!"
[x]
(for [[name var] (ns-publics 'familiar.core)]
[name (:doc (meta var))]))