The docstrings for some Clojure functions (e.g. defrecord) are quite long. When running Clojure at in a terminal window, I'd like to be able to in effect send doc's output through a pager such as more (or less). If someone has written a pager function in Clojure, then I think I could use it along with something like:
(with-out-str (doc defrecord))
Or if there is a standard Java class that implements a pager, I can figure out how to send the output to that.
Alternatively, how can I send the output of doc to a shell command? This doesn't do the job:
(clojure.java.shell/sh "more" :in (with-out-str (doc defrecord))))
[This topic is difficult to search: "more", "less", and "doc" are obviously very common terms, and things like "java pager" bring up pages discussing ways to break text up into pages for formatting documents.]
You can use jline for this. If you call setPaginationEnabled, with true, and use the printColumns method, on your jline ConsoleReader, it will page.
But, if you are trying to do this at a standard Leiningen REPL, things get more complicated. The current version of Leiningen v2 uses REPL-y, which uses jline internally, but doesn't use printColumns, so the jline pagination is ignored.
You can, however, get the current Leiningen REPL height through REPL-y's ConsoleReader, in reply.reader.simple-jline/jline-state, and use it to partition the doc string.
(defmacro doc2 [x]
`(let [h# (-> #reply.reader.simple-jline/jline-state :reader (.. getTerminal getHeight) (- 4))
[s1# s2#] (split-at h# (-> ~x clojure.repl/doc with-out-str clojure.string/split-lines))]
(doseq [x# s1#] (println x#))
(doseq [i# (partition-all h# s2#)]
(println "\n<more>")
(read-line)
(doseq [x# i#] (println x#)))))
You would want to put this macro in your profiles.clj under the :repl profile.
{:user {:plugins [...]}
:repl {:repl-options {:init (defmacro doc2 [x] ...)}}}
This would put the doc2 macro in the user namespace when you load the repl.
echo "(doc defrecord)" |clj|more
Related
I am trying to print the documentation for all functions in a given namespace by invoking the following expression in a REPL:
(doseq
[f (dir-fn 'clojure.repl)]
(doc f))
However the invocation of this expression returns nil without printing the documentation to the REPL. I know this might have to do with doc being a macro, but I'm a Clojure novice and am not entirely sure how to understand the problem.
Why does this expression return nil without printing the documentation?
How can this expression be modified so that it prints the documentation for each function in a given namespace?
Thanks!
Update: Combined both provided answers:
(defn ns-docs [ns']
(doseq [[symbol var] (ns-interns ns')]
(newline)
(println symbol)
(print " ")
(println (:doc (meta var)))))
(ns-docs 'clojure.repl)
I would, instead, start here:
The Clojure CheatSheet
ClojureDocs.org
Clojure-Doc.org (similar name, but different)
The API & Reference sections at Clojure.org
Note that doc is in the namespace clojure.repl, which reflects its intended usage (by a human in a repl). Here is some code that will also iterate on a namespace & print doc strings (using a different technique):
(doseq [[fn-symbol fn-var] (ns-interns 'demo.core)]
(newline)
(println fn-symbol)
(println (:doc (meta fn-var))))
where demo.core is the namespace of interest.
Note that ns-interns gives you both a symbol and var like:
fn-symbol => <#clojure.lang.Symbol -main>
fn-var => <#clojure.lang.Var #'demo.core/-main>
The meta function has lots of other info you may want to use someday:
(meta fn-var) =>
<#clojure.lang.PersistentArrayMap
{ :arglists ([& args]),
:doc "The Main Man!",
:line 9, :column 1,
:file "demo/core.clj",
:name -main,
:ns #object[clojure.lang.Namespace 0x14c35a06 "demo.core"]}>
While this probably won't help you with answering your question, the problem of evaluating macro's comes up a lot when you are learning Clojure.
Macros are responsible for the evaluation of their arguments. In this case clojure.repl/doc will ignore the current lexical context and assume that the symbol f that you're giving it is the name of a function you want to see the documentation for. It does this because it's intended to be used at the REPL, and is assuming you wouldn't want to type quotes all the time.
As f doesn't exist, it prints nothing. Then doseq returns nil, since it exists to do something for side effects only - hence starting in do. In order to pass an argument to a macro that refuses to respect the lexical context like this, you need to write the code for each element in the list.
You can do this by hand, or by constructing the code as data, and passing it to eval to execute. You can do this in an imperative style, using doseq:
(doseq [f (ns-interns 'clojure.repl)]
(eval `(doc ~(symbol "clojure.repl" (str (first f))))))
or in a slightly more Clojurey way (which will allow you to see the code that it would execute by removing eval from the end and running it at the REPL):
(->> (ns-interns 'clojure.repl)
(map #(list 'clojure.repl/doc (symbol "clojure.repl" (str (first %)))))
(cons `do)
eval)
In both of these we use quote and syntax-quote to construct some code from the list of symbols reflected from the namespace, and pass it to eval to actually execute it. This page on Clojure's weird characters should point you in the right direction for understanding what's going on here.
This an example of why you shouldn't write macro's, unless you've got no other options. Macro's do not compose, and are often difficult to work with. For a more in depth discussion, Fogus's talk and Christophe Grand's talk are both good talks.
Why does this expression return nil without printing the documentation?
Because the doc macro is receiving the symbol f from your loop, instead of a function symbol directly.
How can this expression be modified so that it prints the documentation for each function in a given namespace?
(defn ns-docs [ns']
(let [metas (->> (ns-interns ns') (vals) (map meta) (sort-by :name))]
(for [m metas :when (:doc m)] ;; you could filter here if you want fns only
(select-keys m [:name :doc]))))
(ns-docs 'clojure.repl)
=>
({:name apropos,
:doc "Given a regular expression or stringable thing, return a seq of all
public definitions in all currently-loaded namespaces that match the
str-or-pattern."}
...
)
Then you can print those maps/strings if you want.
I want to write a function that returns the names and docstrings of the public functions in my namespace, like so:
(ns familiar.core
(:require [clojure.repl :refer [doc]]
...))
;; various functions with docstrings here
(defn help
"Halp!"
[]
(let [fns (keys (ns-publics 'familiar.core))]
(for [f fns]
[f (with-out-str (doc f))])))
When I call (help) in the REPL, the docstrings don't come with the functions:
familiar.core=> (help)
([save-experiment ""] [load-experiment ""] [add-data ""] [help ""] ... )
But calling (with-out-str (doc add-data)) in the REPL works as I'd expect:
familiar.core=> (with-out-str (doc add-data))
"-------------------------\nfamiliar.core/add-data\n([& coll])\n
Adds instances of variables with values at active date.\n Examp
le:\n (add-data \"mice\" 6 \"cats\" 2 \"dogs\" 0)\n"
What's going on here?
doc is a macro, so it cannot evaluate the local f in the calling context. Instead, you are simply calling (doc f) ovr and over.
The easiest approach to solve this is to go around the doc macro, and look directly at the data that it uses to produce documentation: metadata on vars.
(defn help
"Halp!"
[x]
(for [[name var] (ns-publics 'familiar.core)]
[name (:doc (meta var))]))
I posted before on a huge XML file - it's a 287GB XML with Wikipedia dump I want ot put into CSV file (revisions authors and timestamps). I managed to do that till some point. Before I got the StackOverflow Error, but now after solving the first problem I get: java.lang.OutOfMemoryError: Java heap space error.
My code (partly taken from Justin Kramer answer) looks like that:
(defn process-pages
[page]
(let [title (article-title page)
revisions (filter #(= :revision (:tag %)) (:content page))]
(for [revision revisions]
(let [user (revision-user revision)
time (revision-timestamp revision)]
(spit "files/data.csv"
(str "\"" time "\";\"" user "\";\"" title "\"\n" )
:append true)))))
(defn open-file
[file-name]
(let [rdr (BufferedReader. (FileReader. file-name))]
(->> (:content (data.xml/parse rdr :coalescing false))
(filter #(= :page (:tag %)))
(map process-pages))))
I don't show article-title, revision-user and revision-title functions, because they just simply take data from a specific place in the page or revision hash. Anyone could help me with this - I'm really new in Clojure and don't get the problem.
Just to be clear, (:content (data.xml/parse rdr :coalescing false)) IS lazy. Check its class or pull the first item (it will return instantly) if you're not convinced.
That said, a couple things to watch out for when processing large sequences: holding onto the head, and unrealized/nested laziness. I think your code suffers from the latter.
Here's what I recommend:
1) Add (dorun) to the end of the ->> chain of calls. This will force the sequence to be fully realized without holding onto the head.
2) Change for in process-page to doseq. You're spitting to a file, which is a side effect, and you don't want to do that lazily here.
As Arthur recommends, you may want to open an output file once and keep writing to it, rather than opening & writing (spit) for every Wikipedia entry.
UPDATE:
Here's a rewrite which attempts to separate concerns more clearly:
(defn filter-tag [tag xml]
(filter #(= tag (:tag %)) xml))
;; lazy
(defn revision-seq [xml]
(for [page (filter-tag :page (:content xml))
:let [title (article-title page)]
revision (filter-tag :revision (:content page))
:let [user (revision-user revision)
time (revision-timestamp revision)]]
[time user title]))
;; eager
(defn transform [in out]
(with-open [r (io/input-stream in)
w (io/writer out)]
(binding [*out* out]
(let [xml (data.xml/parse r :coalescing false)]
(doseq [[time user title] (revision-seq xml)]
(println (str "\"" time "\";\"" user "\";\"" title "\"\n")))))))
(transform "dump.xml" "data.csv")
I don't see anything here that would cause excessive memory use.
Unfortunately data.xml/parse is not lazy, it attempts to read the whole file into memory and then parse it.
Instead use the this (lazy) xml library which holds only the part it is currently working on in ram. You will then need to re-structure your code to write the output as it reads the input instead of gathering all the xml, then outputting it.
your line
(:content (data.xml/parse rdr :coalescing false)
will load all the xml into memory and then request the content key from it. which will blow the heap.
a rough outline of a lazy answer would look something like this:
(with-open [input (java.io.FileInputStream. "/tmp/foo.xml")
output (java.io.FileInputStream. "/tmp/foo.csv"]
(map #(write-to-file output %)
(filter is-the-tag-i-want? (parse input))))
Have patience, working with (> data ram) always takes time :)
I don't know about Clojure but in plain Java one could use a SAX event based parser like http://docs.oracle.com/javase/1.4.2/docs/api/org/xml/sax/XMLReader.html
that doesn't need to load the XML to RAM
[this may seem like my problem is with Compojure, but it isn't - it's with Clojure]
I've been pulling my hair out on this seemingly simple issue - but am getting nowhere.
I am playing with Compojure (a light web framework for Clojure) and I would just like to generate a web page showing showing my list of todos that are in a PostgreSQL database.
The code snippets are below (left out the database connection, query, etc - but that part isn't needed because specific issue is that the resulting HTML shows nothing between the <body> and </body> tags).
As a test, I tried hard-coding the string in the call to main-layout, like this:
(html (main-layout "Aki's Todos" "Haircut<br>Study Clojure<br>Answer a question on Stackoverfolw")) - and it works fine.
So the real issue is that I do not believe I know how to build up a string in Clojure. Not the idiomatic way, and not by calling out to Java's StringBuilder either - as I have attempted to do in the code below.
A virtual beer, and a big upvote to whoever can solve it! Many thanks!
=============================================================
;The master template (a very simple POC for now, but can expand on it later)
(defn main-layout
"This is one of the html layouts for the pages assets - just like a master page"
[title body]
(html
[:html
[:head
[:title title]
(include-js "todos.js")
(include-css "todos.css")]
[:body body]]))
(defn show-all-todos
"This function will generate the todos HTML table and call the layout function"
[]
(let [rs (select-all-todos)
sbHTML (new StringBuilder)]
(for [rec rs]
(.append sbHTML (str rec "<br><br>")))
(html (main-layout "Aki's Todos" (.toString sbHTML)))))
=============================================================
Again, the result is a web page but with nothing between the body tags. If I replace the code in the for loop with println statements, and direct the code to the repl - forgetting about the web page stuff (ie. the call to main-layout), the resultset gets printed - BUT - the issue is with building up the string.
Thanks again.
~Aki
for is lazy, and in your function it's never being evaluated. Change for to doseq.
user> (let [rs ["foo" "bar"]
sbHTML (new StringBuilder)]
(for [rec rs]
(.append sbHTML (str rec "<br><br>")))
(.toString sbHTML))
""
user> (let [rs ["foo" "bar"]
sbHTML (new StringBuilder)]
(doseq [rec rs]
(.append sbHTML (str rec "<br><br>")))
(.toString sbHTML))
"foo<br><br>bar<br><br>"
You could also use reduce and interpose, or clojure.string/join from clojure.string, or probably some other options.
user> (let [rs ["foo" "bar"]]
(reduce str (interpose "<br><br>" rs)))
"foo<br><br>bar"
user> (require 'clojure.string)
nil
user> (let [rs ["foo" "bar"]]
(clojure.string/join "<br><br>" rs))
"foo<br><br>bar"
You would like to use the re-gsub like this:
(require 'clojure.contrib.str-utils) ;;put in head for enabling us to use re-gsub later on
(clojure.contrib.str-utils/re-gsub #"\newline" "<br><br>" your-string-with-todos-separated-with-newlines)
This last line will result in the string you like. The require-part is, as you maybe already know, there to enable the compiler to reach the powerful clojure.contrib.str-utils library without importing it to your current namespace (which could potentially lead to unnescessary collisions when the program grows).
re- is for reg-exp, and lets you define a reg-exp of the form #"regexp", which to replace all instances that is hit by the regexp with the argument afterwards, applied to the third argument. The \newline is in this case clojures way of expressing newlines in regexps as well as strings and the character we are looking for.
What I think you really wanted to do is to make a nifty ordered or unordered list in html-format. These can be done with [hiccup-page-helpers][2] (if you don't have them you probably have a compojure from the time before it got splited up in compojure, hiccup and more, since you use the html-function).
If you want to use hiccup-page-helpers, use the command re-split from the clojure.contrib.str-utils mentioned above in this fashion:
(use 'hiccup.page-helpers) ;;watch out for namespace collisions, since all the functions in hiccup.page-helpers got into your current namespace.
(unordered-list (clojure.contrib.str-utils/re-split #"\newline" your-string-with-todos-separated-with-newlines))
which should render a neat
<ul>
<li>todo-item1</li>
<li>todo-item2</li>
</ul>
(and yes, there is an ordered-list command that works the same way!)
In the last line of clojure code above, all you todos gets into a (list "todo1" "todo2") which is immediately consumed by hiccup.page-helpers unordered-list function and is there converted to an html-ized list.
Good luck with compojure and friends!
I'm looking for the ability to have the REPL print the current definition of a function. Is there any way to do this?
For example, given:
(defn foo [] (if true "true"))
I'd like to say something like
(print-definition foo)
and get something along the lines of
(foo [] (if true "true"))
printed.
An alternative to source (which should be available via clojure.repl/source when starting a REPL, as of 1.2.0. If you're working with 1.1.0 or lower, source is in clojure.contrib.repl-utils.), for REPL use, instead of looking at functions defined in a .clj file:
(defmacro defsource
"Similar to clojure.core/defn, but saves the function's definition in the var's
:source meta-data."
{:arglists (:arglists (meta (var defn)))}
[fn-name & defn-stuff]
`(do (defn ~fn-name ~#defn-stuff)
(alter-meta! (var ~fn-name) assoc :source (quote ~&form))
(var ~fn-name)))
(defsource foo [a b] (+ a b))
(:source (meta #'foo))
;; => (defsource foo [a b] (+ a b))
A simple print-definition:
(defn print-definition [v]
(:source (meta v)))
(print-definition #'foo)
#' is just a reader macro, expanding from #'foo to (var foo):
(macroexpand '#'reduce)
;; => (var reduce)
You'll want to import the repl namespace, and use the source function from it:
(ns myns
(:use [clojure.repl :only (source)]))
(defn foo [] (if true "true"))
(source foo)
=> (foo [] (if true "true"))
nil
Though this wouldn't work in the REPL, only where the function is defined in a .clj file on the classpath. Which doesn't answer your question, then: you'd need to have a defn that stores, in the metadata of the fn it defines, the source of the function. Then you'd write a function that recalls that bit of metadata. That shouldn't be terribly difficult.
Clojure doesn't have a decompiler, so that means there's no way to get at the source of an arbitrary function unless it was a defn loaded from disk. However, you can use a neat hack called serializable-fn to create a function that has its source form stored in its metadata: http://github.com/Seajure/serializable-fn
The defsource answer is very similar to this, but this solution works with arbitrary fns, not just top-level defns. It also makes fns print prettily at the repl without a special printing function.
In clojure 1.2's REPL, the source function is immediately available. You can use it this way:
$ java -cp clojure.jar clojure.main
Clojure 1.2.0
user=> (source slurp)
(defn slurp
"Reads the file named by f using the encoding enc into a string
and returns it."
{:added "1.0"}
([f & opts]
(let [opts (normalize-slurp-opts opts)
sb (StringBuilder.)]
(with-open [#^java.io.Reader r (apply jio/reader f opts)]
(loop [c (.read r)]
(if (neg? c)
(str sb)
(do
(.append sb (char c))
(recur (.read r)))))))))
nil
user=>
A few other functions are also automatically imported into the REPL's user namespace from the clojure.repl library. See the API doc here.
However, as pointed out in other answers here, you can't use source as is to print back functions you have defined in the REPL.
I asked exactly this question on the Clojure mailing list recently and the answers included overriding parts of the REPL to stash the input (and output) away for future reference as well as an override of defn to store the source in metadata (which you could then easily retrieve in the REPL).
Read the thread on the Clojure mailing list