Obviously, I can go to the Clojure's Github page or look in my maven repository, this is not a question about literally getting the source code.
I want to programmatically get all the top level forms defined in a namespace. Something like
(get-top-level-forms-of 'clojure.zip)
=>
['(ns ^{:doc "Functional hierarchical zipper, with navigation, editing,
and enumeration. See Huet"
:author "Rich Hickey"}
clojure.zip
(:refer-clojure :exclude (replace remove next)))
'(defn zipper
"Creates a new zipper structure.
branch? is a fn that, given a node, returns true if can have
children, even if it currently doesn't.
children is a fn that, given a branch node, returns a seq of its
children.
make-node is a fn that, given an existing node and a seq of
children, returns a new branch node with the supplied children.
root is the root node."
{:added "1.0"}
[branch? children make-node root]
^{:zip/branch? branch? :zip/children children :zip/make-node make-node}
[root nil])
'(defn seq-zip
"Returns a zipper for nested sequences, given a root sequence"
{:added "1.0"}
[root]
(zipper seq?
identity
.....__ALL_THE_REST_OF_THE_FORMS_IN_clojure.zip_....]
Basically, just getting a ordered sequence of all the forms in a namespace, in the order they were defined. Is this possible?
This will extract top-level forms from the copy of the clojure.zip source bundled inside the Clojure jar:
(require '[clojure.java.io :as io])
(let [rdr (clojure.lang.LineNumberingPushbackReader.
(io/reader (io/resource "clojure/zip.clj")))
sentinel (Object.)]
(take-while #(not (identical? sentinel %))
(repeatedly #(read rdr false sentinel))))
;= ((ns clojure.zip (:refer-clojure :exclude (replace remove next))) ...)
That should do the job:
(keys (ns-publics 'clojure.zip))
Which returns:
(lefts down insert-left up next path children vector-zip append-child zipper branch? end? leftmost edit replace insert-right root insert-child prev seq-zip xml-zip make-node rights node right left remove rightmost)
Other ns-* functions can be found in the namespaces page.
Related
I'm writing a tree ( business process decision tree ) in clojure data structure .
(require clojure.zip :as z)
(z/vector-zip
[ :billed?
[:yes
[:check-bank-account]
[:check-cash] ]
[:send-out-email] ])
when code walk on the first node, it will read the keyword and perform certain actions, the result will be True or False, then I would like it walk into left ( True) or right (False) node .
When my code starts with root node, and call some function associated with :billed? it return a True, how could clojure walk into :yes node, or :send-out-email node ? I thought there is only z/down while, left or rightis only for siblings not for directions of children.
Thank you very much for your time and appreciate any thoughts
Zippers walk through a data structure while keeping track of the current position. To reach different nodes, you have to apply a sequence of moves to the same zipper. When the zipper is created your position is directly above the tree:
(z/node tree)
=> [:billed? [:yes [:check-bank-account] [:check-cash]] [:send-out-email]]
So you can descend into the tree with z/down, and use z/node to get the current node from the zipper's location:
(-> tree
z/down
z/node)
=> :billed?
If you're walking the tree from the top towards some node, you probably only need z/down and z/right, since descending into a child vector will place you at the leftmost child. It's easier to imagine this if you lay out the vector in a flat line and imagine z/right simply moving a cursor to the next element, and z/down moving the cursor to inside a vector.
(-> tree
z/down
z/right
z/node)
=> [:yes [:check-bank-account] [:check-cash]]
(-> tree
z/down
z/right
z/right
z/node)
=> [:send-out-email]
Here's an example of you could walk this tree by evaluating the keys against a map of facts:
(def tree
(z/vector-zip
[:billed?
[:wire-funds?
[:check-bank-account]
[:check-cash]]
[:send-out-email]]))
(defn facts->action [facts]
(loop [curr (z/down tree)]
(let [node (z/node curr)]
(if-let [fact (find facts node)]
(if (val fact)
(recur (-> curr z/right z/down)) ;; descend "left"
(recur (-> curr z/right z/right z/down))) ;; descend "right"
node))))
(facts->action {:billed? false})
=> :send-out-email
(facts->action {:billed? true :wire-funds? true})
=> :check-bank-account
(facts->action {:billed? true :wire-funds? false})
=> :check-cash
Chain multiple zipper movements.
Here's an example:
(require '[clojure.zip :as z])
(def zipper (z/vector-zip
[:billed?
[:yes
[:check-bank-account]
[:check-cash]]
[:send-out-email]]))
(println (-> zipper z/down z/right z/down z/node))
(println (-> zipper z/down z/right z/down z/right z/down z/node))
Consider the following clojurescript code where the specter, reagent and re-frame frameworks are used, an external React.js grid component is used as a view component.
In db.cls :
(def default-db
{:cats [{:id 0 :data {:text "ROOT" :test 17} :prev nil :par nil}
{:id 1 :data {:text "Objects" :test 27} :prev nil :par 0}
{:id 2 :data {:text "Version" :test 37} :prev nil :par 1}
{:id 3 :data {:text "X1" :test 47} :prev nil :par 2}]})
In subs.cls
(register-sub
:cats
(fn [db]
(reaction
(select [ALL :data] (t/tree-visitor (get #db :cats))))))
result from select:
[{:text "ROOT", :test 17}
{:text "Objects", :test 27}
{:text "Version", :test 37}
{:text "X1", :test 47}]
In views.cls
(defn categorymanager []
(let [cats (re-frame/subscribe [:cats])]
[:> Reactable.Table
{:data (clj->js #cats)}]))
The code above works as expected.
Instead of displaying the data with the react.js component I want to go through each of the maps in the :cats vector and display the :text items in html ul / li.
I started as follows:
(defn categorymanager2 []
(let [cats (re-frame/subscribe [:cats])]
[:div
[:ul
(for [category #cats]
;;--- How to continue here ?? ---
)
))
Expected output:
ROOT
Objects
Version
X1
How do I loop through a subscribed collection in re-frame and display the data as a list-item? ( = question for title ).
First, be clear why you use key...
Supplying a key for each item in a list is useful when that list is quite dynamic - when new list items are being regularly added and removed, especially if that list is long, and the items are being added/removed near the top of the list.
keys can deliver big performance gains, because they allow React to more efficiently redraw these changeable lists. Or, more accurately, it allows React to avoid redrawing items which have the same key as last time, and which haven't changed, and which have simply shuffled up or down.
Second, be clear what you should do if the list is quite static (it does not change all the time) OR if there is no unique value associated with each item...
Don't use :key at all. Instead, use into like this:
(defn categorymanager []
(let [cats (re-frame/subscribe [:cats])]
(fn []
[:div
(into [:ul] (map #(vector :li (:text %)) #cats))])))
Notice what has happened here. The list provided by the map is folded into the [:ul] vector. At the end of it, no list in sight. Just nested vectors.
You only get warnings about missing keys when you embed a list into hiccup. Above there is no embedded list, just vectors.
Third, if your list really is dynamic...
Add a unique key to each item (unique amoung siblings). In the example given, the :text itself is a good enough key (I assume it is unique):
(defn categorymanager []
(let [cats (re-frame/subscribe [:cats])]
(fn []
[:div
[:ul (map #(vector :li {:key (:text %)} (:text %)) #cats)]])))
That map will result in a list which is the 1st parameter to the [:ul]. When Reagent/React sees that list it will want to see keys on each item (remember lists are different to vectors in Reagent hiccup) and will print warnings to console were keys to be missing.
So we need to add a key to each item of the list. In the code above we aren't adding :key via metadata (although you can do it that way if you want), and instead we are supplying the key via the 1st parameter (of the [:li]), which normally also carries style data.
Finally - part 1 DO NOT use map-indexed as is suggested in another answer.
key should be a unique value associated with each item. Attaching some arb integer does nothing useful - well, it does get rid of the warnings in the console, but you should use the into technique above if that's all you want.
Finally - part 2 there is no difference between map and for in this context.
They both result in a list. If that list has keys then no warning. But if keys are missing, then lots of warnings. But how the list was created doesn't come into it.
So, this for version is pretty much the same as the map version. Some may prefer it:
(defn categorymanager []
(let [cats (re-frame/subscribe [:cats])]
(fn []
[:div
[:ul (for [i #cats] [:li {:key (:text i)} (:text i)])]])))
Which can also be written using metadata like this:
(defn categorymanager []
(let [cats (re-frame/subscribe [:cats])]
(fn []
[:div
[:ul (for [i #cats] ^{:key (:text i)}[:li (:text i)])]])))
Finally - part 3
mapv is a problem because of this issue:
https://github.com/Day8/re-frame/wiki/Using-%5Bsquare-brackets%5D-instead-of-%28parentheses%29#appendix-2
Edit: For a much more coherent and technically correct explanation of keys and map, see Mike Thompson's answer!
Here's how I would write it:
(defn categorymanager2 []
(let [cats (re-frame/subscribe [:cats])]
(fn []
[:div
[:ul
(map-indexed (fn [n cat] ;;; !!! See https://stackoverflow.com/a/37186230/500207 !!!
^{:key n}
[:li (:text cat)])
#cats)]])))
(defn main-panel []
[:div
[categorymanager2]])
A few points:
See the re-frame readme's Subscribe section, near the end, which says:
subscriptions can only be used in Form-2 components and the subscription must be in the outer setup function and not in the inner render function. So the following is wrong (compare to the correct version above)…
Therefore, your component was ‘wrong’ because it didn't wrap the renderer inside an inner function. The readme has all the details, but in short, not wrapping a component renderer that depends on a subscription inside an inner function is bad because this causes the component to rerender whenever db changes—not what you want! You want the component to only rerender when the subscription changes.
Edit: seriously, see Mike Thompson's answer. For whatever reason, I prefer using map to create a seq of Hiccup tags. You could use a for loop also, but the critical point is that each [:li] Hiccup vector needs a :key entry in its meta-data, which I add here by using the current category's index in the #cats vector. If you don't have a :key, React will complain in the Dev Console. Note that this key should somehow uniquely tie this element of #cats to this tag: if the cats subscription changes and gets shuffled around, the result might not be what you expect because I just used this very simple key. If you can guarantee that category names will be unique, you can just use the :test value, or the :test value, or something else. The point is, the key must be unique and must uniquely identify this element.
(N.B.: don't try and use mapv to make a vector of Hiccup tags—re-frame hates that. Must be a seq like what map produces.)
I also included an example main-panel to emphasize that
parent components don't need the subscriptions that their children component need, and that
you should call categorymanager2 component with square-brackets instead of as a function with parens (see Using [] instead of ()).
Here's an ul / li example:
(defn phone-component
[phone]
[:li
[:span (:name #phone)]
[:p (:snippet #phone)]])
(defn phones-component
[]
(let [phones (re-frame/subscribe [:phones])] ; subscribe to the phones value in our db
(fn []
[:ul (for [phone in #phones] ^{:key phone} [phone-component phone] #phones)])))
I grabbed that code from this reframe tutorial.
Also map is preferable to for when using Reagent. There is a technical reason for this, it is just that I don't know what it is.
I need to turn the following input into output by applying the following two rules:
remove all vectors that have "nope" as last item
remove each map that does not have at least one vector with "ds1" as last item
(def input
[{:simple1 [:from [:simple1 'ds1]]}
{:simple2 [:from-any [[:simple2 'nope] [:simple2 'ds1]]]}
{:walk1 [:from [:sub1 :sub2 'ds1]]}
{:unaffected [:from [:unaffected 'nope]]}
{:replaced-with-nil [:from [:the-original 'ds1]]}
{:concat1 [:concat [[:simple1 'ds1] [:simple2 'ds1]]]}
{:lookup-word [:lookup [:word 'word :word 'ds1]]}])
(def output
[{:simple1 [:from [:simple1 'ds1]]}
{:simple2 [:from-any [[:simple2 'ds1]]]}
{:walk1 [:from [:sub1 :sub2 'ds1]]}
{:replaced-with-nil [:from [:the-original 'ds1]]}
{:concat1 [:concat [[:simple1 'ds1] [:simple2 'ds1]]]}
{:lookup-word [:lookup [:word 'word :word 'ds1]]}])
I was wondering if performing this transformation is possible with zippers?
I'd recommend clojure.walk instead for this kind of general tree transformation. It can take a bit of fiddling to get the replacement functions right but it works nicely with any nesting of Clojure data structures, which AFAIK can be a bit more challenging in a zipper based approach.
We're looking to shrink our tree, so postwalk is my go-to here. It takes a function f and a tree root and goes through the tree, replacing each leaf value with (f leaf), then their parents and their parents etc. until finally replacing the root. (prewalk is similar but proceeds from root and down to leaves, so it's usually more natural when you're growing the tree by splitting branches.)
The strategy here is to somehow construct a function that prunes any branch which meets our removal criteria, but returns any other value unchanged.
(ns shrink-tree
(:require [clojure.walk :refer [postwalk]]))
(letfn[(rule-1 [node]
(and (vector? node)
(= 'nope (last node))))
(rule-2 [node]
(and
(map? node)
(not-any? #(and (vector? %) (= 'ds1 (last %)))
(tree-seq vector? seq (-> node vals first)))))
(remove-marked [node]
(if (coll? node)
(into (empty node) (remove (some-fn rule-1 rule-2) node))
node))]
(= output (postwalk remove-marked input)))
;; => true
Here the fns rule-1 and rule-2 try to turn your rules into predicates and remove-marked:
If a node is a collection, returns the same collection, less any members for which rule1 or rule2 return truthy when called with that member. To check for either one at the same time we combine the predicates with some-fn.
Otherwise returns the same node. This is how we keep values like 'ds1 or :from-any around.
You might also want to consider looking at specter. It supports these sorts of transformations by allowing you to select and transform arbitrarily complex structures.
I've got a recursive function building a list:
(defn- traverse-dir
"Traverses the (source) directory, preorder"
[src-dir dst-root dst-step ffc!]
(let [{:keys [options]} *parsed-args*
uname (:unified-name options)
[dirs files] (list-dir-groomed (fs/list-dir src-dir))
... recursive call of traverse-dir is the last expression of dir-handler
(doall (concat (map-indexed (dir-handler) dirs) (map-indexed (file-handler) files))))) ;; traverse-dir
The list, built by traverse-dir, is recursive, while I want a flat one:
flat-list (->> (flatten recursive-list) (partition 2) (map vec))
Is there a way of building the flat list in the first place? Short of using mutable lists, that is.
I don't quite understand your context with a dir-handler that is called with nothing and returns a function which expects indices and directories, list-dir-groomed and all of that, but I'd recommend a look at tree-seq:
(defn tree-seq
"Returns a lazy sequence of the nodes in a tree, via a depth-first walk.
branch? must be a fn of one arg that returns true if passed a node
that can have children (but may not). children must be a fn of one
arg that returns a sequence of the children. Will only be called on
nodes for which branch? returns true. Root is the root node of the
tree."
{:added "1.0"
:static true}
[branch? children root]
(let [walk (fn walk [node]
(lazy-seq
(cons node
(when (branch? node)
(mapcat walk (children node))))))]
(walk root)))
My go-to use here is
(tree-seq #(.isDirectory %) #(.listFiles %) (clojure.java.io/as-file file-name))
but your context might mean that doesn't work. You can change to different functions for getting child files if you need to sanitize those, or you can just use filter on the output. If that's no good, the same pattern of a local fn from nodes into pre-walks that handles children by recursively mapcatting itself over them seems pretty applicable.
I tried to create function to scrape and tags from HTML page, whose URL I provide to a function, and this works as it should. I get sequence of <h3> and <table> elements, when I try to use select function to extract only table or h3 tags from resulting sequence,
I get (), or if I try to map those tags I get (nil nil nil ...).
Could you please help me to resolve this issue, or explain me what am I doing wrong?
Here is the code:
(ns Test2
(:require [net.cgrand.enlive-html :as html])
(:require [clojure.string :as string]))
(defn get-page
"Gets the html page from passed url"
[url]
(html/html-resource (java.net.URL. url)))
(defn h3+table
"returns sequence of <h3> and <table> tags"
[url]
(html/select (get-page url)
{[:div#wrap :div#middle :div#content :div#prospekt :div#prospekt_container :h3]
[:div#wrap :div#middle :div#content :div#prospekt :div#prospekt_container :table]}
))
(def url "http://www.belex.rs/trgovanje/prospekt/VZAS/show")
This line gives me headache :
(html/select (h3+table url) [:table])
Could you please tell me what am I doing wrong?
Just to clarify my question: is it possible to use enlive's select function to extract only table tags from result of (h3+table url) ?
As #Julien pointed out, you will probably have to work with the deeply nested tree structure that you get from applying (html/select raw-html selectors) on the raw html. It seems like you try to apply html/select multiple times, but this doesn't work. html/select parses html into a clojure datastructure, so you can't apply it on that datastructure again.
I found that parsing the website was actually a little involved, but I thought that it might be a nice use case for multimethods, so I hacked something together, maybe this will get you started:
(The code is ugly here, you can also checkout this gist)
(ns tutorial.scrape1
(:require [net.cgrand.enlive-html :as html]))
(def *url* "http://www.belex.rs/trgovanje/prospekt/VZAS/show")
(defn get-page [url]
(html/html-resource (java.net.URL. url)))
(defn content->string [content]
(cond
(nil? content) ""
(string? content) content
(map? content) (content->string (:content content))
(coll? content) (apply str (map content->string content))
:else (str content)))
(derive clojure.lang.PersistentStructMap ::Map)
(derive clojure.lang.PersistentArrayMap ::Map)
(derive java.lang.String ::String)
(derive clojure.lang.ISeq ::Collection)
(derive clojure.lang.PersistentList ::Collection)
(derive clojure.lang.LazySeq ::Collection)
(defn tag-type [node]
(case (:tag node)
:tr ::CompoundNode
:table ::CompoundNode
:th ::TerminalNode
:td ::TerminalNode
:h3 ::TerminalNode
:tbody ::IgnoreNode
::IgnoreNode))
(defmulti parse-node
(fn [node]
(let [cls (class node)] [cls (if (isa? cls ::Map) (tag-type node) nil)])))
(defmethod parse-node [::Map ::TerminalNode] [node]
(content->string (:content node)))
(defmethod parse-node [::Map ::CompoundNode] [node]
(map parse-node (:content node)))
(defmethod parse-node [::Map ::IgnoreNode] [node]
(parse-node (:content node)))
(defmethod parse-node [::String nil] [node]
node)
(defmethod parse-node [::Collection nil] [node]
(map parse-node node))
(defn h3+table [url]
(let [ws-content (get-page url)
h3s+tables (html/select ws-content #{[:div#prospekt_container :h3]
[:div#prospekt_container :table]})]
(for [node h3s+tables] (parse-node node))))
A few words on what's going on:
content->string takes a data structure and collects its content into a string and returns that so you can apply this to content that may still contain nested subtags (like <br/>) that you want to ignore.
The derive statements establish an ad hoc hierarchy which we will later use in the multi-method parse-node. This is handy because we never quite know which data structures we're going to encounter and we could easily add more cases later on.
The tag-type function is actually a hack that mimics the hierarchy statements - AFAIK you can't create a hierarchy out of non-namespace qualified keywords, so I did it like this.
The multi-method parse-node dispatches on the class of the node and if the node is a map additionally on the tag-type.
Now all we have to do is define the appropriate methods: If we're at a terminal node we convert the contents to a string, otherwise we either recur on the content or map the parse-node function on the collection we're dealing with. The method for ::String is actually not even used, but I left it in for safety.
The h3+table function is pretty much what you had before, I simplified the selectors a bit and put them into a set, not sure if putting them into a map as you did works as intended.
Happy scraping!
Your question is hard to understand, but I think your last line should simply be
(h3+table url)
This will return a deeply nested data structure containing scraped HTML that you can then dive into with the usual Clojure sequence APIs. Good luck.