I am fairly new to Clojure and I am struggling with how I can use a file path to create a tree in Clojure. I get all the files under a directory using file-seq and store them in files. The input is a file path like so:
resources/data/2012/05/02/low.xml
resources/data/2012/05/01/low.xml
I can get all the individual names of the folders and files using this:
(for [x files]
(if (.contains (.getPath x) ".json")
(for [y (str/split (.getPath x) #"\\")] y)))
This gives me lists of all the folders but then I don't know how I can combine them into 1 list to create a tree structure. If any answer could explain how their code works as well, to assist with learning. The desired output for these 2 inputs would be:
(resources (data (2012 (05 (02 (low.xml)) (01 (low.xml))))))
what you would need to build trees is something like this:
(defn as-tree [data]
(map (fn [[k vs]] (cons k (as-tree (keep next vs))))
(group-by first data)))
given a list of parsed paths (or in general any sequences), it would create your structure:
user> (as-tree [["resources" "data" "2012" "05" "02" "low.xml"]
["resources" "data" "2012" "05" "01" "aaa.xml"]
["resources" "data" "2012" "05" "02" "high.xml"]
["resources" "data" "2012" "05" "01" "xsxs.xml"]
["resources" "data" "2012" "06" "01" "bbb.xml"]
["resources" "data" "2012" "05" "01" "ccc.xml"]
["resources" "data" "2012" "02" "some.xml"]
["resources" "data" "2012" "01" "some2.xml"]
["other-resources" "data" "2015" "10" "some100.xml"]])
;; (("resources"
;; ("data"
;; ("2012"
;; ("05"
;; ("02" ("low.xml")
;; ("high.xml"))
;; ("01" ("aaa.xml")
;; ("xsxs.xml")
;; ("ccc.xml")))
;; ("06"
;; ("01" ("bbb.xml")))
;; ("02" ("some.xml"))
;; ("01" ("some2.xml")))))
;; ("other-resources" ("data" ("2015" ("10" ("some100.xml"))))))
so in your case it could look like this (tree for .clj files in project):
(require '[clojure.string :as cs])
(import 'java.io.File)
(->> (File. ".")
file-seq
(map #(.getPath %))
(filter #(cs/ends-with? % ".clj"))
(map #(cs/split % (re-pattern File/separator)))
as-tree
first)
;;=> ("."
;; ("src"
;; ("playground"
;; ("core.clj")))
;; ("test"
;; ("playground"
;; ("core_test.clj")))
;; ("project.clj"))
One way is to walk the directory in order as described here: https://docs.oracle.com/javase/tutorial/essential/io/walk.html
Another way is to accumulate the result after splitting the pathname into its component strings, then repeatedly using assoc-in:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test)
(:require [clojure.string :as str] ))
(defn accum-tree
"Accumulates a file path into a map tree"
[file-elem-tree path-str]
(let [path-elems (str/split path-str #"/")
key-seq (butlast path-elems)
file-name (last path-elems)]
(assoc-in file-elem-tree key-seq file-name)))
each call to accum-tree works like so:
path-elems => ["resources" "data" "2012" "05" "02" "low.xml"]
key-seq => ("resources" "data" "2012" "05" "02")
file-name => "low.xml"
where the unit test shows the final result.
(dotest
(let [file-strings ["resources/data/2012/05/02/low.xml"
"resources/data/2012/05/01/low.xml"]]
(is= (reduce accum-tree {} file-strings)
{"resources"
{"data"
{"2012"
{"05"
{"02" "low.xml",
"01" "low.xml"}}}}})))
With the given file/directory structure:
/tmp/root
├── file1.txt
├── file2.txt
├── sub
│ ├── file5.txt
│ └── file6.txt
└── sub1
├── emptysub
├── file3.txt
├── file4.txt
└── subsub
└── file99.txt
Here's a way to build up (from an empty tree) with a zipper, given those paths:
(def paths
(for [x (file-seq (io/file "/tmp/root"))]
(keep not-empty (str/split (.getPath x) #"/"))))
(defn level-loc [loc v] ;; find node with value at same depth as loc
(loop [l loc]
(when l
(let [n (z/node l)]
(cond
(= n v) l
(and (coll? n) (= (first n) v)) (-> l z/down)
:else (recur (-> l z/right)))))))
(defn graft-path [loc path]
(reduce
(fn [[_ path :as loc] p]
(or (level-loc loc p) ;; find existing node
(if (nil? path)
;; appends at top of tree
(-> loc
(z/append-child p)
z/down)
;; inserts at depth
(-> loc
(z/insert-right (list p))
z/right
z/down))))
loc
path))
(defn paths->tree [paths]
(z/root
(reduce
(comp z/seq-zip z/root graft-path)
(z/seq-zip '())
paths)))
Produces the following output:
(paths->tree paths)
=>
("tmp"
("root"
("sub" ("file6.txt") ("file5.txt"))
("sub1" ("emptysub") ("subsub" ("file99.txt")) ("file4.txt") ("file3.txt"))
("file1.txt")
("file2.txt")))
Related
I am collecting all user defined functions within a project. As a way of testing membership I use:
(defn get-var-namespace
[qualified-var]
{:pre [(var? qualified-var)]}
(-> qualified-var meta :ns))
(defn project-name
"returns a string representation of the root project name.
this is the same as the directory that the project is located in"
[]
(last (str/split (System/getProperty "user.dir") #"/")))
(defn get-project-namespaces
"return all namespaces defined within the project"
[]
(let [p-name (project-name)]
(filter (fn [x]
(let [n (first (str/split (str (ns-name x)) #"\."))]
(= n p-name))) (all-ns))))
(defn user-defined-var? [project-namespaces var]
(some #{(get-var-namespace var)} project-namespaces))
(defn namespace-deps
"returns a map whose keys are user defined functions and vals are sets of
user defined functions."
[project-namespaces ns]
(let [find-vars-in-expr
(fn [x] (let [nodes (ast/nodes x)
top-level (:var (first nodes))
non-recursive-deps (remove #{top-level}
(filter (partial var-has-user-ns project-namespaces)
(filter some? (map :var nodes))))]
{top-level (set non-recursive-deps)}))]
(dissoc (apply merge-with clojure.set/union
(map find-vars-in-expr (jvm/analyze-ns ns))) nil)))
as an example
clj-smart-test.core> (clojure.pprint/pprint (namespace-deps (get-project-namespaces) *ns*))
{#'clj-smart-test.core/foo
#{#'clj-smart-test.baz/baz1 #'clj-smart-test.core/bar
#'clj-smart-test.foo/foo #'clj-smart-test.core/k},
#'clj-smart-test.core/get-project-namespaces
#{#'clj-smart-test.core/project-name},
#'clj-smart-test.core/project-name #{},
#'clj-smart-test.core/var-has-user-ns
#{#'clj-smart-test.core/get-var-namespace},
#'clj-smart-test.core/bar #{},
#'clj-smart-test.core/f #{},
#'clj-smart-test.core/get-var-namespace #{},
#'clj-smart-test.core/namespace-deps
#{#'clj-smart-test.core/var-has-user-ns},
#'clj-smart-test.core/k #{}}
nil
Are there any corner cases that I need to be aware of? I know that a user could just create an arbitrary namespace in a file so I can't always assume that the directory that a project is created for example lein new my-project, which gives my-project as the root dir. This version does not catch defrecord etc.
I define a map of functions:
(ns fs
(:require [folder/a :as a]
[folder/b :as b]
[folder/c :as c])
(def functions {:a a/f :b b/f :c c/f})
(doseq [[_ f] functions] (f))
Now I want to add more namespaces within the folder, and I don't want to modify the above code. How can functions be dynamically populated with the f from each namespace in a folder.
First some helpers:
(defn directory
"Get directory from path"
[path]
(clojure.java.io/file path))
(defn file-names
"Get file names of the files in the directory."
[files]
(map (fn [file] (.getName file)) files))
(defn namespace
"Remove the extension from the file name and prefix with folder-name"
[folder-name file-name]
(->> (clojure.string/split file-name #"\.")
(butlast)
(apply str)
(str folder-name ".")
(symbol)))
Then retrieve all namespaces from your folder, you need the path and the folder name:
(def namespaces
(let [names (->> "/path/to/your/folder-name"
directory
file-seq ;; Gets the tree structure of the directory
rest ;; Get rid of the the directory name
file-names)]
(map (partial namespace "folder-name") names)))
Next get the public functions in every namespace via ns-publics:
(def functions
(->> namespaces
(map (juxt keyword (comp vals ns-publics)))
(into {})))
;; => prints {:namespace-key '(fn-a fn-b) :namespace-key2 '(fn-b fn-c)}
Note that this gets a list of all the public functions in a namesapce after the namespace keys.
We can execute the functions as follows:
(doseq [[_ ns-fns] functions
f ns-fns] (f))
Of course, this only works for functions that have an arity of zero. Otherwise you have to pass arguments to f.
I need to send a string that is a path of a directory to a function.how do I do that in Clojure?
I tried doing the following but it didn't work
(defn make-asm-file [d]
(doseq [f (.listFiles d)]
(if
( and (=(str(last (split (.getName f) #"\."))) "vm") (not (.isDirectory f)))
(translate f d))))
(make-asm-file "~\SimpleAdd")
How about the following,
(defn find-files
[regexp directory]
(filter #(and (.isFile %)
(re-find regexp (str %)))
(.listFiles (clojure.java.io/file directory))))
(doseq [f (find-files #".vm$" "~/SimpleAdd")]
(translate f))
In this case (java.io.File. d) would work instead of (clojure.java.io/file d) as well. You could also use file-seq instead of listfiles, but that would include all *.vm files in subdirs.
I am trying to iterate over a list of files in a given directory, and add an incrementing variable i = {1,2,3.....} to their names.
Here is the code I have for iterating through the files and changing each file's name:
(defn addCounterToExtIn [d]
(def i 0)
(doseq [f (.listFiles (file d)) ] ; make a sequence of all files in d
(if (and (not (.isDirectory f)) ; if file is not a directry and
(= '(\. \i \n) (take-last 3 (.getName f))) ) ; if it ends with .in
(fs/rename f (str d '/ i (.getName f)))))) ; add i to start of its name
I don't know how can I increment i as doseq iterates through each file. Alternatively, is there a better loop to use to achieve the desired result?
use file-seq and map-indexed:
(require '[clojure.java.io :as io])
(dorun
(->>
(file-seq (io/file "/home/eduard/Downloads"))
(filter #(re-find #".+\.pdf$" (.getName %)))
(map-indexed (fn [i v] [i v]))))
Change function in map-indexed to rename and you're done.
The sample output for pdf files:
([0 #<File /home/eduard/Downloads/some.pdf>] ...)
This is the first approach off the top of my head. It's not ideal, but certainly more idiomatic than what the question proposes.
(def rename-one-file! [file counter]
(if (and (not (.isDirectory file))
(= ".in" (str (take-last 3 (.getName file)))))
(fs/rename file (file (parent dir)
(str counter (.getName file)))))
(defn iterate-files-with-counter [fn dir]
(loop [counter 0
remaining-files (.listFiles (file dir))]
(let [current-file (first remaining-files)]
(fn file counter)
(recur (+ counter 1) (rest remaining-files))))
(def add-counter-to-ext-in-dir
(partial iterate-files-with-counter rename-one-file!))
Note that the work of actually performing the rename was split off from the work of iterating over the files. Having a large number of small functions is better than than a small number of large functions in general, and making those functions reusable / independent unless you choose to use them together is even better than that.
Let's say I need to create the following directory structure in Clojure:
a
\--b
| \--b1
| \--b2
\--c
\-c1
Instead of doing procedural things like the following:
(def a (File. "a"))
(.mkdir a)
(def b (File. a "b"))
(.mkdir b)
;; ...
... is there a clever way to somehow represent the above actions as data, declaratively, and then create the hierarchy in one fell swoop?
a quick and simple approach would be to make a vector of dirs to create and map mkdir on to it:
user> (map #(.mkdir (java.io.File. %)) ["a", "a/b" "a/b/c"])
(true true true)
or you can specify your dir structure as a tree and use zippers to walk it making the dirs on the way:
(def dirs ["a" ["b" ["b1" "b2"]] ["c" ["c1"]]])
(defn make-dir-tree [original]
(loop [loc (zip/vector-zip original)]
(if (zip/end? loc)
(zip/root loc)
(recur (zip/next
(do (if (not (vector? (zip/node loc)))
(let [path (apply str (interpose "/" (butlast (map first (zip/path loc)))))
name (zip/node loc)]
(if (empty? path)
(.mkdir (java.io.File. name))
(.mkdir (java.io.File. (str path "/" name))))))
loc))))))
(make-dir-tree dirs)
.
arthur#a:~/hello$ find a
a
a/c
a/c/c1
a/b
a/b/c
a/b/b2
a/b/b1
If you are doing a lot of general systems administration then something heavier may be in order. The pallet project is a library for doing system administration of all sorts on physical and cloud hosted systems (though it tends to lean towards the cloudy stuff). Specifically the directory
Another option if you want to easily handle creating recursive directories is to use .mkdirs
user> (require '[clojure.java.io :as io]')
user> (.mkdirs (io/file "a/b/c/d"))
You can use absolute path eg. /a/b/c/d or else it will be created relative to the path you initiated the repl from.
Also handy to check if given path is not an existing directory
user> (.isDirectory (io/file "a/b/c/d"))