I want to add metadata to a byte array in Clojure. Since this is not allowed, one option I want to try is the simplest object wrapper that could work.
Here is the source code for with-meta.
That made me start looking at Clojure.lang.IObj. I haven't found what I want yet.
Here's how you can create a deftype that supports metadata.
(import '(java.io Writer))
(deftype Box [value _meta]
clojure.lang.IObj
(meta [_] _meta)
(withMeta [_ m] (Box. value m))
clojure.lang.IDeref
(deref [_] value)
Object
(toString [this]
(str (.getName (class this))
": "
(pr-str value))))
(defmethod print-method Box [o, ^Writer w]
(.write w "#<")
(.write w (.getName (class o)))
(.write w ": ")
(.write w (-> o deref pr-str))
(.write w ">"))
(defn box
([value] (box value nil))
([value meta] (Box. value meta)))
And here's some example usage:
user> (def boxed (box (->> (range 5)
(map byte)
(byte-array))
{:stuff :foo}))
#<Var#1acd39b: #<Box#c50aa1: #>>
user> #boxed
[0, 1, 2, 3, 4]
user> (meta boxed)
{:stuff :foo}
user> (meta (with-meta boxed {:stuff :bar}))
{:stuff :bar}
This is the simplest way I can think of to put metadata on a byte array (reify doesn't work with clojure.lang.IObj and records include more unrelated functionality).
Another option (and perhaps simpler depending on the context) would be to store the byte array in a map, with the metadata either right alongside it or as actual metadata.
After discussing with some people on #clojure IRC, I wrote a simple Java class, MetaBox, that implements clojure.lang.IObj. You can use this easily in Clojure with metabox/box and metabox/val as well as the usual metadata functions such as meta and with-meta.
; [metabox "0.1.0"]
(require '[metabox.core :refer (box val)])
(def critical-density (box 0.692 {:uncertainty 0.01}))
(val critical-density) ; 0.692
(meta critical-density) ; {:uncertainty 0.01}
You can find the source code and README over at clj-metabox.
UPDATE: Thanks to a few discussions and suggestion (see below), as of version 0.2.0, the API uses deref instead of val:
; [metabox "0.2.0"]
(require '[metabox.core :refer (box)])
(def critical-density (box 0.692 {:uncertainty 0.01}))
#critical-density ; 0.692
(meta critical-density) ; {:uncertainty 0.01}
Related
Suppose I have a very simple .clj file on disk with the following content:
(def a 2)
(def b 3)
(defn add-two [x y] (+ x y))
(println (add-two a b))
From the context of separate program, I would like to read the above program as a list of S-Expressions, '((def a 2) (def b 3) ... (add-two a b))).
I imagine that one way of doing this involves 1. Using slurp on (io/file file-name.clj) to produce a string containing the file's contents, 2. passing that string to a parser for Clojure code, and 3. injecting the sequence produced by the parser to a list (i.e., (into '() parsed-code)).
However, this approach seems sort of clumsy and error prone. Does anyone know of a more elegant and/or idiomatic way to read a Clojure file as a list of S-Expressions?
Update: Following up on feedback from the comments section, I've decided to try the approach I mentioned on an actual source file using aphyr's clj-antlr as follows:
=> (def file-as-string (slurp (clojure.java.io/file "src/tcl/core.clj")))
=> tcl.core=> (pprint (antlr/parser "src/grammars/Clojure.g4" file-as-string))
{:parser
{:local
#object[java.lang.ThreadLocal 0x5bfcab6 "java.lang.ThreadLocal#5bfcab6"],
:grammar
#object[org.antlr.v4.tool.Grammar 0x5b8cfcb9 "org.antlr.v4.tool.Grammar#5b8cfcb9"]},
:opts
"(ns tcl.core\n (:gen-class)\n (:require [clj-antlr.core :as antlr]))\n\n(def foo 42)\n\n(defn parse-program\n \"uses antlr grammar to \"\n [program]\n ((antlr/parser \"src/grammars/Clojure.g4\") program))\n\n\n(defn -main\n \"I don't do a whole lot ... yet.\"\n [& args]\n (println \"tlc is tcl\"))\n"}
nil
Does anyone know how to transform this output to a list of S-Expressions as originally intended? That is, how might one go about squeezing valid Clojure code/data from the result of parsing with clj-antlr?
(import '[java.io PushbackReader])
(require '[clojure.java.io :as io])
(require '[clojure.edn :as edn])
;; adapted from: http://stackoverflow.com/a/24922859/6264
(defn read-forms [file]
(let [rdr (-> file io/file io/reader PushbackReader.)
sentinel (Object.)]
(loop [forms []]
(let [form (edn/read {:eof sentinel} rdr)]
(if (= sentinel form)
forms
(recur (conj forms form)))))))
(comment
(spit "/tmp/example.clj"
"(def a 2)
(def b 3)
(defn add-two [x y] (+ x y))
(println (add-two a b))")
(read-forms "/tmp/example.clj")
;;=> [(def a 2) (def b 3) (defn add-two [x y] (+ x y)) (println (add-two a b))]
)
Do you need something like this?
(let [exprs (slurp "to_read.clj")]
;; adding braces to form a proper list
(-> (str "(" (str exprs")"))
;; read-string is potentially harmful, since it evals the string
;; there exist non-evaluating readers for clojure but I don't know
;; which one are good
(read-string)
(prn)))
Consider a dataset like this:
(def data [{:url "http://www.url1.com" :type :a}
{:url "http://www.url2.com" :type :a}
{:url "http://www.url3.com" :type :a}
{:url "http://www.url4.com" :type :b}])
The contents of those URL's should be requested in parallel. Depending on the item's :type value those contents should be parsed by corresponding functions. The parsing functions return collections, which should be concatenated, once all the responses have arrived.
So let's assume that there are functions parse-a and parse-b, which both return a collection of strings when they are passed a string containing HTML content.
It looks like core.async could be a good tool for this. One could either have separate channels for each item ore one single channel. I'm not sure which way would be preferable here. With several channels one could use transducers for the postprocessing/parsing. There is also a special promise-chan which might be proper here.
Here is a code-sketch, I'm using a callback based HTTP kit function. Unfortunately, I could not find a generic solution inside the go block.
(defn f [data]
(let [chans (map (fn [{:keys [url type]}]
(let [c (promise-chan (map ({:a parse-a :b parse-b} type)))]
(http/get url {} #(put! c %))
c))
data)
result-c (promise-chan)]
(go (put! result-c (concat (<! (nth chans 0))
(<! (nth chans 1))
(<! (nth chans 2))
(<! (nth chans 3)))))
result-c))
The result can be read like so:
(go (prn (<! (f data))))
I'd say that promise-chan does more harm than good here. The problem is that most of core.async API (a/merge, a/reduce etc.) relies on fact that channels will close at some point, promise-chans in turn never close.
So, if sticking with core.async is crucial for you, the better solution will be not to use promise-chan, but ordinary channel instead, which will be closed after first put!:
...
(let [c (chan 1 (map ({:a parse-a :b parse-b} type)))]
(http/get url {} #(do (put! c %) (close! c)))
c)
...
At this point, you're working with closed channels and things become a bit simpler. To collect all values you could do something like this:
;; (go (put! result-c (concat (<! (nth chans 0))
;; (<! (nth chans 1))
;; (<! (nth chans 2))
;; (<! (nth chans 3)))))
;; instead of above, now you can do this:
(->> chans
async/merge
(async/reduce into []))
UPD (below are my personal opinions):
Seems, that using core.async channels as promises (either in form of promise-chan or channel that closes after single put!) is not the best approach. When things grow, it turns out that core.async API overall is (you may have noticed that) not that pleasant as it could be. Also there are several unsupported constructs, that may force you to write less idiomatic code than it could be. In addition, there is no built-in error handling (if error occurs within go-block, go-block will silently return nil) and to address this you'll need to come up with something of your own (reinvent the wheel). Therefore, if you need promises, I'd recommend to use specific library for that, for example manifold or promesa.
I wanted this functionality as well because I really like core.async but I also wanted to use it in certain places like traditional JavaScript promises. I came up with a solution using macros. In the code below, <? is the same thing as <! but it throws if there's an error. It behaves like Promise.all() in that it returns a vector of all the returned values from the channels if they all are successful; otherwise it will return the first error (since <? will cause it to throw that value).
(defmacro <<? [chans]
`(let [res# (atom [])]
(doseq [c# ~chans]
(swap! res# conj (serverless.core.async/<? c#)))
#res#))
If you'd like to see the full context of the function it's located on GitHub. It's heavily inspired from David Nolen's blog post.
Use pipeline-async in async.core to launch asynchronous operations like http/get concurrently while delivering the result in the same order as the input:
(let [result (chan)]
(pipeline-async
20 result
(fn [{:keys [url type]} ch]
(let [parse ({:a parse-a :b parse-b} type)
callback #(put! ch (parse %)(partial close! ch))]
(http/get url {} callback)))
(to-chan data))
result)
if anyone is still looking at this, adding on to the answer by #OlegTheCat:
You can use a separate channel for errors.
(:require [cljs.core.async :as async]
[cljs-http.client :as http])
(:require-macros [cljs.core.async.macros :refer [go]])
(go (as-> [(http/post <url1> <params1>)
(http/post <url2> <params2>)
...]
chans
(async/merge chans (count chans))
(async/reduce conj [] chans)
(async/<! chans)
(<callback> chans)))
Leonardo Borges has put together a fantastic presentation on Monads in Clojure. In it he describes the reader monad in Clojure using the following code:
;; Reader Monad
(def reader-m
{:return (fn [a]
(fn [_] a))
:bind (fn [m k]
(fn [r]
((k (m r)) r)))})
(defn ask [] identity)
(defn asks [f]
(fn [env]
(f env)))
(defn connect-to-db []
(do-m reader-m
[db-uri (asks :db-uri)]
(prn (format "Connected to db at %s" db-uri))))
(defn connect-to-api []
(do-m reader-m
[api-key (asks :api-key)
env (ask)]
(prn (format "Connected to api with key %s" api-key))))
(defn run-app []
(do-m reader-m
[_ (connect-to-db)
_ (connect-to-api)]
(prn "Done.")))
((run-app) {:db-uri "user:passwd#host/dbname" :api-key "AF167"})
;; "Connected to db at user:passwd#host/dbname"
;; "Connected to api with key AF167"
;; "Done."
The benefit of this is that you're reading values from the environment in a purely functional way.
But this approach looks very similar to the partial function in Clojure. Consider the following code:
user=> (def hundred-times (partial * 100))
#'user/hundred-times
user=> (hundred-times 5)
500
user=> (hundred-times 4 5 6)
12000
My question is: What is the difference between the reader monad and a partial function in Clojure?
The reader monad is a set of rules we can apply to cleanly compose readers. You could use partial to make a reader, but it doesn't really give us a way to put them together.
For example, say you wanted a reader that doubled the value it read. You might use partial to define it:
(def doubler
(partial * 2))
You might also want a reader that added one to whatever value it read:
(def plus-oner
(partial + 1))
Now, suppose you wanted to combine these guys in a single reader that adds their results. You'll probably end up with something like this:
(defn super-reader
[env]
(let [x (doubler env)
y (plus-oner env)]
(+ x y)))
Notice that you have to explicitly forward the environment to those readers. Total bummer, right? Using the rules provided by the reader monad, we can get much cleaner composition:
(def super-reader
(do-m reader-m
[x doubler
y plus-oner]
(+ x y)))
You can use partial to "do" the reader monad. Turn let into a do-reader by doing syntactic transformation on let with partial application of the environment on the right-hand side.
(defmacro do-reader
[bindings & body]
(let [env (gensym 'env_)
partial-env (fn [f] (list `(partial ~f ~env)))
bindings* (mapv #(%1 %2) (cycle [identity partial-env]) bindings)]
`(fn [~env] (let ~bindings* ~#body))))
Then do-reader is to the reader monad as let is to the identity monad (relationship discussed here).
Indeed, since only the "do notation" application of the reader monad was used in Beyamor's answer to your reader monad in Clojure question, the same examples will work as is with m/domonad Reader replaced with do-reader as above.
But, for the sake of variety I'll modify the first example to be just a bit more Clojurish with the environment map and take advantage of the fact that keywords can act as functions.
(def sample-bindings {:count 3, :one 1, :b 2})
(def ask identity)
(def calc-is-count-correct?
(do-reader [binding-count :count
bindings ask]
(= binding-count (count bindings))))
(calc-is-count-correct? sample-bindings)
;=> true
Second example
(defn local [modify reader] (comp reader modify))
(def calc-content-len
(do-reader [content ask]
(count content)))
(def calc-modified-content-len
(local #(str "Prefix " %) calc-content-len))
(calc-content-len "12345")
;=> 5
(calc-modified-content-len "12345")
;=> 12
Note since we built on let, we still have destructing at our disposal. Silly example:
(def example1
(do-reader [a :foo
b :bar]
(+ a b)))
(example1 {:foo 2 :bar 40 :baz 800})
;=> 42
(def example2
(do-reader [[a b] (juxt :foo :bar)]
(+ a b)))
(example2 {:foo 2 :bar 40 :baz 800})
;=> 42
So, in Clojure, you can indeed get the functionality of the do notation of reader monad without introducing monads proper. Analagous to doing a ReaderT transform on the identity monad, we can do a syntactic transformation on let. As you surmised, one way to do so is with partial application of the environment.
Perhaps more Clojurish would be to define a reader-> and reader->> to syntactically insert the environment as the second and last argument respectively. I'll leave those as an exercise for the reader for now.
One take-away from this is that while types and type-classes in Haskell have a lot of benefits and the monad structure is a useful idea, not having the constraints of the type system in Clojure allows us to treat data and programs in the same way and do arbitrary transformations to our programs to implement syntax and control as we see fit.
I have this code and would like to get the metadata transform
(defn truncate
[& {:keys [len]}]
(fn ^:transform [value]
(clojure.string/join (take len value))))
Ex: (meta (var (truncate)) //doesn't work
Something like this is possible? (meta (meta (var truncate))
UPDATE:
I moved it top the function name and solved it this way:
(defn- func-meta [func]
(let [[name-space func-name _] (clojure.string/split (str func) #"\$")]
(meta (ns-resolve (symbol name-space) (symbol func-name)))))
(func-meta (transform/truncate)) ;=> metadata
Attaching this type of metadata to arglists has no particular meaning in Clojure. (Type hints may be attached to arglists, but that's a different matter.) You can, however, attach metadata to the function itself using either of the following methods:
(defn foo []
^:foo (fn [] 1))
(defn foo []
(with-meta (fn [] 1) {:foo 1}))
;; in either case:
(meta (foo))
;= {:foo true}
Also, the var special form gives convenient access to Vars:
(var +)
;= #'clojure.core/+
The #' shorthand notation is used much more frequently.
I want to send var-args of a function to a macro, still as var-args.
Here is my code:
(defmacro test-macro
[& args]
`(println (str "count=" ~(count args) "; args=" ~#args)))
(defn test-fn-calling-macro
[& args]
(test-macro args))
The output of (test-macro "a" "b" "c") is what I want: count=3; args=abc
The output of (test-fn-calling-macro "a" "b" "c") is : count=1; args=("a" "b" "c") because args is sent as a single argument to the macro. How can I expand this args in my function in order to call the macro with the 3 arguments?
I guess I'm just missing a simple core function but I'm not able to find it. Thanks
EDIT 2 - My "real" code, shown in EDIT section below is not a valid situation to use this technique.
As pointed out by #Brian, the macro xml-to-cass can be replaced with a function like this:
(defn xml-to-cass
[zipper table key attr & path]
(doseq [v (apply zf/xml-> zipper path)] (cass/set-attr! table key attr v)))
EDIT - the following section goes beyond my original question but any insight is welcome
The code above is just the most simple I could come with to pinpoint my problem. My real code deals with clj-cassandra and zip-filter. It may also look over-engineering but it's just a toy project and I'm trying to learn the language at the same time.
I want to parse some XML found on mlb.com and insert values found into a cassandra database. Here is my code and the thinking behind it.
Step 1 - Function which works fine but contains code duplication
(ns stats.importer
(:require
[clojure.xml :as xml]
[clojure.zip :as zip]
[clojure.contrib.zip-filter.xml :as zf]
[cassandra.client :as cass]))
(def root-url "http://gd2.mlb.com/components/game/mlb/year_2010/month_05/day_01/")
(def games-table (cass/mk-cf-spec "localhost" 9160 "mlb-stats" "games"))
(defn import-game-xml-1
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))]
(doseq [v (zf/xml-> zipper (zf/attr :type))] (cass/set-attr! games-table game-id :type v))
(doseq [v (zf/xml-> zipper (zf/attr :local_game_time))] (cass/set-attr! games-table game-id :local_game_time v))
(doseq [v (zf/xml-> zipper :team [(zf/attr= :type "home")] (zf/attr :name_full))] (cass/set-attr! games-table game-id :home_team v))))
The parameter to import-game-xml-1 can be for example "gid_2010_05_01_colmlb_sfnmlb_1/". I remove the "gid_" and the trailing slash to make it the key of the ColumnFamily games in my database.
I found that the 3 doseq were a lot of duplication (and there should be more than 3 in the final version). So code templating using a macro seemed appropriate here (correct me if I'm wrong).
Step 2 - Introducing a macro for code templating (still works)
(defmacro xml-to-cass
[zipper table key attr & path]
`(doseq [v# (zf/xml-> ~zipper ~#path)] (cass/set-attr! ~table ~key ~attr v#)))
(defn import-game-xml-2
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))]
(xml-to-cass zipper games-table game-id :type (zf/attr :type))
(xml-to-cass zipper games-table game-id :local_game_time (zf/attr :local_game_time))
(xml-to-cass zipper games-table game-id :home_team :team [(zf/attr= :type "home")] (zf/attr :name_full))))
I believe that's an improvement but I still see some duplication in always reusing the same 3 parameters in my calls to xml-to-cass. That's were I introduced an intermediate function to take care of those.
Step 3 - Adding a function to call the macro (the problem is here)
(defn import-game-xml-3
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))
save-game-attr (fn[key path] (xml-to-cass zipper games-table game-id key path))]
(save-game-attr :type (zf/attr :type)) ; works well because path has only one element
(save-game-attr :local_game_time (zf/attr :local_game_time))
(save-game-attr :home :team [(zf/attr= :type "home"] (zf/attr :name_full))))) ; FIXME this final line doesn't work
Here's a some simple code which may be illuminating.
Macros are about code generation. If you want that to happen at runtime, for some reason, then you have to build and evaluate the code at runtime. This can be a powerful technique.
(defmacro test-macro
[& args]
`(println (str "count=" ~(count args) "; args=" ~#args)))
(defn test-fn-calling-macro
[& args]
(test-macro args))
(defn test-fn-expanding-macro-at-runtime
[& args]
(eval (cons `test-macro args)))
(defmacro test-macro-expanding-macro-at-compile-time
[& args]
(cons `test-macro args))
;; using the splicing notation
(defmacro test-macro-expanding-macro-at-compile-time-2
[& args]
`(test-macro ~#args))
(defn test-fn-expanding-macro-at-runtime-2
[& args]
(eval `(test-macro ~#args)))
(test-macro "a" "b" "c") ;; count=3; args=abc nil
(test-fn-calling-macro "a" "b" "c") ;; count=1; args=("a" "b" "c") nil
(test-fn-expanding-macro-at-runtime "a" "b" "c") ; count=3; args=abc nil
(test-macro-expanding-macro-at-compile-time "a" "b" "c") ; count=3; args=abc nil
(test-macro-expanding-macro-at-compile-time-2 "a" "b" "c") ; count=3; args=abc nil
(test-fn-expanding-macro-at-runtime "a" "b" "c") ; count=3; args=abc nil
If contemplation of the above doesn't prove enlightening, might I suggest a couple of my own blog articles?
In this one I go through macros from scratch, and how clojure's work in particular:
http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-i-getting.html
And in this one I show why run-time code generation might be useful:
http://www.learningclojure.com/2010/09/clojure-faster-than-machine-code.html
The typical way to use a collection as individual arguments to a function is to use (apply function my-list-o-args)
(defn test-not-a-macro [& args]
(print args))
(defn calls-the-not-a-macro [& args]
(apply test-not-a-macro args))
though you wont be able to use apply because test-macro is a macro. to solve this problem you will need to wrap test macro in a function call so you can apply on it.
(defmacro test-macro [& args]
`(println ~#args))
(defn calls-test-macro [& args]
(eval (concat '(test-macro) (args)))) ;you almost never need eval.
(defn calls-calls-test-macro [& args]
(calls-test-macro args))
This is actually a really good example of one of the ways macros are hard to compose. (some would say they cant be composed cleanly, though i think thats an exageration)
Macros are not magic. They are a mechanism to convert code at compile-time to equivalent code; they are not used at run-time. The pain you are feeling is because you are trying to do something you should not be trying to do.
I don't know the library in question, but if cass/set-attr! is a function, I see no reason why the macro you defined has to be a macro; it could be a function instead. You can do what you want to do if you can rewrite your macro as a function instead.
Your requirements aren't clear. I don't see why a macro is necessary here for test-macro, unless you're trying to print the unevaluated forms supplied to your macro.
These functions provide your expected results, but that's because your sample data was self-evaluating.
(defn test-args
[& args]
(println (format "count=%d; args=%s"
(count args)
(apply str args))))
or
(defn test-args
[& args]
(print (format "count=%d; args=" (count args)))
(doseq [a args]
(pr a))
(newline))
You can imagine other variations to get to the same result.
Try calling that function with something that doesn't evaluate to itself, and note the result:
(test-args (+ 1 2) (+ 3 4))
Were you looking to see the arguments printed as "37" or "(+ 1 2)(+ 3 4)"?
If you were instead trying to learn about macros and their expansion in general, as opposed to solving this particular problem, please tune your question to probe further.