I am attempting to learn Clojure and am having trouble with macros. I have a simple macro:
(defmacro add [& x] `(apply + (list ~#x)))
and then I have a function that can call multiple operations (add, subtract, multiply, and divide). I get these from stdin but for simplicity I have created a variable holding a list of numbers:
(defn do-op
[op]
(def numbers (list 1 2 3))
(apply op numbers))
In order to call do-op with the appropriate operation I need to pass the macro to do-op in an anonymous function:
(do-op #(add %&))
When I run this code using arguments passed from stdin I get a confusing error:
Cannot cast clojure.lang.ChunkedCons to java.lang.Number
When I execute this in a REPL I get:
Cannot cast clojure.lang.PersistentList to java.lang.Number
I assume that this has something to do with my lack of understanding of how these variadic arguments are getting handled but I am thoroughly stumped. What am I missing?
You can never ever pass a macro as if it was a function, since it doesn't exist after the code is compiled and ready to run.
When you have;
(do-op #(add %&)) ; which is shorthand for
(do-op (fn [& rest] (add rest))) ; then you expand the macro
(do-op (fn [& rest] (apply + (list (UNQUOTE-SPLICE 'rest)))) ; how do you unsplice the symbol rest to be separate elements?
The short answer is that rest can never be unspliced and thus you are using the macro wrong. However it seems you are just making an alias for + and it can be done by making a function or just making a binding to the original:
(defn add [& args] (apply + args)) ; making a function that calls + with its arguments
(def add +) ; or just make a binding pointing to the same function as + points to
(do-op add) ; ==> 6
You should keep to using functions as abstraction. Whenever you need to write something very verbose, where you cannot make it a function because of too early evaluation, then you have a candidate for a macro. It then translates a shorter way to write it with the long verbose way to write it without knowing the actual data types behind the variable symbols, since the compiler won't know what they are when expanding the macro. Also a macro gets expanded once, while the function that had the macro might get executed many times.
EDIT
In your pastebin code I can confirm you never ever need a macro:
(cond
(= 1 option) (do-op #(add %&))
...)
Should just be
(cond
(= 1 option) (do-op +)
...)
No need to wrap or make macro for this. Pass the function as is and it will do what you want.
Macros run at compile time and transform code data, not runtime data.
There is simply no way to generically apply to a macro a form evaluating to a sequence (like the symbol numbers) because it needs be to evaluated first, which can only happen at runtime (after compile time).
This is the full answer. Hopefully it helps you along understanding macros in Clojure.
If I understand what you are thinking, you should do it more like this:
(ns clj.core
(:require
[tupelo.core :as t]
))
(t/refer-tupelo)
(defn add [& x]
(apply + x))
(def numbers [1 2 3] )
(defn do-op
[op]
(apply op numbers))
(spyx (do-op add))
(defn do-op-2
[op vals]
(apply op vals))
(spyx (do-op-2 add numbers))
to get result:
> lein run
(do-op add) => 6
(do-op-2 add numbers) => 6
Usually we try to avoid having functions access global values like numbers. With do-op-2 we pass the vector of numbers into the function instead.
Note: For (spyx ...) to work you're project.clj must include
:dependencies [
[tupelo "0.9.9"]
Update
Of course, you don't really need do-op, you could do:
(defn add-all [nums-vec]
(apply + nums-vec))
(spyx (add-all numbers))
;=> (add-all numbers) => 6
Related
I find myself writing a lot of clojure in this manner:
(defn my-fun [input]
(let [result1 (some-complicated-procedure input)
result2 (some-other-procedure result1)]
(do-something-with-results result1 result2)))
This let statement seems very... imperative. Which I don't like. In principal, I could be writing the same function like this:
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
The problem with this is that it involves recomputation of some-complicated-procedure, which may be arbitrarily expensive. Also you can imagine that some-complicated-procedure is actually a series of nested function calls, and then I either have to write a whole new function, or risk that changes in the first invocation don't get applied to the second:
E.g. this works, but I have to have an extra shallow, top-level function that makes it hard to do a mental stack trace:
(defn some-complicated-procedure [input] (lots (of (nested (operations input)))))
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
E.g. this is dangerous because refactoring is hard:
(defn my-fun [input]
(do-something-with-results (lots (of (nested (operations (mistake input))))) ; oops made a change here that wasn't applied to the other nested calls
(some-other-procedure (lots (of (nested (operations input))))))))
Given these tradeoffs, I feel like I don't have any alternatives to writing long, imperative let statements, but when I do, I cant shake the feeling that I'm not writing idiomatic clojure. Is there a way I can address the computation and code cleanliness problems raised above and write idiomatic clojure? Are imperitive-ish let statements idiomatic?
The kind of let statements you describe might remind you of imperative code, but there is nothing imperative about them. Haskell has similar statements for binding names to values within bodies, too.
If your situation really needs a bigger hammer, there are some bigger hammers that you can either use or take for inspiration. The following two libraries offer some kind of binding form (akin to let) with a localized memoization of results, so as to perform only the necessary steps and reuse their results if needed again: Plumatic Plumbing, specifically the Graph part; and Zach Tellman's Manifold, whose let-flow form furthermore orchestrates asynchronous steps to wait for the necessary inputs to become available, and to run in parallel when possible. Even if you decide to maintain your present course, their docs make good reading, and the code of Manifold itself is educational.
I recently had this same question when I looked at this code I wrote
(let [user-symbols (map :symbol states)
duplicates (for [[id freq] (frequencies user-symbols) :when (> freq 1)] id)]
(do-something-with duplicates))
You'll note that map and for are lazy and will not be executed until do-something-with is executed. It's also possible that not all (or even not any) of the states will be mapped or the frequencies calculated. It depends on what do-something-with actually requests of the sequence returned by for. This is very much functional and idiomatic functional programming.
i guess the simplest approach to keep it functional would be to have a pass-through state to accumulate the intermediate results. something like this:
(defn with-state [res-key f state]
(assoc state res-key (f state)))
user> (with-state :res (comp inc :init) {:init 10})
;;=> {:init 10, :res 11}
so you can move on to something like this:
(->> {:init 100}
(with-state :inc'd (comp inc :init))
(with-state :inc-doubled (comp (partial * 2) :inc'd))
(with-state :inc-doubled-squared (comp #(* % %) :inc-doubled))
(with-state :summarized (fn [st] (apply + (vals st)))))
;;=> {:init 100,
;; :inc'd 101,
;; :inc-doubled 202,
;; :inc-doubled-squared 40804,
;; :summarized 41207}
The let form is a perfectly functional construct and can be seen as syntactic sugar for calls to anonymous functions. We can easily write a recursive macro to implement our own version of let:
(defmacro my-let [bindings body]
(if (empty? bindings)
body
`((fn [~(first bindings)]
(my-let ~(rest (rest bindings)) ~body))
~(second bindings))))
Here is an example of calling it:
(my-let [a 3
b (+ a 1)]
(* a b))
;; => 12
And here is a macroexpand-all called on the above expression, that reveal how we implement my-let using anonymous functions:
(clojure.walk/macroexpand-all '(my-let [a 3
b (+ a 1)]
(* a b)))
;; => ((fn* ([a] ((fn* ([b] (* a b))) (+ a 1)))) 3)
Note that the expansion doesn't rely on let and that the bound symbols become parameter names in the anonymous functions.
As others write, let is actually perfectly functional, but at times it can feel imperative. It's better to become fully comfortable with it.
You might, however, want to kick the tires of my little library tl;dr that lets you write code like for example
(compute
(+ a b c)
where
a (f b)
c (+ 100 b))
Is there a reasonable way to have multiple def statements happen with destructing the same way that let does it? For Example:
(let [[rtgs pcts] (->> (sort-by second row)
(apply map vector))]
.....)
What I want is something like:
(defs [rtgs pcts] (->> (sort-by second row)
(apply map vector)))
This comes up a lot in the REPL, notebooks and when debugging. Seriously feels like a missing feature so I'd like guidance on one of:
This exists already and I'm missing it
This is a bad idea because... (variable capture?, un-idiomatic?, Rich said so?)
It's just un-needed and I must be suffering from withdrawals from an evil language. (same as: don't mess up our language with your macros)
A super short experiment give me something like:
(defmacro def2 [[name1 name2] form]
`(let [[ret1# ret2#] ~form]
(do (def ~name1 ret1#)
(def ~name2 ret2#))))
And this works as in:
(def2 [three five] ((juxt dec inc) 4))
three ;; => 3
five ;; => 5
Of course and "industrial strength" version of that macro might be:
checking that number of names matches the number of inputs. (return from form)
recursive call to handle more names (can I do that in a macro like this?)
While I agree with Josh that you probably shouldn't have this running in production, I don't see any harm in having it as a convenience at the repl (in fact I think I'll copy this into my debug-repl kitchen-sink library).
I enjoy writing macros (although they're usually not needed) so I whipped up an implementation. It accepts any binding form, like in let.
(I wrote this specs-first, but if you're on clojure < 1.9.0-alpha17, you can just remove the spec stuff and it'll work the same.)
(ns macro-fun
(:require
[clojure.spec.alpha :as s]
[clojure.core.specs.alpha :as core-specs]))
(s/fdef syms-in-binding
:args (s/cat :b ::core-specs/binding-form)
:ret (s/coll-of simple-symbol? :kind vector?))
(defn syms-in-binding
"Returns a vector of all symbols in a binding form."
[b]
(letfn [(step [acc coll]
(reduce (fn [acc x]
(cond (coll? x) (step acc x)
(symbol? x) (conj acc x)
:else acc))
acc, coll))]
(if (symbol? b) [b] (step [] b))))
(s/fdef defs
:args (s/cat :binding ::core-specs/binding-form, :body any?))
(defmacro defs
"Like def, but can take a binding form instead of a symbol to
destructure the results of the body.
Doesn't support docstrings or other metadata."
[binding body]
`(let [~binding ~body]
~#(for [sym (syms-in-binding binding)]
`(def ~sym ~sym))))
;; Usage
(defs {:keys [foo bar]} {:foo 42 :bar 36})
foo ;=> 42
bar ;=> 36
(defs [a b [c d]] [1 2 [3 4]])
[a b c d] ;=> [1 2 3 4]
(defs baz 42)
baz ;=> 42
About your REPL-driven development comment:
I don't have any experience with Ipython, but I'll give a brief explanation of my REPL workflow and you can maybe comment about any comparisons/contrasts with Ipython.
I never use my repl like a terminal, inputting a command and waiting for a reply. My editor supports (emacs, but any clojure editor should do) putting the cursor at the end of any s-expression and sending that to the repl, "printing" the result after the cursor.
I usually have a comment block in the file where I start working, just typing whatever and evaluating it. Then, when I'm reasonably happy with a result, I pull it out of the "repl-area" and into the "real-code".
(ns stuff.core)
;; Real code is here.
;; I make sure that this part always basically works,
;; ie. doesn't blow up when I evaluate the whole file
(defn foo-fn [x]
,,,)
(comment
;; Random experiments.
;; I usually delete this when I'm done with a coding session,
;; but I copy some forms into tests.
;; Sometimes I leave it for posterity though,
;; if I think it explains something well.
(def some-data [,,,])
;; Trying out foo-fn, maybe copy this into a test when I'm done.
(foo-fn some-data)
;; Half-finished other stuff.
(defn bar-fn [x] ,,,)
(keys 42) ; I wonder what happens if...
)
You can see an example of this in the clojure core source code.
The number of defs that any piece of clojure will have will vary per project, but I'd say that in general, defs are not often the result of some computation, let alone the result of a computation that needs to be destructured. More often defs are the starting point for some later computation that will depend on this value.
Usually functions are better for computing a value; and if the computation is expensive, then you can memoize the function. If you feel you really need this functionality, then by all means, use your macro -- that's one of the sellings points of clojure, namely, extensibility! But in general, if you feel you need this construct, consider the possibility that you're relying too much on global state.
Just to give some real examples, I just referenced my main project at work, which is probably 2K-3K lines of clojure, in about 20 namespaces. We have about 20 defs, most of which are marked private and among them, none are actually computing anything. We have things like:
(def path-prefix "/some-path")
(def zk-conn (atom nil))
(def success? #{200})
(def compile* (clojure.core.memoize/ttl compiler {} ...)))
(def ^:private nashorn-factory (NashornScriptEngineFactory.))
(def ^:private read-json (comp json/read-str ... ))
Defining functions (using comp and memoize), enumerations, state via atom -- but no real computation.
So I'd say, based on your bullet points above, this falls somewhere between 2 and 3: it's definitely not a common use case that's needed (you're the first person I've ever heard who wants this, so it's uncommon to me anyway); and the reason it's uncommon is because of what I said above, i.e., it may be a code smell that indicates reliance on too much global state, and hence, would not be very idiomatic.
One litmus test I have for much of my code is: if I pull this function out of this namespace and paste it into another, does it still work? Removing dependencies on external vars allows for easier testing and more modular code. Sometimes we need it though, so see what your requirements are and proceed accordingly. Best of luck!
I'm trying to find a way to thread a value through a list of functions.
Firstly, I had a usual ring-based code:
(defn make-handler [routes]
(-> routes
(wrap-json-body)
(wrap-cors)
;; and so on
))
But this was not optimal as I wanted to write a test to check the routes are actually wrapped with wrap-cors. I decided to extract the wrappers into a def. So the code became as follows:
(def middleware
(list ('wrap-json-body)
('wrap-cors)
;; and so on
))
(defn make-handler [routes]
(-> routes middleware))
This apparently doesn't work and is not supposed to as the -> macro doesn't take a list as the second argument. So I tried to use the apply function to resolve that:
(defn make-handler [routes]
(apply -> routes middleware))
Which eventually bailed out with:
CompilerException java.lang.RuntimeException: Can't take value of a
macro: #'clojure.core/->
So the question arises: How does one pass a list of values to the -> macro (or, say, any other macro) as one would do with apply for a function?
This is an XY Problem.
The main point of -> is to make code easier to read. But if one writes a new macro solely in order to use -> (in code nobody will ever see because it exists only at macro-expansion), it seems to me that this is doing a lot of work for no benefit. Moreover, I believe it obscures, rather than clarifies, the code.
So, in the spirit of never using a macro where functions will do, I suggest the following two equivalent solutions:
Solution 1
(reduce #(%2 %) routes middleware)
Solution 2
((apply comp middleware) routes)
A Better Way
The second solution is easily simplified by changing the definition of middleware from being a list of the functions to being the composition of the functions:
(def middleware
(comp wrap-json-body
wrap-cors
;; and so on
))
(middleware routes)
When I began learning Clojure, I ran across this pattern often enough that many of my early projects have an freduce defined in core:
(defn freduce
"Given an initial input and a collection of functions (f1,..,fn),
This is logically equivalent to ((comp fn ... f1) input)."
[in fs]
(reduce #(%2 %) in fs))
This is totally unnecessary, and some might prefer the direct use of reduce as being more clear. However, if you don't like staring at #(%2 %) in your application code, adding another utility word to your language is fine.
you can make a macro for that:
;; notice that it is better to use a back quote, to qoute function names for macro, as it fully qualifies them.
(def middleware
`((wrap-json-body)
(wrap-cors))
;; and so on
)
(defmacro with-middleware [routes]
`(-> ~routes ~#middleware))
for example this:
(with-middleware [1 2 3])
would expand to this:
(-> [1 2 3] (wrap-json-body) (wrap-cors))
How do I programmatically figure out which Vars may affect the results of a function defined in Clojure?
Consider this definition of a Clojure function:
(def ^:dynamic *increment* 3)
(defn f [x]
(+ x *increment*))
This is a function of x, but also of *increment* (and also of clojure.core/+(1); but I'm less concerned with that). When writing tests for this function, I want to make sure that I control all relevant inputs, so I do something like this:
(assert (= (binding [*increment* 3] (f 1)) 4))
(assert (= (binding [*increment* -1] (f 1)) 0))
(Imagine that *increment* is a configuration value that someone might reasonably change; I don't want this function's tests to need changing when this happens.)
My question is: how do I write an assertion that the value of (f 1) can depend on *increment* but not on any other Var? Because I expect that one day someone will refactor some code and cause the function to be
(defn f [x]
(+ x *increment* *additional-increment*))
and neglect to update the test, and I would like to have the test fail even if *additional-increment* is zero.
This is of course a simplified example – in a large system, there can be lots of dynamic Vars, and they can get referenced through a long chain of function calls. The solution needs to work even if f calls g which calls h which references a Var. It would be great if it didn't claim that (with-out-str (prn "foo")) depends on *out*, but this is less important. If the code being analyzed calls eval or uses Java interop, of course all bets are off.
I can think of three categories of solutions:
Get the information from the compiler
I imagine the compiler does scan function definitions for the necessary information, because if I try to refer to a nonexistent Var, it throws:
user=> (defn g [x] (if true x (+ *foobar* x)))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: *foobar* in this context, compiling:(NO_SOURCE_PATH:24)
Note that this happens at compile time, and regardless of whether the offending code will ever be executed. Thus the compiler should know what Vars are potentially referenced by the function, and I would like to have access to that information.
Parse the source code and walk the syntax tree, and record when a Var is referenced
Because code is data and all that. I suppose this means calling macroexpand and handling each Clojure primitive and every kind of syntax they take. This looks so much like a compilation phase that it would be great to be able to call parts of the compiler, or somehow add my own hooks to the compiler.
Instrument the Var mechanism, execute the test and see which Vars get accessed
Not as complete as the other methods (what if a Var is used in a branch of the code that my test fails to exercise?) but this would suffice. I imagine I would need to redefine def to produce something that acts like a Var but records its accesses somehow.
(1) Actually that particular function doesn't change if you rebind +; but in Clojure 1.2 you can bypass that optimization by making it (defn f [x] (+ x 0 *increment*)) and then you can have fun with (binding [+ -] (f 3)). In Clojure 1.3 attempting to rebind + throws an error.
Regarding your first point you could consider using the analyze library. With it you can quite easily figure out which dynamic vars are used in an expression:
user> (def ^:dynamic *increment* 3)
user> (def src '(defn f [x]
(+ x *increment*)))
user> (def env {:ns {:name 'user} :context :eval})
user> (->> (analyze-one env src)
expr-seq
(filter (op= :var))
(map :var)
(filter (comp :dynamic meta))
set)
#{#'user/*increment*}
I know that this doesn't answer your question, but wouldn't it be a lot less work to just provide two versions of a function where one version has no free variables, and the other version calls the first one with the appropriate top-level defines?
For example:
(def ^:dynamic *increment* 3)
(defn f
([x]
(f x *increment*))
([x y]
(+ x y)))
This way you can write all your tests against (f x y), which doesn't rely on any global state.
All, I'm starting to take a look at the Clojure language, and had a couple questions about something I'm trying to do. The broad objective is to alias the sequence function every? to all?. I'm sure there's a function or macro that does alias-ing (or something along those lines) but I wanted to see if it was possible with some of the basic constructs I know thus far. My approach was going to be to define a function called all? that applies its arguments to the every? implementation.
I'm curious to see if this can be made agnostic, so I wanted to parameter my alias function to take two arguments, the new name (as a Keyword) and the old name (as a function reference). In striving towards this goal, I've encountered two problems.
1) Defining named functions with Keywords throws errors. Apparently it wants clojure.lang.IObj.
user=> (defn :foo "bar")
java.lang.ClassCastException: clojure.lang.Keyword cannot be cast to clojure.lang.IObj (NO_SOURCE_FILE:0)
Is there a function to cast a Keyword to an IObj, or other means to parameterize the name of a newly defined function with some provided value? (In Ruby, define_method amongst other techniques does this)
irb(main)> self.class.instance_eval do
irb(main)* define_method(:foo) { "bar" }
irb(main)> end
=> #<Proc>
irb(main)> foo
=> "bar"
2) Collect all arguments to a function into a single variable. Even basic functions such as (+ 1 2 3 4) take a variable amount of arguments. All the function definition techniques I've seen so far take a specific amount of arguments, with no way to just aggregate everything in a list for handling in the function body. Once again, what I'm going for is done in Ruby like so:
irb(main)> def foo(*args)
irb(main)> p args
irb(main)> end
=> nil
irb(main)> foo(1, 2, 3)
[1, 2, 3]
=> nil
Thanks for any help you can provide me!
I'll answer in bullet points, since the questions can be split neatly into a number of separate issues.
Something which is implicitly contained in what is to follow, but which perhaps warrants a bullet of its own: the top-level objects created by def & Co. (and in particular by defn) are Vars. So what you actually want to do is to alias a Var; functions are just regular values which don't really have names (except they may have a name bound to themselves locally inside their bodies; that's nothing to do with the issue at hand, though).
There is indeed an "aliasing macro" available in Clojure -- clojure.contrib.def/defalias:
(use '[clojure.contrib.def :only [defalias]])
(defalias foo bar)
; => foo can now be used in place of bar
The advantage of this over (def foo bar) is that it copies over metadata (such as the docstring); it even appears to work with macros in the current HEAD, although I recall a bug which prevented that in earlier versions.
Vars are named by symbols, not keywords. Symbol literals in Clojure (and other Lisps) do not start with colons (:foo is a keyword, not a symbol). Thus to define a function called foo you should write
(defn foo [...] ...)
defn is a helper macro easing the creation of new function-holding Vars by allowing the programmer to use a mix of def & fn syntax. So defn is out of question for creating Vars with preexisting values (which might be functions), as is required for creating aliases; use defalias or simply def instead.
To create a variadic function, use the following syntax:
(fn [x y & args] ...)
x and y will be required positional arguments; the rest of the arguments passed to the function (any number of them) will be collected into a seq and available under the name args. You don't have to specify any "required positional arguments" if they are not needed: (fn [& args] ...).
To create a Var holding a variadic function, use
(defn foo [x y & args] ...)
To apply a function to some arguments you've got assembled into a seqable object (such as the args seq in the above examples or perhaps a vector &c.), use apply:
(defn all? [& args]
(apply every? args))
If you want to write a function to create aliases -- as opposed to a macro -- you'll need to investigate the functions intern, with-meta, meta -- and possibly resolve / ns-resolve, depending on whether the function is to accept symbols or Vars. I'll leave filling in the details as an exercise to the reader. :-)
All you need to do is bind the every? function to the all? symbol, which is done via def:
(def all? every?)
For a bit more on this, see Clojure macro to create a synonym for a function
Don't think I can add much to the existing explanations here, except perhaps fill out a couple of blanks in the Ruby traveller's dictionary on argument collection and destructuring:
(defn foo [& args] ; Ruby: def foo(*args)
(println args))
user=> (foo 1 2 3)
(1 2 3)
(defn foo [& args]
(+ args))
user=> (foo 1 2 3)
java.lang.ClassCastException ; + takes numbers, not a list
(defn foo [& args]
(apply + args)) ; apply: as Ruby proc.call(*args)
user=> (foo 1 2 3)
6
(defn foo [& args]
(let [[a b & other] args] ; Ruby: a, b, *other = args
(println a b other)))
user=> (foo 1 2 3)
1 2 (3)