I read that data > functions > macros
Say you want to evaluate code in a postfix fashion.
Which approach would be better?
;; Macro
(defmacro reverse-fn [expression]
(conj (butlast expression) (last expression)))
(reverse-fn ("hello world" println))
; => "hello world"
;; Function and data
(def data ["hello world" println])
(defn reverse-fn [data]
(apply (eval (last data)) (butlast data)))
(reverse-fn ["hello world" println])
; => "hello world"
Thanks!
If you require different evaluation behavior for any kind of data in your code macros are your best choice because they can transform unevaluated data at compile time to the code you'd like to be evaluated instead.
Clojure has a programmatic macro system which allows the compiler to be extended by user code. Macros can be used to define syntactic constructs which would require primitives or built-in support in other languages. (http://clojure.org/macros)
The example you provide has specifically that requirement ("evaluate code in a postfix fashion") which is why a macro is the correct choice.
Your macro is better than your function: a macro is better than a function employing eval.
However, the function need not employ eval. You could write it
(defn reverse-fn [data]
(apply (last data) (butlast data)))
Then, for example, as before,
(reverse-fn [3 inc])
=> 4
On a related topic, you might find this explanation of transducers interesting.
Edit:
Notice that functions are literals, in the sense that a function evaluates to itself:
((eval +) 1 1)
=> 2
In general:
Macros have their use however; macros expand at the point they are encountered so you will have one or more code blocks being inlined in the resulting byte code. They are ideal for encapsulating High Order DSL terminologies.
Functions, on the other hand, are wonderful for reuse where you localize the purity of the function and it can be called from multiple other functions without increasing the code footprint. They are ideal for localized or global usage.
REPL behavior consideration: A single function is easier to rework without worrying about fully evaluating the entire source file(s) to ensure all macro reference expansions get updated.
Hope this helps
Simple rules are the best: Use a function if you can, a macro if you must.
Related
Recently I came across a use for eval within a macro, which I understand is a bit of a faux pas but let's ignore that for now. What I found surprising, was that eval was able to resolve global vars at macroexpansion time. Below is a contrived example, just to illustrate the situation I'm referring to:
(def list-of-things (range 10))
(defmacro force-eval [args]
(apply + (eval args)))
(macroexpand-1 '(force-eval list-of-things))
; => 45
I would have expected args to resolve to the symbol list-of-things inside force-eval, and then list-of-things to be evaluated resulting in an error due to it being unbound:
"unable to resolve symbol list-of-things in this context"
However, instead list-of-things is resolved to (range 10) and no error is thrown - the macroexpansion succeeds.
Contrast this with attempting to perform the same macroexpansion, but within a local binding context:
(defmacro force-eval [args]
(apply + (eval args)))
(let [list-of-things (range 10)]
(macroexpand-1 '(force-eval list-of-things)))
; => Unable to resolve symbol: list-of-thingss in this context
Note in the above examples I'm assuming list-of-things is not previously bound, e.g. a fresh REPL. One final example illustrates why this is important:
(defmacro force-eval [args]
(apply + (eval args)))
(def list-of-things (range 10 20))
(let [list-of-thing (range 10)]
(macroexpand-1 '(force-eval list-of-things)))
; => 145
The above example shows that the locals are ignored, which is expected behavior for eval, but is a bit confusing when you are expecting the global to not be available at macroexpansion time either.
I seem to have a misunderstanding about what exactly is available at macroexpansion time. I had previously thought that essentially any binding, be it global or local, would not be available until runtime. Apparently this is an incorrect assumption. Is the answer to my confusion simply that global vars are available at macroexpansion time? Or am I missing some further nuance here?
Note: this related post closely describes a similar problem, but the focus there is more on how to avoid inappropriate use of eval. I'm mainly interested in understanding why eval works in the first example and by extension what's available to eval at macroexpansion time.
Of course, vars must be visible at compile time. That's where functions like first and + are stored. Without them, you couldn't do anything.
But keep in mind that you have to make sure to refer to them correctly. In the repl, *ns* will be bound, and so a reference to a symbol will look in the current namespace. If you are running a program through -main instead of the repl, *ns* will not be bound, and only properly qualified vars will be found. You can ensure that you qualify them correctly by using
`(force-eval list-of-things)
instead of
'(force-eval list-of-things)
Note I do not distinguish between global vars and non-global vars. All vars in Clojure are global. Local bindings are not called vars. They're called locals, or bindings, or variables, or some combination of those words.
Clojure is designed with an incremental compilation model. This is poorly documented.
In C and other traditional languages, source code must be compiled, then linked with pre-compiled libraries before the final result can be executed. Once execution begins, no changes to the code can occur until the program is terminated, when new source code can be compiled, linked, then executed. Java is normally used in this manner just like C.
With the Clojure REPL, you can start with zero source code in a live executing environment. You can call existing functions like (+ 2 3), or you can define new functions and variables on the fly (both global & local), and redefine existing functions. This is only possible because core Clojure is already available (i.e. clojure.core/+ etc is already "installed"), so you can combine these functions to define your own new functions.
The Clojure "compiler" works just like a giant REPL session. It reads and evaluates forms from your source code files one at a time, incrementally adding them the the global environment. Indeed, it is a design goal/requirement that the result of compiling and executing source code is identical to what would occur if you just pasted each entire source code file into the REPL (in proper dependency order).
Indeed, the simplest mental model for code execution in Clojure is to pretend it is an interpreter instead of a traditional compiler.
And eval in a macro makes no sense.
Because:
a macro already implicitely contains an eval
at the very final step.
If you use macroexpand-1, you make visible how the code was manipulated in the macro before the evocation of the implicite eval inside the macro.
An eval in a macro is an anti-pattern which might indicate that you should use a function instead of a macro - and in your examle this is exactly the case.
So your aim is to dynamically (in run-time) evoke sth in a macro. This you can only do through an eval applied over a macro call OR you should rather use a function.
(defmacro force-eval [args]
(apply + (eval args)))
;; What you actually mean is:
(defn force-eval [args]
(apply + args))
;; because a function in lisp evaluates its arguments
;; - before applying the function body.
;; That means: args in the function body is exactly
;; `(eval args)`!
(def list-of-things (range 10))
(let [lit-of-things (range 10 13)]
(force-eval list-of-things))
;; => 45
;; so this is exactly the behavior you wanted!
The point is, your construct is a "bad" example for a macro.
Because apply is a special function which allows you to
dynamically rearrange function call structures - so it has
some magic of macros inside it - but in run-time.
With apply you can do quite some meta programming in some cases when you just quote some of your input arguments.
(Try (force-eval '(1 2 3)) it returns 6. Because the (1 2 3) is put together with + at its front by apply and then evaluated.)
The second point - I am thinking of this answer I once gave and this to a dynamic macro call problem in Common Lisp.
In short: When you have to control two levels of evaluations inside a macro (often when you want a macro inject some code in runtime into some code), you need too use eval when calling the macro and evaluate those parts in the macro call which then should be processed in the macro.
I'm playing around with clojure macros and I'm finding that a lot of macro behavior I can just replicate with function composition.
A good example of this is the threading macro:
(defn add1 [n] (+ n 1))
(defn mult10 [n] (* n 10))
(defn threadline [arg]
(-> arg
add1
mult10))
I can replicate this easily with a higher order function like pipe:
(defn pipe [& fns]
(reduce (fn [f g] (fn [arg] (g(f arg)))) fns))
(def pipeline
(pipe
#(+ % 1)
#(* % 10)))
There must be instances where a macro can not be replaced by a function. I was wondering if someone had some good examples of these sorts of situations, and the reoccurring themes involved.
One important advantage of macros is their ability to transform code at compile-time without evaluating any of it. Macros receive code as data during compilation, but functions receive values at run-time. Macros allow you to extend the compiler in a sense.
For example, Clojure's and and or are implemented as recursive macros that expand into nested if forms. This allows lazy evaluation of the and/or's inner forms i.e. if the first or form is truthy, its value will be returned and none of the others will be evaluated. If you wrote and/or as a function, all its arguments would be evaluated before they could be examined.
Short-circuiting control flow isn't an issue in your pipe function example, but pipe adds considerable run-time complexity compared to -> which simply unrolls to nested forms. A more interesting macro to try to implement as a function might be some->.
I'm finding that a lot of macro behavior I can just replicate with function composition
If your functions are amenable to it, you can certainly replace a simple threading macro with function composition with comp, similar to "point free" style in other functional languages: #(-> % inc str) is functionally equivalent to (comp str inc) and #(str (inc %)).
It's generally advised to prefer functions when possible, and even when writing a macro you can usually farm out most of the "work" to function(s).
The first macro I ever learned is a good example of a macro that can't be written as a plain function:
(defmacro infix [[arg1 f arg2]]
(list f arg1 arg2))
(infix (1 + 2))
=> 3
Of course this exact macro would never be used in the wild, but it sets the stage for more useful macros that act as readability helpers. It should also be noted that while you can replicate a lot of basic macro's behavior with plain functions, should you? It would be hard to argue that your pipe example leads to easier to read/write code than, say, as->.
The "reoccurring themes" you're looking for are cases where you're manipulating data at compile-time ("data" being the code itself), not run-time. Any case that requires the function to take its argument unevaluated must be a macro. You can partially "cheat" in some cases and just wrap the code in a function to delay evaluation, but that doesn't work for all cases (like the infix example).
Macros are not interchangeable with functions, but you examples are:
(macroexpand '#(+ % 1))
; ==> (fn* [p1__1#] (+ p1__1# 1))
The reason why it works is because the argument expect a function and you use a macro that becomes a function. However I know that cond is a macro. It cannot be replaced with a function implementation since the arguments of a function gets evaluated and the whole point of cond is to only evaluate some parts of the arguments in a specific order based on evaluation of their predicates. eg. making a recursive function with that would never terminate since the default case will also always be called before the body of the function cond is evaluated.
The whole point of macros is to expand the lamguage and since the evaluation is controlled by the result you can make all sorts of new features that would be impossible with function except if one passed all arguments as functions to delay evaluation.
In any language, macros -- compile-time functions from code to code -- let you do three things:
Define new binding forms (e.g. Clojure's destructuring let).
Change the order of evaluation (e.g. or, and).
Present a domain-specific language (e.g. Instaparse).
You can debate 3 -- whether implementing DSLs truly requires macros. Sure you can do code generators that are functions from text files to text files. Or, sure you can do Ruby style runtime DSLs. But if you want a DSL that's integrated into the compiler at compile-time, then macros are effectively your "compiler API".
Having said that, it makes sense to use macros only for these special purposes. Use functions and/or data-driven code as much as possible. Including to do work behind the "facade" provide by a macro.
The two big things macros do is control evaluation of their arguments and transform code at compile time. You can do both with functions by requiring calling code to quote their arguments.
For instance, you could write a version of defn that is called this way:
(defn 'name '[arg1 arg2]
'(expression1)
'(expression2)
'etc)
Then you could eval arguments at will, evaluating them or not, changing the order of execution, or altering forms before you evaluate them, exactly the things macros are good for.
What macros can do that functions can't is gain this ability without any cooperation from the calling code. Users can call macros as if they were plain functions and don't have to treat their arguments any differently.
That's how macros allow you to extend the language: you don't have to treat macro code any differently than regular code, unlike, say, JavaScript, Ruby, or Python, where the language can only be extended with new control flow constructs by doing what you've done in your example, wrapping code in a block, lambda, or function.
I find it hard to reason about macro-expansion and was wondering what the best practices were for testing them.
So if I have a macro, I can perform one level of macro expansion via macroexpand-1.
(defmacro incf-twice (n)
`(progn
(incf ,n)
(incf ,n)))
for example
(macroexpand-1 '(incf-twice n))
evaluates to
(PROGN (INCF N) (INCF N))
It seems simple enough to turn this into a test for the macro.
(equalp (macroexpand-1 '(incf-twice n))
'(progn (incf n) (incf n)))
Is there an established convention for organizing tests for macros? Also, is there a library for summarizing differences between s-expressions?
Generally testing macros is not one of the strong parts of Lisp and Common Lisp. Common Lisp (and Lisp dialects in general) uses procedural macros. The macros can depend on the runtime context, the compile-time context, the implementation and more. They also can have side effects (like registering things in the compile-time environment, registering things in the development environment and more).
So one might want to test:
that the right code gets generated
that the generated code actually does the right thing
that the generated code actually works in code contexts
that the macro arguments are actually parsed correctly in case of complex macros. Think loop, defstruct, ... macros.
that the macro detects wrongly formed argument code. Again, think of macros like loop and defstruct.
the side effects
From above list on can infer that it is best to minimize all these problem areas when developing a macro. BUT: there are really really complex macros out there. Really scary ones. Especially those who are used to implemented new domain specific languages.
Using something like equalp to compare code works only for relatively simple macros. Macros often introduce new, uninterned and unique symbols. Thus equalp will fail to work with those.
Example: (rotatef a b) looks simple, but the expansion is actually complicated:
CL-USER 28 > (pprint (macroexpand-1 '(rotatef a b)))
(PROGN
(LET* ()
(LET ((#:|Store-Var-1234| A))
(LET* ()
(LET ((#:|Store-Var-1233| B))
(PROGN
(SETQ A #:|Store-Var-1233|)
(SETQ B #:|Store-Var-1234|))))))
NIL)
#:|Store-Var-1233| is a symbol, which is uninterned and newly created by the macro.
Another simple macro form with a complex expansion would be (defstruct s b).
Thus one would need a s-expression pattern matcher to compare the expansions. There are a few available and they would be useful here. One needs to make sure in the test patterns that the generated symbols are identical, where needed.
There are also s-expression diff tools. For example diff-sexp.
I agree with Rainer Joswig's answer; in general, this is a very difficult task to solve because macros can do a whole lot. However, I would point out that in many cases, the easiest way to unit test your macros is by making the macros do as little as possible. In many cases, the easiest implementation of a macro is just syntactic sugar around a simpler function. E.g., there's a typical pattern of with-… macros in Common Lisp (e.g., with-open-file), where the macro simply encapsulates some boilerplate code:
(defun make-frob (frob-args)
;; do something and return the resulting frob
(list 'frob frob-args))
(defun cleanup-frob (frob)
(declare (ignore frob))
;; release the resources associated with the frob
)
(defun call-with-frob (frob-args function)
(let ((frob (apply 'make-frob frob-args)))
(unwind-protect (funcall function frob)
(cleanup-frob frob))))
(defmacro with-frob ((var &rest frob-args) &body body)
`(call-with-frob
(list ,#frob-args)
(lambda (,var)
,#body)))
The first two functions here, make-frob and cleanup-frob are relatively straightforward to unit test. The call-with-frob is a bit harder. The idea is that it's supposed to handle the boilerplate code of creating the frob and ensuring that the cleanup call happens. That's a bit harder to check, but if the boilerplate only depends on some well defined interfaces, then you'll probably be able to create mock up a frob that can detect whether it's cleaned up correctly. Finally, the with-frob macro is so simple that you can probably test it the way you've been considering, i.e., checking its expansion. Or you might say that it's simple enough that you don't need to test it.
On the other hand, if you're looking at a much more complex macro, such as loop, which is really a kind of compiler in its own right, you're almost certainly already going to have the expansion logic in some separate functions. E.g., you might have
(defmacro loop (&body body)
(compile-loop body))
in which case you really don't need to test loop, you need to test compile-loop, and then you're back in the realm of your usual unit testing.
I'd generally just test the functionality, not the shape of the expansion.
Yes, there are all kinds of contexts and surroundings that might influence what happens, but if you rely on such things, it should be no problem to set them up the same for your test.
Some common cases:
binding macros: test that the variables are bound as intended inside and that any shadowed outside variables are unaffected
unwind-protect wrappers: provoke a nonlocal exit from the inside and check that the cleanup is working
definition/registration: test that you can define/register what you want and use it afterwards
This is not my 'production code' but a simplication of the problem for illustration purposes. Also, the title of this question is misleading because it brings to mind the ~# expansion, which I understand, and which may not necessarily be the problem. Please suggest a better question title if you can.
Given a macro with the following form:
(defmacro my-add [x & ys] `(+ ~x ~#ys))
Now let's say we have a list:
(def my-lst '(2 3))
Now I want a function that uses my-add that I can pass my-lst to as an arg, i.e.
(call-my-add 1 my-lst)
I define the function in what would seem to be the obvious way:
(defn call-my-add [x ys]
(apply my-add (cons x ys)))
But:
java.lang.Exception: Can't take value of a macro: #'user/call-my-add (repl-1:60)
I've tried all sorts of wild tricks to get the call-my-add function to work using evals, applies, and even defining call-my-add as a macro, but they all give similar ClassCastExceptions.
Is there any way out of this?
No. Macros do not, cannot, will never, have access to the actual run-time values contained in their arguments, so cannot splice them into the expansion. All they get is the symbol(s) you pass to them, in this case my-list. The "way around this" is to define my-add as a function, and then (optionally) have a macro that calls that function in order to generate its code.
I wrote a blog post about this semi-recently that you might find enlightening.
You could do it with evals if you wanted to, but that is a terrible idea in almost every case:
(let [my-list '(1 2)]
(eval `(my-add 5 ~#my-list)))
What a great example showing that macros are not first class citizens in Clojure (or any Lisp that I know of). They cannot be applied to functions, stored in containers, or passed to functions etc. In exchange for this they get to control when and if their arguments are evaluated.
What happens at macro expansion time must stay in macro expansion time. So if my-add is being evaluated at macro expansion time and you want to use apply then you need... another macro; to do the applying.
(defmacro call-my-add [x ys]
`(my-add ~#(cons x ys)))
Macros are somewhat contagious in this way.
PS: I'm not at my repl so please edit if you see a bug in this example (or I'll fix it when I get back)
I cannot use logical functions on a range of booleans in Clojure (1.2). Neither of the following works due to logical functions being macros:
(reduce and [... sequence of bools ...])
(apply or [... sequence of bools ...])
The error message says that I "can't take value of a macro: #'clojure.core/and". How to apply these logical functions (macros) without writing boilerplate code?
Don't -- use every? and some instead.
Michal's answer is already spot on, but the following alternative approach can be useful in similar situations whenever you want to use a macro as a function:
(reduce #(and %1 %2) [... sequence of bools ...])
Basically you just wrap the macro in an anonymous function.
There are a couple of good reasons to consider this approach:
There are situations where a handy function like some or every? does not exist
You may get better performance (reduce is likely to benefit from some very good optimisations in the future, for example applying the function directly to a vector rather than converting the vector into a sequence)