Clojure, being a Lisp dialect, inherited Lisp's homoiconicity. Homoiconicity makes metaprogramming easier, since code can be treated as data: reflection in the language (examining the program's entities at runtime) depends on a single, homogeneous structure, and it does not have to handle several different structures that would appear in a complex syntax [1].
The downside of a more homogeneous language structure is that language constructs, such as loops, nested ifs, function calls or switches, etc, are more similar to each other.
In clojure:
;; if:
(if (chunked-seq? s)
(chunk-cons (chunk-first s) (concat (chunk-rest s) y))
(cons (first s) (concat (rest s) y)))
;; function call:
(repaint (chunked-seq? s)
(chunk-cons (chunk-first s) (concat (chunk-rest s) y))
(cons (first s) (concat (rest s) y)))
The difference between the two constructs is just a word. In a non homoiconic language:
// if:
if (chunked-seq?(s))
chunk-cons(chunk-first(s), concat(chunk-rest(s), y));
else
cons(first(s), concat(rest(s), y));
// function call:
repaint(chunked-seq?(s),
chunk-cons(chunk-first(s), concat(chunk-rest(s), y)),
cons(first(s), concat(rest(s), y));
Is there a way to make these program constructs easier to identify (more conspicuous) in Clojure? Maybe some recommended code format or best practice?
Besides using an IDE that supports syntax highlighting for the different cases, no, there isn't really a way to differentiate between them in the code itself.
You could try and use formatting to differentiate between function calls and macros:
(for [a b]
[a a])
(some-func [a b] [a a])
But then that prevents you from using a one-line list comprehension using for; sometimes they can neatly fit on one line. This also prevents you from breaking up large function calls into several lines. Unless the reducing function is pre-defined, most of my calls to reduce take the form:
(reduce (fn [a b] ...)
starting-acc
coll)
There are just too many scenarios to try to limit how calls are formatted. What about more complicated macros like cond?
I think a key thing to understand is that the operation of a form depends entirely on the first symbol in the form. Instead of relying on special syntaxes to differentiate between them, train your eyes to snap to the first symbol in the form, and do a quick "lookup" in your head.
And really, there are only a few cases that need to be considered:
Special forms like if and let (actually let*). These are fundamental constructs of the language, so you will be exposed to them constantly.
I don't think these should pose a problem. Your brain should immediately know what's going on when you see if. There are so few special forms that plain memorization is the best route.
Macros with "unusual" behavior like threading macros and cond. There are still some instances where I'll be looking over someone's code, and because they're using a macro that I don't have a lot of familiarity with, it'll take me a second to figure out the flow of the code.
This is remedied though by just getting some practice with the macro. Learning a new macro extends your capabilities when writing Clojure anyway, so this should always be considered. As with special forms, there really aren't that many mind-bending macros, so memorizing the main ones (the basic threading macros, and conditional macros) is simple.
Functions. If it's not either of the above, it must be a function and follow typical function calling syntax.
Related
Macros do not evaluate their arguments until explicitly told to do so, however functions do. In the following code:
(defmacro foo [xs]
(println xs (type xs)) ;; unquoted list
(blah xs))
(defn blah [xs] ;; xs is unquoted list, yet not evaluated
(println xs)
xs)
(foo (+ 1 2 3))
It seems that blah does not evaluate xs, since we still have the entire list: (+ 1 2 3) bound to xs in the body of blah.
I have basically just memorized this interaction between helper functions within macros and their evaluation of arguments, but to be honest it goes against what my instincts are (that xs would be evaluated before entering the body since function arguments are always evaluated).
My thinking was basically: "ok, in this macro body I have xs as the unevaluated list, but if I call a function with xs from within the macro it should evaluate that list".
Clearly I have an embarassingly fundamental misunderstanding here of how things work. What am I missing in my interpretation? How does the evaluation actually occur?
EDIT
I've thought on this a bit more and it seems to me that maybe viewing macro arguments as "implicitly quoted" would solve some confusion on my part.
I think I just got mixed up in the various terminologies, but given that quoted forms are synonymous with unevaluated forms, and given macro arguments are unevaluated, they are implicitly quoted.
So in my above examples, saying xs is unquoted is somewhat misleading. For example, this macro:
(defmacro bluh [xs]
`(+ 1 2 ~xs))
Is basically the same as the below macro (excluding namespacing on the symbols). Resolving xs in the call to list gives back an unevaluated (quoted?) list.
(defmacro bleh [xs]
(list '+ '1 '2 xs)) ;; xs resolves to a quoted list (or effectively quoted)
Calling bleh (or bluh) is the same as saying:
(list '+ '1 '2 '(+ 1 2 3))
;; => (+ 1 2 (+ 1 2 3))
If xs did not resolve to a quoted list, then we would end up with:
(list '+ '1 '2 (+ 1 2 3))
;; => (+ 1 2 6)
So, in short, macro arguments are quoted.
I thnk part of my confusion came from thinking about the syntax quoted forms as templates with slots filled in e.g. (+ 1 2 ~xs) I would mentally expand to (+ 1 2 (+ 1 2 3)), and seeing that (+ 1 2 3) was not quoted in that expansion, I found it confusing that function calls using xs (in the first example above blah) would not evalute immediately to 6.
The template metaphor is helpful, but if I instead look at it as a
shortcut for (list '+ '1 '2 xs) it becomes obvious that xs must be a quoted list otherwise the expansion would include 6 and not the entire list.
I'm not sure why I found this so confusing... have I got this right or did I just go down the wrong path entirely?
[This answer is an attempt to explain why macros and functions which don't evaluate their arguments are different things. I believe this applies to macros in Clojure but I am not an expert on Clojure. It's also much too long, sorry.]
I think you are confused between what Lisp calls macros and a construct which modern Lisps don't have but which used to be called FEXPRs.
There are two interesting, different, things you might want:
functions which, when called, do not immediately evaluate their arguments;
syntax transformers, which are called macros in Lisp.
I'll deal with them in order.
Functions which do not immediately evaluate their arguments
In a conventional Lisp, a form like (f x y ...), where f is a function, will:
establish that f is a function and not some special thing;
get the function corresponding to f and evaluate x, y, & the rest of the arguments in some order specified by the language (which may be 'in an unspecified order');
call f with the results of evaluating the arguments.
Step (1) is needed initially because f might be a special thing (like, say if, or quote), and it might be that the function definition is retrieved in (1) as well: all of this, as well as the order that things happen in in (2) is something the language needs to define (or, in the case of Scheme say, leave explicitly undefined).
This ordering, and in particular the ordering of (2) & (3) is known as applicative order or eager evaluation (I'll call it applicative order below).
But there are other possibilities. One such is that the arguments are not evaluated: the function is called, and only when the values of the arguments are needed are they evaluated. There are two approaches to doing this.
The first approach is to define the language so that all functions work this way. This is called lazy evaluation or normal order evaluation (I'll call it normal order below). In a normal order language function arguments are evaluated, by magic, at the point they are needed. If they are never needed then they may never be evaluated at all. So in such a language (I am inventing the syntax for function definition here so as not to commit CL or Clojure or anything else):
(def foo (x y z)
(if x y z))
Only one of y or z will be evaluated in a call to foo.
In a normal order language you don't need to explicitly care about when things get evaluated: the language makes sure that they are evaluated by the time they're needed.
Normal order languages seem like they'd be an obvious win, but they tend to be quite hard to work with, I think. There are two problems, one obvious and one less so:
side-effects happen in a less predictable order than they do in applicative order languages and may not happen at all, so people used to writing in an imperative style (which is most people) find them hard to deal with;
even side-effect-free code can behave differently than in an applicative order language.
The side-effect problem could be treated as a non-problem: we all know that code with side-effects is bad, right, so who cares about that? But even without side-effects things are different. For instance here's a definition of the Y combinator in a normal order language (this is kind of a very austere, normal order subset of Scheme):
(define Y
((λ (y)
(λ (f)
(f ((y y) f))))
(λ (y)
(λ (f)
(f ((y y) f))))))
If you try to use this version of Y in an applicative order language -- like ordinary Scheme -- it will loop for ever. Here's the applicative order version of Y:
(define Y
((λ (y)
(λ (f)
(f (λ (x)
(((y y) f) x)))))
(λ (y)
(λ (f)
(f (λ (x)
(((y y) f) x)))))))
You can see it's kind of the same, but there are extra λs in there which essentially 'lazify' the evaluation to stop it looping.
The second approach to normal order evaluation is to have a language which is mostly applicative order but in which there is some special mechanism for defining functions which don't evaluate their arguments. In this case there often would need to be some special mechanism for saying, in the body of the function, 'now I want the value of this argument'. Historically such things were called FEXPRs, and they existed in some very old Lisp implementations: Lisp 1.5 had them, and I think that both MACLISP & InterLisp had them as well.
In an applicative order language with FEXPRs, you need somehow to be able to say 'now I want to evaluate this thing', and I think this is the problem are running up against: at what point does the thing decide to evaluate the arguments? Well, in a really old Lisp which is purely dynamically scoped there's a disgusting hack to do this: when defining a FEXPR you can just pass in the source of the argument and then, when you want its value, you just call EVAL on it. That's just a terrible implementation because it means that FEXPRs can never really be compiled properly, and you have to use dynamic scope so variables can never really be compiled away. But this is how some (all?) early implementations did it.
But this implementation of FEXPRs allows an amazing hack: if you have a FEXPR which has been given the source of its arguments, and you know that this is how FEXPRs work, then, well, it can manipulate that source before calling EVAL on it: it can call EVAL on something derived from the source instead. And, in fact, the 'source' it gets given doesn't even need to be strictly legal Lisp at all: it can be something which the FEXPR knows how to manipulate to make something that is. That means you can, all of a sudden, extend the syntax of the language in pretty general ways. But the cost of being able to do that is that you can't compile any of this: the syntax you construct has to be interpreted at runtime, and the transformation happens each time the FEXPR is called.
Syntax transformers: macros
So, rather than use FEXPRs, you can do something else: you could change the way that evaluation works so that, before anything else happens, there is a stage during which the code is walked over and possibly transformed into some other code (simpler code, perhaps). And this need happen only once: once the code has been transformed, then the resulting thing can be stashed somewhere, and the transformation doesn't need to happen again. So the process now looks like this:
code is read in and structure built from it;
this initial structure is possibly transformed into other structure;
(the resulting structure is possibly compiled);
the resulting structure, or the result of compiling it is evaluated, probably many times.
So now the process of evaluation is divided into several 'times', which don't overlap (or don't overlap for a particular definition):
read time is when the initial structure is built;
macroexpansion time is when it is transformed;
compile time (which may not happen) is when the resulting thing is compiled;
evaluation time is when it is evaluated.
Well, compilers for all languages probably do something like this: before actually turning your source code into something that the machine understands they will do all sorts of source-to-source transformations. But these things are in the guts of the compiler and are operating on some representation of the source which is idiosyncratic to that compiler and not defined by the language.
Lisp opens this process to users. The language has two features which make this possible:
the structure that is created from source code once it has been read is defined by the language and the language has a rich set of tools for manipulating this structure;
the structure created is rather 'low commitment' or austere -- it does not particularly predispose you to any interpretation in many cases.
As an example of the second point, consider (in "my.file"): that's a function call of a function called in, right? Well, may be: (with-open-file (in "my.file") ...) almost certainly is not a function call, but binding in to a filehandle.
Because of these two features of the language (and in fact some others I won't go into) Lisp can do a wonderful thing: it can let users of the language write these syntax-transforming functions -- macros -- in portable Lisp.
The only thing that remains is to decide how these macros should be notated in source code. And the answer is the same way as functions are: when you define some macro m you use it just as (m ...) (some Lisps support more general things, such as CL's symbol macros. At macroexpansion time -- after the program is read but before it is (compiled and) run -- the system walks over the structure of the program looking for things which have macro definitions: when it finds them it calls the function corresponding to the macro with the source code specified by its arguments, and the macro returns some other chunk of source code, which gets walked in turn until there are no macros left (and yes, macros can expand to code involving other macros, and even to code involving themselves). Once this process is complete then the resulting code can be (compiled and) run.
So although macro look like function calls in the code, they are not just functions which don't evaluate their arguments, like FEXPRs were: instead they are functions which take a bit of Lisp source code and return another bit of Lisp source code: they're syntax transformers, or function which operate on source code (syntax) and return other source code. Macros run at macroexpansion time which is properly before evaluation time (see above).
So, in fact macros are functions, written in Lisp, and the functions they call evaluate their arguments perfectly conventionally: everything is perfectly ordinary. But the arguments to macros are programs (or the syntax of programs represented as Lisp objects of some kind) and their results are (the syntax of) other programs. Macros are functions at the meta-level, if you like. So a macro if a function which computes (parts of) programs: those programs may later themselves be run (perhaps much later, perhaps never) at which point the evaluation rules will be applied to them. But at the point a macro is called what it's dealing with is just the syntax of programs, not evaluating parts of that syntax.
So, I think your mental model is that macros are something like FEXPRs in which case the 'how does the argument get evaluated' question is an obvious thing to ask. But they're not: they're functions which compute programs, and they run properly before the program they compute is run.
Sorry this answer has been so long and rambling.
What happened to FEXPRs?
FEXPRs were always pretty problematic. For instance what should (apply f ...) do? Since f might be a FEXPR, but this can't generally be known until runtime it's quite hard to know what the right thing to do is.
So I think that two things happened:
in the cases where people really wanted normal order languages, they implemented those, and for those languages the evaluation rules dealt with the problems FEXPRs were trying to deal with;
in applicative order languages then if you want to not evaluate some argument you now do it by explicitly saying that using constructs such as delay to construct a 'promise' and force to force evaluation of a promise -- because the semantics of the languages improved it became possible to implement promises entirely in the language (CL does not have promises, but implementing them is essentially trivial).
Is the history I've described correct?
I don't know: I think it may be but it may also be a rational reconstruction. I certainly, in very old programs in very old Lisps, have seen FEXPRs being used the way I describe. I think Kent Pitman's paper, Special Forms in Lisp may have some of the history: I've read it in the past but had forgotten about it until just now.
A macro definition is a definition of a function that transforms code. The input for the macro function are the forms in the macro call. The return value of the macro function will be treated as code inserted where the macro form was. Clojure code is made of Clojure data structures (mostly lists, vectors, and maps).
In your foo macro, you define the macro function to return whatever blah did to your code. Since blah is (almost) the identity function, it just returns whatever was its input.
What is happening in your case is the following:
The string "(foo (+ 1 2 3))" is read, producing a nested list with two symbols and three integers: (foo (+ 1 2 3)).
The foo symbol is resolved to the macro foo.
The macro function foo is invoked with its argument xs bound to the list (+ 1 2 3).
The macro function (prints and then) calls the function blah with the list.
blah (prints and then) returns that list.
The macro function returns the list.
The macro is thus “expanded” to (+ 1 2 3).
The symbol + is resolved to the addition function.
The addition function is called with three arguments.
The addition function returns their sum.
If you wanted the macro foo to expand to a call to blah, you need to return such a form. Clojure provides a templating convenience syntax using backquote, so that you do not have to use list etc. to build the code:
(defmacro foo [xs]
`(blah ~xs))
which is like:
(defmacro foo [xs]
(list 'blah xs))
I'm playing around with clojure macros and I'm finding that a lot of macro behavior I can just replicate with function composition.
A good example of this is the threading macro:
(defn add1 [n] (+ n 1))
(defn mult10 [n] (* n 10))
(defn threadline [arg]
(-> arg
add1
mult10))
I can replicate this easily with a higher order function like pipe:
(defn pipe [& fns]
(reduce (fn [f g] (fn [arg] (g(f arg)))) fns))
(def pipeline
(pipe
#(+ % 1)
#(* % 10)))
There must be instances where a macro can not be replaced by a function. I was wondering if someone had some good examples of these sorts of situations, and the reoccurring themes involved.
One important advantage of macros is their ability to transform code at compile-time without evaluating any of it. Macros receive code as data during compilation, but functions receive values at run-time. Macros allow you to extend the compiler in a sense.
For example, Clojure's and and or are implemented as recursive macros that expand into nested if forms. This allows lazy evaluation of the and/or's inner forms i.e. if the first or form is truthy, its value will be returned and none of the others will be evaluated. If you wrote and/or as a function, all its arguments would be evaluated before they could be examined.
Short-circuiting control flow isn't an issue in your pipe function example, but pipe adds considerable run-time complexity compared to -> which simply unrolls to nested forms. A more interesting macro to try to implement as a function might be some->.
I'm finding that a lot of macro behavior I can just replicate with function composition
If your functions are amenable to it, you can certainly replace a simple threading macro with function composition with comp, similar to "point free" style in other functional languages: #(-> % inc str) is functionally equivalent to (comp str inc) and #(str (inc %)).
It's generally advised to prefer functions when possible, and even when writing a macro you can usually farm out most of the "work" to function(s).
The first macro I ever learned is a good example of a macro that can't be written as a plain function:
(defmacro infix [[arg1 f arg2]]
(list f arg1 arg2))
(infix (1 + 2))
=> 3
Of course this exact macro would never be used in the wild, but it sets the stage for more useful macros that act as readability helpers. It should also be noted that while you can replicate a lot of basic macro's behavior with plain functions, should you? It would be hard to argue that your pipe example leads to easier to read/write code than, say, as->.
The "reoccurring themes" you're looking for are cases where you're manipulating data at compile-time ("data" being the code itself), not run-time. Any case that requires the function to take its argument unevaluated must be a macro. You can partially "cheat" in some cases and just wrap the code in a function to delay evaluation, but that doesn't work for all cases (like the infix example).
Macros are not interchangeable with functions, but you examples are:
(macroexpand '#(+ % 1))
; ==> (fn* [p1__1#] (+ p1__1# 1))
The reason why it works is because the argument expect a function and you use a macro that becomes a function. However I know that cond is a macro. It cannot be replaced with a function implementation since the arguments of a function gets evaluated and the whole point of cond is to only evaluate some parts of the arguments in a specific order based on evaluation of their predicates. eg. making a recursive function with that would never terminate since the default case will also always be called before the body of the function cond is evaluated.
The whole point of macros is to expand the lamguage and since the evaluation is controlled by the result you can make all sorts of new features that would be impossible with function except if one passed all arguments as functions to delay evaluation.
In any language, macros -- compile-time functions from code to code -- let you do three things:
Define new binding forms (e.g. Clojure's destructuring let).
Change the order of evaluation (e.g. or, and).
Present a domain-specific language (e.g. Instaparse).
You can debate 3 -- whether implementing DSLs truly requires macros. Sure you can do code generators that are functions from text files to text files. Or, sure you can do Ruby style runtime DSLs. But if you want a DSL that's integrated into the compiler at compile-time, then macros are effectively your "compiler API".
Having said that, it makes sense to use macros only for these special purposes. Use functions and/or data-driven code as much as possible. Including to do work behind the "facade" provide by a macro.
The two big things macros do is control evaluation of their arguments and transform code at compile time. You can do both with functions by requiring calling code to quote their arguments.
For instance, you could write a version of defn that is called this way:
(defn 'name '[arg1 arg2]
'(expression1)
'(expression2)
'etc)
Then you could eval arguments at will, evaluating them or not, changing the order of execution, or altering forms before you evaluate them, exactly the things macros are good for.
What macros can do that functions can't is gain this ability without any cooperation from the calling code. Users can call macros as if they were plain functions and don't have to treat their arguments any differently.
That's how macros allow you to extend the language: you don't have to treat macro code any differently than regular code, unlike, say, JavaScript, Ruby, or Python, where the language can only be extended with new control flow constructs by doing what you've done in your example, wrapping code in a block, lambda, or function.
I want to 1) create a list of symbols with the function below; then 2) create atoms with these symbols/names so that the atoms can be modified from other functions. This is the function to generate symbols/names:
(defn genVars [ dist ]
(let [ nms (map str (range dist)) neigs (map #(apply str "neig" %) nms) ]
(doseq [ v neigs ]
(intern *ns* (symbol v) [ ] ))
))
If dist=3, then 3 symbols, neig0, ... neig2 are created each bound with an empty vector. If it is possible to functionally create atoms with these symbols so that they are accessible from other functions. Any help is much appreciated, even if there are other ways to accomplish this.
your function seems to be correct, just wrap the value in the intern call with atom call. Also I would rather use dotimes.
user>
(defn gen-atoms [amount prefix]
(dotimes [i amount]
(intern *ns* (symbol (str prefix i)) (atom []))))
#'user/gen-atoms
user> (gen-atoms 2 "x")
nil
user> x0
#atom[[] 0x30f1a7b]
user> x1
#atom[[] 0x2149efef]
The desire to generate names suggests you would be better served by a single map instead:
(def neighbours (atom (make-neighbours)))
Where the definition of make-neigbours might look something like this:
(defn make-neighbours []
(into {} (for [i (range 10)]
[(str "neig" i) {:age i}])))
Where the other namespace would look values up using something like:
(get-in #data/neighbours ["neig0" :age])
Idiomatic Clojure tends to avoid creating many named global vars, preferring instead to collocating state into one or a few vars governed by Clojure's concurrency primitives (atom/ref/agent). I encourage you to think about whether your problem can be solved with a single atom in this way instead of requiring defining multiple vars.
Having said that, if you really really need multiple atoms, consider storing them all in a single map var instead of creating many global vars. Personally, I have never encountered a situation where creating many atoms was better than a single big atom (so I would be interested to hear about situations where this would be important).
If you really really need many vars, be aware that defining vars inside a function is actually bad style (https://github.com/bbatsov/clojure-style-guide#dont-def-vars-inside-fns). With good reason too! The beauty of using functions and data comes from the purity of the functions. def inside a function is particularly nasty as it is not only a side-effect, but is an potentially execution flow altering side-effect.
Of course yes there is a way to achieve it, as another answer points out.
Where it comes to defining things that goes beyond def and defn, there is quite a lot of precedence to using macros. For example defroutes from compojure, defschema from Schema, deftest from clojure.test. Generally anything that is a convenience form for creating vars. You could use a macro solution to create defs for your atoms:
(defmacro defneighbours [n]
`(do
~#(for [sym (for [i (range n)]
(symbol (str "neig" i)))]
`(def ~sym (atom {}))))
In my opinion this is actually less offensive than a functional version, only because it is creating global defs. It is a little more obvious about creating global defs by using the regular def syntax. But I only bring it up as a strawman, because this is still bad.
The reason functions and data work best is because they compose.
There are tangible considerations that make a single atom governing state very convenient. You can iterate over all neighbors conveniently, you can add new ones dynamically. Also you can do things like concatenating neighbors with other neighbors etc. Basically there are lots of function/data abstractions that you lock yourself out of if you create many global vars.
This is the reason that macros are generally considered useful for syntactic tricks, but best avoided in favor of functions and data. And it has a real impact on the flexibility of your code. For example going back to compojure; the macro syntax is actually very limiting, and for that reason I prefer not to use defroutes at all.
In summary:
Don't make lots of global defs if you can avoid it.
Prefer 1 atom over many atoms where possible.
Don't def inside a function.
Macros are best avoided in favor of functions and data.
Regardless of these guidelines, it is always good to explore what is possible, and I can't know your circumstances, so above all I hope you overcome your immediate problem and find Clojure a pleasant language to use.
I find it hard to reason about macro-expansion and was wondering what the best practices were for testing them.
So if I have a macro, I can perform one level of macro expansion via macroexpand-1.
(defmacro incf-twice (n)
`(progn
(incf ,n)
(incf ,n)))
for example
(macroexpand-1 '(incf-twice n))
evaluates to
(PROGN (INCF N) (INCF N))
It seems simple enough to turn this into a test for the macro.
(equalp (macroexpand-1 '(incf-twice n))
'(progn (incf n) (incf n)))
Is there an established convention for organizing tests for macros? Also, is there a library for summarizing differences between s-expressions?
Generally testing macros is not one of the strong parts of Lisp and Common Lisp. Common Lisp (and Lisp dialects in general) uses procedural macros. The macros can depend on the runtime context, the compile-time context, the implementation and more. They also can have side effects (like registering things in the compile-time environment, registering things in the development environment and more).
So one might want to test:
that the right code gets generated
that the generated code actually does the right thing
that the generated code actually works in code contexts
that the macro arguments are actually parsed correctly in case of complex macros. Think loop, defstruct, ... macros.
that the macro detects wrongly formed argument code. Again, think of macros like loop and defstruct.
the side effects
From above list on can infer that it is best to minimize all these problem areas when developing a macro. BUT: there are really really complex macros out there. Really scary ones. Especially those who are used to implemented new domain specific languages.
Using something like equalp to compare code works only for relatively simple macros. Macros often introduce new, uninterned and unique symbols. Thus equalp will fail to work with those.
Example: (rotatef a b) looks simple, but the expansion is actually complicated:
CL-USER 28 > (pprint (macroexpand-1 '(rotatef a b)))
(PROGN
(LET* ()
(LET ((#:|Store-Var-1234| A))
(LET* ()
(LET ((#:|Store-Var-1233| B))
(PROGN
(SETQ A #:|Store-Var-1233|)
(SETQ B #:|Store-Var-1234|))))))
NIL)
#:|Store-Var-1233| is a symbol, which is uninterned and newly created by the macro.
Another simple macro form with a complex expansion would be (defstruct s b).
Thus one would need a s-expression pattern matcher to compare the expansions. There are a few available and they would be useful here. One needs to make sure in the test patterns that the generated symbols are identical, where needed.
There are also s-expression diff tools. For example diff-sexp.
I agree with Rainer Joswig's answer; in general, this is a very difficult task to solve because macros can do a whole lot. However, I would point out that in many cases, the easiest way to unit test your macros is by making the macros do as little as possible. In many cases, the easiest implementation of a macro is just syntactic sugar around a simpler function. E.g., there's a typical pattern of with-… macros in Common Lisp (e.g., with-open-file), where the macro simply encapsulates some boilerplate code:
(defun make-frob (frob-args)
;; do something and return the resulting frob
(list 'frob frob-args))
(defun cleanup-frob (frob)
(declare (ignore frob))
;; release the resources associated with the frob
)
(defun call-with-frob (frob-args function)
(let ((frob (apply 'make-frob frob-args)))
(unwind-protect (funcall function frob)
(cleanup-frob frob))))
(defmacro with-frob ((var &rest frob-args) &body body)
`(call-with-frob
(list ,#frob-args)
(lambda (,var)
,#body)))
The first two functions here, make-frob and cleanup-frob are relatively straightforward to unit test. The call-with-frob is a bit harder. The idea is that it's supposed to handle the boilerplate code of creating the frob and ensuring that the cleanup call happens. That's a bit harder to check, but if the boilerplate only depends on some well defined interfaces, then you'll probably be able to create mock up a frob that can detect whether it's cleaned up correctly. Finally, the with-frob macro is so simple that you can probably test it the way you've been considering, i.e., checking its expansion. Or you might say that it's simple enough that you don't need to test it.
On the other hand, if you're looking at a much more complex macro, such as loop, which is really a kind of compiler in its own right, you're almost certainly already going to have the expansion logic in some separate functions. E.g., you might have
(defmacro loop (&body body)
(compile-loop body))
in which case you really don't need to test loop, you need to test compile-loop, and then you're back in the realm of your usual unit testing.
I'd generally just test the functionality, not the shape of the expansion.
Yes, there are all kinds of contexts and surroundings that might influence what happens, but if you rely on such things, it should be no problem to set them up the same for your test.
Some common cases:
binding macros: test that the variables are bound as intended inside and that any shadowed outside variables are unaffected
unwind-protect wrappers: provoke a nonlocal exit from the inside and check that the cleanup is working
definition/registration: test that you can define/register what you want and use it afterwards
This is not my 'production code' but a simplication of the problem for illustration purposes. Also, the title of this question is misleading because it brings to mind the ~# expansion, which I understand, and which may not necessarily be the problem. Please suggest a better question title if you can.
Given a macro with the following form:
(defmacro my-add [x & ys] `(+ ~x ~#ys))
Now let's say we have a list:
(def my-lst '(2 3))
Now I want a function that uses my-add that I can pass my-lst to as an arg, i.e.
(call-my-add 1 my-lst)
I define the function in what would seem to be the obvious way:
(defn call-my-add [x ys]
(apply my-add (cons x ys)))
But:
java.lang.Exception: Can't take value of a macro: #'user/call-my-add (repl-1:60)
I've tried all sorts of wild tricks to get the call-my-add function to work using evals, applies, and even defining call-my-add as a macro, but they all give similar ClassCastExceptions.
Is there any way out of this?
No. Macros do not, cannot, will never, have access to the actual run-time values contained in their arguments, so cannot splice them into the expansion. All they get is the symbol(s) you pass to them, in this case my-list. The "way around this" is to define my-add as a function, and then (optionally) have a macro that calls that function in order to generate its code.
I wrote a blog post about this semi-recently that you might find enlightening.
You could do it with evals if you wanted to, but that is a terrible idea in almost every case:
(let [my-list '(1 2)]
(eval `(my-add 5 ~#my-list)))
What a great example showing that macros are not first class citizens in Clojure (or any Lisp that I know of). They cannot be applied to functions, stored in containers, or passed to functions etc. In exchange for this they get to control when and if their arguments are evaluated.
What happens at macro expansion time must stay in macro expansion time. So if my-add is being evaluated at macro expansion time and you want to use apply then you need... another macro; to do the applying.
(defmacro call-my-add [x ys]
`(my-add ~#(cons x ys)))
Macros are somewhat contagious in this way.
PS: I'm not at my repl so please edit if you see a bug in this example (or I'll fix it when I get back)