Why does this clojure macro need `'~? - clojure

Question

(Apologies if this is a duplicate of another question, my search for all those fancy special characters didn't yield anything.)
I'm reading Mastering Clojure Macros and have trouble understanding the following example:
(defmacro inspect-caller-locals []
(->> (keys &env)
(map (fn [k] [`'~k k]))
(into {})))
=> #'user/inspect-caller-locals
(let [foo "bar" baz "quux"]
(inspect-caller-locals))
=> {foo "bar", baz "quux"}
What is the difference between the following and the much simpler 'k?
`'~k
As far as I understand, the innermost unquote ~ should simply reverts the effect of the outermost syntax-quote `, but a short experiment reveals that there's more to it:
(defmacro inspect-caller-locals-simple []
(->> (keys &env)
(map (fn [k] ['k k]))
(into {})))
=> #'user/inspect-caller-locals-simple
(let [foo "bar" baz "quux"]
(inspect-caller-locals-simple))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: k in this context, compiling:(/tmp/form-init4400591386630133028.clj:2:3)
Unfortunately, my usual investigation approach doesn't apply here:
(macroexpand '(let [foo "bar" baz "quux"]
(inspect-caller-locals)))
=> (let* [foo "bar" baz "quux"] (inspect-caller-locals))
(let [foo "bar" baz "quux"]
(macroexpand '(inspect-caller-locals)))
=> {}
What am I missing here?

Answer 1

Let's first establish what the k inside the macro is:
(defmacro inspect-caller-locals []
(mapv (fn [k]
(println (class k)))
(keys &env))
nil)
(let [x 1]
(inspect-caller-locals))
;; Prints:
;; clojure.lang.Symbol
So you each k inside the function is a symbol. If you return a symbol from a macro (ie generate code from it), clojure will lookup the value that it refers to and print it. For instance you could do this:
(defmacro inspect-caller-locals []
(mapv (fn [k]
[(quote x) k]) ;; not the "hard coded" x
(keys &env)))
(let [x 1]
(inspect-caller-locals))
;; Prints:
;; [[1 1]]
What you want however is the actual symbol. The problem (as you noted) is that quote is a special form that DOES NOT EVALUTE whatever you pass it. Ie, the k will not obtain the function parameter but stay k which is not usually defined:
(defmacro inspect-caller-locals []
(mapv (fn [k]
[(quote k) k])
(keys &env)))
(let [x 1]
(inspect-caller-locals))
;; => Error
(let [k 1]
(inspect-caller-locals))
;; Prints:
;; [[1 1]]
You somehow need to evaluate what you pass into quote, this is not however possible since that isn't what quote does. Other functions, such as str don't have that problem:
(defmacro inspect-caller-locals []
(mapv (fn [k]
[(str k) k])
(keys &env)))
(let [x 1]
(inspect-caller-locals))
;; Prints:
;; [["x" 1]]
The trick is to go one level deeper and quote the quote itself so you can pass the symbol to it:
(defmacro inspect-caller-locals []
(mapv (fn [k]
[;; This will evaluate k first but then generate code that
;; wraps that symbol with a quote:
(list (quote quote) k)
;; Or equivalently and maybe easier to understand:
(list 'quote k)
k])
(keys &env)))
(let [x 1]
(inspect-caller-locals))
;; Prints:
;; [[x x 1]]
Or by using the reader that can do this for you:
(defmacro inspect-caller-locals []
(mapv (fn [k]
[`(quote ~k)
`'~k
k])
(keys &env)))
(let [x 1]
(inspect-caller-locals))
;; Prints:
;; [[x x 1]]
Because after all:
(read-string "`'~k")
=> (clojure.core/seq (clojure.core/concat (clojure.core/list (quote quote)) (clojure.core/list k)))
(defmacro inspect-caller-locals []
(mapv (fn [k]
[(clojure.core/seq (clojure.core/concat (clojure.core/list (quote quote)) (clojure.core/list k)))
k])
(keys &env)))
(let [x 1]
(inspect-caller-locals))
;; Prints:
;; [[x 1]]

Answer 2

Some alternative, and equivalent, ways of writing
`'~k
are:
`(quote ~k) ;; expands the ' reader macro to the quote special form
(list 'quote k) ;; avoids syntax quote entirely
You are pretty much right to think that
the innermost unquote ~ should simply reverts the effect of the outermost syntax-quote
The only thing missing from your description there is that you can't pull quote outside of a syntax-quoted expression, since quote is a special form and changes the meaning of what's inside. Otherwise,
'`~k
would be equivalent to 'k - and as you noticed, it's not!
I'll echo #amalloy's general advice, that trying syntax-quoted stuff in the REPL, outside of the context of macros/macroexpansion, is the best way to get your head around these things.
p.s. Also, I'll make a note that I need to fix this confusion by explaining better in a future book edition ;)