In Clojure,
(def x 3)
(eval '(prn x))
prints 3, whereas
(let [y 3]
(eval '(prn y)))
and
(binding [z 3] (eval '(prn z)))
generate an 'Unable to resolve var' exception.
According to http://clojure.org/evaluation, eval, load-string, etc generate temporary namespaces to evaluate their contents. Therefore, I'd expect neither of the above code samples to work, since (def x 3) is done in my current namespace, not the one created by eval.
Why does the first code sample work and not the last two?
How can I eval a form with bound variables without using def?
Thanks!
1.:
The reason this doesn't work is (more or less) given on the page you linked:
It is an error if there is no global var named by the symbol […]
And:
[…]
A lookup is done in the current namespace to see if there is a mapping
from the symbol to a var. If so, the
value is the value of the binding of
the var referred-to by the symbol.
It is an error.
eval evaluates forms in an empty (null in CL-lingo) lexical environment. This means, that you cannot access lexical variable bindings from the caller's scope. Also, binding creates new bindings for existing vars, which is why you cannot use it "by itself", without having declared or defed the variables you try to bind. Besides, lexical variables (at least in CL, but I would be surprised if this wasn't the case for Clojure) already ceased to exist at runtime – They are translated to addresses or values.
See also my older post about this topic.
2.:
So, you have to use dynamic variables. You can avoid the explicit def, but you still at least need to declare them (which defs var names without bindings):
user=> (declare ^:dynamic x)
#'user/x
user=> (binding [x 10] (eval '(prn x)))
10
nil
By the way: I suppose you know why you need eval, and that its use is considered evil when other solutions would be appropriate.
Related
Recently I came across a use for eval within a macro, which I understand is a bit of a faux pas but let's ignore that for now. What I found surprising, was that eval was able to resolve global vars at macroexpansion time. Below is a contrived example, just to illustrate the situation I'm referring to:
(def list-of-things (range 10))
(defmacro force-eval [args]
(apply + (eval args)))
(macroexpand-1 '(force-eval list-of-things))
; => 45
I would have expected args to resolve to the symbol list-of-things inside force-eval, and then list-of-things to be evaluated resulting in an error due to it being unbound:
"unable to resolve symbol list-of-things in this context"
However, instead list-of-things is resolved to (range 10) and no error is thrown - the macroexpansion succeeds.
Contrast this with attempting to perform the same macroexpansion, but within a local binding context:
(defmacro force-eval [args]
(apply + (eval args)))
(let [list-of-things (range 10)]
(macroexpand-1 '(force-eval list-of-things)))
; => Unable to resolve symbol: list-of-thingss in this context
Note in the above examples I'm assuming list-of-things is not previously bound, e.g. a fresh REPL. One final example illustrates why this is important:
(defmacro force-eval [args]
(apply + (eval args)))
(def list-of-things (range 10 20))
(let [list-of-thing (range 10)]
(macroexpand-1 '(force-eval list-of-things)))
; => 145
The above example shows that the locals are ignored, which is expected behavior for eval, but is a bit confusing when you are expecting the global to not be available at macroexpansion time either.
I seem to have a misunderstanding about what exactly is available at macroexpansion time. I had previously thought that essentially any binding, be it global or local, would not be available until runtime. Apparently this is an incorrect assumption. Is the answer to my confusion simply that global vars are available at macroexpansion time? Or am I missing some further nuance here?
Note: this related post closely describes a similar problem, but the focus there is more on how to avoid inappropriate use of eval. I'm mainly interested in understanding why eval works in the first example and by extension what's available to eval at macroexpansion time.
Of course, vars must be visible at compile time. That's where functions like first and + are stored. Without them, you couldn't do anything.
But keep in mind that you have to make sure to refer to them correctly. In the repl, *ns* will be bound, and so a reference to a symbol will look in the current namespace. If you are running a program through -main instead of the repl, *ns* will not be bound, and only properly qualified vars will be found. You can ensure that you qualify them correctly by using
`(force-eval list-of-things)
instead of
'(force-eval list-of-things)
Note I do not distinguish between global vars and non-global vars. All vars in Clojure are global. Local bindings are not called vars. They're called locals, or bindings, or variables, or some combination of those words.
Clojure is designed with an incremental compilation model. This is poorly documented.
In C and other traditional languages, source code must be compiled, then linked with pre-compiled libraries before the final result can be executed. Once execution begins, no changes to the code can occur until the program is terminated, when new source code can be compiled, linked, then executed. Java is normally used in this manner just like C.
With the Clojure REPL, you can start with zero source code in a live executing environment. You can call existing functions like (+ 2 3), or you can define new functions and variables on the fly (both global & local), and redefine existing functions. This is only possible because core Clojure is already available (i.e. clojure.core/+ etc is already "installed"), so you can combine these functions to define your own new functions.
The Clojure "compiler" works just like a giant REPL session. It reads and evaluates forms from your source code files one at a time, incrementally adding them the the global environment. Indeed, it is a design goal/requirement that the result of compiling and executing source code is identical to what would occur if you just pasted each entire source code file into the REPL (in proper dependency order).
Indeed, the simplest mental model for code execution in Clojure is to pretend it is an interpreter instead of a traditional compiler.
And eval in a macro makes no sense.
Because:
a macro already implicitely contains an eval
at the very final step.
If you use macroexpand-1, you make visible how the code was manipulated in the macro before the evocation of the implicite eval inside the macro.
An eval in a macro is an anti-pattern which might indicate that you should use a function instead of a macro - and in your examle this is exactly the case.
So your aim is to dynamically (in run-time) evoke sth in a macro. This you can only do through an eval applied over a macro call OR you should rather use a function.
(defmacro force-eval [args]
(apply + (eval args)))
;; What you actually mean is:
(defn force-eval [args]
(apply + args))
;; because a function in lisp evaluates its arguments
;; - before applying the function body.
;; That means: args in the function body is exactly
;; `(eval args)`!
(def list-of-things (range 10))
(let [lit-of-things (range 10 13)]
(force-eval list-of-things))
;; => 45
;; so this is exactly the behavior you wanted!
The point is, your construct is a "bad" example for a macro.
Because apply is a special function which allows you to
dynamically rearrange function call structures - so it has
some magic of macros inside it - but in run-time.
With apply you can do quite some meta programming in some cases when you just quote some of your input arguments.
(Try (force-eval '(1 2 3)) it returns 6. Because the (1 2 3) is put together with + at its front by apply and then evaluated.)
The second point - I am thinking of this answer I once gave and this to a dynamic macro call problem in Common Lisp.
In short: When you have to control two levels of evaluations inside a macro (often when you want a macro inject some code in runtime into some code), you need too use eval when calling the macro and evaluate those parts in the macro call which then should be processed in the macro.
I'm always a bit confused about Symbols and Vars in Clojure.
For example, is it safe to say that + is a symbol which is used to denote a var, and this var points to a value which is a function that can add numbers?
So what happens, step by step when I just enter "+" in a REPL?
The symbol gets qualified to a namespace, in this case clojure.core
Then in some symbol table there is the information that + refers to a var
When this var is evaluated, the result is a function-value?
There's a symbol + that you can talk about by quoting it:
user=> '+
+
user=> (class '+)
clojure.lang.Symbol
user=> (resolve '+)
#'clojure.core/+
So it resolves to #'+, which is a Var:
user=> (class #'+)
clojure.lang.Var
The Var references the function object:
user=> (deref #'+)
#<core$_PLUS_ clojure.core$_PLUS_#55a7b0bf>
user=> ##'+
#<core$_PLUS_ clojure.core$_PLUS_#55a7b0bf>
(The # sign is just shorthand for deref.) Of course the usual way to get to the function is to not quote the symbol:
user=> +
#<core$_PLUS_ clojure.core$_PLUS_#55a7b0bf>
Note that lexical bindings are a different mechanism, and they can shadow Vars, but you can bypass them by referring to the Var explicitly:
user=> (let [+ -] [(+ 1 2) (##'+ 1 2)])
[-1 3]
In that last example the deref can even be left out:
user=> (let [+ -] [(+ 1 2) (#'+ 1 2)])
[-1 3]
This is because Var implements IFn (the interface for Clojure functions) by calling deref on itself, casting the result to IFn and delegating the function call to that.
The visibility mechanism used when you define private functions with defn- is based on metadata on the symbol. You can bypass it by referring directly to the Var, as above:
user=> (ns foo)
nil
foo=> (defn- private-function [] :secret)
#'foo/private-function
foo=> (in-ns 'user)
#<Namespace user>
user=> (foo/private-function)
java.lang.IllegalStateException: var: #'foo/private-function is not public (NO_SOURCE_FILE:36)
user=> (#'foo/private-function)
:secret
See the documentation for namespaces:
Namespaces are mappings from simple (unqualified) symbols to Vars and/or Classes. Vars can be interned in a namespace, using def or any of its variants, in which case they have a simple symbol for a name and a reference to their containing namespace, and the namespace maps that symbol to the same var. A namespace can also contain mappings from symbols to vars interned in other namespaces by using refer or use, or from symbols to Class objects by using import.
So basically your steps 1 and 2 are unified: the namespaces are the symbol tables.
And regarding step 3: I like the definition for variables that they're combinations of names of values. The symbol is the variable's name, and evaluating it will result in its value.
This answer is not very different from the other ones, it just doesn't assume you originally wish to learn several new functions and concepts just to understand what's going on:
+ is a symbol sitting in clojure.core which is by default accessible to your code.
When used in your code without any very advanced intents such as quoting it or finding out its class ― clojure will seek the Var which it points to.
If this Var is a function, when + is used in head position of a list,
clojure will try to call that function (NullPointerException if this Var happened not to point at a function). If supplied as an argument to
another function, that function may do the same to call it. That's
how function calling works.
further comments to round up:
Most or all languages use symbol tables. Being a somewhat dynamic language, Clojure uses this extra layer of indirection (Symbol → Var → function, rather than only Symbol → function) so that dynamically rewriting which function is tied to which symbol ― is more feasible and elegant, and this is sometimes a source of curiosity for beginners.
As other answers have somewhat overly emphasized, you may otherwise perform fancy stuff like quote it ('+) to avoid its evaluation, or even inspect it with class and/or resolve as if you are interested in verifying what it is (class), or which namespace it sits in (resolve). You can also poke at the var it points at via var or #'. You'll typically do those fancy things if you are writing macros or if you're very experimentally inclined, especially when working in the repl; depending on which style you write your macros, you might actually quote quite a lot in them.
and a fancy illustration for the extra-exploratory person:
Being a somewhat flexible language, clojure exposes api for taking the Symbol → Var → function walk on your own. You'd not typically ever do that for just using a function, because obviously that would be boring and redundant, but it can be used here to illustrate the process:
(deref (resolve '+))
Namely, the symbol is first resolved to its Var, then the thing that the Var points to, is arrived at. This just illustrates the 2-step process for arriving at a function (Symbol → Var → function), which happens behind the scenes. I hope you've avoided reading this extra part.
TL;DR
the answer to the original question is simply: Yes.
I find an understanding of the difference between symbols, functions, literals, and vars is necessary to grok what's happening.
(def one (fn [] 1)) ⇒ #'example/one. It references the function #function[example/one]
(def x one) ⇒ #'example/x references the function #function[example/one]
(def y 'one) ⇒ #'example/y references the symbol one
(def z #'one) ⇒ #'example/z references the var #'example/one
To put it precicely one is a symbol that resolves to the var #'example/one. The var references a function, #function[example/one], which returns the literal 1.
Each def yields a var. The var is denoted by the #' syntax in #'example/x. Each var references a value.
According to Clojure's own documentation, symbols resolve to a value, a special form, or an error when evaluated. So it can be a little confusing because there is no mention of a var:
(type one) ⇒ example$one
(type x) ⇒ example$one
(type y) ⇒ clojure.lang.Symbol
(type z) ⇒ clojure.lang.Var
In the cases above, the value "is the value of the binding of the global var named by the symbol." As another answer stated more succinctly: symbol → var → value.
(one) ⇒ 1
(x) ⇒ 1
(y) ⇒ unrelated associative lookup error
(z) ⇒ 1
Remember: #'example/z references the var #'example/one. #'example/x and #'example/y both reference #function[example/one]. The significance of this difference is revealed when the original symbol is interned to a new function: (def one (fn [] 2)).
(one) ⇒ 2 ← new value
(x) ⇒ 1 ← original value
(y) ⇒ unrelated associative lookup error
(z) ⇒ 2 ← new value
Following the logic of symbol → var → value:
x → (var x) → (defn one [] 1)
x resolves to (var x) ("the global var named by the symbol")
(var x) dereferences the current binding of x: (fn [] 1) ("the binding of the global var named by the symbol")
z → (var z) → (var one).
z resolves to its var, which dereferences to (var one)
one resolves to its var, which dereferences to (fn [] 2)
One final note. #'example/one still evaluates and returns a literal (1 or 2) because the var is in the first position of a list ((z)). This behavior is similar to placing a keyword in the first position of a list and a map in the second: (:two {:one 1 :two 2}) ⇒ 2.
call
(ns-map *ns*)
to get a map of all available symbols in your namespace and vars they are pointing to
The code below prints 10 as expected.
(def x 10)
(let [y 30] (eval (symbol "x")))
The code below generates an exception:
(let [y 20] (eval (symbol "y")))
Unable to resolve symbol: y in this context
which is expected (but confusing!). According to the documentation, symbols defined by let do not belong to any namespace and so can not be resolved through namespace mechanism.
So the question is: what should be an equivalent of function symbol for local variables.
Additionally:
I thought that Clojure compiler internally calls function symbol for each identifier to "intern" it, but as the example above shows it is not the case. Curious what the compiler is actually doing with the local identifiers. I assumed that when I enter x in REPL
x
is is essentially processed as this:
(deref (resolve (symbol "x")))
but again evidently it is not the case for local variables.
PS: Symbols in Clojure does not cover local variables.
All input to the clojure compiler is read, to form lists, symbols, keywords, numbers, and other readable data (eg. if you use a hash map literal, the compiler will get a hash map).
For example:
user=> (eval '(let [y 20] y))
20
In this case, we give the compiler a list starting with the symbol let (which resolves to a var macro wrapping a special form).
When you ask "what should be an equivalent of function symbol for local variables", my first thought is that you are misunderstanding what the function symbol is for. The following is equivalent to my initial example:
user=> (eval (list (symbol "let") [(symbol "y") 20] (symbol "y")))
20
symbol is only incidentally useful for getting a var from a string. In fact this usage is usually a hack, and a sign you are doing something wrong. Its primary purpose is to construct input for the compiler. Writing a form that gets a binding from its lexical scope is best done by writing a function, and having the user pass in the value to be used. History shows us that implicitly using locals in the callers environment is messy and error prone, and it is not a feature that Clojure explicitly supports (though there are definitely hacks that will work, which will be based on implementation details that are not guaranteed to behave properly the next release of the language).
I'm always a bit confused about Symbols and Vars in Clojure.
For example, is it safe to say that + is a symbol which is used to denote a var, and this var points to a value which is a function that can add numbers?
So what happens, step by step when I just enter "+" in a REPL?
The symbol gets qualified to a namespace, in this case clojure.core
Then in some symbol table there is the information that + refers to a var
When this var is evaluated, the result is a function-value?
There's a symbol + that you can talk about by quoting it:
user=> '+
+
user=> (class '+)
clojure.lang.Symbol
user=> (resolve '+)
#'clojure.core/+
So it resolves to #'+, which is a Var:
user=> (class #'+)
clojure.lang.Var
The Var references the function object:
user=> (deref #'+)
#<core$_PLUS_ clojure.core$_PLUS_#55a7b0bf>
user=> ##'+
#<core$_PLUS_ clojure.core$_PLUS_#55a7b0bf>
(The # sign is just shorthand for deref.) Of course the usual way to get to the function is to not quote the symbol:
user=> +
#<core$_PLUS_ clojure.core$_PLUS_#55a7b0bf>
Note that lexical bindings are a different mechanism, and they can shadow Vars, but you can bypass them by referring to the Var explicitly:
user=> (let [+ -] [(+ 1 2) (##'+ 1 2)])
[-1 3]
In that last example the deref can even be left out:
user=> (let [+ -] [(+ 1 2) (#'+ 1 2)])
[-1 3]
This is because Var implements IFn (the interface for Clojure functions) by calling deref on itself, casting the result to IFn and delegating the function call to that.
The visibility mechanism used when you define private functions with defn- is based on metadata on the symbol. You can bypass it by referring directly to the Var, as above:
user=> (ns foo)
nil
foo=> (defn- private-function [] :secret)
#'foo/private-function
foo=> (in-ns 'user)
#<Namespace user>
user=> (foo/private-function)
java.lang.IllegalStateException: var: #'foo/private-function is not public (NO_SOURCE_FILE:36)
user=> (#'foo/private-function)
:secret
See the documentation for namespaces:
Namespaces are mappings from simple (unqualified) symbols to Vars and/or Classes. Vars can be interned in a namespace, using def or any of its variants, in which case they have a simple symbol for a name and a reference to their containing namespace, and the namespace maps that symbol to the same var. A namespace can also contain mappings from symbols to vars interned in other namespaces by using refer or use, or from symbols to Class objects by using import.
So basically your steps 1 and 2 are unified: the namespaces are the symbol tables.
And regarding step 3: I like the definition for variables that they're combinations of names of values. The symbol is the variable's name, and evaluating it will result in its value.
This answer is not very different from the other ones, it just doesn't assume you originally wish to learn several new functions and concepts just to understand what's going on:
+ is a symbol sitting in clojure.core which is by default accessible to your code.
When used in your code without any very advanced intents such as quoting it or finding out its class ― clojure will seek the Var which it points to.
If this Var is a function, when + is used in head position of a list,
clojure will try to call that function (NullPointerException if this Var happened not to point at a function). If supplied as an argument to
another function, that function may do the same to call it. That's
how function calling works.
further comments to round up:
Most or all languages use symbol tables. Being a somewhat dynamic language, Clojure uses this extra layer of indirection (Symbol → Var → function, rather than only Symbol → function) so that dynamically rewriting which function is tied to which symbol ― is more feasible and elegant, and this is sometimes a source of curiosity for beginners.
As other answers have somewhat overly emphasized, you may otherwise perform fancy stuff like quote it ('+) to avoid its evaluation, or even inspect it with class and/or resolve as if you are interested in verifying what it is (class), or which namespace it sits in (resolve). You can also poke at the var it points at via var or #'. You'll typically do those fancy things if you are writing macros or if you're very experimentally inclined, especially when working in the repl; depending on which style you write your macros, you might actually quote quite a lot in them.
and a fancy illustration for the extra-exploratory person:
Being a somewhat flexible language, clojure exposes api for taking the Symbol → Var → function walk on your own. You'd not typically ever do that for just using a function, because obviously that would be boring and redundant, but it can be used here to illustrate the process:
(deref (resolve '+))
Namely, the symbol is first resolved to its Var, then the thing that the Var points to, is arrived at. This just illustrates the 2-step process for arriving at a function (Symbol → Var → function), which happens behind the scenes. I hope you've avoided reading this extra part.
TL;DR
the answer to the original question is simply: Yes.
I find an understanding of the difference between symbols, functions, literals, and vars is necessary to grok what's happening.
(def one (fn [] 1)) ⇒ #'example/one. It references the function #function[example/one]
(def x one) ⇒ #'example/x references the function #function[example/one]
(def y 'one) ⇒ #'example/y references the symbol one
(def z #'one) ⇒ #'example/z references the var #'example/one
To put it precicely one is a symbol that resolves to the var #'example/one. The var references a function, #function[example/one], which returns the literal 1.
Each def yields a var. The var is denoted by the #' syntax in #'example/x. Each var references a value.
According to Clojure's own documentation, symbols resolve to a value, a special form, or an error when evaluated. So it can be a little confusing because there is no mention of a var:
(type one) ⇒ example$one
(type x) ⇒ example$one
(type y) ⇒ clojure.lang.Symbol
(type z) ⇒ clojure.lang.Var
In the cases above, the value "is the value of the binding of the global var named by the symbol." As another answer stated more succinctly: symbol → var → value.
(one) ⇒ 1
(x) ⇒ 1
(y) ⇒ unrelated associative lookup error
(z) ⇒ 1
Remember: #'example/z references the var #'example/one. #'example/x and #'example/y both reference #function[example/one]. The significance of this difference is revealed when the original symbol is interned to a new function: (def one (fn [] 2)).
(one) ⇒ 2 ← new value
(x) ⇒ 1 ← original value
(y) ⇒ unrelated associative lookup error
(z) ⇒ 2 ← new value
Following the logic of symbol → var → value:
x → (var x) → (defn one [] 1)
x resolves to (var x) ("the global var named by the symbol")
(var x) dereferences the current binding of x: (fn [] 1) ("the binding of the global var named by the symbol")
z → (var z) → (var one).
z resolves to its var, which dereferences to (var one)
one resolves to its var, which dereferences to (fn [] 2)
One final note. #'example/one still evaluates and returns a literal (1 or 2) because the var is in the first position of a list ((z)). This behavior is similar to placing a keyword in the first position of a list and a map in the second: (:two {:one 1 :two 2}) ⇒ 2.
call
(ns-map *ns*)
to get a map of all available symbols in your namespace and vars they are pointing to
How do I programmatically figure out which Vars may affect the results of a function defined in Clojure?
Consider this definition of a Clojure function:
(def ^:dynamic *increment* 3)
(defn f [x]
(+ x *increment*))
This is a function of x, but also of *increment* (and also of clojure.core/+(1); but I'm less concerned with that). When writing tests for this function, I want to make sure that I control all relevant inputs, so I do something like this:
(assert (= (binding [*increment* 3] (f 1)) 4))
(assert (= (binding [*increment* -1] (f 1)) 0))
(Imagine that *increment* is a configuration value that someone might reasonably change; I don't want this function's tests to need changing when this happens.)
My question is: how do I write an assertion that the value of (f 1) can depend on *increment* but not on any other Var? Because I expect that one day someone will refactor some code and cause the function to be
(defn f [x]
(+ x *increment* *additional-increment*))
and neglect to update the test, and I would like to have the test fail even if *additional-increment* is zero.
This is of course a simplified example – in a large system, there can be lots of dynamic Vars, and they can get referenced through a long chain of function calls. The solution needs to work even if f calls g which calls h which references a Var. It would be great if it didn't claim that (with-out-str (prn "foo")) depends on *out*, but this is less important. If the code being analyzed calls eval or uses Java interop, of course all bets are off.
I can think of three categories of solutions:
Get the information from the compiler
I imagine the compiler does scan function definitions for the necessary information, because if I try to refer to a nonexistent Var, it throws:
user=> (defn g [x] (if true x (+ *foobar* x)))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: *foobar* in this context, compiling:(NO_SOURCE_PATH:24)
Note that this happens at compile time, and regardless of whether the offending code will ever be executed. Thus the compiler should know what Vars are potentially referenced by the function, and I would like to have access to that information.
Parse the source code and walk the syntax tree, and record when a Var is referenced
Because code is data and all that. I suppose this means calling macroexpand and handling each Clojure primitive and every kind of syntax they take. This looks so much like a compilation phase that it would be great to be able to call parts of the compiler, or somehow add my own hooks to the compiler.
Instrument the Var mechanism, execute the test and see which Vars get accessed
Not as complete as the other methods (what if a Var is used in a branch of the code that my test fails to exercise?) but this would suffice. I imagine I would need to redefine def to produce something that acts like a Var but records its accesses somehow.
(1) Actually that particular function doesn't change if you rebind +; but in Clojure 1.2 you can bypass that optimization by making it (defn f [x] (+ x 0 *increment*)) and then you can have fun with (binding [+ -] (f 3)). In Clojure 1.3 attempting to rebind + throws an error.
Regarding your first point you could consider using the analyze library. With it you can quite easily figure out which dynamic vars are used in an expression:
user> (def ^:dynamic *increment* 3)
user> (def src '(defn f [x]
(+ x *increment*)))
user> (def env {:ns {:name 'user} :context :eval})
user> (->> (analyze-one env src)
expr-seq
(filter (op= :var))
(map :var)
(filter (comp :dynamic meta))
set)
#{#'user/*increment*}
I know that this doesn't answer your question, but wouldn't it be a lot less work to just provide two versions of a function where one version has no free variables, and the other version calls the first one with the appropriate top-level defines?
For example:
(def ^:dynamic *increment* 3)
(defn f
([x]
(f x *increment*))
([x y]
(+ x y)))
This way you can write all your tests against (f x y), which doesn't rely on any global state.