I want to embed a Java object (in this case a BufferedImage) in Clojure code that can be evald later.
Creating the code works fine:
(defn f [image]
`(.getRGB ~image 0 0))
=> #'user/f
(f some-buffered-image)
=> (.getRGB #<BufferedImage BufferedImage#5527f4f9: type = 2 DirectColorModel: rmask=ff0000 gmask=ff00 bmask=ff amask=ff000000 IntegerInterleavedRaster: width = 256 height = 256 #Bands = 4 xOff = 0 yOff = 0 dataOffset[0] 0> 0 0)
However you get an exception when trying to eval it:
(eval (f some-buffered-image))
=> CompilerException java.lang.RuntimeException: Can't embed object in code, maybe print-dup not defined: BufferedImage#612dcb8c: type = 2 DirectColorModel: rmask=ff0000 gmask=ff00 bmask=ff amask=ff000000 IntegerInterleavedRaster: width = 256 height = 256 #Bands = 4 xOff = 0 yOff = 0 dataOffset[0] 0, compiling:(NO_SOURCE_PATH:1)
Is there any way to make something like this work?
EDIT:
The reason I am trying to do this is that I want to be able to generate code that takes samples from an image. The image is passed to the function that does the code generation (equivalent to f above), but (for various reasons) can't be passed as a parameter to the compiled code later.
I need to generate quoted code because this is part of a much larger code generation library that will apply further transforms to the generated code, hence I can't just do something like:
(defn f [image]
(fn [] (.getRGB image 0 0)))
Not sure what you need it for, but you can create code that evals to an arbitrary object using the following cheat:
(def objs (atom []))
(defn make-code-that-evals-to [x]
(let [
nobjs (swap! objs #(conj % x))
i (dec (count nobjs))]
`(nth ~i #objs)))
Then you can:
> (eval (make-code-that-evals-to *out*))
#<PrintWriter java.io.PrintWriter#14aed2c>
This is just a proof of concept and it leaks the objects being produced - you could produce code that removes the reference on eval but then you could eval it only once.
Edit: The leaks can be prevented by the following (evil!) hack:
The above code bypasses eval's compiler by storing the object reference externally at the time the code is generated. This can be deferred. The object reference can be stored in the generated code with the compiler being bypassed by a macro. Storing the reference in the code means the garbage collector works normally.
The key is the macro that wraps the object. It does what the original solution did (i.e. store the object externally to bypass the compiler), but just before compilation. The generated expression retrieves the external reference, then deletes to prevent leaks.
Note: This is evil. Expansion-time of macros is the least desirable place for global side-effects to occur and this is exactly what this solution does.
Now for the code:
(def objs (atom {}))
Here's where will temporarily store the objects, keyed by unique keys (dictionary).
(defmacro objwrap [x sym]
(do
(swap! objs #(assoc % sym x) ) ; Global side-effect in macro expansion
`(let [o# (#objs ~sym)]
(do
(swap! objs #(dissoc % ~sym))
o#))))
This is the evil macro that sits in the generated code, keeping the object reference in x and a unique key in sym. Before compilation it stores the object in the external dictionary under the key sym and generates code that retrieves it, deletes the external reference and returns the retrieved object.
(defn make-code-that-evals-to [x]
(let [s 17] ; please replace 17 with a generated unique key. I was lazy here.
`(objwrap ~x ~s)))
Nothing fancy, just wrap the object in the evil macro, together with a unique key.
Of course if you only expand the macro without evaluating its result, you'll still get a leak.
I guess you would need to write a macro that takes the object (or a way to create the required object) at compile time, serialize that object in binary format (byte array) and the output of the macro should be - a symbol that refer to the byte array and a function that can be used to get the object from the serialized data by de-serializing it.
why not:
(defmacro m [img] `(.getRGB ~img 0 0))
then u can write: (m some-buffered-image)
The difference is that f is a function, so its argument will be evaluated before its body being evaluated. Thus the image object itself will be placed within the generated code. But for macros, their arguments will not be evaluated. So only the symbol some-buffered-image will be placed within the code. The generated code will be: (.getRGB some-buffered-image 0 0). Just like u write the source code directly. I think that's what u want.
But why can't I place an object in the generated code? The answer is: yes, u can. What the exception message says is not the truth. u can embed some kinds of objects in generated code, but not all kinds of them. They include symbol, number, character, string, regex patten, keyword, boolean, list, map, set etc. All these objects will be understood by the Clojure compiler. They are like key words, operators and literals in other languages. u can't require the Clojure compiler knows all kinds of objects, just like u can't require a C or Java compiler knows all words not included by its syntax.
Related
I wrote the following function to print a value for 10 times:
(defn test-let []
(loop [index 0]
(when (< index 10)
(let [x 2]
(println "Value of x: " x))
(recur (inc index)))))
My question is:
Did because of let inside the loop, 10 variables (representing the value 2) and only 1 symbol (x, that represent the variable that holds value 2) got created?
If 10 variables were created, at what point they qualify for garbage collection?
I think you are mixing up values and variables. In your example above, there are 10 values created each time in the loop and are bound to x each time. In Clojure, the local symbols (aka names) created by a let form are not variables (for example, like those top level symbols created by def). Inside a let the symbols are just directly bound to the values. See https://clojure.org/reference/special_forms#let for more. The Clojure compiler does locals clearing, and will generate code to remove references to local values once the locus of control leaves the local scope. See https://clojure.org/reference/compilation#_locals_clearing for more details. In your particular case of a value of 2 it probably does not mater since Clojure will use the primitive value directly. I presume you were asking for the more general case where the value that is bound to x may be bigger than just a primitive.
Recently I came across a use for eval within a macro, which I understand is a bit of a faux pas but let's ignore that for now. What I found surprising, was that eval was able to resolve global vars at macroexpansion time. Below is a contrived example, just to illustrate the situation I'm referring to:
(def list-of-things (range 10))
(defmacro force-eval [args]
(apply + (eval args)))
(macroexpand-1 '(force-eval list-of-things))
; => 45
I would have expected args to resolve to the symbol list-of-things inside force-eval, and then list-of-things to be evaluated resulting in an error due to it being unbound:
"unable to resolve symbol list-of-things in this context"
However, instead list-of-things is resolved to (range 10) and no error is thrown - the macroexpansion succeeds.
Contrast this with attempting to perform the same macroexpansion, but within a local binding context:
(defmacro force-eval [args]
(apply + (eval args)))
(let [list-of-things (range 10)]
(macroexpand-1 '(force-eval list-of-things)))
; => Unable to resolve symbol: list-of-thingss in this context
Note in the above examples I'm assuming list-of-things is not previously bound, e.g. a fresh REPL. One final example illustrates why this is important:
(defmacro force-eval [args]
(apply + (eval args)))
(def list-of-things (range 10 20))
(let [list-of-thing (range 10)]
(macroexpand-1 '(force-eval list-of-things)))
; => 145
The above example shows that the locals are ignored, which is expected behavior for eval, but is a bit confusing when you are expecting the global to not be available at macroexpansion time either.
I seem to have a misunderstanding about what exactly is available at macroexpansion time. I had previously thought that essentially any binding, be it global or local, would not be available until runtime. Apparently this is an incorrect assumption. Is the answer to my confusion simply that global vars are available at macroexpansion time? Or am I missing some further nuance here?
Note: this related post closely describes a similar problem, but the focus there is more on how to avoid inappropriate use of eval. I'm mainly interested in understanding why eval works in the first example and by extension what's available to eval at macroexpansion time.
Of course, vars must be visible at compile time. That's where functions like first and + are stored. Without them, you couldn't do anything.
But keep in mind that you have to make sure to refer to them correctly. In the repl, *ns* will be bound, and so a reference to a symbol will look in the current namespace. If you are running a program through -main instead of the repl, *ns* will not be bound, and only properly qualified vars will be found. You can ensure that you qualify them correctly by using
`(force-eval list-of-things)
instead of
'(force-eval list-of-things)
Note I do not distinguish between global vars and non-global vars. All vars in Clojure are global. Local bindings are not called vars. They're called locals, or bindings, or variables, or some combination of those words.
Clojure is designed with an incremental compilation model. This is poorly documented.
In C and other traditional languages, source code must be compiled, then linked with pre-compiled libraries before the final result can be executed. Once execution begins, no changes to the code can occur until the program is terminated, when new source code can be compiled, linked, then executed. Java is normally used in this manner just like C.
With the Clojure REPL, you can start with zero source code in a live executing environment. You can call existing functions like (+ 2 3), or you can define new functions and variables on the fly (both global & local), and redefine existing functions. This is only possible because core Clojure is already available (i.e. clojure.core/+ etc is already "installed"), so you can combine these functions to define your own new functions.
The Clojure "compiler" works just like a giant REPL session. It reads and evaluates forms from your source code files one at a time, incrementally adding them the the global environment. Indeed, it is a design goal/requirement that the result of compiling and executing source code is identical to what would occur if you just pasted each entire source code file into the REPL (in proper dependency order).
Indeed, the simplest mental model for code execution in Clojure is to pretend it is an interpreter instead of a traditional compiler.
And eval in a macro makes no sense.
Because:
a macro already implicitely contains an eval
at the very final step.
If you use macroexpand-1, you make visible how the code was manipulated in the macro before the evocation of the implicite eval inside the macro.
An eval in a macro is an anti-pattern which might indicate that you should use a function instead of a macro - and in your examle this is exactly the case.
So your aim is to dynamically (in run-time) evoke sth in a macro. This you can only do through an eval applied over a macro call OR you should rather use a function.
(defmacro force-eval [args]
(apply + (eval args)))
;; What you actually mean is:
(defn force-eval [args]
(apply + args))
;; because a function in lisp evaluates its arguments
;; - before applying the function body.
;; That means: args in the function body is exactly
;; `(eval args)`!
(def list-of-things (range 10))
(let [lit-of-things (range 10 13)]
(force-eval list-of-things))
;; => 45
;; so this is exactly the behavior you wanted!
The point is, your construct is a "bad" example for a macro.
Because apply is a special function which allows you to
dynamically rearrange function call structures - so it has
some magic of macros inside it - but in run-time.
With apply you can do quite some meta programming in some cases when you just quote some of your input arguments.
(Try (force-eval '(1 2 3)) it returns 6. Because the (1 2 3) is put together with + at its front by apply and then evaluated.)
The second point - I am thinking of this answer I once gave and this to a dynamic macro call problem in Common Lisp.
In short: When you have to control two levels of evaluations inside a macro (often when you want a macro inject some code in runtime into some code), you need too use eval when calling the macro and evaluate those parts in the macro call which then should be processed in the macro.
In Clojure,
(def x 3)
(eval '(prn x))
prints 3, whereas
(let [y 3]
(eval '(prn y)))
and
(binding [z 3] (eval '(prn z)))
generate an 'Unable to resolve var' exception.
According to http://clojure.org/evaluation, eval, load-string, etc generate temporary namespaces to evaluate their contents. Therefore, I'd expect neither of the above code samples to work, since (def x 3) is done in my current namespace, not the one created by eval.
Why does the first code sample work and not the last two?
How can I eval a form with bound variables without using def?
Thanks!
1.:
The reason this doesn't work is (more or less) given on the page you linked:
It is an error if there is no global var named by the symbol […]
And:
[…]
A lookup is done in the current namespace to see if there is a mapping
from the symbol to a var. If so, the
value is the value of the binding of
the var referred-to by the symbol.
It is an error.
eval evaluates forms in an empty (null in CL-lingo) lexical environment. This means, that you cannot access lexical variable bindings from the caller's scope. Also, binding creates new bindings for existing vars, which is why you cannot use it "by itself", without having declared or defed the variables you try to bind. Besides, lexical variables (at least in CL, but I would be surprised if this wasn't the case for Clojure) already ceased to exist at runtime – They are translated to addresses or values.
See also my older post about this topic.
2.:
So, you have to use dynamic variables. You can avoid the explicit def, but you still at least need to declare them (which defs var names without bindings):
user=> (declare ^:dynamic x)
#'user/x
user=> (binding [x 10] (eval '(prn x)))
10
nil
By the way: I suppose you know why you need eval, and that its use is considered evil when other solutions would be appropriate.
I've been into Clojure lately and have avoided macros up until now, so this is my first exposure to them. I've been reading "Mastering Clojure Macros", and on Chapter 3, page 28, I encountered the following example:
user=> (defmacro square [x] `(* ~x ~x))
;=> #'user/square
user=> (map (fn [n] (square n)) (range 10))
;=> (0 1 4 9 16 25 36 49 64 81)
The context is the author is explaining that while simply passing the square macro to map results in an error (can't take value of a macro), wrapping it in a function works because:
when the anonymous function (fn [n] (square n)) gets compiled, the
square expression gets macroexpanded, to (fn [n] (clojure.core/* n
n)). And this is a perfectly reasonable function, so we don’t have any
problems with the compiler
This makes sense to me if we assume the body of the function is evaluated before runtime (at compile, or "definition" time) thus expanding the macro ahead of runtime. However, I always thought that function bodys were not evaluated until runtime, and at compile time you would basically just have a function object with some knowledge of it's lexical scope (but no knowledge of its body).
I'm clearly mixed up on the compile/runtime semantics here, but when I look at this sample I keep thinking that square won't be expanded until map forces it's call, since it's in the body of the anonymous function, which I thought would be unevaluated until runtime. I know my thinking is wrong, because if that was the case, then n would be bound to each number in (range 10), and there wouldn't be an issue.
I know it's a pretty basic question, but macros are proving to be pretty tricky for me to fully wrap my head around at first exposure!
Generally speaking function bodies aren't evaluated at compile time, but macros are always evaluated at compile time because they're always expanded at compile time whether inside a function or not.
You can write a macro that expands to a function, but you still can't refer/pass the macro around as if it were a function:
(defmacro inc-macro [] `(fn [x#] (inc x#)))
=> #'user/inc-macro
(map (inc-macro) [1 2 3])
=> (2 3 4)
defmacro is expanded at compile time, so you can think of it as a function executed during compilation. This will replace every occurrence of the macro "call" with the code it "returns".
You may consider a macros as a syntax rule. For example, in Scheme, a macros is declared with define-syntax form. There is a special step in compiler that substitutes all the macros calls into their content before compiling the code. As a result, there won't be any square calls in your final code. Say, if you wrote something like
(def value (square 3))
the final version after expansion would be
(def value (clojure.core/* 3 3))
There is a special way to check what will be the body of your macros after being expanded:
user=> (defmacro square [x] `(* ~x ~x))
#'user/square
user=> (macroexpand '(square 3))
(clojure.core/* 3 3)
That's why a macros is an ephemeral thing that lives only in source code but not in the compiled version of it. That's why it cannot be passed as a value or referenced somehow.
The best rule regarding macroses is: try to avoid them until you really need them in your work.
In Clojure, hash-maps and vectors implement invoke, so that they can be used as functions, for example
(let [dict {:species "Ursus horribilis"
:ornery :true
:diet "You"}]
(dict :diet))
lein> "You"
or, for vectors,
(let [v [42 613 28]]
(v 1))
lein> 613
One can make callable objects in Clojure by having them implement IFn. I'm new-ish to Common Lisp -- are callable objects possible and if so what would implementing that involve? I'd really like to be able to do things like
(let ((A (make-array (list n n) ...)))
(loop for i from 0 to n
for j from 0 to m
do (setf (A i j) (something i j)))
A)
rather than have code littered with aref. Likewise, it would be cool if you could access entries of other data structures, e.g. dictionaries, the same way.
I've looked at the wiki entry on function objects in Lisp/Scheme and it seems as if having a separate function namespace will complicate matters for CL, whereas in Scheme you can just do this with closures.
Example of callable objects in a precursor of Common Lisp
Callable objects have been provided before. For example in Lisp Machine Lisp:
Command: ("abc" 1) ; doesn't work in Common Lisp
#\b
Bindings in Common Lisp
Common Lisp has separate namespaces of names for functions and values. So (array 10 1 20) would only make sense, when array would be a symbol denoting a function in the function namespace. Thus the function value then would be a callable array.
Making values bound to variables act as functions mostly defeats the purpose of the different namespaces for functions and values.
(let ((v #(1 2 3)))
(v 10)) ; doesn't work in Common Lisp
Above makes no sense in a language with different namespaces for functions and values.
FLET is used for functions instead of LET.
(flet ((v #(1 2 3 4 5 6 7))) ; doesn't work in Common Lisp
(v 4))
This would then mean we would put data into the function namespace. Do we want that? Not really.
Literal data as functions in function calls.
One could also think of at least allowing literal data act as functions in direct function calls:
(#(1 2 3 4 5 6 7) 4) ; doesn't work in Common Lisp
instead of
(aref #(1 2 3 4 5 6 7) 4)
Common Lisp does not allow that in any trivial or relatively simple way.
Side remark:
One can implement something in the direction of integrating functions and values with CLOS, since CLOS generic functions are also CLOS instances of the class STANDARD-GENERIC-FUNCTION and it's possible to have and use user-defined subclasses of that. But that's usually not exploited.
Recommendation
So, best to adjust to a different language style and use CL as it is. In this case Common Lisp is not flexible enough to easily incorporate such a feature. It is general CL style to not omit symbols for minor code optimizations. The danger is obfuscation and write-only code, because a lot of information is not directly in the source code, then.
Although there may not be a way to do exactly what you want to do, there are some ways to hack together something similar. One option is define a new binding form, with-callable, that allows us to bind functions locally to callable objects. For example we could make
(with-callable ((x (make-array ...)))
(x ...))
be roughly equivalent to
(let ((x (make-array ...)))
(aref x ...))
Here is a possible definition for with-callable:
(defmacro with-callable (bindings &body body)
"For each binding that contains a name and an expression, bind the
name to a local function which will be a callable form of the
value of the expression."
(let ((gensyms (loop for b in bindings collect (gensym))))
`(let ,(loop for (var val) in bindings
for g in gensyms
collect `(,g (make-callable ,val)))
(flet ,(loop for (var val) in bindings
for g in gensyms
collect `(,var (&rest args) (apply ,g args)))
,#body))))
All that's left is to define different methods for make-callable that return closures for accessing into the objects. For example here is a method that would define it for arrays:
(defmethod make-callable ((obj array))
"Make an array callable."
(lambda (&rest indices)
(apply #'aref obj indices)))
Since this syntax is kind of ugly we can use a macro to make it prettier.
(defmacro defcallable (type args &body body)
"Define how a callable form of TYPE should get access into it."
`(defmethod make-callable ((,(car args) ,type))
,(format nil "Make a ~A callable." type)
(lambda ,(cdr args) ,#body)))
Now to make arrays callable we would use:
(defcallable array (obj &rest indicies)
(apply #'aref obj indicies))
Much better. We now have a form, with-callable, which will define local functions that allow us to access into objects, and a macro, defcallable, that allows us to define how to make callable versions of other types. One flaw with this strategy is that we have to explicitly use with-callable every time we want to make an object callable.
Another option that is similar to callable objects is Arc's structure accessing ssyntax. Basically x.5 accesses the element at index five in x. I was able to implement this in Common Lisp. You can see the code I wrote for it here, and here. I also have tests for it so you can see what using it looks like here.
How my implementation works is I wrote a macro w/ssyntax which looks at all of the symbols in the body and defines macros and symbol-macros for some of them. For example the symbol-macro for x.5 would be (get x 5), where get is a generic function I defined that accesses into structures. The flaw with this is I always have to use w/ssyntax anywhere I want to use ssyntax. Fortunately I am able to hide it away inside a macro def which acts like defun.
I agree with Rainer Joswig's advice: It would be better to become comfortable with Common Lisp's way of doing things--just as it's better for a Common Lisp programmer to become comfortable with Clojure's way of doing things, when switching to Clojure. However, it is possible to do part of what you want, as malisper's sophisticated answer shows. Here is the start of a simpler strategy:
(defun make-array-fn (a)
"Return a function that, when passed an integer i, will
return the element of array a at index i."
(lambda (i) (aref a i)))
(setf (symbol-function 'foo) (make-array-fn #(4 5 6)))
(foo 0) ; => 4
(foo 1) ; => 5
(foo 2) ; => 6
symbol-function accesses the function cell of the symbol foo, and setf puts the function object created by make-array-fn into it. Since this function is then in the function cell, foo can be used in the function position of a list. If you wanted, you could wrap up the whole operation into a macro, e.g. like this:
(defmacro def-array-fn (sym a)
"Define sym as a function that is the result of (make-array-fn a)."
`(setf (symbol-function ',sym)
(make-array-fn ,a)))
(def-array-fn bar #(10 20 30 40))
(bar 0) ; => 10
(bar 1) ; => 20
(bar 3) ; => 40
Of course, an "array" defined this way no longer looks like an array. I suppose you could do something fancy with CL's printing routines. It's also possible to allow setting values of the array as well, but this would probably require a separate symbols.