I've just started learning Clojure and trying to understand the difference between 2 approaches which at first sight seem very identical.
(def func0 (delay (do
(println "did some work")
100)))
so.core=> (force my-delay2)
did some work
100
so.core=> (force my-delay2)
100
(defn vanilla-func [] (println "did some work") 100)
(def func1 (memoize vanilla-func))
so.core=> (func1)
did some work
100
so.core=> (func1)
100
Both approaches do some sort of function memoization. What am I missing?
I've tried to find the explanation on https://clojuredocs.org/clojure.core/delay & https://clojuredocs.org/clojure.core/memoize but couldn't.
delay holds one result and you have to deref to get the result.
memoize is an unbound cache, that caches the result depending on the
input arguments. E.g.
user=> (def myinc (memoize (fn [x] (println x) (inc x))))
#'user/myinc
user=> (myinc 1)
1
2
user=> (myinc 1)
2
In your (argument-less) example the only difference is that you can use
the result directly (no deref needed)
Classic use-cases for delay are things needed later, that would block
or delay startup. Or if you want to "hide" top-level defs from
the compiler (e.g. they do side-effects).
memoize is a classic cache and is best used if the calculation is
expensive and the set of input arguments is not excessive. There are
other caching options in the clojure-verse, that allow better
configurations (e.g. they are not unbound).
Related
I find myself writing a lot of clojure in this manner:
(defn my-fun [input]
(let [result1 (some-complicated-procedure input)
result2 (some-other-procedure result1)]
(do-something-with-results result1 result2)))
This let statement seems very... imperative. Which I don't like. In principal, I could be writing the same function like this:
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
The problem with this is that it involves recomputation of some-complicated-procedure, which may be arbitrarily expensive. Also you can imagine that some-complicated-procedure is actually a series of nested function calls, and then I either have to write a whole new function, or risk that changes in the first invocation don't get applied to the second:
E.g. this works, but I have to have an extra shallow, top-level function that makes it hard to do a mental stack trace:
(defn some-complicated-procedure [input] (lots (of (nested (operations input)))))
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
E.g. this is dangerous because refactoring is hard:
(defn my-fun [input]
(do-something-with-results (lots (of (nested (operations (mistake input))))) ; oops made a change here that wasn't applied to the other nested calls
(some-other-procedure (lots (of (nested (operations input))))))))
Given these tradeoffs, I feel like I don't have any alternatives to writing long, imperative let statements, but when I do, I cant shake the feeling that I'm not writing idiomatic clojure. Is there a way I can address the computation and code cleanliness problems raised above and write idiomatic clojure? Are imperitive-ish let statements idiomatic?
The kind of let statements you describe might remind you of imperative code, but there is nothing imperative about them. Haskell has similar statements for binding names to values within bodies, too.
If your situation really needs a bigger hammer, there are some bigger hammers that you can either use or take for inspiration. The following two libraries offer some kind of binding form (akin to let) with a localized memoization of results, so as to perform only the necessary steps and reuse their results if needed again: Plumatic Plumbing, specifically the Graph part; and Zach Tellman's Manifold, whose let-flow form furthermore orchestrates asynchronous steps to wait for the necessary inputs to become available, and to run in parallel when possible. Even if you decide to maintain your present course, their docs make good reading, and the code of Manifold itself is educational.
I recently had this same question when I looked at this code I wrote
(let [user-symbols (map :symbol states)
duplicates (for [[id freq] (frequencies user-symbols) :when (> freq 1)] id)]
(do-something-with duplicates))
You'll note that map and for are lazy and will not be executed until do-something-with is executed. It's also possible that not all (or even not any) of the states will be mapped or the frequencies calculated. It depends on what do-something-with actually requests of the sequence returned by for. This is very much functional and idiomatic functional programming.
i guess the simplest approach to keep it functional would be to have a pass-through state to accumulate the intermediate results. something like this:
(defn with-state [res-key f state]
(assoc state res-key (f state)))
user> (with-state :res (comp inc :init) {:init 10})
;;=> {:init 10, :res 11}
so you can move on to something like this:
(->> {:init 100}
(with-state :inc'd (comp inc :init))
(with-state :inc-doubled (comp (partial * 2) :inc'd))
(with-state :inc-doubled-squared (comp #(* % %) :inc-doubled))
(with-state :summarized (fn [st] (apply + (vals st)))))
;;=> {:init 100,
;; :inc'd 101,
;; :inc-doubled 202,
;; :inc-doubled-squared 40804,
;; :summarized 41207}
The let form is a perfectly functional construct and can be seen as syntactic sugar for calls to anonymous functions. We can easily write a recursive macro to implement our own version of let:
(defmacro my-let [bindings body]
(if (empty? bindings)
body
`((fn [~(first bindings)]
(my-let ~(rest (rest bindings)) ~body))
~(second bindings))))
Here is an example of calling it:
(my-let [a 3
b (+ a 1)]
(* a b))
;; => 12
And here is a macroexpand-all called on the above expression, that reveal how we implement my-let using anonymous functions:
(clojure.walk/macroexpand-all '(my-let [a 3
b (+ a 1)]
(* a b)))
;; => ((fn* ([a] ((fn* ([b] (* a b))) (+ a 1)))) 3)
Note that the expansion doesn't rely on let and that the bound symbols become parameter names in the anonymous functions.
As others write, let is actually perfectly functional, but at times it can feel imperative. It's better to become fully comfortable with it.
You might, however, want to kick the tires of my little library tl;dr that lets you write code like for example
(compute
(+ a b c)
where
a (f b)
c (+ 100 b))
I would like to define a predicate that, taking as input some predicates
with corresponding inputs (they could be given as a lazy sequence of calls),
runs them in parallel and computes the logical or of the results,
in such a way that, the moment a predicate call terminates returning true,
the whole computation also terminates (returning true).
Apart from offering time optimization, this would also help avoiding
non-termination in some cases (some predicate calls may not terminate).
Actually, interpreting non-termination as a third undefined value,
this predicate simulates the or operation in Kleene's K3 logic
(the join in the initial centered Kleene algebra).
Something similar is presented here for the Haskell family.
Is there any (preferably simple) way to do this in Clojure?
EDIT: I decided to add some clarifications after reading the comments.
(a) First of all, what happens after the thread pool gets exhausted is of less importance. I think creating a thread pool large enough for our needs is a reasonable convention.
(b) The most crucial requirement is that the predicate calls start running in parallel and, once a predicate call terminates returning true, all the other threads running get interrupted. The intended behavior is that:
if there is a predicate call returning true: the parallel or returns true
else if there is a predicate call that does not terminate: the parallel or does not terminate
else: the parallel or returns false
In other words, it behaves like the join in the 3-element lattice given by false<undefined<true, with undefined representing non-termination.
(c) The parallel or should be able to take as input many predicates and many predicate-inputs (each one corresponding to a predicate). But it would be even better if it took as input a lazy sequence. Then, naming the parallel or pany (for "parallel any"), we could have calls like the following:
(pany (map (comp eval list) predicates inputs))
(pany (map (comp eval list) predicates (repeat input)))
(pany (map (comp eval list) (repeat predicate) inputs)) which is equivalent to (pany (map predicate (unchunk inputs)))
As a final remark, I think that it is quite natural to ask for things like pany, a dual pall or a mechanism for building such early-terminating parallel reductions to be easily implementable or even built-in in a parallelism-oriented language like Clojure.
I will define our predicates in terms of a reducing function. Practically, we could reimplement all of the Clojure iteration functions to support this parallel operation, but I'll just use reduce as an example.
I'll define a computation function. I'll just use the same one, but nothing stopping you from having many. The function is "true" if it accumulates 1000.
(defn computor [acc val]
(let [new (+' acc val)] (if (> new 1000) (reduced new) new)))
(reduce computor 0 (range))
;; =>
1035
(reduce computor 0 (range Long/MIN_VALUE 0))
;; =>
;; ...this is a proxy for a non-returning computation
;; wrap these up in a form suitable for application of reduction
(def predicates [[computor 0 (range)]
[computor 0 (range Long/MIN_VALUE 0)]])
Now let's get to the meat of this. I want to take a step in each computation, and if one of the computations completes, I want to return it. In actual fact one step at a time using pmap is very slow - the units of work are too small to be worth threading. Here I've changed things to do 1000 iterations of each unit of work before moving on. You'd probably tune this based on your workload and the cost of a step.
(defn p-or-reducer* [reductions]
(let [splits (map #(split-at 1000 %) reductions) ;; do at least 1000 iterations per cycle
complete (some #(if (empty? (second %)) (last (first %))) splits)]
(or complete (recur (map second splits)))))
I then wrap this in a driver.
(defn p-or [s]
(p-or-reducer* (map #(apply reductions %) s)))
(p-or predicates)
;; =>
1035
Where to insert the CPU parallelism? s/map/pmap/ in p-or-reducer* should do it. I suggest just parallelising the first operation, as this will drive the reducing sequences to compute.
(defn p-or-reducer* [reductions]
(let [splits (pmap #(split-at 1000 %) reductions) ;; do at least 1000 iterations per cycle
complete (some #(if (empty? (second %)) (last (first %))) splits)]
(or complete (recur (map second splits)))))
(def parallelism-tester (conj (vec (repeat 40000 [computor 0 (range Long/MIN_VALUE 0)]))
[computor 0 (range)]))
(p-or parallelism-tester) ;; terminates even though the first 40K predicates will not
It's extremely hard to define a performant generic version of this. Without knowing the cost per iteration an efficient parallelism strategy is hard to derive - if one iteration take 10s then we'd probably take a single step at a time. If it takes 100ns then we need to take many steps at a time.
Will you consider adopting core.async to handle parallel tasks with async/go or async/thread, and early return with async/alts!?
For example, to turn the core or function from serial into parallel. We can create a macro (I called it por) to wrap the input functions (or predicates) into async/thread and then do a socket select async/alts! on top of them:
(defmacro por [& fns]
`(let [[v# c#] (async/alts!!
[~#(for [f fns]
(list `async/thread f))])]
v#))
(time
(por (do (println "running a") (Thread/sleep 30) :a)
(do (println "running b") (Thread/sleep 20) :b)
(do (println "running c") (Thread/sleep 10) :c)))
;; running a
;; running b
;; running c
;; "Elapsed time: 11.919169 msecs"
;; => :c
In comparison with the original or (which run in serial):
(time
(or (do (println "running a") (Thread/sleep 30) :a)
(do (println "running b") (Thread/sleep 20) :b)
(do (println "running c") (Thread/sleep 10) :c)))
;; running a
;; => :a
;; "Elapsed time: 31.642506 msecs"
I have occasionally used memoize function. In usually the following form:
(defn- sqrt-denom [iterations]
(/ 1 (if (= iterations 0)
2
(+ 2 (sqrt-denom (dec iterations))))))
(def sqrt-denom (memoize sqrt-denom))
I assumed that it is "proper" to reuse the function name when memoizing. Is this a good practice? Or should I use different names for the non-memoized and memoized functions?
I would never re-use the name of a top-level def, especially when self-referencing. Two choices:
(defn ^:no-doc sqrt-denom-impl [iterations]
(/ 1 (if (= iterations 0)
2
(+ 2 (sqrt-denom (dec iterations))))))
(def sqrt-denom (memoize sqrt-denom-impl))
or even simpler:
(def sqrt-denom
(memoize (fn [iterations]
(/ 1 (if (= iterations 0)
2
(+ 2 (sqrt-denom (dec iterations))))))
Like Alan Thompson, I've often used (def sqrt-denom (memoize (fn ...))), but it's awkward to split a simple defn into a def and fn just to wrap the latter. The syntactic transformation makes it difficult to switch back and forth between the memoized and the unmemoized versions of the function, and memoize is the one function that made me wish Clojure had something like Python decorators.
Then, recently, I discovered that Clojure does have decorators:
(defn sqrt-denom ...)
(alter-var-root #'sqrt-denom memoize)
This is very similar to your example, but it avoids the confusion of two vars named the same thing, and it clearly states the author's intent. This is now my preferred method for memoizing functions.
Is there a reasonable way to have multiple def statements happen with destructing the same way that let does it? For Example:
(let [[rtgs pcts] (->> (sort-by second row)
(apply map vector))]
.....)
What I want is something like:
(defs [rtgs pcts] (->> (sort-by second row)
(apply map vector)))
This comes up a lot in the REPL, notebooks and when debugging. Seriously feels like a missing feature so I'd like guidance on one of:
This exists already and I'm missing it
This is a bad idea because... (variable capture?, un-idiomatic?, Rich said so?)
It's just un-needed and I must be suffering from withdrawals from an evil language. (same as: don't mess up our language with your macros)
A super short experiment give me something like:
(defmacro def2 [[name1 name2] form]
`(let [[ret1# ret2#] ~form]
(do (def ~name1 ret1#)
(def ~name2 ret2#))))
And this works as in:
(def2 [three five] ((juxt dec inc) 4))
three ;; => 3
five ;; => 5
Of course and "industrial strength" version of that macro might be:
checking that number of names matches the number of inputs. (return from form)
recursive call to handle more names (can I do that in a macro like this?)
While I agree with Josh that you probably shouldn't have this running in production, I don't see any harm in having it as a convenience at the repl (in fact I think I'll copy this into my debug-repl kitchen-sink library).
I enjoy writing macros (although they're usually not needed) so I whipped up an implementation. It accepts any binding form, like in let.
(I wrote this specs-first, but if you're on clojure < 1.9.0-alpha17, you can just remove the spec stuff and it'll work the same.)
(ns macro-fun
(:require
[clojure.spec.alpha :as s]
[clojure.core.specs.alpha :as core-specs]))
(s/fdef syms-in-binding
:args (s/cat :b ::core-specs/binding-form)
:ret (s/coll-of simple-symbol? :kind vector?))
(defn syms-in-binding
"Returns a vector of all symbols in a binding form."
[b]
(letfn [(step [acc coll]
(reduce (fn [acc x]
(cond (coll? x) (step acc x)
(symbol? x) (conj acc x)
:else acc))
acc, coll))]
(if (symbol? b) [b] (step [] b))))
(s/fdef defs
:args (s/cat :binding ::core-specs/binding-form, :body any?))
(defmacro defs
"Like def, but can take a binding form instead of a symbol to
destructure the results of the body.
Doesn't support docstrings or other metadata."
[binding body]
`(let [~binding ~body]
~#(for [sym (syms-in-binding binding)]
`(def ~sym ~sym))))
;; Usage
(defs {:keys [foo bar]} {:foo 42 :bar 36})
foo ;=> 42
bar ;=> 36
(defs [a b [c d]] [1 2 [3 4]])
[a b c d] ;=> [1 2 3 4]
(defs baz 42)
baz ;=> 42
About your REPL-driven development comment:
I don't have any experience with Ipython, but I'll give a brief explanation of my REPL workflow and you can maybe comment about any comparisons/contrasts with Ipython.
I never use my repl like a terminal, inputting a command and waiting for a reply. My editor supports (emacs, but any clojure editor should do) putting the cursor at the end of any s-expression and sending that to the repl, "printing" the result after the cursor.
I usually have a comment block in the file where I start working, just typing whatever and evaluating it. Then, when I'm reasonably happy with a result, I pull it out of the "repl-area" and into the "real-code".
(ns stuff.core)
;; Real code is here.
;; I make sure that this part always basically works,
;; ie. doesn't blow up when I evaluate the whole file
(defn foo-fn [x]
,,,)
(comment
;; Random experiments.
;; I usually delete this when I'm done with a coding session,
;; but I copy some forms into tests.
;; Sometimes I leave it for posterity though,
;; if I think it explains something well.
(def some-data [,,,])
;; Trying out foo-fn, maybe copy this into a test when I'm done.
(foo-fn some-data)
;; Half-finished other stuff.
(defn bar-fn [x] ,,,)
(keys 42) ; I wonder what happens if...
)
You can see an example of this in the clojure core source code.
The number of defs that any piece of clojure will have will vary per project, but I'd say that in general, defs are not often the result of some computation, let alone the result of a computation that needs to be destructured. More often defs are the starting point for some later computation that will depend on this value.
Usually functions are better for computing a value; and if the computation is expensive, then you can memoize the function. If you feel you really need this functionality, then by all means, use your macro -- that's one of the sellings points of clojure, namely, extensibility! But in general, if you feel you need this construct, consider the possibility that you're relying too much on global state.
Just to give some real examples, I just referenced my main project at work, which is probably 2K-3K lines of clojure, in about 20 namespaces. We have about 20 defs, most of which are marked private and among them, none are actually computing anything. We have things like:
(def path-prefix "/some-path")
(def zk-conn (atom nil))
(def success? #{200})
(def compile* (clojure.core.memoize/ttl compiler {} ...)))
(def ^:private nashorn-factory (NashornScriptEngineFactory.))
(def ^:private read-json (comp json/read-str ... ))
Defining functions (using comp and memoize), enumerations, state via atom -- but no real computation.
So I'd say, based on your bullet points above, this falls somewhere between 2 and 3: it's definitely not a common use case that's needed (you're the first person I've ever heard who wants this, so it's uncommon to me anyway); and the reason it's uncommon is because of what I said above, i.e., it may be a code smell that indicates reliance on too much global state, and hence, would not be very idiomatic.
One litmus test I have for much of my code is: if I pull this function out of this namespace and paste it into another, does it still work? Removing dependencies on external vars allows for easier testing and more modular code. Sometimes we need it though, so see what your requirements are and proceed accordingly. Best of luck!
I'm using this basic macro as a building block for other timing macros:
(defmacro time-pure
"Evaluates expr and returns the time it took.
Modified the native time macro to return the time taken."
[expr]
`(let [start# (current-nano-timestamp)
ret# ~expr]
(/ (double (- (current-nano-timestamp) start#)) 1000000.0)))
I've tested this, and used it in other macros, so I know it works fine.
My problem can be described by the following snippet:
(defmacro time-each [& exprs]
`(mapv #(time-pure %) '~exprs))
I would expect this to give each expression to time-each, where it executes and times it; returning the result. When I test it however, it finishes instantly:
(time-each
(Thread/sleep 500)
(Thread/sleep 1000))
[0.036571 0.0]
I'm confused by this since I know that (time-pure (Thread/sleep 1000)) will take around a second to return, and all this macro does it delegate to time-pure.
What's causing this? I really have no idea how to properly debug a macro. I used macro-expand-1 to inspect the generated code:
(clojure.pprint/pprint
(macroexpand-1
'(time-each
(Thread/sleep 500)
(Thread/sleep 1000))))
(clojure.core/mapv
(fn*
[p1__1451__1452__auto__]
(helpers.general-helpers/time-pure p1__1451__1452__auto__))
'((Thread/sleep 500) (Thread/sleep 1000)))
But nothing really stands out to me.
What's going on here?
(Note, this is a dupe of a question I posted a few minutes ago. I realized the case I was showing was convoluted, so I made up a better example.)
Solution
I think what happens here is that your mapv executes after compile time, meaning at run time, this is why it cant just "paste" the code list as an sexp. I think a better approach would be to keep the mapv out of the syntax quoting:
(defmacro each-time-mine [& exprs]
(mapv #(time-pure %) exprs))
(each-time-mine
(Thread/sleep 500)
(Thread/sleep 1000))
[501.580465 1001.196752]
Original
Although eval is usually frowned upon, it seems to solve the problem in this situation:
(defmacro time-pure
[expr]
`(let [start# (current-nano-timestamp)
ret# (eval ~expr)]
(/ (double (- (current-nano-timestamp) start#)) 1000000.0)))
(defmacro time-each [& exprs]
`(mapv #(time-pure %) '~exprs))
(time-each
(Thread/sleep 500)
(Thread/sleep 1000))
[501.249249 1001.242522]
What happens is that mapv sees the sexps as lists, and at the moment time-pure wants to execute them, it simply assigns ret# the value of the list. So that list needs to be evaled.
There might be better ways to achieve this though.
(defmacro time-each [& exprs]
`(list ~#(for [expr exprs]
`(time-pure ~expr))))
You need to be careful not to evaluate an expression outside of a time-pure context, as is done in Arthur's answer, which evaluates each expression and then calls time-pure on the (very fast) operation of "looking at the result". Rather, wrap time-pure around each expression before evaluating it.
"I'm still trying to don't wrap my head around macros."
It takes a while. :) One thing that helped me was macro-expanding:
(macroexpand-1 '(time-each (+ 2 2 )(+ 3 3)))
That produces:
(clojure.core/mapv
(fn* [p1__24013__24014__auto__]
(rieclj.core/time-pure p1__24013__24014__auto__))
(quote ((+ 2 2) (+ 3 3))))
The alarming thing there is that time-pure is being passed a quoted list, so it is all just symbols at run time, not macroexpansion time.
Then another thing that helped me was that the beauty of Lispy macros is that they are normal functions that just run at a different time (while the reader is processing the source). But because they are normal functions we can probe them with print statements just as we probe normal code.
I have modified time-pure to print out its macroexpansion-time argument expr and, in the generate code, to have it print out the evaluated input, which we now suspect is a list of symbols, or '(+ 2 2) in the first case.
(defmacro time-pure
"Evaluates expr and returns the time it took.
Modified the native time macro to return the time taken."
[expr]
(prn :time-pure-sees expr)
`(let [start# (now)
ret# ~expr]
(prn :ret-is ret# (type (first ret#)))
(/ (double (- (now) start#)) 1000000.0)))
I print the type of the first of ret to drive home that + is a symbol, not a function. Evaluating:
(time-each (+ 2 2 )(+ 3 3))
yields:
:time-pure-sees p1__24013__24014__auto__
:ret-is (+ 2 2) clojure.lang.Symbol
:ret-is (+ 3 3) clojure.lang.Symbol
Seeing (+ 2 2) might make you think all is well, but the key is that the rightside is '(+ 2 2) so ret# gets bound to that symbolic expression instead of the intended computation.
Again, printing the type of the first hopefully makes clear that time-pure was working on a list of symbols at runtime.
Sorry if all that was confusing, but the moral is simple: use macroexpand-1 and embedded print statements when macros mess with your head. In time you will internalize the two different times, macroexpansion and run.