Using let style destructuring for def - clojure

Is there a reasonable way to have multiple def statements happen with destructing the same way that let does it? For Example:
(let [[rtgs pcts] (->> (sort-by second row)
(apply map vector))]
.....)
What I want is something like:
(defs [rtgs pcts] (->> (sort-by second row)
(apply map vector)))
This comes up a lot in the REPL, notebooks and when debugging. Seriously feels like a missing feature so I'd like guidance on one of:
This exists already and I'm missing it
This is a bad idea because... (variable capture?, un-idiomatic?, Rich said so?)
It's just un-needed and I must be suffering from withdrawals from an evil language. (same as: don't mess up our language with your macros)
A super short experiment give me something like:
(defmacro def2 [[name1 name2] form]
`(let [[ret1# ret2#] ~form]
(do (def ~name1 ret1#)
(def ~name2 ret2#))))
And this works as in:
(def2 [three five] ((juxt dec inc) 4))
three ;; => 3
five ;; => 5
Of course and "industrial strength" version of that macro might be:
checking that number of names matches the number of inputs. (return from form)
recursive call to handle more names (can I do that in a macro like this?)

While I agree with Josh that you probably shouldn't have this running in production, I don't see any harm in having it as a convenience at the repl (in fact I think I'll copy this into my debug-repl kitchen-sink library).
I enjoy writing macros (although they're usually not needed) so I whipped up an implementation. It accepts any binding form, like in let.
(I wrote this specs-first, but if you're on clojure < 1.9.0-alpha17, you can just remove the spec stuff and it'll work the same.)
(ns macro-fun
(:require
[clojure.spec.alpha :as s]
[clojure.core.specs.alpha :as core-specs]))
(s/fdef syms-in-binding
:args (s/cat :b ::core-specs/binding-form)
:ret (s/coll-of simple-symbol? :kind vector?))
(defn syms-in-binding
"Returns a vector of all symbols in a binding form."
[b]
(letfn [(step [acc coll]
(reduce (fn [acc x]
(cond (coll? x) (step acc x)
(symbol? x) (conj acc x)
:else acc))
acc, coll))]
(if (symbol? b) [b] (step [] b))))
(s/fdef defs
:args (s/cat :binding ::core-specs/binding-form, :body any?))
(defmacro defs
"Like def, but can take a binding form instead of a symbol to
destructure the results of the body.
Doesn't support docstrings or other metadata."
[binding body]
`(let [~binding ~body]
~#(for [sym (syms-in-binding binding)]
`(def ~sym ~sym))))
;; Usage
(defs {:keys [foo bar]} {:foo 42 :bar 36})
foo ;=> 42
bar ;=> 36
(defs [a b [c d]] [1 2 [3 4]])
[a b c d] ;=> [1 2 3 4]
(defs baz 42)
baz ;=> 42
About your REPL-driven development comment:
I don't have any experience with Ipython, but I'll give a brief explanation of my REPL workflow and you can maybe comment about any comparisons/contrasts with Ipython.
I never use my repl like a terminal, inputting a command and waiting for a reply. My editor supports (emacs, but any clojure editor should do) putting the cursor at the end of any s-expression and sending that to the repl, "printing" the result after the cursor.
I usually have a comment block in the file where I start working, just typing whatever and evaluating it. Then, when I'm reasonably happy with a result, I pull it out of the "repl-area" and into the "real-code".
(ns stuff.core)
;; Real code is here.
;; I make sure that this part always basically works,
;; ie. doesn't blow up when I evaluate the whole file
(defn foo-fn [x]
,,,)
(comment
;; Random experiments.
;; I usually delete this when I'm done with a coding session,
;; but I copy some forms into tests.
;; Sometimes I leave it for posterity though,
;; if I think it explains something well.
(def some-data [,,,])
;; Trying out foo-fn, maybe copy this into a test when I'm done.
(foo-fn some-data)
;; Half-finished other stuff.
(defn bar-fn [x] ,,,)
(keys 42) ; I wonder what happens if...
)
You can see an example of this in the clojure core source code.

The number of defs that any piece of clojure will have will vary per project, but I'd say that in general, defs are not often the result of some computation, let alone the result of a computation that needs to be destructured. More often defs are the starting point for some later computation that will depend on this value.
Usually functions are better for computing a value; and if the computation is expensive, then you can memoize the function. If you feel you really need this functionality, then by all means, use your macro -- that's one of the sellings points of clojure, namely, extensibility! But in general, if you feel you need this construct, consider the possibility that you're relying too much on global state.
Just to give some real examples, I just referenced my main project at work, which is probably 2K-3K lines of clojure, in about 20 namespaces. We have about 20 defs, most of which are marked private and among them, none are actually computing anything. We have things like:
(def path-prefix "/some-path")
(def zk-conn (atom nil))
(def success? #{200})
(def compile* (clojure.core.memoize/ttl compiler {} ...)))
(def ^:private nashorn-factory (NashornScriptEngineFactory.))
(def ^:private read-json (comp json/read-str ... ))
Defining functions (using comp and memoize), enumerations, state via atom -- but no real computation.
So I'd say, based on your bullet points above, this falls somewhere between 2 and 3: it's definitely not a common use case that's needed (you're the first person I've ever heard who wants this, so it's uncommon to me anyway); and the reason it's uncommon is because of what I said above, i.e., it may be a code smell that indicates reliance on too much global state, and hence, would not be very idiomatic.
One litmus test I have for much of my code is: if I pull this function out of this namespace and paste it into another, does it still work? Removing dependencies on external vars allows for easier testing and more modular code. Sometimes we need it though, so see what your requirements are and proceed accordingly. Best of luck!

Related

Functional alternative to "let"

I find myself writing a lot of clojure in this manner:
(defn my-fun [input]
(let [result1 (some-complicated-procedure input)
result2 (some-other-procedure result1)]
(do-something-with-results result1 result2)))
This let statement seems very... imperative. Which I don't like. In principal, I could be writing the same function like this:
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
The problem with this is that it involves recomputation of some-complicated-procedure, which may be arbitrarily expensive. Also you can imagine that some-complicated-procedure is actually a series of nested function calls, and then I either have to write a whole new function, or risk that changes in the first invocation don't get applied to the second:
E.g. this works, but I have to have an extra shallow, top-level function that makes it hard to do a mental stack trace:
(defn some-complicated-procedure [input] (lots (of (nested (operations input)))))
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
E.g. this is dangerous because refactoring is hard:
(defn my-fun [input]
(do-something-with-results (lots (of (nested (operations (mistake input))))) ; oops made a change here that wasn't applied to the other nested calls
(some-other-procedure (lots (of (nested (operations input))))))))
Given these tradeoffs, I feel like I don't have any alternatives to writing long, imperative let statements, but when I do, I cant shake the feeling that I'm not writing idiomatic clojure. Is there a way I can address the computation and code cleanliness problems raised above and write idiomatic clojure? Are imperitive-ish let statements idiomatic?
The kind of let statements you describe might remind you of imperative code, but there is nothing imperative about them. Haskell has similar statements for binding names to values within bodies, too.
If your situation really needs a bigger hammer, there are some bigger hammers that you can either use or take for inspiration. The following two libraries offer some kind of binding form (akin to let) with a localized memoization of results, so as to perform only the necessary steps and reuse their results if needed again: Plumatic Plumbing, specifically the Graph part; and Zach Tellman's Manifold, whose let-flow form furthermore orchestrates asynchronous steps to wait for the necessary inputs to become available, and to run in parallel when possible. Even if you decide to maintain your present course, their docs make good reading, and the code of Manifold itself is educational.
I recently had this same question when I looked at this code I wrote
(let [user-symbols (map :symbol states)
duplicates (for [[id freq] (frequencies user-symbols) :when (> freq 1)] id)]
(do-something-with duplicates))
You'll note that map and for are lazy and will not be executed until do-something-with is executed. It's also possible that not all (or even not any) of the states will be mapped or the frequencies calculated. It depends on what do-something-with actually requests of the sequence returned by for. This is very much functional and idiomatic functional programming.
i guess the simplest approach to keep it functional would be to have a pass-through state to accumulate the intermediate results. something like this:
(defn with-state [res-key f state]
(assoc state res-key (f state)))
user> (with-state :res (comp inc :init) {:init 10})
;;=> {:init 10, :res 11}
so you can move on to something like this:
(->> {:init 100}
(with-state :inc'd (comp inc :init))
(with-state :inc-doubled (comp (partial * 2) :inc'd))
(with-state :inc-doubled-squared (comp #(* % %) :inc-doubled))
(with-state :summarized (fn [st] (apply + (vals st)))))
;;=> {:init 100,
;; :inc'd 101,
;; :inc-doubled 202,
;; :inc-doubled-squared 40804,
;; :summarized 41207}
The let form is a perfectly functional construct and can be seen as syntactic sugar for calls to anonymous functions. We can easily write a recursive macro to implement our own version of let:
(defmacro my-let [bindings body]
(if (empty? bindings)
body
`((fn [~(first bindings)]
(my-let ~(rest (rest bindings)) ~body))
~(second bindings))))
Here is an example of calling it:
(my-let [a 3
b (+ a 1)]
(* a b))
;; => 12
And here is a macroexpand-all called on the above expression, that reveal how we implement my-let using anonymous functions:
(clojure.walk/macroexpand-all '(my-let [a 3
b (+ a 1)]
(* a b)))
;; => ((fn* ([a] ((fn* ([b] (* a b))) (+ a 1)))) 3)
Note that the expansion doesn't rely on let and that the bound symbols become parameter names in the anonymous functions.
As others write, let is actually perfectly functional, but at times it can feel imperative. It's better to become fully comfortable with it.
You might, however, want to kick the tires of my little library tl;dr that lets you write code like for example
(compute
(+ a b c)
where
a (f b)
c (+ 100 b))

testing if something is an empty list

Which way should I prefer to test if an object is an empty list in Clojure? Note that I want to test just this and not if it is empty as a sequence. If it is a "lazy entity" (LazySeq, Iterate, ...) I don't want it to get realized?.
Below I give some possible tests for x.
;0
(= clojure.lang.PersistentList$EmptyList (class x))
;1
(and (list? x) (empty? x))
;2
(and (list? x) (zero? (count x)))
;3
(identical? () x)
Test 0 is a little low level and relies on "implementation details". My first version of it was (instance? clojure.lang.PersistentList$EmptyList x), which gives an IllegalAccessError. Why is that so? Shouldn't such a test be possible?
Tests 1 and 2 are higher level and more general, since list? checks if something implements IPersistentList. I guess they are slightly less efficient too. Notice that the order of the two sub-tests is important as we rely on short-circuiting.
Test 3 works under the assumption that every empty list is the same object. The tests I have done confirm this assumption but is it guaranteed to hold? Even if it is so, is it a good practice to rely on this fact?
All this may seem trivial but I was a bit puzzled not finding a completely straightforward solution (or even a built-in function) for such a simple task.
update
Perhaps I did not formulate the question very well. In retrospect, I realized that what I wanted to test was if something is a non-lazy empty sequence. The most crucial requirement for my use case is that, if it is a lazy sequence, it does not get realized, i.e. no thunk gets forced.
Using the term "list" was a little confusing. After all what is a list? If it is something concrete like PersistentList, then it is non-lazy. If it is something abstract like IPersistentList (which is what list? tests and probably the correct answer), then non-laziness is not exactly guaranteed. It just so happens that Clojure's current lazy sequence types do not implement this interface.
So first of all I need a way to test if something is a lazy sequence. The best solution I can think of right now is to use IPending to test for laziness in general:
(def lazy? (partial instance? clojure.lang.IPending))
Although there are some lazy sequence types (e.g. chunked sequences like Range and LongRange) that do not implement IPending, it seems reasonable to expect that lazy sequences implement it in general. LazySeq does so and this is what really matters in my specific use case.
Now, relying on short-circuiting to prevent realization by empty? (and to prevent giving it an unacceptable argument), we have:
(defn empty-eager-seq? [x] (and (not (lazy? x)) (seq? x) (empty? x)))
Or, if we know we are dealing with sequences like in my case, we can use the less restrictive:
(defn empty-eager? [x] (and (not (lazy? x)) (empty? x)))
Of course we can write safe tests for more general types like:
(defn empty-eager-coll? [x] (and (not (lazy? x)) (coll? x) (empty? x)))
(defn empty-eager-seqable? [x] (and (not (lazy? x)) (seqable? x) (empty? x)))
That being said, the recommended test 1 also works for my case, thanks to short-circuiting and the fact that LazySeq does not implement IPersistentList. Given this and that the question's formulation was suboptimal, I will accept Lee's succinct answer and thank Alan Thompson for his time and for the helpful mini-discussion we had with an upvote.
Option 0 should be avoided since it relies on a class within clojure.lang that is not part of the public API for the package: From the javadoc for clojure.lang:
The only class considered part of the public API is IFn. All other
classes should be considered implementation details.
Option 1 uses functions from the public API and avoids iterating the entire input sequence if it is non-empty
Option 2 iterates the entire input sequence to obtain the count which is potentially expensive.
Option 3 does not appear to be guaranteed and can be circumvented with reflection:
(identical? '() (.newInstance (first (.getDeclaredConstructors (class '()))) (into-array [{}])))
=> false
Given these I'd prefer option 1.
Just use choice (1):
(ns tst.demo.core
(:use tupelo.core tupelo.test) )
(defn empty-list? [arg] (and (list? arg)
(not (seq arg))))
(dotest
(isnt (empty-list? (range)))
(isnt (empty-list? [1 2 3]))
(isnt (empty-list? (list 1 2 3)))
(is (empty-list? (list)))
(isnt (empty-list? []))
(isnt (empty-list? {}))
(isnt (empty-list? #{})))
with result:
-------------------------------
Clojure 1.10.1 Java 13
-------------------------------
Testing tst.demo.core
Ran 2 tests containing 7 assertions.
0 failures, 0 errors.
As you can see by the first test with (range), the infinite lazy seq didn't get realized by empty?.
Update
Choice 0 depends on implementation details (unlikely to change, but why bother?). Also, it is noisier to read.
Choice 2 will blow up for infinite lazy seq's.
Choice 3 is not guaranteed to work. You could have more than one list with zero elements.
Update #2
OK, you are correct re (2). We get:
(type (range)) => clojure.lang.Iterate
Notice that it is not a Lazy-Seq as both you and I expected.
So you are relying on a (non-obvious) detail to prevent getting to count, which will blow up for an infinite lazy seq. Too subtle for my taste. My motto: Keep it as obvious as possible
Re choice (3), again it relies on the implementation detail of (the current release of) Clojure. I could almost make it fail except that clojure.lang.PersistentList$EmptyList is a package-protected inner class, so I would have to really try hard (subvert Java inheritance) to make a duplicate instance of the class, which would then fail.
However, I can come close:
(defn el3? [arg] (identical? () arg))
(dotest
(spyx (type (range)))
(isnt (el3? (range)))
(isnt (el3? [1 3 3]))
(isnt (el3? (list 1 3 3)))
(is (el3? (list)))
(isnt (el3? []))
(isnt (el3? {}))
(isnt (el3? #{}))
(is (el3? ()))
(is (el3? '()))
(is (el3? (list)))
(is (el3? (spyxx (rest [1]))))
(let [jull (LinkedList.)]
(spyx jull)
(spyx (type jull))
(spyx (el3? jull))) ; ***** contrived, but it fails *****
with result
jull => ()
(type jull) => java.util.LinkedList
(el3? jull) => false
So, I again make a plea to keep it obvious and simple.
There are two ways of constructing a software design. One way is to
make it so simple that there are obviously no deficiencies. And the
other way is to make it so complicated that there are no obvious
deficiencies.
---C.A.R. Hoare

What is the recommended way to use memoize?

I have occasionally used memoize function. In usually the following form:
(defn- sqrt-denom [iterations]
(/ 1 (if (= iterations 0)
2
(+ 2 (sqrt-denom (dec iterations))))))
(def sqrt-denom (memoize sqrt-denom))
I assumed that it is "proper" to reuse the function name when memoizing. Is this a good practice? Or should I use different names for the non-memoized and memoized functions?
I would never re-use the name of a top-level def, especially when self-referencing. Two choices:
(defn ^:no-doc sqrt-denom-impl [iterations]
(/ 1 (if (= iterations 0)
2
(+ 2 (sqrt-denom (dec iterations))))))
(def sqrt-denom (memoize sqrt-denom-impl))
or even simpler:
(def sqrt-denom
(memoize (fn [iterations]
(/ 1 (if (= iterations 0)
2
(+ 2 (sqrt-denom (dec iterations))))))
Like Alan Thompson, I've often used (def sqrt-denom (memoize (fn ...))), but it's awkward to split a simple defn into a def and fn just to wrap the latter. The syntactic transformation makes it difficult to switch back and forth between the memoized and the unmemoized versions of the function, and memoize is the one function that made me wish Clojure had something like Python decorators.
Then, recently, I discovered that Clojure does have decorators:
(defn sqrt-denom ...)
(alter-var-root #'sqrt-denom memoize)
This is very similar to your example, but it avoids the confusion of two vars named the same thing, and it clearly states the author's intent. This is now my preferred method for memoizing functions.

'Repeatedly' in Core.Async

Consider the following snippet:
(require '[clojure.core.async :refer :all])
(def my-chan (chan (buffer 10)))
(go (while true
(>! my-chan (rand))))
This basically provides a buffered channel, which always contains some 10 random numbers. When the channel is consumed, the buffer is filled again.
Is there an abstraction for this in core.async? As there are transducers for manipulating the consumption of channels, there might be something for the production of them as well:
For sequences one would go for something like this:
(def my-seq
(map (fn [_] (rand)) (range)))
or, just:
(def my-seq (repeatedly rand))
Which of course is not buffered, but it might give an idea of what I'm looking for.
Transducers don't manipulate the consumption of channels -- they affect the values, but they don't affect the consumption of the data on the channel.
You seem to be asking of a way to abstract the creation of a channel, and then get values off of it as a sequence. Here are some ideas, though I'm not convinced that core.async really offers anything above normal clojure.core functionality in this case.
Abstraction is done here the way it usually is done -- with functions. This will call f and put its result on the channel. The implication here is of course that f will be side-effecting, and impure, otherwise it would be quite a boring channel to consume from, with every value being identical.
(defn chan-factory
[f buf]
(let [c (chan buf)]
(go-loop []
(>! c (f))
(recur))
c))
If you then wanted to create a lazy sequence from this, you could do:
(defn chan-to-seq [c]
(lazy-seq
(cons (<!! c) (chan-to-seq c))))
Define your seq:
(def rand-from-chan (chan-to-seq (chan-factory rand 10)))
(take 5 rand-from-chan)
=>
(0.6873518531956767
0.6940302424998631
0.07293052906941855
0.7264083273536271
0.4670275072317531)
However, you can accomplish this same thing by doing:
(def rand-nums (repeatedly rand))
So, while what you're doing is a great thought experiment, it may be more helpful to find some concrete use cases, and then maybe you will receive more specific ideas. Good luck!

Which Vars affect a Clojure function?

How do I programmatically figure out which Vars may affect the results of a function defined in Clojure?
Consider this definition of a Clojure function:
(def ^:dynamic *increment* 3)
(defn f [x]
(+ x *increment*))
This is a function of x, but also of *increment* (and also of clojure.core/+(1); but I'm less concerned with that). When writing tests for this function, I want to make sure that I control all relevant inputs, so I do something like this:
(assert (= (binding [*increment* 3] (f 1)) 4))
(assert (= (binding [*increment* -1] (f 1)) 0))
(Imagine that *increment* is a configuration value that someone might reasonably change; I don't want this function's tests to need changing when this happens.)
My question is: how do I write an assertion that the value of (f 1) can depend on *increment* but not on any other Var? Because I expect that one day someone will refactor some code and cause the function to be
(defn f [x]
(+ x *increment* *additional-increment*))
and neglect to update the test, and I would like to have the test fail even if *additional-increment* is zero.
This is of course a simplified example – in a large system, there can be lots of dynamic Vars, and they can get referenced through a long chain of function calls. The solution needs to work even if f calls g which calls h which references a Var. It would be great if it didn't claim that (with-out-str (prn "foo")) depends on *out*, but this is less important. If the code being analyzed calls eval or uses Java interop, of course all bets are off.
I can think of three categories of solutions:
Get the information from the compiler
I imagine the compiler does scan function definitions for the necessary information, because if I try to refer to a nonexistent Var, it throws:
user=> (defn g [x] (if true x (+ *foobar* x)))
CompilerException java.lang.RuntimeException: Unable to resolve symbol: *foobar* in this context, compiling:(NO_SOURCE_PATH:24)
Note that this happens at compile time, and regardless of whether the offending code will ever be executed. Thus the compiler should know what Vars are potentially referenced by the function, and I would like to have access to that information.
Parse the source code and walk the syntax tree, and record when a Var is referenced
Because code is data and all that. I suppose this means calling macroexpand and handling each Clojure primitive and every kind of syntax they take. This looks so much like a compilation phase that it would be great to be able to call parts of the compiler, or somehow add my own hooks to the compiler.
Instrument the Var mechanism, execute the test and see which Vars get accessed
Not as complete as the other methods (what if a Var is used in a branch of the code that my test fails to exercise?) but this would suffice. I imagine I would need to redefine def to produce something that acts like a Var but records its accesses somehow.
(1) Actually that particular function doesn't change if you rebind +; but in Clojure 1.2 you can bypass that optimization by making it (defn f [x] (+ x 0 *increment*)) and then you can have fun with (binding [+ -] (f 3)). In Clojure 1.3 attempting to rebind + throws an error.
Regarding your first point you could consider using the analyze library. With it you can quite easily figure out which dynamic vars are used in an expression:
user> (def ^:dynamic *increment* 3)
user> (def src '(defn f [x]
(+ x *increment*)))
user> (def env {:ns {:name 'user} :context :eval})
user> (->> (analyze-one env src)
expr-seq
(filter (op= :var))
(map :var)
(filter (comp :dynamic meta))
set)
#{#'user/*increment*}
I know that this doesn't answer your question, but wouldn't it be a lot less work to just provide two versions of a function where one version has no free variables, and the other version calls the first one with the appropriate top-level defines?
For example:
(def ^:dynamic *increment* 3)
(defn f
([x]
(f x *increment*))
([x y]
(+ x y)))
This way you can write all your tests against (f x y), which doesn't rely on any global state.