Is it possible to create a ref with a transducer in Clojure, in a way analogous to creating a chan with a transducer?
i.e., when you create a chan with a transducer, it filters/maps all the inputs into the outputs.
I'd expect there's also a way to create a ref such that whatever you set, it can either ignore or modify the input. Is this possible to do?
Adding a transducer to a channel modifies the contents as they pass through, which is roughly analogous to adding a watch to a ref that applies it's own change each time the value changes. This change it's self then triggers the watch again so be careful not to blow the stack if they are recursive.
user> (def r (ref 0))
#'user/r
user> (add-watch r :label
(fn [label the-ref old-state new-state]
(println "adding that little something extra")
(if (< old-state 10) (dosync (commute the-ref inc)))))
#<Ref#1af618c2: 0>
user> (dosync (alter r inc))
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
adding that little something extra
1
user> #r
11
You could even apply a transducer to the state of the atom if you wanted.
This is an interesting idea, but the wrong way to go about it for at least a couple reasons. You'd lose some relationships you'd expect to hold:
(alter r identity) =/= r
(alter r f)(alter r f) =/= (alter r (comp f f))
(alter r f) =/= (ref-set r (f #r))
Also some transducers are side-effecting volatiles, and have no business in a dosync block. i.e. if you use (take n) as your transducer then if your dosync fails, then it'll retry as though invoked with (take (dec n)), which violates dosync body requirements.
The problem is a ref lets you read and write as separate steps. If instead there was something foundational that let you "apply" an input to a hidden "state" and collect the output all in one step, consistently with the STM, then that'd be something to work with.
Related
I find myself writing a lot of clojure in this manner:
(defn my-fun [input]
(let [result1 (some-complicated-procedure input)
result2 (some-other-procedure result1)]
(do-something-with-results result1 result2)))
This let statement seems very... imperative. Which I don't like. In principal, I could be writing the same function like this:
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
The problem with this is that it involves recomputation of some-complicated-procedure, which may be arbitrarily expensive. Also you can imagine that some-complicated-procedure is actually a series of nested function calls, and then I either have to write a whole new function, or risk that changes in the first invocation don't get applied to the second:
E.g. this works, but I have to have an extra shallow, top-level function that makes it hard to do a mental stack trace:
(defn some-complicated-procedure [input] (lots (of (nested (operations input)))))
(defn my-fun [input]
(do-something-with-results (some-complicated-procedure input)
(some-other-procedure (some-complicated-procedure input)))))
E.g. this is dangerous because refactoring is hard:
(defn my-fun [input]
(do-something-with-results (lots (of (nested (operations (mistake input))))) ; oops made a change here that wasn't applied to the other nested calls
(some-other-procedure (lots (of (nested (operations input))))))))
Given these tradeoffs, I feel like I don't have any alternatives to writing long, imperative let statements, but when I do, I cant shake the feeling that I'm not writing idiomatic clojure. Is there a way I can address the computation and code cleanliness problems raised above and write idiomatic clojure? Are imperitive-ish let statements idiomatic?
The kind of let statements you describe might remind you of imperative code, but there is nothing imperative about them. Haskell has similar statements for binding names to values within bodies, too.
If your situation really needs a bigger hammer, there are some bigger hammers that you can either use or take for inspiration. The following two libraries offer some kind of binding form (akin to let) with a localized memoization of results, so as to perform only the necessary steps and reuse their results if needed again: Plumatic Plumbing, specifically the Graph part; and Zach Tellman's Manifold, whose let-flow form furthermore orchestrates asynchronous steps to wait for the necessary inputs to become available, and to run in parallel when possible. Even if you decide to maintain your present course, their docs make good reading, and the code of Manifold itself is educational.
I recently had this same question when I looked at this code I wrote
(let [user-symbols (map :symbol states)
duplicates (for [[id freq] (frequencies user-symbols) :when (> freq 1)] id)]
(do-something-with duplicates))
You'll note that map and for are lazy and will not be executed until do-something-with is executed. It's also possible that not all (or even not any) of the states will be mapped or the frequencies calculated. It depends on what do-something-with actually requests of the sequence returned by for. This is very much functional and idiomatic functional programming.
i guess the simplest approach to keep it functional would be to have a pass-through state to accumulate the intermediate results. something like this:
(defn with-state [res-key f state]
(assoc state res-key (f state)))
user> (with-state :res (comp inc :init) {:init 10})
;;=> {:init 10, :res 11}
so you can move on to something like this:
(->> {:init 100}
(with-state :inc'd (comp inc :init))
(with-state :inc-doubled (comp (partial * 2) :inc'd))
(with-state :inc-doubled-squared (comp #(* % %) :inc-doubled))
(with-state :summarized (fn [st] (apply + (vals st)))))
;;=> {:init 100,
;; :inc'd 101,
;; :inc-doubled 202,
;; :inc-doubled-squared 40804,
;; :summarized 41207}
The let form is a perfectly functional construct and can be seen as syntactic sugar for calls to anonymous functions. We can easily write a recursive macro to implement our own version of let:
(defmacro my-let [bindings body]
(if (empty? bindings)
body
`((fn [~(first bindings)]
(my-let ~(rest (rest bindings)) ~body))
~(second bindings))))
Here is an example of calling it:
(my-let [a 3
b (+ a 1)]
(* a b))
;; => 12
And here is a macroexpand-all called on the above expression, that reveal how we implement my-let using anonymous functions:
(clojure.walk/macroexpand-all '(my-let [a 3
b (+ a 1)]
(* a b)))
;; => ((fn* ([a] ((fn* ([b] (* a b))) (+ a 1)))) 3)
Note that the expansion doesn't rely on let and that the bound symbols become parameter names in the anonymous functions.
As others write, let is actually perfectly functional, but at times it can feel imperative. It's better to become fully comfortable with it.
You might, however, want to kick the tires of my little library tl;dr that lets you write code like for example
(compute
(+ a b c)
where
a (f b)
c (+ 100 b))
(def alice-height
(ref 3))
(def right-hand-bites
(ref 10))
(defn eat-from-right-hand []
(dosync
(when (pos? #right-hand-bites)
(alter right-hand-bites dec)
(alter alice-height #(+ % 24)))))
This code is from the book Living Clojure. In the book, the author also gave an example with alter replaced by commute. I'm wondering with the pos? test at the beginning, can we really do this replacement?
No, replacing alter with commute when decrementing right-hand-bites is not correct.
The intention of the conditional is apparently to prevent right-hand-bites from becoming negative. The decrement is only valid under the assumption that right-hand-bites won’t change until the end of the transaction. While, like alter, commute has its own snapshot view of the ref world, it will re-read and re-apply the commute function to the ref at commit time, and that would be a mistake in this program.
So, with commute it is possible to commit a negative value to right-hand-bites.
Either stick with alter, or use ensure instead of # (though that makes the whole commute exercise rather pointless).
What is the simplest way to trigger a side-effecting function to be called only when an atom's value changes?
If I were using a ref, I think I could just do this:
(defn transform-item [x] ...)
(defn do-side-effect-on-change [] nil)
(def my-ref (ref ...))
(when (dosync (let [old-value #my-ref
_ (alter! my-ref transform-item)
new-value #my-ref]
(not= old-value new-value)))
(do-side-effect-on-change))
But this seems seems a bit roundabout, since I'm using a ref even though I am not trying to coordinate changes across multiple refs. Essentially I am using it just to conveniently access the old and new value within a successful transaction.
I feel like I should be able to use an atom instead. Is there a solution simpler than this?
(def my-atom (atom ...))
(let [watch-key ::side-effect-watch
watch-fn (fn [_ _ old-value new-value]
(when (not= old-value new-value)
(do-side-effect-on-change)))]
(add-watch my-atom watch-key watch-fn)
(swap! my-atom transform-item)
(remove-watch watch-key))
This also seems roundabout, because I am adding and removing the watch around every call to swap!. But I need this, because I don't want a watch hanging around that causes the side-effecting function to be triggered when other code modifies the atom.
It is important that the side-effecting function be called exactly once per mutation to the atom, and only when the transform function transform-item actually returns a new value. Sometimes it will return the old value, yielding new change.
(when (not= #a (swap! a transform))
(do-side-effect))
But you should be very clear about what concurrency semantics you need. For example another thread may modify the atom between reading it and swapping it:
a = 1
Thread 1 reads a as 1
Thread 2 modifies a to 2
Thread 1 swaps a from 2 to 2
Thread 1 determines 1 != 2 and calls do-side-effect
It is not clear to me from the question whether this is desirable or not desirable. If you do not want this behavior, then an atom just will not do the job unless you introduce concurrency control with a lock.
Seeing as you started with a ref and asked about an atom, I think you have probably given some thought to concurrency already. It seems like from your description the ref approach is better:
(when (dosync (not= #r (alter r transform))
(do-side-effect))
Is there a reason you don't like your ref solution?
If the answer is "because I don't have concurrency" Then I would encourage you to use a ref anyway. There isn't really a downside to it, and it makes your semantics explicit. IMO programs tend to grow and to a point where concurrency exists, and Clojure is really great at being explicit about what should happen when it exists. (For example oh I'm just calculating stuff, oh I'm just exposing this stuff as a web service now, oh now I'm concurrent).
In any case, bear in mind that functions like alter and swap! return the value, so you can make use of this for concise expressions.
I'm running into the same situation and just come up 2 solutions.
state field :changed?
Keeping a meanless :changed mark in atom to track swap function. And take the return value of swap! to see if things changed. For example:
(defn data (atom {:value 0 :changed? false}))
(let [{changed? :changed?} (swap! data (fn [data] (if (change?)
{:value 1 :changed? true}
{:value 0 :change? false})))]
(when changed? (do-your-task)))
exception based
You can throw an Exception in swap function, and catch it outside:
(try
(swap! data (fn [d] (if (changed?) d2 (ex-info "unchanged" {})))
(do-your-task)
(catch Exception _
))
What's the difference between?
1.
(def x (ref 0))
(dosync
(commute x f))
2.
(def x (atom 0))
(swap! x f))
These two examples work equally
If you have only one ref and one atom, there actually is no big difference. I think you might learn something about the difference of atoms and refs.
commute is a function which can be called in any place in the dosync block. Clojure itself chooses when to execute an commute inside this block.
Maybe I explan it with a small example:
(def x (ref 0))
(def y (ref 0))
(dosync (alter x inc)
(alter y dec))
alter changes the value of a ref inside a dosync block. Because refs are coordinated and synchronous, we need to place them inside this block. If here occurs an error while altering one of the refs, the whole block fails and your refs have the same value as they had before calling the alters.
The difference to commute is the following: If we have the following code block
(def x (ref 0))
(def y (ref 0))
(dosync (commute x inc)
(alter y dec))
If an error occurs here while changing the value of your refs, commute is still called and it changes your ref. It does not care about the result of the alter and even if the alter fails, commute does increment x.
Clojure can choose when to call commute, so it can be executed even if an error occurs.
The big difference to atoms is that atoms are not coordinated. If there are multiple swap! on one atom, maybe just the first swap! will be executed. One of my projects got stuck on this issue, I had to ensure that the first swap! has terminated before the next swap! is called.
So again: Think of what you really need. Do you need an atom or some refs? If you choose refs, you have the choice between alter, which is executed at the position you want it to, or commute where Clojure can decide when to call it (the call is then commutative).
If you choose an atom, you can only change one value while this atom is executed. If another call trys to swap! the value, it is rejected.
In Clojure, if I have a function f,
(defn f [& r] ... )
and I have a seq args with the arguments I want to call f with, I can easily use apply:
(apply f args)
Now, say I have another function g, which is designed to take any of a number of optional, named arguments - that is, where the rest argument is destructured as a map:
(defn g [& {:keys [a b] :as m}] ... )
I'd normally call g by doing something like
(g :a 1 :b 2)
but if I happen to have a map my-map with the value {:a 1 :b 2}, and I want to "apply" g to my-map - in other words, get something that would end up as the above call, then I naturally couldn't use apply, since it would be equivalent to
(g [:a 1] [:b 2])
Is there a nice way to handle this? May I have gone off track in my design to end up with this? The best solution I can find would be
(apply g (flatten (seq my-map)))
but I surely don't like it. Any better solutions?
EDIT: A slight improvement to the suggested solution might be
(apply g (mapcat seq my-map))
which at least removes one function call, but it may still not be very clear what's going on.
I have stumbled into this problem myself and ended up defining functions to expect one map. A map can have a variable amount of key/value pairs, and if flexible enough, so there is no need for & rest arguments. Also there is no pain with apply. Makes life a lot easier!
(defn g [{:keys [a b] :as m}] ... )
There is no better direct way than converting to a seq.
You are done. You have done all you can.
It's just not really clojurish to have Common Lisp style :keyword arg functions. If you look around Clojure code you will find that almost no functions are written that way.
Even the great RMS is not a fan of them:
"One thing I don't like terribly much is keyword arguments (8). They don't seem quite Lispy to me; I'll do it sometimes but I minimize the times when I do that." (Source)
At the moment where you have to break a complete hash map into pieces just to pass all of them as keyword mapped arguments you should question your function design.
I find that in the case where you want to pass along general options like :consider-nil true you are probably never going to invoke the function with a hash-map {:consider-nil true}.
In the case where you want to do an evaluation based on some keys of a hash map you are 99% of the time having a f ([m & args]) declaration.
When I started out defining functions in Clojure I hit the same problem. However after thinking more about the problems I tried to solve I noticed myself using destructoring in function declaration almost never.
Here is a very simplistic function which may be used exactly as apply, except that the final arg (which should be a map) will be expanded out to :key1 val1 :key2 val2 etc.
(defn mapply
[f & args]
(apply f (reduce concat (butlast args) (last args))))
I'm sure there are more efficient ways to do it, and whether or not you'd want to end up in a situation where you'd have to use such a function is up for debate, but it does answer the original question. Mostly, I'm childishly satisfied with the name...
Nicest solution I have found:
(apply g (apply concat my-map))