I'm developing a library in clojure, which - in a way - needs to be stateful. In order to not spending too many word in abstract descriptions, here would be an OOP example of how I'd imagine the API of the library could be used.
mylib = new lib();
state1 = mylib.getState();
mylib.continue();
state2 = mylib.getState();
mylib.continue();
state3 = mylib.getState();
[..]
(whereas obviously state1 != state2 != state3)
Ok, how could this be done in a functional language like Clojure?
One approach, that comes to my mind:
(require '(lib.core :as mylib))
(def state1 (mylib/start-state))
(def state2 (mylib/continue state1))
(def state3 (mylib/continue state2))
[..]
This approach does not allocate the state-keeping to the library-side. The problem I have with this is the following: my state does keep information, that should be public to the API user. But also it keeps information that is important for the generation of the next state, which is however not relevant to the public.
Well, there could be another function (mylib/extract-relevant-data state1) which could postprocess the state in oder for a "clean usage".
I would be really intersted in learning in which ways I could approach this in Clojure.
Method 1: reimplement OOP. I'm treating last-time-called as private data and relevant state as what you want to show to people.
(defn start-state []
(let [last-time-called (atom (new java.util.Date))
relevant-state (atom 1)
private-update (fn [] (swap! last-time-called (fn [_] (new java.util.Date))))
get-state (fn [] (do (private-update)
#relevant-state))
continue (fn [] (do (private-update)
(swap! relevant-state inc)
nil))]
{:get-state get-state :continue continue}))
Demonstration:
stack-prj.hiddenState> (def mylib (start-state))
#'stack-prj.hiddenState/mylib
stack-prj.hiddenState> ((mylib :get-state))
1
stack-prj.hiddenState> ((mylib :continue))
nil
stack-prj.hiddenState> ((mylib :get-state))
2
stack-prj.hiddenState> ((mylib :continue))
nil
Note that get-state and continue have access to last-time-called, if they need it.
Method 2: pure functions on simple data.
(defn new-lib []
{:relevant-state 1
:last-time-called (new java.util.Date)})
(defn get-state [lib]
(lib :relevant-state))
(defn lib-continue [lib]
{:relevant-state (inc (lib :relevant-state))
:last-time-called (new java.util.Date)})
Demonstration:
stack-prj.noHiddenState> (def mylib (new-lib))
#'stack-prj.noHiddenState/mylib
stack-prj.noHiddenState> (get-state mylib)
1
stack-prj.noHiddenState> (def ml2 (lib-continue mylib))
#'stack-prj.noHiddenState/ml2
stack-prj.noHiddenState> (get-state ml2)
2
Note that with method 2, you cannot update the object's private variables when the user accesses the state via get-state. If you need this functionality, then method 1 meets your needs better; if you don't, then method 2 offers more clean, idiomatic, and maintainable code than the first method.
Related
I want to figure out how best to create an async component, or accommodate async code in a Component-friendly way. This is the best I can come up with, and... it just doesn't feel quite right.
The Gist: take words, uppercase them and reverse them, finally print them.
Problem 1: I can't get the system to stop at the end. I expect to see a println of the individual c-chans stopping, but don't.
Problem 2: How do I properly inject deps. into the producer/consumer fns? I mean, they're not components, and I think they should not be components since they have no sensible lifecycle.
Problem 3: How do I idiomatically handle the async/pipeline-creating side-effects named a>b, and b>c? Should a pipeline be a component?
(ns pipelines.core
(:require [clojure.core.async :as async
:refer [go >! <! chan pipeline-blocking close!]]
[com.stuartsierra.component :as component]))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; PIPELINES
(defn a>b [a> b>]
(pipeline-blocking 4
b>
(map clojure.string/upper-case)
a>))
(defn b>c [b> c>]
(pipeline-blocking 4
c>
(map (comp (partial apply str)
reverse))
b>))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; PRODUCER / CONSUMER
(defn producer [a>]
(doseq [word ["apple" "banana" "carrot"]]
(go (>! a> word))))
(defn consumer [c>]
(go (while true
(println "Your Word Is: " (<! c>)))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; SYSTEM
(defn pipeline-system [config-options]
(let [c-chan (reify component/Lifecycle
(start [this]
(println "starting chan: " this)
(chan 1))
(stop [this]
(println "stopping chan: " this)
(close! this)))]
(-> (component/system-map
:a> c-chan
:b> c-chan
:c> c-chan)
(component/using {}))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; RUN IT!
(def system (atom nil))
(let [_ (reset! system (component/start (pipeline-system {})))
_ (a>b (:a> #system) (:b> #system))
_ (b>c (:b> #system) (:c> #system))
_ (producer (:a> #system))
_ (consumer (:c> #system))
_ (component/stop #system)])
EDIT:
I started thinking about the following, but I'm not quite sure if it's closing properly...
(extend-protocol component/Lifecycle
clojure.core.async.impl.channels.ManyToManyChannel
(start [this]
this)
(stop [this]
(close! this)))
I rewrote your example a little to make it reloadable:
Reloadable Pipeline
(ns pipeline
(:require [clojure.core.async :as ca :refer [>! <!]]
[clojure.string :as s]))
(defn upverse [from to]
(ca/pipeline-blocking 4
to
(map (comp s/upper-case
s/reverse))
from))
(defn produce [ch xs]
(doseq [word xs]
(ca/go (>! ch word))))
(defn consume [ch]
(ca/go-loop []
(when-let [word (<! ch)]
(println "your word is:" word)
(recur))))
(defn start-engine []
(let [[from to] [(ca/chan) (ca/chan)]]
(upverse to from)
(consume from)
{:stop (fn []
(ca/close! to)
(ca/close! from)
(println "engine is stopped"))
:process (partial produce to)}))
this way you can just do (start-engine) and use it to process word sequences:
REPL time
boot.user=> (require '[pipeline])
boot.user=> (def engine (pipeline/start-engine))
#'boot.user/engine
running with it
boot.user=> ((engine :process) ["apple" "banana" "carrot"])
your word is: TORRAC
your word is: ANANAB
your word is: ELPPA
boot.user=> ((engine :process) ["do" "what" "makes" "sense"])
your word is: OD
your word is: SEKAM
your word is: ESNES
your word is: TAHW
stopping it
boot.user=> ((:stop engine))
engine is stopped
;; engine would not process anymore
boot.user=> ((engine :process) ["apple" "banana" "carrot"])
nil
State Management
Depending on how you intend to use this pipeline, a state management framework like Component might not be needed at all: no need to add anything "just in case", starting and stopping the pipeline in this case is a matter of calling two functions.
However in case this pipeline is used within a larger app with more states you could definitely benefit from a state management library.
I am not a fan of Component primarily because it requires a full app buyin (which makes it a framework), but I do respect other people using it.
mount
I would recommend to either not use anything specific in case the app is small: you, for example could compose this pipeline with other pipelines / logic and kick it off from -main, but if the app is any bigger and has more unrelated states, here is all you need to do to add mount to it:
(defstate engine :start (start-engine)
:stop ((:stop engine)))
starting pipeline
boot.user=> (mount/start)
{:started ["#'pipeline/engine"]}
running with it
boot.user=> ((engine :process) ["do" "what" "makes" "sense"])
your word is: OD
your word is: SEKAM
your word is: ESNES
your word is: TAHW
stopping it
boot.user=> (mount/stop)
engine is stopped
{:stopped ["#'pipeline/engine"]}
Here is a gist with a full example that includes build.boot.
You can just download and play with it via boot repl
[EDIT]: to answer the comments
In case you are already hooked on Component, this should get you started:
(defrecord WordEngine []
component/Lifecycle
(start [component]
(merge component (start-engine)))
(stop [component]
((:stop component))
(assoc component :process nil :stop nil)))
This, on start, would create a WordEngine object that would have a :process method.
You won't be able to call it as you would a normal Clojure function: i.e. from REPL or any namespace just by :requireing it, unless you pass a reference to the whole system around which is not recommended.
So in order to call it, this WordEngine would need to be plugged into a Component system, and injected into yet another Component which can then destructure the :process function and call it.
I have this small game world state, something like the following:
(defn odds [percentage]
(< (rand-int 100) percentage))
(defn world []
{:entities []})
(defn make-bird []
{:pos [(rand-int 100) (rand-int 100)]
:age 0
:dir (vec/dir (rand (. Math PI)))})
(defn generate-entities [entities]
(if (odds 10)
(conj entities (make-bird))
entities))
(defn update-entity [entity]
(-> entity
(update :pos (partial vec/add (:dir entity)))
(update :age inc)))
(defn update-entities [entities]
(vec (map update-entity entities)))
(defn old? [{age :age}]
(> age 10))
(defn prune-entities [entities]
(vec (filter #(not (old? %)) entities)))
(defn update-world [world]
(-> world
(update :entities generate-entities)
(update :entities update-entities)
(update :entities prune-entities)))
So update-world goes through three steps. First there's a 1/10 chance of generating a new bird entity, which flies in a random direction. Then it updates all birds, updating their position and incrementing their age. Then it prunes all old birds.
I use this same technique for generating particles systems. You can do fun stuff like (iterate update-world (world)) to get a lazy list of world states which you can consume at whatever frame rate you want.
However, I now have a game world with autonomous entities which roam around and do stuff, kind of like the birds. But I want to get a textual representation of what happened when evaluating update-world. For example, update-world would ideally return a tuple of the new world state and a vector of strings - ["A bird was born at [12, 8].", "A bird died of old age at [1, 2]."].
But then I really can't use (iterate update-world (world)) anymore. I can't really see how to do this.
Is this something you'd use with-out-string for?
If you want to enhance only your top-level function (update-world) in your case you can just create a wrapper function that you can use in iterate. A simple example:
(defn increment [n]
(inc n))
(defn logging-increment [[_ n]]
(let [new-n (increment n)]
[(format "Old: %s New: %s" n new-n) new-n]))
(take 3 (iterate logging-increment [nil 0]))
;; => ([nil 0] ["Old: 0 New: 1" 1] ["Old: 1 New: 2" 2])
In case you want to do it while collecting data at multiple level and you don't want to modify the signatures of your existing functions (e.g. you want to use it only for debugging), then using dynamic scope seems like a reasonable option.
Alternatively you can consider using some tracing tools, like clojure/tools.trace. You could turn on and off logging of your function calls by simply changing defn to deftrace or using trace-ns or trace-vars.
There are two potential issues with using with-out-str
It returns a string, not a vector. If you need to use a vector, you'll need to use something else.
Only the string is returned. If you are using with-out-str to wrap a side-effect (e.g., swap!), this might be fine.
For debugging purposes, I usually just use println. You can use with-out if you want control over where the output goes. You could even implement a custom stream that collects the output into a vector of strings if you wanted. You could get similar results with a dynamically bound vector that you accumulate (via set!) the output string (or wrap the vector in an atom and use swap!).
If the accumulated vector is part of the computation per se, and you want to remain pure, you might consider using a monad.
What about using clojure.data/diff to generate the string representation of changes? You could do something like this:
(defn update-world [[world mutations]]
(let [new-world (-> world
(update :entities generate-entities)
(update :entities update-entities)
(update :entities prune-entities))]
[new-world (mutations (clojure.data/diff world new-world))]))
Then you could do something like (iterate update-world [(world) []]) to get the ball rolling.
I am implementing an app using Stuart Sierra component. As he states in the README :
Having a coherent way to set up and tear down all the state associated
with an application enables rapid development cycles without
restarting the JVM. It can also make unit tests faster and more
independent, since the cost of creating and starting a system is low
enough that every test can create a new instance of the system.
What would be the preferred strategy here ? Something similar to JUnit oneTimeSetUp / oneTimeTearDown , or really between each test (similar to setUp / tearDown) ?
And if between each test, is there a simple way to start/stop a system for all tests (before and after) without repeating the code every time ?
Edit : sample code to show what I mean
(defn test-component-lifecycle [f]
(println "Setting up test-system")
(let [s (system/new-test-system)]
(f s) ;; I cannot pass an argument here ( https://github.com/clojure/clojure/blob/master/src/clj/clojure/test.clj#L718 ), so how can I pass a system in parameters of a test ?
(println "Stopping test-system")
(component/stop s)))
(use-fixtures :once test-component-lifecycle)
Note : I am talking about unit-testing here.
I would write a macro, which takes a system-map and starts all components before running tests and stop all components after testing.
For example:
(ns de.hh.new-test
(:require [clojure.test :refer :all]
[com.stuartsierra.component :as component]))
;;; Macro to start and stop component
(defmacro with-started-components [bindings & body]
`(let [~(bindings 0) (component/start ~(bindings 1))]
(try
(let* ~(destructure (vec (drop 2 bindings)))
~#body)
(catch Exception e1#)
(finally
(component/stop ~(bindings 0))))))
;; Test Component
(defprotocol Action
(do-it [self]))
(defrecord TestComponent [state]
component/Lifecycle
(start [self]
(println "====> start")
(assoc self :state (atom state)))
(stop [self]
(println "====> stop"))
Action
(do-it [self]
(println "====> do action")
#(:state self)))
;: TEST
(deftest ^:focused component-test
(with-started-components
[system (component/system-map :test-component (->TestComponent"startup-state"))
test-component (:test-component system)]
(is (= "startup-state" (do-it test-component)))))
Running Test you should see the out put like this
====> start
====> do action
====> stop
Ran 1 tests containing 1 assertions.
0 failures, 0 errors.
I have a set of functions that all have the same first parameter.
(defn get-file [dir filename] ...)
(defn write-file [dir filename] ...)
I'd like to partially apply all of them at once, basically. Seems like I could wrap them all in a function like this:
(defn get-fns [dir]
{:get-file (fn [filename] ...)
:write-file (fn [filename] ...)})
But that seems like accessing the functions would be kind of annoying.
(let [fns (get-fns dir-name)]
((fns :get-file) filename)))
I suppose I could use a mutable var as well, but that doesn't seem very, well, functional. Is there a canonical/idiomatic way to do this?
Let's think it through: you've got a list of things, and you want to apply something to each of them, so map. What do you want to apply to them? You want to partially apply the first argument, so partial. Then you want to have a unique reference to each of those values, so use deconstruction on the list.
Adding all that up, assuming dir, get-file, and write-file are all defined, you'd do
(let [[get-file-here write-file-here] (map #(partial % dir) [get-file write-file])]
...)
Here's a full example
(let [[add-to-3 sub-from-3] (map #(partial % 3) [+ -])]
(prn (add-to-3 2)) ; 5
(prn (sub-from-3 5))) ; -2
So we want a series of partial applications that we can use in a local binding context.
(defn get-fns
[dir]
{:get-f (partial get-file dir)
:write-f (partial write-file dir)})
Then, in a local binding, we can use them
(let [{:keys [get-f write-f]} (get-fns dir)]
(get-f file-name))
(you are of course free to use your original keywords, by changing the keywords I make it unambiguous that get-f is coming from get-fns and is not the globally bound var).
First, I have no experience with CS and Clojure is my first language, so pardon if the following problem has a solution, that is immediately apparent for a programmer.
The summary of the question is as follows: one needs to create atoms at will with unknown yet symbols at unknown times. My approach revolves around a) storing temporarily the names of the atoms as strings in an atom itself; b) changing those strings to symbols with a function; c) using a function to add and create new atoms. The problem pertains to step "c": calling the function does not create new atoms, but using its body does create them.
All steps taken in the REPL are below (comments follow code blocks):
user=> (def atom-pool
#_=> (atom ["a1" "a2"]))
#'user/atom-pool
'atom-pool is the atom that stores intermediate to-be atoms as strings.
user=> (defn atom-symbols []
#_=> (mapv symbol (deref atom-pool)))
#'user/atom-symbols
user=> (defmacro populate-atoms []
#_=> (let [qs (vec (remove #(resolve %) (atom-symbols)))]
#_=> `(do ~#(for [s qs]
#_=> `(def ~s (atom #{}))))))
#'user/populate-atoms
'populate-atoms is the macro, that defines those atoms. Note, the purpose of (remove #(resolve %) (atom-symbols)) is to create only yet non-existing atoms. 'atom-symbols reads 'atom-pool and turns its content to symbols.
user=> (for [s ['a1 'a2 'a-new]]
#_=> (resolve s))
(nil nil nil)
Here it is confirmed that there are no 'a1', 'a2', 'a-new' atoms as of yet.
user=> (defn new-atom [a]
#_=> (do
#_=> (swap! atom-pool conj a)
#_=> (populate-atoms)))
#'user/new-atom
'new-atom is the function, that first adds new to-be atom as string to `atom-pool. Then 'populate-atoms creates all the atoms from 'atom-symbols function.
user=> (for [s ['a1 'a2 'a-new]]
#_=> (resolve s))
(#'user/a1 #'user/a2 nil)
Here we see that 'a1 'a2 were created as clojure.lang.Var$Unbound just by defining a function, why?
user=> (new-atom "a-new")
#'user/a2
user=> (for [s ['a1 'a2 'a-new]]
#_=> (resolve s))
(#'user/a1 #'user/a2 nil)
Calling (new-atom "a-new") did not create the 'a-new atom!
user=> (do
#_=> (swap! atom-pool conj "a-new")
#_=> (populate-atoms))
#'user/a-new
user=> (for [s ['a1 'a2 'a-new]]
#_=> (resolve s))
(#'user/a1 #'user/a2 #'user/a-new)
user=>
Here we see that resorting explicitly to 'new-atom's body did create the 'a-new atom. 'a-new is a type of clojure.lang.Atom, but 'a1 and 'a2 were skipped due to already being present in the namespace as clojure.lang.Var$Unbound.
Appreciate any help how to make it work!
EDIT: Note, this is an example. In my project the 'atom-pool is actually a collection of maps (atom with maps). Those maps have keys {:name val}. If a new map is added, then I create a corresponding atom for this map by parsing its :name key.
"The summary of the question is as follows: one needs to create atoms at will with unknown yet symbols at unknown times. "
This sounds like a solution looking for a problem. I would generally suggest you try another way of achieving whatever the actual functionality is without generating vars at runtime, but if you must, you should use intern and leave out the macro stuff.
You cannot solve this with macros since macros are expanded at compile time, meaning that in
(defn new-atom [a]
(do
(swap! atom-pool conj a)
(populate-atoms)))
populate-atoms is expanded only once; when the (defn new-atom ...) form is compiled, but you're attempting to change its expansion when new-atom is called (which necessarily happens later).
#JoostDiepenmaat is right about why populate-atoms is not behaving as expected. You simply cannot do this using macros, and it is generally best to avoid generating vars at runtime. A better solution would be to define your atom-pool as a map of keywords to atoms:
(def atom-pool
(atom {:a1 (atom #{}) :a2 (atom #{})}))
Then you don't need atom-symbols or populate-atoms because you're not dealing with vars at compile-time, but typical data structures at run-time. Your new-atom function could look like this:
(defn new-atom [kw]
(swap! atom-pool assoc kw (atom #{})))
EDIT: If you don't want your new-atom function to override existing atoms which might contain actual data instead of just #{}, you can check first to see if the atom exists in the atom-pool:
(defn new-atom [kw]
(when-not (kw #atom-pool)
(swap! atom-pool assoc kw (atom #{}))))
I've already submitted one answer to this question, and I think that that answer is better, but here is a radically different approach based on eval:
(def atom-pool (atom ["a1" "a2"]))
(defn new-atom! [name]
(load-string (format "(def %s (atom #{}))" name)))
(defn populate-atoms! []
(doseq [x atom-pool]
(new-atom x)))
format builds up a string where %s is substituted with the name you're passing in. load-string reads the resulting string (def "name" (atom #{})) in as a data structure and evals it (this is equivalent to (eval (read-string "(def ...)
Of course, then we're stuck with the problem of only defining atoms that don't already exist. We could change the our new-atom! function to make it so that we only create an atom if it doesn't already exist:
(defn new-atom! [name]
(when-not (resolve (symbol name))
(load-string (format "(def %s (atom #{}))" name name))))
The Clojure community seems to be against using eval in most cases, as it is usually not needed (macros or functions will do what you want in 99% of cases*), and eval can be potentially unsafe, especially if user input is involved -- see Brian Carper's answer to this question.
*After attempting to solve this particular problem using macros, I came to the conclusion that it either cannot be done without relying on eval, or my macro-writing skills just aren't good enough to get the job done with a macro!
At any rate, I still think my other answer is a better solution here -- generally when you're getting way down into the nuts & bolts of writing macros or using eval, there is probably a simpler approach that doesn't involve metaprogramming.