I have the code bellow for my Pedestal component. When Stuart Sierra's library starts my system-map, the method start implemented in the Pedestal defrecord gets called and it returns an updated version of my component with :pedestal-server associated. Shouldn't the lifecycle manager propagate the updated component so it could be used by the stop method? Whenever I try to stop the server by calling (component/stop (system)) in the REPL, nothing happens because the :pedestal-server key is set to nil.
(defrecord Pedestal [service-map pedestal-server]
component/Lifecycle
(start [this]
(if pedestal-server
this
(assoc this :pedestal-server
(-> service-map
http/create-server
http/start))))
(stop [this]
(when pedestal-server
(http/stop pedestal-server))
(assoc this :pedestal-server nil)))
(defn new-pedestal []
(map->Pedestal {}))
You should note that calling (com.stuartsierra.component/start) on a component the function returns a started copy of the component but it does not modify the component itself. Likewise calling (com.stuartsierra.component/stop) will return a stopped copy of the component.
I think that the value of :pedestal-server key is nil because you did not store the returned value of the (start) call but you called it on the original (unstarted) component.
You need to store the state of the application in some storage, like an atom or a var. Then you can update the state of the storage with start and stop.
For example:
;; first we create a new component and store it in system.
(def system (new-pedestal))
;; this function starts the state and saves it:
(defn start-pedestal! [] (alter-var-root #'system component/start))
;; this function stops the running state:
(defn stop-pedestal! [] (alter-var-root #'system component/stop))
Related
I'm fetching thousands of entities from an API one at a time using http requests. As next step in the pipeline I want to shovel all of them into a database.
(->> ids
(pmap fetch-entity)
(pmap store-entity)
(doall))
fetch-entity expects a String id and tries to retrieve an entity using an http request and either returns a Map or throws an exception (e.g. because of a timeout).
store-entity expects a Map and tries to store it in a database. It possibly throws an exception (e.g. if the Map doesn't match the database schema or if it didn't receive a Map at all).
Inelegant Error Handling
My first "solution" was to write wrapper functions fetch-entity' and store-entity' to catch exceptions of their respective original functions.
fetch-entity' returns its input on failure, basically passing along a String id if the http request failed. This ensures that the whole pipeline keeps on trucking.
store-entity' checks the type of its argument. If the argument is a Map (fetch entity was successful and returned a Map) it attempts to store it in the database.
If the attempt of storing to the database throws an exception or if store-entity' got passed a String (id) instead of a Map it will conj to an external Vector of error_ids.
This way I can later use error_ids to figure out how often there was a failure and which ids were affected.
It doesn't feel like the above is a sensible way to achieve what I'm trying to do. For example the way I wrote store-entity' complects the function with the previous pipeline step (fetch-entity') because it behaves differently based on whether the previous pipeline step was successful or not.
Also having store-entity' be aware of an external Vector called error_ids does not feel right at all.
Is there an idiomatic way to handle these kinds of situations where you have multiple pipeline steps where some of them can throw exceptions (e.g. because they are I/O) where I can't easily use predicates to make sure the function will behave predictable and where I don't want to disturb the pipeline and only later check in which cases it went wrong?
It is possible to use a type of Try monad, for example from the cats library:
It represents a computation that may either result in an exception or return a successfully computed value. Is very similar to the Either monad, but is semantically different.It consists of two types: Success and Failure. The Success type is a simple wrapper, like Right of the Either monad. But the Failure type is slightly different from Left, because it always wraps an instance of Throwable (or any value in cljs since you can throw arbitrary values in the JavaScript host).(...)It is an analogue of the try-catch block: it replaces try-catch’s stack-based error handling with heap-based error handling. Instead of having an exception thrown and having to deal with it immediately in the same thread, it disconnects the error handling and recovery.
Heap-based error-handling is what you want.
Below I made an example of fetch-entity and store-entity. I made fetch-entity throw an ExceptionInfo on the first id (1) and store-entity throws a DivideByZeroException on the second id (0).
(ns your-project.core
(:require [cats.core :as cats]
[cats.monad.exception :as exc]))
(def ids [1 0 2]) ;; `fetch-entity` throws on 1, `store-entity` on 0, 2 works
(defn fetch-entity
"Throws an exception when the id is 1..."
[id]
(if (= id 1)
(throw (ex-info "id is 1, help!" {:id id}))
id))
(defn store-entity
"Unfortunately this function still needs to be aware that it receives a Try.
It throws a `DivideByZeroException` when the id is 0"
[id-try]
(if (exc/success? id-try) ; was the previous step a success?
(exc/try-on (/ 1 (exc/extract id-try))) ; if so: extract, apply fn, and rewrap
id-try)) ; else return original for later processing
(def results
(->> ids
(pmap #(exc/try-on (fetch-entity %)))
(pmap store-entity)))
Now you can filter results on successes or failures with respectively success? or failure? and retrieve the values via cats-extract
(def successful-results
(->> results
(filter exc/success?)
(mapv cats/extract)))
successful-results ;; => [1/2]
(def error-messages
(->> results
(filter exc/failure?)
(mapv cats/extract) ; gets exceptions without raising them
(mapv #(.getMessage %))))
error-messages ;; => ["id is 1, help!" "Divide by zero"]
Note that if you want to only loop over the errors or successful-results once you can use a transducer as follows:
(transduce (comp
(filter exc/success?)
(map cats/extract))
conj
results))
;; => [1/2]
My first thought is to combine fetch-entity and store-entity into a single operation:
(defn fetch-and-store [id]
(try
(store-entity (fetch-entity id))
(catch ... <log error msg> )))
(doall (pmap fetch-and-store ids))
Would something like this work?
I am creating an agent for writing changes back to a database (as discussed in the agent-based write-behind log in Clojure Programming).
This is working fine, but I am struggling to create the agent late. I don't want to create it as a def, as I don't want it created when my tests are running (I see the pool starting up when the tests load the forms even though I use with-redefs to set a test value).
The code I started with is (using c3p0 pooling):
(def dba (agent (pool/make-datasource-spec (u/load-config "db.edn"))))
I tried making the agent nil, and investigated how I could set it in the main of my application, when it's really needed. But there doesn't seem to be an equivalent reset! function as there is with an atom. And the following code also failed saying the agent wasn't in error so didn't need restarting:
(when (not #dba)
(restart-agent dba (create-db-pool)))
So at the moment, I have an atom containing the agent, where I then do:
(def dba (atom nil))
;; ...
(defn init-db! []
(when (not #dba)
(log/info "Creating agent for pooled connection")
(reset! dba (agent (create-db-pool))))
But the very fact I'm having to do ##dba to reference the contents of the agent (i.e. the pool) makes me think this is insane.
Is there a more obvious way of creating the pool agent lazily?
delay is useful for cases like this. It causes the item to be created the first time it is read. so if your tests don't read it it will not be created.
user=> (def my-agent (delay (do (println "im making the agent now") (agent 0))))
#'user/my-agent
user=>
user=> #my-agent
im making the agent now
#object[clojure.lang.Agent 0x2cd73ca1 {:status :ready, :val 0}]
Is it ok to use binding with core.async? I'm using ClojureScript so core.async is very different.
(def ^:dynamic token "no-token")
(defn call
[path body]
(http-post (str host path) (merge {:headers {"X-token" token}} body)))) ; returns a core.async channel
(defn user-settings
[req]
(call "/api/user/settings" req))
; elsewhere after I've logged in
(let [token (async/<! (api/login {:user "me" :pass "pass"}))]
(binding
[token token]
(user-settings {:all-settings true})))
In ClojureScript1, binding is basically with-redefs plus an extra check that the Vars involved are marked :dynamic. On the other hand, gos get scheduled for execution1 in chunks (that is, they may be "parked" and later resumed, and interleaving between go blocks is arbitrary). These models don't mesh very well at all.
In short, no, please use explicitly-passed arguments instead.
1 The details are different in Clojure, but the conclusion remains the same.
2 Using the fastest mechanism possible, setTimeout with a time of 0 if nothing better is available.
Now, I need some kind of code reloading in the clojure/java mixing system.
I know tools.namespace lib can be used to refresh clj code. But when I try to reload code which is used in a thread, it is not updated.
This is the example clojure code I need to update: (just switch between upper case and lower case.)
(ns com.test.test-util)
(defn get-word [sentence]
(let [a-word (rand-nth (clojure.string/split sentence #"\s"))
;;a-word (clojure.string/upper-case a-word)
]
(println a-word)
a-word))
I used a future to start a thread and call the code above.
In main method, in my first test, I just create a future and refresh the code after a while. But it is not updated. So, I also tested to create another thread after refresh. But the code is not updated.
(defn start-job []
(future (
(println "start worker job.")
(while #job-cond
(do (get-word (rand-nth sentences))
(Thread/sleep 2000)) )
(println "return!")
)))
(defn -main []
(let [work-job (start-job)]
(println "modify clj file...")
(Thread/sleep 10000)
(swap! job-cond not)
(future-cancel work-job)
(swap! job-cond not)
(refresh)
(Thread/sleep 3000)
(let [new-work-job (start-job)]
(Thread/sleep 10000)
(swap! job-cond not)
(future-cancel new-work-job))))
Thanks.
In general you don't want code-reloading to affect runtime. Rather you should try to seek for clean ways to boot and shutdown the running application code using e. g. Stuart Sierras library component or juxt/jig by Malcolm Sparks. Then reload namespaces only when your application is not up and running (or at least not the parts of running code that depend on namespaces being reloaded).
That being said, here is why your example does not work:
start-job can not automatically refer to a reloaded version of get-word in its lifetime because the version of get-work it uses is resolved when it is compiled. So it will still use the old version.
Explicitly resolve the var of get-word using
(resolve `get-word)
and invoke the returned var as a function instead of using get-word directly in the future.
What will happen then?
After you made the changes and refresh has been invoked, -main will still refer to the old-version of start-job (so if you change that, it will not show effect in runtime). However within that get-word will be resolved as described above and the new version of get-word will be used.
Short explanation: You can't expect a running compiled function to replace itself with reloaded code on the fly (no matter whether it runs on a separate thread or not). That would (unless you are using resolve in it) be a necessity though for it to invoke reloaded functions.
My problem is that with the doc and examples providede i can't understand the meaning of :key parameter or its possible values
This is the official doc page of the function that I'm referring:
http://clojuredocs.org/clojure_core/clojure.core/add-watch
add-watch clojure.core
(add-watch reference key fn)
Adds a watch function to an agent/atom/var/ref reference. The watch fn
must be a fn of 4 args: a key, the reference, its old-state, its
new-state. Whenever the reference's state might have been changed, any
registered watches will have their functions called. The watch fn will
be called synchronously, on the agent's thread if an agent, before any
pending sends if agent or ref. Note that an atom's or ref's state may
have changed again prior to the fn call, so use old/new-state rather
than derefing the reference. Note also that watch fns may be called
from multiple threads simultaneously. Var watchers are triggered only
by root binding changes, not thread-local set!s. Keys must be unique
per reference, and can be used to remove the watch with remove-watch,
but are otherwise considered opaque by the watch mechanism.
Thanks
It's basically just an identifier that you can use in calling code to identify the watch, in case you have more than one watches per reference. It's something that should have significance to your application code, but will be passed through by Clojure.
For instance:
user> (def a (atom 0))
#'user/a
user> (add-watch a
:count-to-3
(fn [k r old-state new-state]
(println "changed from" old-state "to" new-state)
(when (>= new-state 3)
(remove-watch a :count-to-3))))
#<Atom#3287a10: 0>
user> (dotimes [_ 5] (swap! a inc))
changed from 0 to 1
changed from 1 to 2
changed from 2 to 3
nil
user> #a
5
The answer's right there:
Keys must be unique per reference, and can be used to remove
the watch with remove-watch, but are otherwise considered opaque by
the watch mechanism.
In other words, the actual watch mechanism doesn't care what you set the key to (as long as it's unique among the handlers set on the given ref), but you'll need to hang on to it if you ever want to call remove-watch to get rid of your handler