The following Clojure code causes an NPE. It seems from the stacktrace this is caused when >= is called. I can't seem to figure out why this is happening.
(def service-thresholds
[{:service "cpu idle" :invert true :warning 50 :critital 10}
{:service "cpu user" :invert false :warning 80 :critical 90}])
(defn threshold-check
[thresholds]
(fn [e]
(or (first (for [threshold thresholds
:when (= (:service threshold) (:service e))]
(assoc e :state
(if (not (:invert threshold))
(condp <= (:metric e)
(:critical threshold) "critical"
(:warning threshold) "warning"
"ok")
(condp >= (:metric e)
(:critical threshold) "critical"
(:warning threshold) "warning"
"ok")))))
e)))
(println ((threshold-check service-thresholds) {:service "cpu idle" :metric 20 :state nil}))
:critital is not a correct spelling of :critical.
Related
In my Clojure project I'm trying to make a list of http calls to an API that has a rate limiter that only allows n calls per minute. I want each of the responses to be returned once all the http calls are finished for further processing. I am new to Clojure's Core Async, but thought it would be a good fit, but because I need to run each call n seconds apart I am also trying to use the Chime library. In Chime's library it has examples using Core Async, but the examples all call the same function at each time interval which won't work for this use case.
While there is probably a way to use chime-async that better serves this use case, all of my attempts at that have failed so I've tried simply wrapping Chime calls with core async, but I am probably more baffled by Core Async than Chime.
This is an example of my name space.
(ns mp.util.schedule
(:require [chime.core :as chime]
[clojure.core.async :as a]
[tick.alpha.api :as tick]))
(defn schedule-fns
"Takes a list of functions and a duration in seconds then runs each function in the list `sec` seconds apart
optionally provide an inst to start from"
[fs sec & [{:keys [inst] :or {inst (tick/now)}}]]
(let [ch (a/chan (count fs))
chime-times (map-indexed
(fn mapped-fn [i f]
(a/put! ch (chime/chime-at [(.plusSeconds inst (* i sec))]
(fn wrapped-fn [_] (f)))))
fs)]
(doseq [chi chime-times]
(a/<!! chi))))
; === Test Code ===
; simple test function
(defn sim-fn
"simple function that prints a message and value, then returns the value"
[v m]
(println m :at (tick/now))
v)
; list of test functions
(def fns [#(sim-fn 1 :one)
#(sim-fn 2 :two)
#(sim-fn 3 :three)])
What I want to happen when calling (schedule-fns fns 2) is for each function in fns to run n seconds from each other and for schedule-fns to return (1 2 3) (the return values of the functions), but this isn't what it is doing. It is calling each of the functions at the correct times (which I can see from the log statements) but it isn't returning anything and there's an error I don't understand. I'm getting:
(schedule-fns fns 2)
:one :at #time/instant "2021-03-05T23:31:52.565Z"
Execution error (IllegalArgumentException) at clojure.core.async.impl.protocols/eval11496$fn$G (protocols.clj:15).
No implementation of method: :take! of protocol: #'clojure.core.async.impl.protocols/ReadPort found for class: java.lang.Boolean
:two :at #time/instant "2021-03-05T23:31:54.568Z"
:three :at #time/instant "2021-03-05T23:31:56.569Z"
If I could get help getting my code to use Core Async properly (with or without Chime) I'd really appreciate it. Thanks.
Try this:
(defn sim-fn
"simple function that prints a message and value, then returns the value"
[v m]
(println m)
v)
; list of test functions
(def fns [#(sim-fn 1 :one)
#(sim-fn 2 :two)
#(sim-fn 3 :three)])
(defn schedule-fns [fns sec]
(let [program (interpose #(Thread/sleep (* sec 1000))
fns)]
(remove #(= % nil)
(for [p program]
(p)))))
Then call:
> (schedule-fns fns 2)
:one
:two
:three
=> (1 2 3)
I came up with a way to get what I want...with some caveats.
(def results (atom []))
(defn schedule-fns
"Takes a list of functions and a duration in seconds then runs each function in the list `sec` seconds apart
optionally provide an inst to start from"
[fs sec]
(let [ch (chan (count fs))]
(go-loop []
(swap! results conj (<! ch))
(recur))
(map-indexed (fn [i f]
(println :waiting (* i sec) :seconds)
(go (<! (timeout (* i sec 1000)))
(>! ch (f))))
fs)))
This code has the timing and behavior that I want, but I have to use an atom to store the responses. While I can add a watcher to determine when all the results are in, I still feel like I shouldn't have to do that.
I guess I'll use this for now, but at some point I'll keep working on this and if anyone has something better than this approach I'd love to see it.
I had a couple friends look at this and they each came up with different answers. These are certainly better than what I was doing.
(defn schedule-fns [fs secs]
(let [ret (atom {})
sink (a/chan)]
(doseq [[n f] (map-indexed vector fs)]
(a/thread (a/<!! (a/timeout (* 1000 n secs)))
(let [val (f)
this-ret (swap! ret assoc n val)]
(when (= (count fs) (count this-ret))
(a/>!! sink (mapv (fn [i] (get this-ret i)) (range (count fs))))))))
(a/<!! sink)))
and
(defn schedule-fns
[fns sec]
(let [concurrent (count fns)
output-chan (a/chan)
timedout-coll (map-indexed (fn [i f]
#(do (println "Waiting")
(a/<!! (a/timeout (* 1000 i sec)))
(f))) fns)]
(a/pipeline-blocking concurrent
output-chan
(map (fn [f] (f)))
(a/to-chan timedout-coll))
(a/<!! (a/into [] output-chan))))
If your objective is to work around the rate limiter, you can consider implementing it in the async channel. Below is one sample implementation - the function takes a channel, throttled its input with a token based limiter and pipe it to an output channel.
(require '[clojure.core.async :as async])
(defn rate-limiting-ch [input xf rate]
(let [tokens (numerator rate)
period (denominator rate)
ans (async/chan tokens xf)
next (fn [] (+ period (System/currentTimeMillis)))]
(async/go-loop [c tokens
t (next)]
(if (zero? c)
(do
(async/<! (async/timeout (- t (System/currentTimeMillis))))
(recur tokens (next)))
(when-let [x (async/<! input)]
(async/>! ans x)
(recur (dec c) t))))
ans))
And here is a sample usage:
(let [start (System/currentTimeMillis)
input (async/to-chan (range 10))
output (rate-limiting-ch input
;; simulate an api call with roundtrip time of ~300ms
(map #(let [wait (rand-int 300)
ans {:time (- (System/currentTimeMillis) start)
:wait wait
:input %}]
(Thread/sleep wait)
ans))
;; rate limited to 2 calls per 1000ms
2/1000)]
;; consume the output
(async/go-loop []
(when-let [x (async/<! output)]
(println x)
(recur))))
Output:
{:time 4, :wait 63, :input 0}
{:time 68, :wait 160, :input 1}
{:time 1003, :wait 74, :input 2}
{:time 1079, :wait 151, :input 3}
{:time 2003, :wait 165, :input 4}
{:time 2169, :wait 182, :input 5}
{:time 3003, :wait 5, :input 6}
{:time 3009, :wait 18, :input 7}
{:time 4007, :wait 138, :input 8}
{:time 4149, :wait 229, :input 9}
I'm implementing Sleeping barber using core.async. My current code is:
(def workingtime 10000)
(defn barber [in waiting-room]
(go-loop [served-customers 0]
(let [[v] (alts! [waiting-room in])]
(if (= v :close)
served-customers
(do (Thread/sleep 20)
(recur (inc served-customers)))))))
(defn customers [in waiting-room]
(go-loop [customers-overall 0]
(let [customer-arrival-interval (timeout (+ 10 (rand-int 20)))
[v] (alts! [in customer-arrival-interval])]
(if (= v :close)
customers-overall
(do (>! waiting-room :customer)
(recur (inc customers-overall)))))))
(defn -main [& args]
(let [in (chan)
waiting-room (chan (dropping-buffer 3))
barber-ch (barber in waiting-room)
customers-ch (customers in waiting-room)]
(println "opening the shop for 10 seconds...")
(Thread/sleep workingtime)
(>!! in :close)
(>!! in :close)
(println "closing the shop...")
(println (str "Served " (<!! barber-ch) " customers"))
(println (str "Overall " (<!! customers-ch) " customers came"))))
Is it a correct solution? Can it be improved to make it more Clojure-like?
I wanted to use alt! instead of alts! which makes code easier to read:
(defn barber [in]
(go-loop [served-customers 0]
(alt!
waiting-room (do (Thread/sleep 20)
(recur (inc served-customers)))
in served-customers)))
Runtime throws an exception: Can only recur from tail position. Can I still use alt!?
You could solve the alt!/recur problem by rewriting to:
(defn barber [in]
(go-loop [served-customers 0]
(if (= :waiting-room
(a/alt!
waiting-room ([result] :waiting-room) ;; you could also use result if needed
in ([result] :in))) ;; same here
(do (Thread/sleep 20)
(recur (inc served-customers)))
served-customers)))
I am trying to show a graph on riemann-dashboard using query
"pingDelay > 0" .
I already have indexed my data using following code
(let [index (index)]
(defn write-dht-metric [e]
(let [dhtstate (re-find #"dht_status: health\.(\S+), msg count (\d+) \((\d+) bytes\).*peak \{ping = (\d+)" (:pgmsg e))]
(if (not= dhtstate nil)
(do
(prn "RESULT>" dhtstate)
(index {:host "dht-info"
:service (:service e)
:time (unix-time)
:dhtStatus (get dhtstate 1)
:msgCount (get dhtstate 2)
:pingDelay (get dhtstate 3)}
)
)
)
)
)
)
However, I am not getting anything on graph. Earlier, I thought that perhaps its because my "pingDelay" is in string "12345", so, i also tried ":pingDelay #(Long. (get dhtstate 3))" without any success.
Can anyone please help me about what I must do to make it work?
Regards
defining top level forms in function calls is a little odd. It works only because defining a var returns that var to the calling form. It's more typical to write it like:
(defn write-dht-metric [e]
(let [dhtstate (re-find #"dht_status: health\.(\S+), msg count (\d+) \((\d+) bytes\).*peak \{ping = (\d+)" (:pgmsg e))]
(if (not= dhtstate nil)
(do
(prn "RESULT>" dhtstate)
(index {:host "dht-info"
:service (:service e)
:time (unix-time)
:dhtStatus (get dhtstate 1)
:msgCount (get dhtstate 2)
:pingDelay (get dhtstate 3)})))))
(let [index (index)]
(streams
write-dht-metric))
there are several other ways of writing it:
(defn write-dht-metric [e]
(let [dhstate (:dhstate e)]
(prn "RESULT>" dhtstate)
(index {:host "dht-info"
:service (:service e)
:time (unix-time)
:dhtStatus (get dhtstate 1)
:msgCount (get dhtstate 2)
:pingDelay (get dhtstate 3)})))
(let [index (index)]
(streams
(with :dhstate (re-find #"dht_status: health\.(\S+), msg count (\d+) \((\d+) bytes\).*peak \{ping = (\d+)" (:pgmsg event))
(when :dhstate
write-dht-metric)))
It turned out that I had to write value of my pingDelay in ":metric field". It starts working now.
Here is an example from http://www.braveclojure.com/core-async/ :
(defn hotdog-machine-v2 [hotdog-count]
(let [in (chan) out (chan)]
(go (loop [hc hotdog-count]
(if (> hc 0)
(let [input (<! in)]
(if (= 3 input)
(do
(>! out "hotdog")
(recur (dec hc)) )
(do
(>! out (Exception. "Not enough payment!"))
(recur hc))))
(do
(close! in)
(close! out)))))
[in out]))
(let [[in out] (hotdog-machine-v2 2)]
(>!! in "pocket lint")
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
)
If I leave out the channel-closing part, then this code hangs forever, why?
If I run this in the normal case with the close it prints this:
user> (let [[in out] (hotdog-machine-v2 2)]
(>!! in "pocket lint")
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out)))
#error {
:cause Not enough payment!
:via
[{:type java.lang.Exception
:message Not enough payment!
... stack trace here ...
[java.lang.Thread run Thread.java 724]]}
hotdog
hotdog
nil ;; <---- look here
That last nil is the result of the last read from out reading the value nil, which is being sent as a result of the channel closing because the machine is out of hotdogs. Without the close that last put (>!! in 3) blocks waiting for somthing to read from the chan which nobody is going to do. By default writes to core.async chans don't succeed in writing until someone is ready to read that value
If I take out the close, and take out the last write it does not print that nil, and does not block:
user> (let [[in out] (hotdog-machine-v2 2)]
(>!! in "pocket lint")
(println (<!! out))
(>!! in 3)
(println (<!! out))
(>!! in 3)
(println (<!! out))
)
#error {
:cause Not enough payment!
:via
[{:type java.lang.Exception
:message Not enough payment!
:at ... stack trace here ...}
hotdog
hotdog
I am trying to execute a func several times before giving up upon exceptions.
But it is not valid in Clojure to recur from catch block.
How can this be achieved ?
(loop [tries 10]
(try
(might-throw-exception)
(catch Exception e
(when (pos? tries) (recur (dec tries))))))
java.lang.UnsupportedOperationException: Cannot recur from catch/finally
The best I could find is the following clumsy solution (wrapping in func and calling it)
(defn do-it []
(try
(might-throw-exception)
(catch Exception e nil)))
(loop [times 10]
(when (and (nil? (do-it)) (pos? times))
(recur (dec times))))
Macros are calling...
How about this:
(defn try-times*
"Executes thunk. If an exception is thrown, will retry. At most n retries
are done. If still some exception is thrown it is bubbled upwards in
the call chain."
[n thunk]
(loop [n n]
(if-let [result (try
[(thunk)]
(catch Exception e
(when (zero? n)
(throw e))))]
(result 0)
(recur (dec n)))))
(defmacro try-times
"Executes body. If an exception is thrown, will retry. At most n retries
are done. If still some exception is thrown it is bubbled upwards in
the call chain."
[n & body]
`(try-times* ~n (fn [] ~#body)))
kotarak's idea is the way to go, but this question tickled my fancy so I'd like to provide a riff on the same theme that I prefer because it doesn't use loop/recur:
(defn try-times* [thunk times]
(let [res (first (drop-while #{::fail}
(repeatedly times
#(try (thunk)
(catch Throwable _ ::fail)))))]
(when-not (= ::fail res)
res)))
And leave the try-times macro as it is.
If you want to allow the thunk to return nil, you can drop the let/when pair, and let ::fail represent "the function failed n times", while nil means "the function returned nil". This behavior would be more flexible but less convenient (the caller has to check for ::fail to see if it worked rather than just nil), so perhaps it would be best implemented as an optional second parameter:
(defn try-times* [thunk n & fail-value]
(first (drop-while #{fail-value} ...)))
A try-times macro is elegant, but for a one-off, just pull your when out of the try block:
(loop [tries 10]
(when (try
(might-throw-exception)
false ; so 'when' is false, whatever 'might-throw-exception' returned
(catch Exception e
(pos? tries)))
(recur (dec tries))))
My proposal:
(defmacro try-times
"Retries expr for times times,
then throws exception or returns evaluated value of expr"
[times & expr]
`(loop [err# (dec ~times)]
(let [[result# no-retry#] (try [(do ~#expr) true]
(catch Exception e#
(when (zero? err#)
(throw e#))
[nil false]))]
(if no-retry#
result#
(recur (dec err#))))))
Will print "no errors here" once:
(try-times 3 (println "no errors here") 42)
Will print "trying" 3 times, then throw Divide by zero:
(try-times 3 (println "trying") (/ 1 0))
One more solution, without macro
(defn retry [& {:keys [fun waits ex-handler]
:or {ex-handler #(log/error (.getMessage %))}}]
(fn [ctx]
(loop [[time & rem] waits]
(let [{:keys [res ex]} (try
{:res (fun ctx)}
(catch Exception e
(when ex-handler
(ex-handler e))
{:ex e}))]
(if-not ex
res
(do
(Thread/sleep time)
(if (seq rem)
(recur rem)
(throw ex))))))))
This allows catching multiple more then one exception and provides some feedback about the causes for the retries.
(defmacro try-n-times
"Try running the body `n` times, catching listed exceptions."
{:style/indent [2 :form :form [1]]}
[n exceptions & body]
`(loop [n# ~n
causes# []]
(if (> n# 0)
(let [result#
(try
~#body
~#(map (partial apply list 'catch) exceptions (repeat `(e# e#))))]
(if (some #(instance? % result#) ~exceptions)
(recur (dec n#) (conj causes# result#))
result#))
(throw (ex-info "Maximum retries exceeded!"
{:retries ~n
:causes causes#})))))
If you add a result arg to your loop, you can nest the (try) block inside of the (recur). I solved it like this:
(loop [result nil tries 10]
(cond (some? result) result
(neg? tries) nil
:else (recur (try (might-throw-exception)
(catch Exception e nil))
(dec tries))))
Here's yet another approach:
(loop [tries 10]
(let [res (try
(might-throw-exception)
(catch Exception e
(if (pos? tries)
::retry
(throw e))))]
(if (#{::retry} res)
(recur (dec tries))
res)))
But may I also recommend a cool little trick, instead of having a number of retries, provide a seq of times to sleep for:
(loop [tries [10 10 100 1000]]
(let [res (try
(might-throw-exception)
(catch Exception e
(if tries
::retry
(throw e))))]
(if (#{::retry} res)
(do
(Thread/sleep (first tries))
(recur (next tries)))
res)))
And finally put it all into a macro if you want it to be less verbose:
(defmacro with-retries
[retries & body]
`(loop [retries# ~retries]
(let [res# (try ~#body
(catch Exception e#
(if retries#
'retry#
(throw e#))))]
(if (= 'retry# res#)
(do (Thread/sleep (first retries#))
(recur (next retries#)))
res#))))
(with-retries [10 10 100 1000]
(might-throw-exception))