Trying to use https://github.com/grammarly/perseverance,
why does the following fails? Ie. the last call still produces an exception
(defn fail-n
"returns a function that will fail to process for the first n times is is called"
[n]
(let [cnt (atom 0)]
(fn []
(if (< #cnt n)
(do (swap! cnt inc)
(throw (RuntimeException. "Failed")))
:success))))
(defn safe-fail-n [n]
(p/retriable {
;;:catch [RuntimeException]
} (fail-n n)))
;;(def f (fail-n 1))
(def f (safe-fail-n 2))
(p/retry {;;:strategy (p/constant-retry-strategy 0)
;;:catch [Exception]
;;:log-fn (fn [& a] (println "Retrying "a))
}
(f))
checkout the github readme, you have to catch the RE in the p/retriable form
otherwise nothing is catched - or retried
user=> (let [x (fail-n 5)]
(p/retry {}
(p/retriable {:catch [RuntimeException]} (x))))
java.lang.RuntimeException: Failed, retrying in 0,5 seconds...
java.lang.RuntimeException: Failed, retrying in 0,5 seconds...
java.lang.RuntimeException: Failed, retrying in 0,5 seconds...
java.lang.RuntimeException: Failed, retrying in 1,0 seconds...
java.lang.RuntimeException: Failed, retrying in 2,0 seconds...
:success
relevant doc:
:catch — should be a list of Exception classes that are going to be caught by retriable. The default value is [java.io.IOException]. Perseverance doesn’t catch all exceptions intentionally to avoid retrying the errors that aren’t IO-related, which would circumvent the proper error handling in your program. Yet you can always provide :catch [Exception] if you are sure that any potential exception inside is retriable
Related
I have 100 workers (agents) that share one ref that contains collection of tasks. While this collection have tasks, each worker get one task from this collection (in dosync block), print it and sometimes put it back in the collection (in dosync block):
(defn have-tasks?
[tasks]
(not (empty? #tasks)))
(defn get-task
[tasks]
(dosync
(let [task (first #tasks)]
(alter tasks rest)
task)))
(defn put-task
[tasks task]
(dosync (alter tasks conj task))
nil)
(defn worker
[& {:keys [tasks]}]
(agent {:tasks tasks}))
(defn worker-loop
[{:keys [tasks] :as state}]
(while (have-tasks? tasks)
(let [task (get-task tasks)]
(println "Task: " task)
(when (< (rand) 0.1)
(put-task tasks task))))
state)
(defn create-workers
[count & options]
(->> (range 0 count)
(map (fn [_] (apply worker options)))
(into [])))
(defn start-workers
[workers]
(doseq [worker workers] (send-off worker worker-loop)))
(def tasks (ref (range 1 10000000)))
(def workers (create-workers 100 :tasks tasks))
(start-workers workers)
(apply await workers)
When i run this code, the last value printed by agents is (after several tries):
435445,
4556294,
1322061,
3950017.
But never 9999999 what I expect.
And every time the collection is really empty at the end.
What I'm doing wrong?
Edit:
I rewrote worker-loop as simple as possible:
(defn worker-loop
[{:keys [tasks] :as state}]
(loop []
(when-let [task (get-task tasks)]
(println "Task: " task)
(recur)))
state)
But problem is still there.
This code behaves as expected when create one and only one worker.
The problem here has nothing to do with agents and barely anything to do with laziness. Here's a somewhat reduced version of the original code that still exhibits the problem:
(defn f [init]
(let [state (ref init)
task (fn []
(loop [last-n nil]
(if-let [n (dosync
(let [n (first #state)]
(alter state rest)
n))]
(recur n)
(locking :out
(println "Last seen:" last-n)))))
workers (->> (range 0 5)
(mapv (fn [_] (Thread. task))))]
(doseq [w workers] (.start w))
(doseq [w workers] (.join w))))
(defn r []
(f (range 1 100000)))
(defn i [] (f (->> (iterate inc 1)
(take 100000))))
(defn t []
(f (->> (range 1 100000)
(take Integer/MAX_VALUE))))
Running this code shows that both i and t, both lazy, reliably work, whereas r reliably doesn't. The problem is in fact a concurrency bug in the class returned by the range call. Indeed, that bug is documented in this Clojure ticket and is fixed as of Clojure version 1.9.0-alpha11.
A quick summary of the bug in case the ticket is not accessible for some reason: in the internals of the rest call on the result of range, there was a small opportunity for a race condition: the "flag" that says "the next value has already been computed" was set before the actual value itself, which meant that a second thread could see that flag as true even though the "next value" is still nil. The call to alter would then fix that nil value on the ref. It's been fixed by swapping the two assignment lines.
In cases where the result of range was either forcibly realized in a single thread or wrapped in another lazy seq, that bug would not appear.
I asked this question on the Clojure Google Group and it helped me to find the answer.
The problem is that I used a lazy sequence within the STM transaction.
When I replaced this code:
(def tasks (ref (range 1 10000000)))
by this:
(def tasks (ref (into [] (range 1 10000000))))
it worked as expected!
In my production code where the problem occurred, I used the Korma framework that also returns a lazy collection of tuples, as in my example.
Conclusion: Avoid the use of lazy data structures within the STM transaction.
When the last number in the range is reached, there a are still older numbers being held by the workers. Some of these will be returned to the queue, to be processed again.
In order to better see what is happening, you can change worker-loop to print the last task handled by each worker:
(defn worker-loop
[{:keys [tasks] :as state}]
(loop [last-task nil]
(if (have-tasks? tasks)
(let [task (get-task tasks)]
;; (when (< (rand) 0.1)
;; (put-task tasks task)
(recur task))
(when last-task
(println "Last task:" last-task))))
state)
This also shows the race condition in the code, where tasks seen by have-tasks? often is taken by others when get-task is called near the end of the processing of the tasks.
The race condition can be solved by removing have-tasks? and instead using the return value of nil from get-task as a signal that no more tasks are available (at the moment).
Updated:
As observed, this race conditions does not explain the problem.
Neither is the problem solved by removing a possible race condition in get-task like this:
(defn get-task [tasks]
(dosync
(first (alter tasks rest))))
However changing get-task to use an explicit lock seems to solve the problem:
(defn get-task [tasks]
(locking :lock
(dosync
(let [task (first #tasks)]
(alter tasks rest)
task))))
I have implemented retry policy for update function (talking to database) - if update throws exception I retry it up to 10 times. I am mocking the update function with midje. I want to simulate that first time it fails and second time it succeeds. I tried this :
(fact "update is retried when failed"
(ud/ensure-data {:username ..username.. :data :h}) => ..result..
(provided
(ud/get-raw-user-data ..username..) => example-user-data-raw
(ud/update-user-data {:username ..username..
:version 1
:userdata {:data {:h {}}}}) =throws=> (Exception.)
(ud/update-user-data {:username ..username..
:version 1
:userdata {:data {:h {}}}}) => ..result..))
but this doesn't seem to work...... Response is :
These calls were not made the right number of times:
(ud/update-user-data {:username ..username.., :version 1, :userdata {:homebases {:h {:sensors []}}}}) [expected at least once, actually never called]
I also found streams (https://github.com/marick/Midje/wiki/Variant-prerequisite-arrows) but I don't know how to combine Exceptions with success calls with streams.
I haven't got a clear understanding of how to use Midje streams either. So my personal solution is, in a word, not to use provided or 'streams', but use with-redefs and 'stubs'.
(defn fn0 [] -1) ;; function we want to mock
;; fn1 retries fn0 if there is an exception
(defn fn1 [] (try
(fn0)
(catch Exception e
(do
(prn "Exception Caught, try again...")
(fn0)))))
(def call-count (atom -1)) ;; counts how many times fn0 is called
;; stub fn0 by returning different result
(defn make-stub [& result-seqs]
(fn [& _]
(swap! call-count inc)
(let [result (nth result-seqs #call-count)]
(if (instance? Throwable result)
(throw result)
result))))
(fact "fn1 ignores first exception and returns second call to fn0"
(with-redefs [fn0 (make-stub (Exception. "stubbed error") 100)]
(fn1) => 100))
This question already has an answer here:
Closed 10 years ago.
Possible Duplicate:
Clojure error on thread: java.lang.IllegalArgumentException: Key must be integer
This code is so simple, I am confused how it could go wrong. I have:
(defn create-server [port]
(let [ss (new ServerSocket port)]
(start-thread (fn [ss]
(while (not (. ss (isClosed)))
(try (listen-and-respond ss)
(catch SocketException e)))))))
(defn -main [& args]
(println "Server is starting")
(let [port (Integer/parseInt (first args))]
(println "port: " port)
(create-server port)))
I compile this, then uberjar it, then start it on the command line. These lines:
(println "Server is starting")
(println "port: " port)
prints out:
Server is starting
port: 3457
On the next line, create-server is called and I get this error:
Exception in thread "Thread-1" clojure.lang.ArityException: Wrong number of args (0) passed to: core$create-server$fn
at clojure.lang.AFn.throwArity(AFn.java:437)
at clojure.lang.AFn.invoke(AFn.java:35)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:680)
Clearly, the line in -main can not be a problem, because I know that "port" has a value of 3457 the line before the first call to create-server. I also notice this error is in Thread-1, so I am thinking that somehow this code recurs in a way that I don't understand.
Any thoughts?
If start-thread ultimately calls into new Thread(Runnable) then the error is actually because public void run() of Runnable does not have any arguments.
Because ss is declared in the enclosing let expression, it will be visible to your anonymous function you passed to start-thread.
Edit Reproducing your exact problem at the REPL using Java interop (*1 in the REPL means "the value returned from the previous evaluation):
user=> (doto (new Thread (fn run [s] (println "blah"))))
#<Thread Thread[Thread-14,5,main]>
user=> (.start *1)
Exception in thread "Thread-14" nil
clojure.lang.ArityException: Wrong number of args (0) passed to: user$eval318$run
at clojure.lang.AFn.throwArity(AFn.java:437)
at clojure.lang.AFn.invoke(AFn.java:35)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:679)
user=> (doto (new Thread (fn run [] (println "blah"))))
#<Thread Thread[Thread-15,5,main]>
user=> (.start *1)
nilblah
I don't know how to implement this piece of Python code in Clojure
for i in range(3):
try:
......
except e:
if i == 2:
raise e
else:
continue
else:
break
I wonder why something so simple in Python is so hard in Clojure. I think the difficulty is because Clojure is a functional programming language and thus is not suitable for such an imperative task. This is my attempt:
(first
(remove #(instance? Exception %)
(for [i (range 3)]
(try (......)
(catch Exception e
(if (== i 2)
(throw e)
e)))))))
It is very ugly, and worse, it doesn't work as expected. The for loop is actually evaluated fully instead of lazily (I realized this when I put a println inside).
If anyone has a better idea to implement that, please enlighten me.
Similar to Marcyk's answer, but no macro trickery:
(defn retry
[retries f & args]
(let [res (try {:value (apply f args)}
(catch Exception e
(if (zero? retries)
(throw e)
{:exception e})))]
(if (:exception res)
(recur (dec retries) f args)
(:value res))))
Slightly complicated because you can't recur inside a catch clause. Note that this takes a function:
(retry 3 (fn []
(println "foo")
(if (zero? (rand-int 2))
(throw (Exception. "foo"))
2)))
=>
foo ;; one or two or three of these
foo
2
Here's one approach:
(defmacro retry
"Evaluates expr up to cnt + 1 times, retrying if an exception
is thrown. If an exception is thrown on the final attempt, it
is allowed to bubble up."
[cnt expr]
(letfn [(go [cnt]
(if (zero? cnt)
expr
`(try ~expr
(catch Exception e#
(retry ~(dec cnt) ~expr)))))]
(go cnt)))
Example from the REPL:
user> (retry 2 (do (println :foo) (throw (RuntimeException. "foo"))))
:foo
:foo
:foo
; Evaluation aborted.
(Passing 2 to retry asks expr to be retried twice it if fails the first time round, for a total of three attempts. Three :foos are printed, because the println occurs before the throw in the do form passed to retry. The final ; Evaluation aborted. means an exception was thrown.)
Also, about the for loop from your snippet:
If you try looping over a longer range (replace (range 3) with (range 10), say), the output will end after i reaches 3. Also, if you put in a println before the form which throws the exception, it will of course print out whatever you pass to it; if the println occurs after the exception-throwing form, there will be no printout. In any case, at most three calls to println will be executed (assuming an exception is thrown on every iteration).
(cond (every? nil? (for [x (range (inc retry)) :while (not #tmp-doc)]
...do sth) )
;all failed
:else
;at least one success
You can do like this:
(defn retry
"Tries at most n times, return first try satisfying pred or nil"
[times pred? lazy-seq]
(let [successful-trial (drop-while (complement pred?) (take times lazy-seq))]
(if (empty? successful-trial)
nil
(first successful-trial))))
Then you could use the function as such:
(when-not (retry 3 pos? (repeatedly #(rand-nth [-1 -2 -3 2 1]))
(throw (Exception. "my exception message"))
This would try at most three times to take a positive number at random from the vector and if it doesn't succeed, throws an exception.
Retry with sleep and another implementation based on this question's answers:
(defn retry [retries sleep-ms func url options]
(let [{:keys [status headers body error] :as resp} #(func url options)]
(if error
(do
(log/error (str "Retry " retries " returned error: " e))
(Thread/sleep sleep-ms)
(if (= retries 1)
resp
(retry (dec retries) sleep-ms func url options)))
resp)))
What is the proper way to do the following in clojure?
(ns todo.test.models.task
(:use [clojure.test]))
(deftest main-test
(is (thrown? Exception (throw Exception "stuff")))
(is (not (thrown? Exception (+ 2 3))))
)
First testcase runs fine but the whole snippet returns "Unable to resolve symbol: thrown?"
is is a macro that looks for the symbol thrown? in its body and build tests.
thrown? is not actually a function you can call. The default behaviour of is fails the test if an exception is thrown that was not beeing looked for, so you can just remove the (not (thrown? from the above example and get the result you are looking for.
thrown? is a special assertion that must show up after is, so you can't nest it in other expressions, so in the context of the is macro, the second assertion will not understand the symbol thrown?.
You could just say:
(deftest main-test
(is (thrown? Exception (throw (Exception. "stuff"))))
(is (= 5 (+ 2 3))))
If an exception is thrown in (+ 2 3), clojure.test will report 1 :error and 0 :fail and dump the stack trace.
Also note that your (throw Exception "stuff") is incorrect - you need to construct the Exception correctly inside the throw.
Use doseq if you want to do it for many statements:
(testing "bla"
(doseq [x [1 2 3 4]]
(my-dangerous-func! x)))
I know this is an old question but..
In addition to the above answers, if you really want a not-thrown? assertion, you can extend the is macro by adding your own e.g.
(defmethod assert-expr 'not-thrown? [msg form]
;; (is (not-thrown? c expr))
;; Asserts that evaluating expr does not throws an exception of class c.
;; Returns the exception thrown.
(let [klass (second form)
body (nthnext form 2)]
`(try ~#body
(do-report {:type :pass, :message ~msg,
:expected '~form, :actual nil})
(catch ~klass e#
(do-report {:type :fail, :message ~msg,
:expected '~form, :actual e#})
e#))))
This should then work as your original expectations
((deftest main-test
(is (thrown? Exception (throw (Exception. "stuff"))))
(is (not-thrown? Exception (+ 2 3)))))
However, please note that clojure.test will always report an error if an exception occurs in your function but if you have a special use-case for this, there you go.