retrying something 3 times before throwing an exception - in clojure - clojure

I don't know how to implement this piece of Python code in Clojure
for i in range(3):
try:
......
except e:
if i == 2:
raise e
else:
continue
else:
break
I wonder why something so simple in Python is so hard in Clojure. I think the difficulty is because Clojure is a functional programming language and thus is not suitable for such an imperative task. This is my attempt:
(first
(remove #(instance? Exception %)
(for [i (range 3)]
(try (......)
(catch Exception e
(if (== i 2)
(throw e)
e)))))))
It is very ugly, and worse, it doesn't work as expected. The for loop is actually evaluated fully instead of lazily (I realized this when I put a println inside).
If anyone has a better idea to implement that, please enlighten me.

Similar to Marcyk's answer, but no macro trickery:
(defn retry
[retries f & args]
(let [res (try {:value (apply f args)}
(catch Exception e
(if (zero? retries)
(throw e)
{:exception e})))]
(if (:exception res)
(recur (dec retries) f args)
(:value res))))
Slightly complicated because you can't recur inside a catch clause. Note that this takes a function:
(retry 3 (fn []
(println "foo")
(if (zero? (rand-int 2))
(throw (Exception. "foo"))
2)))
=>
foo ;; one or two or three of these
foo
2

Here's one approach:
(defmacro retry
"Evaluates expr up to cnt + 1 times, retrying if an exception
is thrown. If an exception is thrown on the final attempt, it
is allowed to bubble up."
[cnt expr]
(letfn [(go [cnt]
(if (zero? cnt)
expr
`(try ~expr
(catch Exception e#
(retry ~(dec cnt) ~expr)))))]
(go cnt)))
Example from the REPL:
user> (retry 2 (do (println :foo) (throw (RuntimeException. "foo"))))
:foo
:foo
:foo
; Evaluation aborted.
(Passing 2 to retry asks expr to be retried twice it if fails the first time round, for a total of three attempts. Three :foos are printed, because the println occurs before the throw in the do form passed to retry. The final ; Evaluation aborted. means an exception was thrown.)
Also, about the for loop from your snippet:
If you try looping over a longer range (replace (range 3) with (range 10), say), the output will end after i reaches 3. Also, if you put in a println before the form which throws the exception, it will of course print out whatever you pass to it; if the println occurs after the exception-throwing form, there will be no printout. In any case, at most three calls to println will be executed (assuming an exception is thrown on every iteration).

(cond (every? nil? (for [x (range (inc retry)) :while (not #tmp-doc)]
...do sth) )
;all failed
:else
;at least one success

You can do like this:
(defn retry
"Tries at most n times, return first try satisfying pred or nil"
[times pred? lazy-seq]
(let [successful-trial (drop-while (complement pred?) (take times lazy-seq))]
(if (empty? successful-trial)
nil
(first successful-trial))))
Then you could use the function as such:
(when-not (retry 3 pos? (repeatedly #(rand-nth [-1 -2 -3 2 1]))
(throw (Exception. "my exception message"))
This would try at most three times to take a positive number at random from the vector and if it doesn't succeed, throws an exception.

Retry with sleep and another implementation based on this question's answers:
(defn retry [retries sleep-ms func url options]
(let [{:keys [status headers body error] :as resp} #(func url options)]
(if error
(do
(log/error (str "Retry " retries " returned error: " e))
(Thread/sleep sleep-ms)
(if (= retries 1)
resp
(retry (dec retries) sleep-ms func url options)))
resp)))

Related

Clojure program works fine when debugged, fails in repl

I'm learning core.async and have written a simple producer consumer code:
(ns webcrawler.parallel
(:require [clojure.core.async :as async
:refer [>! <! >!! <!! go chan buffer close! thread alts! alts!! timeout]]))
(defn consumer
[in out f]
(go (loop [request (<! in)]
(if (nil? request)
(close! out)
(do (print f)
(let [result (f request)]
(>! out result))
(recur (<! in)))))))
(defn make-consumer [in f]
(let [out (chan)]
(consumer in out f)
out))
(defn process
[f s no-of-consumers]
(let [in (chan (count s))
consumers (repeatedly no-of-consumers #(make-consumer in f))
out (async/merge consumers)]
(map #(>!! in %1) s)
(close! in)
(loop [result (<!! out)
results '()]
(if (nil? result)
results
(recur (<!! out)
(conj results result))))))
This code works fine when I step in through the process function in debugger supplied with Emacs' cider.
(process (partial + 1) '(1 2 3 4) 1)
(5 4 3 2)
However, if I run it by itself (or hit continue in the debugger) I get an empty result.
(process (partial + 1) '(1 2 3 4) 1)
()
My guess is that in the second case for some reason producer doesn't wait for consumers before exiting, but I'm not sure why. Thanks for help!
The problem is that your call to map is lazy, and will not run until something asks for the results. Nothing does this in your code.
There are 2 solutions:
(1) Use the eager function mapv:
(mapv #(>!! in %1) items)
(2) Use the doseq, which is intended for side-effecting operations (like putting values on a channel):
(doseq [item items]
(>!! in item))
Both will work and produce output:
(process (partial + 1) [1 2 3 4] 1) => (5 4 3 2)
P.S. You have a debug statement in (defn consumer ...)
(print f)
that produces a lot of noise in the output:
<#clojure.core$partial$fn__5561 #object[clojure.core$partial$fn__5561 0x31cced7
"clojure.core$partial$fn__5561#31cced7"]>
That is repeated 5 times back to back. You probably want to avoid that, as printing function "refs" is pretty useless to a human reader.
Also, debug printouts in general should normally use println so you can see where each one begins and ends.
I'm going to take a safe stab that this is being caused by the lazy behavior of map, and this line that's carrying out side effects:
(map #(>!! in %1) s)
Because you never explicitly use the results, it never runs. Change it to use mapv, which is strict, or more correctly, use doseq. Never use map to run side effects. It's meant to lazily transform a list, and abuse of it leads to behaviour like this.
So why is it working while debugging? I'm going to guess because the debugger forces evaluation as part of its operation, which is masking the problem.
As you can read from docstring map returns a lazy sequence. And I think the best way is to use dorun. Here is an example from clojuredocs:
;;map a function which makes database calls over a vector of values
user=> (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"])
JdbcSQLException The object is already closed [90007-170] org.h2.message.DbE
xception.getJdbcSQLException (DbException.java:329)
;;database connection was closed before we got a chance to do our transactions
;;lets wrap it in dorun
user=> (dorun (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"]))
DEBUG :db insert into person values name = 'Fred'
DEBUG :db insert into person values name = 'Ethel'
DEBUG :db insert into person values name = 'Lucy'
DEBUG :db insert into person values name = 'Ricardo'
nil

midje - how to expect different results for two identical function calls (including exception to be throwed)

I have implemented retry policy for update function (talking to database) - if update throws exception I retry it up to 10 times. I am mocking the update function with midje. I want to simulate that first time it fails and second time it succeeds. I tried this :
(fact "update is retried when failed"
(ud/ensure-data {:username ..username.. :data :h}) => ..result..
(provided
(ud/get-raw-user-data ..username..) => example-user-data-raw
(ud/update-user-data {:username ..username..
:version 1
:userdata {:data {:h {}}}}) =throws=> (Exception.)
(ud/update-user-data {:username ..username..
:version 1
:userdata {:data {:h {}}}}) => ..result..))
but this doesn't seem to work...... Response is :
These calls were not made the right number of times:
(ud/update-user-data {:username ..username.., :version 1, :userdata {:homebases {:h {:sensors []}}}}) [expected at least once, actually never called]
I also found streams (https://github.com/marick/Midje/wiki/Variant-prerequisite-arrows) but I don't know how to combine Exceptions with success calls with streams.
I haven't got a clear understanding of how to use Midje streams either. So my personal solution is, in a word, not to use provided or 'streams', but use with-redefs and 'stubs'.
(defn fn0 [] -1) ;; function we want to mock
;; fn1 retries fn0 if there is an exception
(defn fn1 [] (try
(fn0)
(catch Exception e
(do
(prn "Exception Caught, try again...")
(fn0)))))
(def call-count (atom -1)) ;; counts how many times fn0 is called
;; stub fn0 by returning different result
(defn make-stub [& result-seqs]
(fn [& _]
(swap! call-count inc)
(let [result (nth result-seqs #call-count)]
(if (instance? Throwable result)
(throw result)
result))))
(fact "fn1 ignores first exception and returns second call to fn0"
(with-redefs [fn0 (make-stub (Exception. "stubbed error") 100)]
(fn1) => 100))

Using with-redefs inside a doseq

I have a Ring handler which uses several functions to build the response. If any of these functions throw an exception, it should be caught so a custom response body can be written before returning a 500.
I'm writing the unit test for the handler, and I want to ensure that an exception thrown by any of these functions will be handled as described above. My instinct was to use with-redefs inside a doseq:
(doseq [f [ns1/fn1 ns1/fn2 ns2/fn1]]
(with-redefs [f (fn [& args] (throw (RuntimeException. "fail!"))]
(let [resp (app (request :get "/foo")))))]
(is (= (:status resp) 500))
(is (= (:body resp) "Something went wrong")))))
Of course, given that with-redefs wants to change the root bindings of vars, it is treating f as a symbol. Is there any way to get it to rebind the var referred to by f? I suspect that I'm going to need a macro to accomplish this, but I'm hoping that someone can come up with a clever solution.
with-redefs is just sugar over repeated calls to alter-var-root, so you can just write the desugared form yourself, something like:
(doseq [v [#'ns1/fn1 #'ns1/fn2 #'ns2/fn1]]
(let [old #v]
(try
(alter-var-root v (constantly (fn [& args]
(throw (RuntimeException. "fail!")))))
(let [resp (app (request :get "/foo")))
(is (= (:status resp) 500))
(is (= (:body resp) "Something went wrong")))
(finally (alter-var-root v (constantly old))))))
Following on from amalloy's great answer, here's a utility function that I wrote to use in my tests:
(defn with-mocks
"Iterates through a list of functions to-mock, rebinding each to mock-fn, and calls
test-fn with the optional test-args.
Example:
(defn foobar [a b]
(try (+ (foo a) (bar b))
(catch Exception _ 1)))
(deftest test-foobar
(testing \"Exceptions are handled\"
(with-mocks
[#'foo #'bar]
(fn [& _] (throw (RuntimeException. \"\")))
(fn [a b] (is (= 1 (foobar a b)))) 1 2)))"
[to-mock mock-fn test-fn & test-args]
(doseq [f to-mock]
(let [real-fn #f]
(try
(alter-var-root f (constantly mock-fn))
(apply test-fn test-args)
(finally
(alter-var-root f (constantly real-fn)))))))

How to read clojure exceptions in REPL?

Many times when I try to write some function I get exception. That is normal. In Java you can find place and reason why exception occures but in clojure exception texts just make me crazy. Is there some tips how to read exceptions in clojure and how to find where in code exception happens and why?
For example I will take some code:
(do
(list?)
(list? [])
(list? '(1 2 3))
(list? (defn f [] (do ())))
(list? "a"))
when I call this function in REPL I will get
java.lang.IllegalArgumentException: Wrong number of args (0) passed to: core$list-QMARK- (NO_SOURCE_FILE:46)
which do not help me much to find the problem in second line. In little more complicated code it will give almost no information. (Of course it tells look at list? in some there is wrong number of arguments.) Is it wrong that I try to write code in REPL? How to read exception mesages in REPL? Is there way how to get better information about exceptions in REPL?
You can use clojure.stacktrace: http://richhickey.github.com/clojure/clojure.stacktrace-api.html
Usage:
(use 'clojure.stacktrace)
(/ 1 0)
(e)
Output:
java.lang.ArithmeticException: Divide by zero
at clojure.lang.Numbers.divide (Numbers.java:156)
clojure.lang.Numbers.divide (Numbers.java:3691)
user$eval954.invoke (NO_SOURCE_FILE:1)
clojure.lang.Compiler.eval (Compiler.java:6511)
clojure.lang.Compiler.eval (Compiler.java:6477)
clojure.core$eval.invoke (core.clj:2797)
clojure.main$repl$read_eval_print__6405.invoke (main.clj:245)
clojure.main$repl$fn__6410.invoke (main.clj:266)
nil
Acquire org.clojure/tools.trace.
user=> (use 'clojure.tools.trace)
Let's try a dotrace (changed up the order to make things more interesting):
user=> (dotrace [list?]
#_=> (do
#_=> (list? [])
#_=> (list? '(1 2 3))
#_=> (list?)
#_=> (list? (defn f [] (do ())))
#_=> (list? "a"))
#_=> )
IllegalStateException Can't dynamically bind non-dynamic var: clojure.core/list?
clojure.lang.Var.pushThreadBindings (Var.java:353)
Hmm...
user=> (.setDynamic #'list?)
#'clojure.core/list?
Let's try that again:
user=> (dotrace [list?]
#_=> (do
#_=> (list? [])
#_=> (list? '(1 2 3))
#_=> (list?)
#_=> (list? (defn f [] (do ())))
#_=> (list? "a")))
TRACE t1216: (list? [])
TRACE t1216: => false
TRACE t1217: (list? (1 2 3))
TRACE t1217: => true
TRACE t1218: (list?)
ArityException Wrong number of args (0) passed to: core$list-QMARK-
clojure.lang.AFn.throwArity (AFn.java:437)
Aha! Made it to (list?) before the exception.

How to use clojure.edn/read to get a sequence of objects in a file?

Clojure 1.5 introduced clojure.edn, which includes a read function that requires a PushbackReader.
If I want to read the first five objects, I can do:
(with-open [infile (java.io.PushbackReader. (clojure.java.io/reader "foo.txt"))]
(binding [*in* infile]
(let [edn-seq (repeatedly clojure.edn/read)]
(dorun (take 5 (map println edn-seq))))))
How can I instead print out all of the objects? Considering that some of them may be nils, it seems like I need to check for the EOF, or something similar. I want to have a sequence of objects similar to what I would get from line-seq.
Use :eof key
http://clojure.github.com/clojure/clojure.edn-api.html
opts is a map that can include the following keys: :eof - value to
return on end-of-file. When not supplied, eof throws an exception.
edit: sorry, that wasn't enough detail! here y'go:
(with-open [in (java.io.PushbackReader. (clojure.java.io/reader "foo.txt"))]
(let [edn-seq (repeatedly (partial edn/read {:eof :theend} in))]
(dorun (map println (take-while (partial not= :theend) edn-seq)))))
that should do it
I looked at this again. Here is what I came up with:
(defn edn-seq
"Returns the objects from stream as a lazy sequence."
([]
(edn-seq *in*))
([stream]
(edn-seq {} stream))
([opts stream]
(lazy-seq (cons (clojure.edn/read opts stream) (edn-seq opts stream)))))
(defn swallow-eof
"Ignore an EOF exception raised when consuming seq."
[seq]
(-> (try
(cons (first seq) (swallow-eof (rest seq)))
(catch java.lang.RuntimeException e
(when-not (= (.getMessage e) "EOF while reading")
(throw e))))
lazy-seq))
(with-open [stream (java.io.PushbackReader. (clojure.java.io/reader "foo.txt"))]
(dorun (map println (swallow-eof (edn-seq stream)))))
edn-seq has the same signature as clojure.edn/read, and preserves all of the existing behavior, which I think is important given that people may use the :eof option in different ways. A separate function to contain the EOF exception seemed like a better choice, though I'm not sure how best to capture it since it shows up just as a java.lang.RuntimeException.