Assume I'm sending an email, asynchronously, and want my program to continue its execution. So far, I've been doing this with futures, but unfortunately, when the email fails to send, no exception is raised.
I understand that dereferencing the future will raise an ExecutionException, but derefing would defeat the point.
Is there a better way to "fire and go", without losing exception information?
I'd go with agents for this, using handler functions to handle exceptions thrown by agent actions:
(agent initial-state :error-handler handler-fn)
See (doc agent), (doc set-error-handler!), (doc set-error-mode!) for details. initial-state here might simply be nil, or perhaps a structure holding some logging data.
To make this convenient, you'd want to have an email function usable with send (send-off, send-via):
(defn email [agent-state message] ...)
If the main thread needs to be notified that something is amiss, it will need to pay attention to communications over some channel. (Java queues are one possibility, the channels of core.async are another.) The handler function can then push messages over that channel.
Without trying...
(def foo (future (Thread/sleep 5000)
(throw (Exception. "bananas!"))
(Thread/sleep 5000)))
(future-done? foo)
;=> false
5 seconds later...
(future-done? foo)
;=> true
#foo
;=> Exception bananas! user/fn...
With trying...
(def bar (future (try (Thread/sleep 5000)
(throw (Exception. "bananas!"))
(Thread/sleep 5000)
(catch Exception e (println "Oh, " (.getMessage e))))))
(future-done? bar)
;=> false
5 seconds later...
;=>Oh, bananas!
#bar
;=> nil
Related
I have a Ring-based server which has an atom for storing application state which is periodically fetched from database every 10 seconds for frequently changing info and every 60 seconds for the rest.
(defn set-world-update-interval
[f time-in-ms]
(let [stop (async/chan)]
(async/go-loop []
(async/alt!
(async/timeout time-in-ms) (do (async/<! (async/thread (f)))
(recur))
stop :stop))
stop))
(mount/defstate world-listener
:start (set-world-update-interval #(do (println "Checking data in db") (reset! world-atom (fetch-world-data)) ) 10000)
:stop (async/close! world-listener))
It works pretty good. RAM usage is pretty stable. But I'm wondering if this is an improper use of core.async?
Perhaps it should be a regular Thread instead like this?
(doto (Thread. (fn []
(loop []
(Thread/sleep 1000)
(println "Checking data in db")
(reset! world-atom (fetch-world-data))
(recur))))
(.setUncaughtExceptionHandler
(reify Thread$UncaughtExceptionHandler
(uncaughtException [this thread exception]
(println "Cleaning up!"))))
(.start))
While there's nothing wrong with your core.async implementation of this pattern, I'd suggest using a java.util.concurrent.ScheduledExecutorService for this. It gives you precise control over the thread pool and the scheduling.
Try something like this:
(ns your-app.world
(:require [clojure.tools.logging :as log]
[mount.core :as mount])
(:import
(java.util.concurrent Executors ScheduledExecutorService ThreadFactory TimeUnit)))
(defn ^ThreadFactory create-thread-factory
[thread-name-prefix]
(let [thread-number (atom 0)]
(reify ThreadFactory
(newThread [_ runnable]
(Thread. runnable (str thread-name-prefix "-" (swap! thread-number inc)))))))
(defn ^ScheduledExecutorService create-single-thread-scheduled-executor
[thread-name-prefix]
(let [thread-factory (create-thread-factory thread-name-prefix)]
(Executors/newSingleThreadScheduledExecutor thread-factory)))
(defn schedule
[executor runnable interval unit]
(.scheduleWithFixedDelay executor runnable 0 interval unit))
(defn shutdown-executor
"Industrial-strength executor shutdown, modify/simplify according to need."
[^ScheduledExecutorService executor]
(if (.isShutdown executor)
(log/info "Executor already shut down")
(do
(log/info "Shutting down executor")
(.shutdown executor) ;; Disable new tasks from being scheduled
(try
;; Wait a while for currently running tasks to finish
(if-not (.awaitTermination executor 10 TimeUnit/SECONDS)
(do
(.shutdownNow executor) ;; Cancel currently running tasks
(log/info "Still waiting to shut down executor. Sending interrupt to tasks.")
;; Wait a while for tasks to respond to being cancelled
(when-not (.awaitTermination executor 10 TimeUnit/SECONDS)
(throw (ex-info "Executor could not be shut down" {}))))
(log/info "Executor shutdown completed"))
(catch InterruptedException _
(log/info "Interrupted while shutting down. Sending interrupt to tasks.")
;; Re-cancel if current thread also interrupted
(.shutdownNow executor)
;; Preserve interrupt status
(.interrupt (Thread/currentThread)))))))
(defn world-updating-fn
[]
(log/info "Updating world atom")
;; Do your thing here
)
(mount/defstate world-listener
:start (doto (create-single-thread-scheduled-executor "world-listener")
(schedule world-updating-fn 10 TimeUnit/MINUTES))
:stop (shutdown-executor world-listener))
It seems a little silly to me to use go-loop to create a goroutine that parks on a timeout, when all the work you actually want to do is IO-intensive, and therefore (rightly) done in a separate thread. The result of this is that you're going through threads every cycle. These threads are pooled by core.async, so you're not doing the expensive work of creating new threads from nothing, but there's still some overhead to getting them out of the pool and interacting with core.async for the timeout. I'd just keep it simple by leaving core.async out of this operation.
I have implemented retry policy for update function (talking to database) - if update throws exception I retry it up to 10 times. I am mocking the update function with midje. I want to simulate that first time it fails and second time it succeeds. I tried this :
(fact "update is retried when failed"
(ud/ensure-data {:username ..username.. :data :h}) => ..result..
(provided
(ud/get-raw-user-data ..username..) => example-user-data-raw
(ud/update-user-data {:username ..username..
:version 1
:userdata {:data {:h {}}}}) =throws=> (Exception.)
(ud/update-user-data {:username ..username..
:version 1
:userdata {:data {:h {}}}}) => ..result..))
but this doesn't seem to work...... Response is :
These calls were not made the right number of times:
(ud/update-user-data {:username ..username.., :version 1, :userdata {:homebases {:h {:sensors []}}}}) [expected at least once, actually never called]
I also found streams (https://github.com/marick/Midje/wiki/Variant-prerequisite-arrows) but I don't know how to combine Exceptions with success calls with streams.
I haven't got a clear understanding of how to use Midje streams either. So my personal solution is, in a word, not to use provided or 'streams', but use with-redefs and 'stubs'.
(defn fn0 [] -1) ;; function we want to mock
;; fn1 retries fn0 if there is an exception
(defn fn1 [] (try
(fn0)
(catch Exception e
(do
(prn "Exception Caught, try again...")
(fn0)))))
(def call-count (atom -1)) ;; counts how many times fn0 is called
;; stub fn0 by returning different result
(defn make-stub [& result-seqs]
(fn [& _]
(swap! call-count inc)
(let [result (nth result-seqs #call-count)]
(if (instance? Throwable result)
(throw result)
result))))
(fact "fn1 ignores first exception and returns second call to fn0"
(with-redefs [fn0 (make-stub (Exception. "stubbed error") 100)]
(fn1) => 100))
In the following example I can see, that published messages arrive to the subscribed channel, although, they are published before subscription is made.
(let [in (async/chan)
out (async/chan)
pub (async/pub in :key)]
(async/go
(>! in {:id 1 :key :k1})
(>! in {:id 2 :key :k1})
(>! in {:id 3 :key :k1}))
(async/sub pub :k1 out)
(async/go-loop []
(println (<! out))
(recur)))
Is this expected behavior? As far as I can see in the documentation, it clearly states:
Items received when there are no matching subs get dropped.
I get same results in both Clojure and ClojureScript.
Added:
with mult/tap I see similar behavior
You don't know that the messages are published before the subscription is made. Because go is asynchronous, it's very possible that the subscription happens before the first message has been put into the channel. It's a race condition, really.
Try putting a (Thread/sleep [some value]) before the suscription and see what happens.
I apologize in advance for the length of this note. I spent considerable time making it shorter, and this was as small as I could get it.
I have a mystery and would be grateful for your help. This mystery comes from the behavior of an rxjava observer I wrote in Clojure over a couple of straightforward observables cribbed from online samples.
One observable synchronously sends messages to the onNext handlers of its observers, and my supposedly principled observer behaves as expected.
The other observable asynchronously does the same, on another thread, via a Clojure future. The exact same observer does not capture all events posted to its onNext; it just seems to lose a random number of messages at the tail.
There is an intentional race in the following between the expiration of a wait for the promised onCompleted and the expiration of a wait for all events sent to an agent collector. If the promise wins, I expect to see false for onCompleted and a possibly short queue in the agent. If the agent wins, I expect to see true for onCompleted and all messages from the agent's queue. The one result I DO NOT expect is true for onCompleted AND a short queue from the agent. But, Murphy doesn't sleep, and that's exactly what I see. I don't know whether garbage-collection is at fault, or some internal queuing to Clojure's STM, or my stupidity, or something else altogether.
I present the source in the order of its self-contained form, here, so that it can be run directly via lein repl. There are three cermonials to get out of the way: first, the leiningen project file, project.clj, which declares dependency on the 0.9.0 version of Netflix's rxjava:
(defproject expt2 "0.1.0-SNAPSHOT"
:description "FIXME: write description"
:url "http://example.com/FIXME"
:license {:name "Eclipse Public License"
:url "http://www.eclipse.org/legal/epl-v10.html"}
:dependencies [[org.clojure/clojure "1.5.1"]
[com.netflix.rxjava/rxjava-clojure "0.9.0"]]
:main expt2.core)
Now, the namespace and a Clojure requirement and the Java imports:
(ns expt2.core
(:require clojure.pprint)
(:refer-clojure :exclude [distinct])
(:import [rx Observable subscriptions.Subscriptions]))
Finally, a macro for output to the console:
(defmacro pdump [x]
`(let [x# ~x]
(do (println "----------------")
(clojure.pprint/pprint '~x)
(println "~~>")
(clojure.pprint/pprint x#)
(println "----------------")
x#)))
Finally, to my observer. I use an agent to collect the messages sent by any observable's onNext. I use an atom to collect a potential onError. I use a promise for the onCompleted so that consumers external to the observer can wait on it.
(defn- subscribe-collectors [obl]
(let [;; Keep a sequence of all values sent:
onNextCollector (agent [])
;; Only need one value if the observable errors out:
onErrorCollector (atom nil)
;; Use a promise for 'completed' so we can wait for it on
;; another thread:
onCompletedCollector (promise)]
(letfn [;; When observable sends a value, relay it to our agent"
(collect-next [item] (send onNextCollector (fn [state] (conj state item))))
;; If observable errors out, just set our exception;
(collect-error [excp] (reset! onErrorCollector excp))
;; When observable completes, deliver on the promise:
(collect-completed [ ] (deliver onCompletedCollector true))
;; In all cases, report out the back end with this:
(report-collectors [ ]
(pdump
;; Wait for everything that has been sent to the agent
;; to drain (presumably internal message queues):
{:onNext (do (await-for 1000 onNextCollector)
;; Then produce the results:
#onNextCollector)
;; If we ever saw an error, here it is:
:onError #onErrorCollector
;; Wait at most 1 second for the promise to complete;
;; if it does not complete, then produce 'false'.
;; I expect if this times out before the agent
;; times out to see an 'onCompleted' of 'false'.
:onCompleted (deref onCompletedCollector 1000 false)
}))]
;; Recognize that the observable 'obl' may run on another thread:
(-> obl
(.subscribe collect-next collect-error collect-completed))
;; Therefore, produce results that wait, with timeouts, on both
;; the completion event and on the draining of the (presumed)
;; message queue to the agent.
(report-collectors))))
Now, here is a synchronous observable. It pumps 25 messages down the onNext throats of its observers, then calls their onCompleteds.
(defn- customObservableBlocking []
(Observable/create
(fn [observer] ; This is the 'subscribe' method.
;; Send 25 strings to the observer's onNext:
(doseq [x (range 25)]
(-> observer (.onNext (str "SynchedValue_" x))))
; After sending all values, complete the sequence:
(-> observer .onCompleted)
; return a NoOpSubsription since this blocks and thus
; can't be unsubscribed (disposed):
(Subscriptions/empty))))
We subscribe our observer to this observable:
;;; The value of the following is the list of all 25 events:
(-> (customObservableBlocking)
(subscribe-collectors))
It works as expected, and we see the following results on the console
{:onNext (do (await-for 1000 onNextCollector) #onNextCollector),
:onError #onErrorCollector,
:onCompleted (deref onCompletedCollector 1000 false)}
~~>
{:onNext
["SynchedValue_0"
"SynchedValue_1"
"SynchedValue_2"
"SynchedValue_3"
"SynchedValue_4"
"SynchedValue_5"
"SynchedValue_6"
"SynchedValue_7"
"SynchedValue_8"
"SynchedValue_9"
"SynchedValue_10"
"SynchedValue_11"
"SynchedValue_12"
"SynchedValue_13"
"SynchedValue_14"
"SynchedValue_15"
"SynchedValue_16"
"SynchedValue_17"
"SynchedValue_18"
"SynchedValue_19"
"SynchedValue_20"
"SynchedValue_21"
"SynchedValue_22"
"SynchedValue_23"
"SynchedValue_24"],
:onError nil,
:onCompleted true}
----------------
Here is an asynchronous observable that does exactly the same thing, only on a future's thread:
(defn- customObservableNonBlocking []
(Observable/create
(fn [observer] ; This is the 'subscribe' method
(let [f (future
;; On another thread, send 25 strings:
(doseq [x (range 25)]
(-> observer (.onNext (str "AsynchValue_" x))))
; After sending all values, complete the sequence:
(-> observer .onCompleted))]
; Return a disposable (unsubscribe) that cancels the future:
(Subscriptions/create #(future-cancel f))))))
;;; For unknown reasons, the following does not produce all 25 events:
(-> (customObservableNonBlocking)
(subscribe-collectors))
But, surprise, here is what we see on the console: true for onCompleted, implying that the promise DID NOT TIME-OUT; but only some of the asynch messages. The actual number of messages we see varies from run to run, implying that there is some concurrency phenomenon at play. Clues appreciated.
----------------
{:onNext (do (await-for 1000 onNextCollector) #onNextCollector),
:onError #onErrorCollector,
:onCompleted (deref onCompletedCollector 1000 false)}
~~>
{:onNext
["AsynchValue_0"
"AsynchValue_1"
"AsynchValue_2"
"AsynchValue_3"
"AsynchValue_4"
"AsynchValue_5"
"AsynchValue_6"],
:onError nil,
:onCompleted true}
----------------
The await-for on agent means Blocks the current thread until all actions dispatched thus
far (from this thread or agent) to the agents have occurred, which means that it may happen that after your await is over there is still some other thread that can send messages to the agent and that is what is happening in your case. After your await on agent is over and you have deref its value in the :onNext key in the map, then you wait for the on completed promise which turns out to be true after the wait but in the mean time some other messages were dispatched to the agent to be collected into the vector.
You can solve this by having the :onCompleted key as the first key in the map which basically means wait for the completion and then wait for the agents coz by that time there is no more send calls on the agent can happen after as have already received onCompleted.
{:onCompleted (deref onCompletedCollector 1000 false)
:onNext (do (await-for 0 onNextCollector)
#onNextCollector)
:onError #onErrorCollector
}
What would be an idiomatic way of executing a function within a time limit? Something like,
(with-timeout 5000
(do-somthing))
Unless do-something returns within 5000 throw an exception or return nil.
EDIT: before someone points it out there is,
clojure (with-timeout ... macro)
but with that the future keeps executing that does not work in my case.
I think you can do this reasonably reliably by using the timeout capability within futures:
(defmacro with-timeout [millis & body]
`(let [future# (future ~#body)]
(try
(.get future# ~millis java.util.concurrent.TimeUnit/MILLISECONDS)
(catch java.util.concurrent.TimeoutException x#
(do
(future-cancel future#)
nil)))))
A bit of experimenting verified that you need to do a future-cancel to stop the future thread from continuing to execute....
What about?
(defn timeout [timeout-ms callback]
(let [fut (future (callback))
ret (deref fut timeout-ms ::timed-out)]
(when (= ret ::timed-out)
(future-cancel fut))
ret))
(timeout 100 #(Thread/sleep 1000))
;=> :user/timed-out
This isn't something you can do 100% reliably on the JVM. The only way to stop something after a while is to give it a new thread, and then send that thread an exception when you want it to stop. But their code can catch the exception, or they can spin up another thread that you don't control, or...
But most of the time, and especially if you control the code that's being timed out, you can do something like we do in clojail:
If you wanted to make that prettier you could define a macro like
(defmacro with-timeout [time & body]
`(thunk-timeout (fn [] ~#body) ~time))
It's a quite a breeze using clojure's channel facilities
https://github.com/clojure/core.async
require respective namespace
(:require [clojure.core.async :refer [>! alts!! timeout chan go]])
the function wait takes a timeout [ms], a function [f] and optional parameters [args]
(defn wait [ms f & args]
(let [c (chan)]
(go (>! c (apply f args)))
(first (alts!! [c (timeout ms)]))))
third line pops off the call to f to another thread. fourth line consumes the result of the function call or (if faster) the timeout.
consider the following example calls
(wait 1000 (fn [] (do (Thread/sleep 100) 2)))
=> 2
but
(wait 50 (fn [] (do (Thread/sleep 100) 2)))
=> nil
You can probably use an agent, and then await-for it.
Adding a possible (macro-less) alternative to the mix (though the macro isn't required in the accepted answer of course)
(defn with-timeout [f ms]
(let [p (promise)
h (future
(deliver p (f)))
t (future
(Thread/sleep ms)
(future-cancel h)
(deliver p nil))]
#p))
Requires two threads, but just an idea.