I am writing a program that allows two futures to transfer money from account A to account B. Each future will try to transfer its amount and sleep for a while after that. My program compiled with No Error, so I don't know where to start debugging it. It is supposed to print out text, but it does not. Can someone tell me what is happening with my program?
;here are initial amounts of balance A and B
(def balanceA {ref 1000})
(def balanceB {ref 2000})
;agent will count a number of complete transfer
(def agentCount {agent 1})
; this func will do the transfer with the waitingTime/sleep
(defn transfer [balanceA balanceB amount futureNum waitingTime]
)
; Two futures will repeat 10 times doing the transactions and print out the balances
(dotimes [n 10](def futureA (future transfer(balanceA balanceB 20 1 (rand-int 100)) (prn "result" #balanceA #balanceB))))
(dotimes [n 10](def futureB (future transfer(balanceA balanceB 15 2 (rand-int 40))(prn "result" #balanceA #balanceB))))
(shutdown-agents)
UPDATED: thanks to #noisesmith comment/correction!
Only you have a displaced parenthesis on your transfer call and an error on your ref and agent definitions
The error definition: {ref 1000} or {agent 1} instead of (ref 1000) (agent 1)
(def balanceA (ref 1000))
(def balanceB (ref 2000))
(def agentCount (agent 1))
and the parenthesis ...
(future transfer ( balanceA ... => (future transfer balanceA ...
You have to change this:
(dotimes [n 10](def futureA (future transfer(balanceA balanceB 20 1 (rand-int 100)) (prn "result" #balanceA #balanceB))))
(dotimes [n 10](def futureB (future transfer(balanceA balanceB 15 2 (rand-int 40))(prn "result" #balanceA #balanceB))))
With this:
(dotimes [n 10](def futureA (future (transfer balanceA balanceB 20 1 (rand-int 100)) (print "result" #balanceA #balanceB))))
(dotimes [n 10](def futureB (future (transfer balanceA balanceB 15 2 (rand-int 40))(prn "result" #balanceA #balanceB))))
Related
In my Clojure project I'm trying to make a list of http calls to an API that has a rate limiter that only allows n calls per minute. I want each of the responses to be returned once all the http calls are finished for further processing. I am new to Clojure's Core Async, but thought it would be a good fit, but because I need to run each call n seconds apart I am also trying to use the Chime library. In Chime's library it has examples using Core Async, but the examples all call the same function at each time interval which won't work for this use case.
While there is probably a way to use chime-async that better serves this use case, all of my attempts at that have failed so I've tried simply wrapping Chime calls with core async, but I am probably more baffled by Core Async than Chime.
This is an example of my name space.
(ns mp.util.schedule
(:require [chime.core :as chime]
[clojure.core.async :as a]
[tick.alpha.api :as tick]))
(defn schedule-fns
"Takes a list of functions and a duration in seconds then runs each function in the list `sec` seconds apart
optionally provide an inst to start from"
[fs sec & [{:keys [inst] :or {inst (tick/now)}}]]
(let [ch (a/chan (count fs))
chime-times (map-indexed
(fn mapped-fn [i f]
(a/put! ch (chime/chime-at [(.plusSeconds inst (* i sec))]
(fn wrapped-fn [_] (f)))))
fs)]
(doseq [chi chime-times]
(a/<!! chi))))
; === Test Code ===
; simple test function
(defn sim-fn
"simple function that prints a message and value, then returns the value"
[v m]
(println m :at (tick/now))
v)
; list of test functions
(def fns [#(sim-fn 1 :one)
#(sim-fn 2 :two)
#(sim-fn 3 :three)])
What I want to happen when calling (schedule-fns fns 2) is for each function in fns to run n seconds from each other and for schedule-fns to return (1 2 3) (the return values of the functions), but this isn't what it is doing. It is calling each of the functions at the correct times (which I can see from the log statements) but it isn't returning anything and there's an error I don't understand. I'm getting:
(schedule-fns fns 2)
:one :at #time/instant "2021-03-05T23:31:52.565Z"
Execution error (IllegalArgumentException) at clojure.core.async.impl.protocols/eval11496$fn$G (protocols.clj:15).
No implementation of method: :take! of protocol: #'clojure.core.async.impl.protocols/ReadPort found for class: java.lang.Boolean
:two :at #time/instant "2021-03-05T23:31:54.568Z"
:three :at #time/instant "2021-03-05T23:31:56.569Z"
If I could get help getting my code to use Core Async properly (with or without Chime) I'd really appreciate it. Thanks.
Try this:
(defn sim-fn
"simple function that prints a message and value, then returns the value"
[v m]
(println m)
v)
; list of test functions
(def fns [#(sim-fn 1 :one)
#(sim-fn 2 :two)
#(sim-fn 3 :three)])
(defn schedule-fns [fns sec]
(let [program (interpose #(Thread/sleep (* sec 1000))
fns)]
(remove #(= % nil)
(for [p program]
(p)))))
Then call:
> (schedule-fns fns 2)
:one
:two
:three
=> (1 2 3)
I came up with a way to get what I want...with some caveats.
(def results (atom []))
(defn schedule-fns
"Takes a list of functions and a duration in seconds then runs each function in the list `sec` seconds apart
optionally provide an inst to start from"
[fs sec]
(let [ch (chan (count fs))]
(go-loop []
(swap! results conj (<! ch))
(recur))
(map-indexed (fn [i f]
(println :waiting (* i sec) :seconds)
(go (<! (timeout (* i sec 1000)))
(>! ch (f))))
fs)))
This code has the timing and behavior that I want, but I have to use an atom to store the responses. While I can add a watcher to determine when all the results are in, I still feel like I shouldn't have to do that.
I guess I'll use this for now, but at some point I'll keep working on this and if anyone has something better than this approach I'd love to see it.
I had a couple friends look at this and they each came up with different answers. These are certainly better than what I was doing.
(defn schedule-fns [fs secs]
(let [ret (atom {})
sink (a/chan)]
(doseq [[n f] (map-indexed vector fs)]
(a/thread (a/<!! (a/timeout (* 1000 n secs)))
(let [val (f)
this-ret (swap! ret assoc n val)]
(when (= (count fs) (count this-ret))
(a/>!! sink (mapv (fn [i] (get this-ret i)) (range (count fs))))))))
(a/<!! sink)))
and
(defn schedule-fns
[fns sec]
(let [concurrent (count fns)
output-chan (a/chan)
timedout-coll (map-indexed (fn [i f]
#(do (println "Waiting")
(a/<!! (a/timeout (* 1000 i sec)))
(f))) fns)]
(a/pipeline-blocking concurrent
output-chan
(map (fn [f] (f)))
(a/to-chan timedout-coll))
(a/<!! (a/into [] output-chan))))
If your objective is to work around the rate limiter, you can consider implementing it in the async channel. Below is one sample implementation - the function takes a channel, throttled its input with a token based limiter and pipe it to an output channel.
(require '[clojure.core.async :as async])
(defn rate-limiting-ch [input xf rate]
(let [tokens (numerator rate)
period (denominator rate)
ans (async/chan tokens xf)
next (fn [] (+ period (System/currentTimeMillis)))]
(async/go-loop [c tokens
t (next)]
(if (zero? c)
(do
(async/<! (async/timeout (- t (System/currentTimeMillis))))
(recur tokens (next)))
(when-let [x (async/<! input)]
(async/>! ans x)
(recur (dec c) t))))
ans))
And here is a sample usage:
(let [start (System/currentTimeMillis)
input (async/to-chan (range 10))
output (rate-limiting-ch input
;; simulate an api call with roundtrip time of ~300ms
(map #(let [wait (rand-int 300)
ans {:time (- (System/currentTimeMillis) start)
:wait wait
:input %}]
(Thread/sleep wait)
ans))
;; rate limited to 2 calls per 1000ms
2/1000)]
;; consume the output
(async/go-loop []
(when-let [x (async/<! output)]
(println x)
(recur))))
Output:
{:time 4, :wait 63, :input 0}
{:time 68, :wait 160, :input 1}
{:time 1003, :wait 74, :input 2}
{:time 1079, :wait 151, :input 3}
{:time 2003, :wait 165, :input 4}
{:time 2169, :wait 182, :input 5}
{:time 3003, :wait 5, :input 6}
{:time 3009, :wait 18, :input 7}
{:time 4007, :wait 138, :input 8}
{:time 4149, :wait 229, :input 9}
Why does Alpha stop early, when I expect it to behave like Beta? The only difference between Alpha and Beta is >! and put!, as commented below.
Alpha:
user=> (def q (chan))
#'user/q
user=> (def counter (atom 0))
#'user/counter
user=> (defn mg [event-queue]
#_=> (go-loop [event (<! event-queue)]
#_=> (swap! counter inc)
#_=> (when (< #counter 4)
#_=> (println "counter: " #counter)
#_=> (>! event-queue {:a #counter}) ;; Here's the only difference
#_=> (println "event: " event)
#_=> (recur (<! event-queue)))))
#'user/mg
user=> (mg q)
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x3a1ffd56 "clojure.core.async.impl.channels.ManyToManyChannel#3a1ffd56"]
user=> (put! q "hi")
counter: true
1
user=>
Beta:
user=> (def q (chan))
#'user/q
user=> (def counter (atom 0))
#'user/counter
user=> (defn mg [event-queue]
#_=> (go-loop [event (<! event-queue)]
#_=> (swap! counter inc)
#_=> (when (< #counter 4)
#_=> (println "counter: " #counter)
#_=> (put! event-queue {:a #counter}) ;; Here's the only difference
#_=> (println "event: " event)
#_=> (recur (<! event-queue)))))
#'user/mg
user=> (mg q)
#object[clojure.core.async.impl.channels.ManyToManyChannel 0x72c9b65a "clojure.core.async.impl.channels.ManyToManyChannel#72c9b65a"]
user=> (put! q "hi")
true
counter: 1
event: hi
counter: 2
event: {:a 1}
counter: 3
event: {:a 2}
user=>
It's also interesting that, after executing Alpha, the channel #'user/q was properly enqueued:
user=> (take! q println)
event: hi
{:a 1}
nil
user=>
The same results occur in both Clojure and Clojurescript. Is this some sort of deadlock, or is the suppose to happen?
This is expected.
The channel q is created without a buffer, so when a value is placed with >!, it will block (park) the go-loop until another thread is ready to consume the value with <!.
One way to work around this is to give q a 1-slot buffer with (def q (chan 1)). The buffer allows 1 value to be placed in the channel without blocking the sender.
Beta behaves differently because put! is asynchronous wrt. the caller -- it uses a separate thread to place the new value in the channel. This avoids blocking the current go-loop, allowing the channel to be read and progress to continue.
(ns learnclojure.core)
(def acct1 (atom 1000 :validator #(>= % 0)))
(def acct2 (atom 1000 :validator #(>= % 0)))
(defn transfer [from-ac to-ac amt]
(swap! to-ac + amt)
(swap! from-ac - amt))
(dotimes [_ 10]
(future (transfer acct2 acct1 100)))
(deref acct1)
(deref acct2)
(def acct1 (ref 1000 :validator #(>= % 0)))
(def acct2 (ref 1000 :validator #(>= % 0)))
(defn transfer [from-ac to-ac amt]
(dosync
(alter to-ac + amt)
(alter from-ac - amt)))
(dotimes [_ 10]
(future (transfer acct2 acct1 100)))
(deref acct1)
(deref acct2)
I have two Clojure code changing states concurrently.
The first one that uses atom (line 3 - 14) seems to be working fine, whereas the second one that uses ref (line 17 and 29) shows random results. What might be wrong?
The last (deref acct1) (deref acct2) forms are evaluated before the futures are done executing.
What's more, the result is inconsistent because the reads are not coordinated; if you had written something like (dosync [(deref acct1) (deref acct2)]) the sum would always be 2000.
By the way, I strongly recommend you do not re-define the #'transfer, #'acct1 and #'acct2 vars for this kind of concurrency experiment; choose different names :)
Is there any way to poll whether Clojure's STM transactions are being retried, and at what rate?
You can observe the history count of a ref which will indicate that there is contention on it:
user=> (def my-ref (ref 0 :min-history 1))
#'user/my-ref
user=> (ref-history-count my-ref)
0
user=> (dosync (alter my-ref inc))
1
user=> (ref-history-count my-ref)
1
The history count does not directly represent contention. Instead it represents the number of past values that have been maintained in order to service concurrent reads.
The size of the history is limited by min and max values. By default those are 0 and 10, respectively, but you can change them when creating the ref (see above). Since min-history is 0 by default, you won't usually see ref-history-count return non-zero values, unless there is contention on the ref.
See more discussion on history count here: https://groups.google.com/forum/?fromgroups#!topic/clojure/n_MKCoa870o
I don't think there is any way, provided by clojure.core, to observe the rate of STM transactions at the moment. You can of course do something similar to what #Chouser did in his history stress test:
(dosync
(swap! try-count inc)
...)
i.e. increment a counter inside the transaction. The increment will happen every time the transaction is tried. If try-count is larger than 1, the transaction was retried.
By introducing named dosync blocks and commit counts (the times a named dosync has succeeded), one can quite easily keep track of the times threads have retried a given transaction.
(def ^{:doc "ThreadLocal<Map<TxName, Map<CommitNumber, TriesCount>>>"}
local-tries (let [l (ThreadLocal.)]
(.set l {})
l))
(def ^{:doc "Map<TxName, Int>"}
commit-number (ref {}))
(def history ^{:doc "Map<ThreadId, Map<TxName, Map<CommitNumber, TriesCount>>>"}
(atom {}))
(defn report [_ thread-id tries]
(swap! history assoc thread-id tries))
(def reporter (agent nil))
(defmacro dosync [tx-name & body]
`(clojure.core/dosync
(let [cno# (#commit-number ~tx-name 0)
tries# (update-in (.get local-tries) [~tx-name] update-in [cno#] (fnil inc 0))]
(.set local-tries tries#)
(send reporter report (.getId (Thread/currentThread)) tries#))
~#body
(alter commit-number update-in [~tx-name] (fnil inc 0))))
Given the following example...
(def foo (ref {}))
(def bar (ref {}))
(defn x []
(dosync :x ;; `:x`: the tx-name.
(let [r (rand-int 2)]
(alter foo assoc r (rand))
(Thread/sleep (rand-int 400))
(alter bar assoc (rand-int 2) (#foo r)))))
(dotimes [i 4]
(future
(dotimes [i 10]
(x))))
...#history evaluates to:
;; {thread-id {tx-name {commit-number tries-count}}}
{40 {:x {3 1, 2 4, 1 3, 0 1}}, 39 {:x {2 1, 1 3, 0 1}}, ...}
This additional implementation is substantially simpler.
;; {thread-id retries-of-latest-tx}
(def tries (atom {}))
;; The max amount of tries any thread has performed
(def max-tries (atom 0))
(def ninc (fnil inc 0))
(def reporter (agent nil))
(defn report [_ tid]
(swap! max-tries #(max % (get #tries tid 0)))
(swap! tries update-in [tid] (constantly 0)))
(defmacro dosync [& body]
`(clojure.core/dosync
(swap! tries update-in [(.getId (Thread/currentThread))] ninc)
(commute commit-id inc)
(send reporter report (.getId (Thread/currentThread)))
~#body))
I can launch two threads and they work, but synchronously. What am I missing to get these threads independently launched?
main, thread, and output
(defn -main
[& args]
(do
(let [grid-dim-in [0 5]
mr1-pos [\N 2 4]
mr2-pos [\N 1 5]
mr1-movs "LMLMMRMM"
mr2-movs "RMRMMMLM"]
(reset! grid-dim grid-dim-in)
(reset! mr1-id {:mr1 mr1-pos})
(reset! mr2-id {:mr2 mr2-pos})
(.start (Thread. (rover-thread mr1-id mr1-movs update-work-block)))
(.start (Thread. (rover-thread mr2-id mr2-movs update-work-block))))))
(defn rover-thread [id movs update-ref]
(let [id-key (keys #id)
id-vals (vals #id)]
(doseq [mov movs]
(println "Rover " id-key " is moving ")
(let [new-mov (determine-rover-move (first id-vals) mov)]
(move-rover id new-mov update-ref)
(print "Rover ")
(print (first id-key))
(print " is at ")
(println new-mov)
(Thread/sleep (rand 1000)))))
Rover :mr1 is at [E 2 4]
Rover (:mr1) is moving
Rover :mr1 is at [N 2 5]
Rover (:mr1) is moving
Rover :mr1 is at [N 2 5]
Finished on Thread[main,5,main]
Rover (:mr2) is moving
Rover :mr2 is at [E 1 5]
Rover (:mr2) is moving
Rover :mr2 is at [N 1 6]
Take a close look at these two lines:
(.start (Thread. (rover-thread mr1-id mr1-movs update-work-block)))
(.start (Thread. (rover-thread mr2-id mr2-movs update-work-block))))))
This code evaluates the (rover-thread mr1-id mr1-movs update-work-block) first, and passes the result of that to the constructor of Thread, which is not what you want.
Here's a simple function to illustrate the principle. This doesn't work, because the (f ...) is evaluated before its result it passed to the Thread constructor:
(defn run-thread-thing-wrong []
(let [f (fn [n s]
(doseq [i (range n)]
(prn s i)
(Thread/sleep (rand 1000))))]
(.start (Thread. (f 10 "A")))
(.start (Thread. (f 10 "B"))))
nil)
Here's a version that does work. A function is passed to the Thread constructor instead:
(defn run-thread-thing []
(let [f (fn [n s]
(doseq [i (range n)]
(prn s i)
(Thread/sleep (rand 1000))))]
(.start (Thread. (fn [] (f 10 "A"))))
(.start (Thread. (fn [] (f 10 "B")))))
nil)
Note: instead of (fn [] ....) you can use the short form #(....) for anonymous functions.
Here's another version that does the same, but with a future instead of manually creating threads:
(defn run-thread-thing []
(let [f (fn [n s]
(doseq [i (range n)]
(prn s i)
(Thread/sleep (rand 1000))))]
(future (f 10 "A"))
(future (f 10 "B")))
nil)
Note that in this case, you pass a form to future instead of a function.
This seems like a really good place to use Clojure's agent feature. I am not qualified to fully explain how to use them, but a really good example of their usage can be found here. Starting threads using agents is dead-easy, and I think it is more idiomatic.
The code would look something like,
(def rover1 (agent [mr1-posn mr1-movs mr1-id]))
(def rover2 (agent [mr2-posn mr2-movs mr2-id]))
(defn rover-behave [[posn movs id]]
(send-off *agent* #'rover-behave)
(. Thread (sleep 1000))
(let [new-mov (determine-rover-move posn movs id)
new-posn (posn-after-move posn new-mov)]
;return value updates state of agent
[new-posn movs id]
)
)
(send-off rover1 rover-behave)
(send-off rover2 rover-behave)