I am using jetty "7.6.8.v20121106" as a part of https://github.com/ring-clojure/ring/tree/master/ring-jetty-adapter with my server.
I am making calls using http://http-kit.org/ with following code. Essentially I am making server calls but ignoring the response. What I am finding is that all the server threads become blocked/deadlocked after that. This seems like a really easy way to bring to server down and wanted to understand what is going on here.
Code from client is:
(require '[org.httpkit.client :as hk-client])
(defn hget [id]
(hk-client/get (str "http://localhost:5000/v1/pubapis/auth/ping?ping=" id)))
(doall (map hget (take 100 (range))))) ; Gives problem
(doall (map deref (map hget (take 100 (range)))))) ; Doesn't give problem
Threads blocked at
sun.nio.cs.StreamEncoder.write(StreamEncoder.java:118)
and deadlocked at
java.io.PrintStream.write(PrintStream.java:479)
Would really appreciate if someone can help with what is going on over here.
Finally found what the problem was. Took lot of digging through and starting with a sample project to find this.
When I started learning clojure and copied the following from somewhere for logging:
(defn log [msg & vals]
(let [line (apply format msg vals)]
(locking System/out (println line))))
The line locking over there was causing dead lock in some situation. Do not know enough about concurrency to solve this. Will create a separate question for that.
Removing this line fixes the problem.
Related
Hello and thank you for any help. I am just starting to learn Clojure, and think its amazing. Below is my codes for the sleeping barber problem. I thought that dropping-buffers from core/async would be perfect for this problem, and while it seems to work it never stops.
The haircuts and dropping buffer seem to work right.
---Edited
It does stop now. But I get an error trying to check customer-num for nil (I've noted the line in the code below). It seems like it can't do an if on a nil because it's nil!
(if (not (nil? customer-num)) ;; throws error => Cannot invoke "clojure.lang.IFn.invoke()" because the return value of "clojure.lang.IFn.invoke(Object)" is null
---End of edit
Also, how to get the return value of the number of haircuts to the calling operate-shop?
Sleeping barber problem as written up in Seven Languages in Seven Weeks. It was created by Edsger Dijkstra in 1965.
A barber shop takes customers.
Customers arrive at random intervals, from ten to thirty milliseconds.
The barber shop has three chairs in the waiting room.
The barber shop has one barber and one barber chair.
When the barber’s chair is empty, a customer sits in the chair, wakes up the barber, and gets a haircut.
If the chairs are occupied, all new customers will turn away.
Haircuts take twenty milliseconds.
After a customer receives a haircut, he gets up and leaves.
Determine how many haircuts a barber can give in ten seconds.
(ns sleepbarber
(:require [clojure.core.async
:as a
:refer [>! <! >!! <!! go go-loop chan dropping-buffer close! thread
alt! alts! alts!! timeout]]))
(def barber-shop (chan (dropping-buffer 3))) ;; no more than 3 customers waiting
(defn cut-hair []
(go-loop [haircuts 0]
(let [customer-num (<! barber-shop)]
(if (not (nil? customer-num)) ;; throws error => Cannot invoke "clojure.lang.IFn.invoke()" because the return value of "clojure.lang.IFn.invoke(Object)" is null
(do (<! (timeout 20)) ;; wait for haircut to finish
(println haircuts "haircuts!" (- customer-num haircuts) "customers turned away!!")
(recur (inc haircuts)))
haircuts))))
(defn operate-shop [open-time]
((let [[_ opening] (alts!! [(timeout open-time)
(go-loop [customer 0]
(<! (timeout (+ 10 (rand-int 20)))) ;; wait for random arrival of customers
(>! barber-shop customer)
(recur (+ customer 1)))])]
(close! barber-shop)
(close! opening)
)))
(cut-hair)
(operate-shop 2000)
Without running your code to confirm my suspicions, I see two problems with your implementation.
The first is that the body of operate-shop starts with ((, which you appear to intend as a grouping mechanism. But of course, in Clojure, (f x y) is how you call the function f with arguments x y. So your implementation calls alts!, then calls close!, then calls shutdown-agents - all intended so far - but then calls the result of alts! (which surely is not a function) with two nil arguments. So you should get a ClassCastException once your shop closes. Normally I would recommend just removing the outer parens, but since you're using core.async you should wrap the body in go, as in (go x y z). Is this your real code? If you call alts! outside of a go context, as your snippet suggests, you can only get a runtime error.
The second is that your first go-loop has no termination condition. You treat customer-num as if it were a number, but if the channel is closed, it will be nil: that's how you can tell a channel is closed. Involving it in subtraction should throw some kind of exception. Instead, you should check whether the result is nil, and if so, exit the loop as the shop is closed.
let's say I've got a channel out (chan). I need to take values that are put into the channel and add them. The number of values is undetermined (thus cannot use traditional loop with an end case with (<! out)) and comes from an external IO. I'm using a fixed timeout with alts! but that doesn't seem like the best way to approach the problem. So far, I've got the following (which I got from https://gist.github.com/schaueho/5726a96641693dce3e47)
(go-loop
[[v ch] (alts! [out (timeout 1000)])
acc 0]
(if-not v
(do (close! out)
(deliver p acc))
(do
(>! task-ch (as/progress-tick))
(recur (alts! [out (timeout 1000)]) (+ acc v)))))
The problem I've got is that a timeout of 1000 is sometimes not enough and causes the go-loop to exit prematurely (as it may take more than 1000ms for the IO operation to complete and put the val in the out channel). I do not think increasing the timeout value is such a good idea as it may cause me to wait longer than necessary.
What is the best way to guarantee all reads from the out channel and exit out correctly from the loop?
Update:
Why am I using timeout?
Because the number of values being put in the channel is not fixed; which means, I cannot create an exit case. W/o the exit case, the go-loop will park indefinely waiting ((<! out)) for values to be put in the channel out. If you have a solve without the timeout, that'd be really awesome.
How do i know I've read the last value?
I dont. That's the problem. That's why I'm using timeout and alts!! to exit the go-loop.
What do you want to do w/ the result?
Simple addition for now. However, that's not the important bit.
Update Final:
I figured out a way to get the number of values I'd be dealing with. So I modified my logic to make use of that. I'm still going to use the timeout and alts! to prevent any locking.
(go-loop
[[v _] (alts! [out (timeout 1000)])
i 0
acc 0]
(if (and v (not= n i))
(do
(>! task-ch (as/progress-tick))
(recur (alts! [out (timeout 1000)]) (inc i) (+ acc v)))
(do (close! out)
(deliver p* (if (= n i) acc nil)))))
I think your problem is a bit higher-up in your design, not a core-async specific one:
On one hand, you have an undetermined amount of values coming in a channel — there could be 0, there could be 10, there could be 1,000,000.
On the other hand, you want to read all of them, do some calculation and then return. This is impossible to do — unless there is some other signal that you can use to say "I think I'm done now".
If that signal is the timing of the values, then your approach of using alts! is the correct one, albeit I believe the code can be simplified a bit.
Update: Do you have access to the "upstream" IO? Can you put a sentinel value (e.g. something like ::closed) to the channel when the IO operation finishes?
The 'best' way is to wait for either a special batch ending message from out or for out to be closed by the sender to mark the end of the inputs.
Either way, the solution rests with the sender communicating something about the inputs.
I am launching a few hundreds concurrent http-kit.client/get requests provided with a callback to write results to a single file.
What would be a good way to deal with thread-safety? Using chanand <!! from core.asyc?
Here's the code I would consider :
(defn launch-async [channel url]
(http/get url {:timeout 5000
:user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:10.0) Gecko/20100101 Firefox/10.0"}
(fn [{:keys [status headers body error]}]
(if error
(put! channel (json/generate-string {:url url :headers headers :status status}))
(put! channel (json/generate-string body))))))
(defn process-async [channel func]
(when-let [response (<!! channel)]
(func response)))
(defn http-gets-async [func urls]
(let [channel (chan)]
(doall (map #(launch-async channel %) urls))
(process-async channel func)))
Thanks for your insights.
Since you are already using core.async in your example, I thought I'd point out a few issues and how you can address them. The other answer mentions using a more basic approach, and I agree wholeheartedly that a simpler approach is just fine. However, with channels, you have a simple way of consuming the data which does not involve mapping over a vector, which will also grow large over time if you have many responses. Consider the following issues and how we can fix them:
(1) Your current version will crash if your url list has more than 1024 elements. There's an internal buffer for puts and takes that are asynchronous (i.e., put! and take! don't block but always return immediately), and the limit is 1024. This is in place to prevent unbounded asynchronous usage of the channel. To see for yourself, call (http-gets-async println (repeat 1025 "http://blah-blah-asdf-fakedomain.com")).
What you want to do is to only put something on the channel when there's room to do so. This is called back-pressure. Taking a page from the excellent wiki on go block best practices, one clever way to do this from your http-kit callback is to use the put! callback option to launch your next http get; this will only happen when the put! immediately succeeds, so you will never have a situation where you can go beyond the channel's buffer:
(defn launch-async
[channel [url & urls]]
(when url
(http/get url {:timeout 5000
:user-agent "Mozilla"}
(fn [{:keys [status headers body error]}]
(let [put-on-chan (if error
(json/generate-string {:url url :headers headers :status status})
(json/generate-string body))]
(put! channel put-on-chan (fn [_] (launch-async channel urls))))))))
(2) Next, you seem to be only processing one response. Instead, use a go-loop:
(defn process-async
[channel func]
(go-loop []
(when-let [response (<! channel)]
(func response)
(recur))))
(3) Here's your http-gets-async function. I see no harm in adding a buffer here, as it should help you fire off a nice burst of requests at the beginning:
(defn http-gets-async
[func urls]
(let [channel (chan 1000)]
(launch-async channel urls)
(process-async channel func)))
Now, you have the ability to process an infinite number of urls, with back-pressure. To test this, define a counter, and then make your processing function increment this counter to see your progress. Using a localhost URL that is easy to bang on (wouldn't recommend firing off hundreds of thousands of requests to, say, google, etc.):
(def responses (atom 0))
(http-gets-async (fn [_] (swap! responses inc))
(repeat 1000000 "http://localhost:8000"))
As this is all asynchronous, your function will return immediately and you can look at #responses grow.
One other interesting thing you can do is instead of running your processing function in process-async, you could optionally apply it as a transducer on the channel itself.
(defn process-async
[channel]
(go-loop []
(when-let [_ (<! channel)]
(recur))))
(defn http-gets-async
[func urls]
(let [channel (chan 10000 (map func))] ;; <-- transducer on channel
(launch-async channel urls)
(process-async channel)))
There are many ways to do this, including constructing it so that the channel closes (note that above, it stays open). You have java.util.concurrent primitives to help in this regard if you like, and they are quite easy to use. The possibilities are very numerous.
This is simple enough that I wouldn't use core.async for it. You can do this with an atom storing use a vector of the responses, then have a separate thread reading the contents of atom until it's seen all of the responses. Then, in your http-kit callback, you could just swap! the response into the atom directly.
If you do want to use core.async, I'd recommend a buffered channel to keep from blocking your http-kit thread pool.
If I wrap a function in a future and invoke this from leiningen on the commandline, it adds 1 full minute to the runtime. Can any one tell me why this is? Can you also tell me how to stop this behavior? I'd like to avoid this extra minute.
Example code:
(ns futest.core
(:gen-class))
(defn testme []
(Thread/sleep 2000)
42)
(defn -main
[& args]
(if (= (nth args 0) "a")
;; either run a simple function that takes 2 seconds
(do
(println "no future sleep")
(let [t (testme)]
(println t))
(println "done."))
;; or run a simple function that takes 2 seconds as a future
(do
(println "future sleep")
(let [t (future (testme))]
(println #t))
(println "done.")
;; soo, why does it wait for 1 minute here?
)))
this is because agents uses two threadpools, the first is a fixed threadpool and the second a cached threadpool. cached threadpool terminates running threads that were inactive for a certain duration, the default being 60 seconds. This is the reason you see the 60 seconds delay. Of course, if you manually call shutdown-agents both these threadpools terminate leaving no non-daemon threads that blocks your exit.
As noted in the answer to this question you need to call shutdown-agents at the end of your -main method.
I'm posting this as self-answered Q&A since that question doesn't mention future, so it didn't turn up on my google searches. Sure enough, if I add:
;; soo, why does it wait for 1 minute here?
(shutdown-agents)
)))
the problem goes away.
As I tested, a separate thread is used for each new agent, when I create them.
Could several agents be run in one thread?
My idea is to create 10K+ light-weight agents (like actors in erlang), so is it a challenge for Clojure?
Thanks
This is incorrect. Agents use a thread pool which is the number of core + 2 in size. So on a quad core machine even 10k+ agents will only use 6 worker threads.
With send, that is. With send-off new threads will be started.
Consider using a j.u.c.DelayQueue
Here's a sketch of how it would work,
the (delayed-function is a bit cumbersome here, but it basically constructs an instance of j.u.c.Delayed for submission to the queue.)
(import [java.util.concurrent Delayed DelayQueue TimeUnit])
(defn delayed-function [f]
(let [execute-time (+ 5000 (System/currentTimeMillis))
remaining-delay (fn [t] (.convert t
(- execute-time
(System/currentTimeMillis))
TimeUnit/MILLISECONDS))]
(reify
Delayed (getDelay [_ t] (remaining-delay t))
Runnable (run [_] (f))
Comparable (compareTo [_ _] 0))))
;;; Use java's DelayQueue from clojure.
;;; See http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/DelayQueue.html
(def q (DelayQueue.))
(defn delayed
"put the function f on the queue, it will only execute after the delay
expires"
[f]
(.offer q (delayed-function f)))
(defn start-processing
"starts a thread that endlessly reads from the delay queue and
executes the function found in the queue"
[]
(.start
(Thread.
#(while true
(.run (.take q))))))
user> (start-processing)
user> (delayed #(println "Hello"))
; 5 seconds passes
Hello
the at function of the at-at library that was developed to support the (in my opinion fantastic) Overtone music synthesizer provides a nice clean interfase for running functions at a specific point in time.
(use 'overtone.at-at)
(def my-pool (mk-pool))
(after 1000 #(println "hello from the past!") my-pool)