I am trying to send requests to a server. Each request is referenced by an integer. The server will only respond to requests that come in ascending order - that is, if I send request 7 and then request 6, it will disregard request 6. I am working in a multi-threaded environment where several threads can send requests concurrently. In Java, I solved the problem this way:
synchronized(this){
r = requestId.incrementAndGet();//requestId is an AtomicInteger
socket.sendRequest(r, other_parameters);
}
In Clojure, I thought about defining request-id as an Atom and doing the following:
(send-request socket (swap! request-id inc) other-parameters)
Does that work or is it possible thread 1 increments the atom, but by the time the send-request function sends the request, thread 2 increments the atom again and somehow contacts the server first? What is the best way to avoid such a scenario?
Thank you,
Clojure's equivalent construct to synchronized is locking, which can be used in basically the same way:
(locking some-lock
(let [r (.incrementAndGet requestId)]
(.sendRequest socket r other_parameters)))
Where some-lock is the object you're locking on. I'm not sure what you'd want that to be in the absence of this.
Related
I have created small example which highlight the problem:
(->> (range 0 4)
(mapv (fn [i]
(http/get "http://http-kit.org/"
(fn [res]
(info "first callback")
(let [res2 #(http/get "http://http-kit.org/")]
(info "second callback ")))))))
It's stuck on printing the 4s first callback msg's.
If I change the range for 0..3 it will work, the sync version also works.
Update:
The (info) is a taoensso.timbre logging library
My current hypothesis is that you get into a deadlock by exhausting your thread-pool:
You create a thread per outer http/get
If you create less requests than available threads in the thread pool, there is room to service at least one inner http/get (which will require a new thread)
Or if your first request is completed before you exhaust the thread-pool
Once there are no more threads in the thread-pool, the inner http/get cannot
be serviced
Since the inner request cannot be completed, the outers are stuck forever
You can check the status of the thread-pool http-kit uses peeking http/default-pool. There you can see things like:
#object[java.util.concurrent.ThreadPoolExecutor 0x5a99e5c "java.util.concurrent.ThreadPoolExecutor#5a99e5c[Running, pool size = 8, active threads = 0, queued tasks = 0, completed tasks = 24]"]
when you did not get into the deadlock. Or
#object[java.util.concurrent.ThreadPoolExecutor 0x5a99e5c "java.util.concurrent.ThreadPoolExecutor#5a99e5c[Running, pool size = 8, active threads = 8, queued tasks = 8, completed tasks = 28]"]
when you did.
I have tested this in my machine (shows 8 as (.availableProcessors (Runtime/getRuntime))) and I got the results above. I walked into a deadlock when I run more than 8 requests.
Regards
It looks like the issue caused because the http-kit client thread pool doesn't relieve the thread until callback function is finished. And so it ends up with run out of threads.
So I started to think how to make callback function quicker and came up with this solution:
I have created async wrapper function for http-kit client get which using clojure.core.async/chan in callback to quickly put the result in channel and then wait for it results and execute the heavy callback:
(defn async-http-get
[url opts callback]
(let [channel (chan)]
(http/get url opts #(go (>! channel %)))
(go (callback (<! channel)))
nil))
So now using async-http-get instead of http/get solve the problem for me.
(->> (range 0 4)
(mapv (fn [i]
(async-http-get "http://http-kit.org/"
(fn [res]
(info "first callback")
(let [res2 #(http/get "http://http-kit.org/")]
(info "second callback ")))))))
I have a function which processes a stream of values from a shared channel and looks for a value which satisfies a specific predicate:
(defn find-my-object [my-channel]
(go-loop []
(when-some [value (<! my-channel)]
(if (some-predicate? value)
value
(recur)
Now I want this function to return some 'failure' value after waiting for a timeout, which I have implemented like this:
(alts! [(find-my-object my-channel) (timeout 1000)])
The problem with this is that the go-loop above continues to execute after the timeout. I want find-my-object to drain values from my-channel while the search is ongoing, but I don't want to close my-channel on timeout, since it's used in some other places. What is the idiomatic way to implement this functionality?
Read from the timeout channel inside of find-my-object, not outside of it. You can hardcode that if you like, or make the function take a second channel argument, which is the "please stop reading things" channel. Then you can pass it a timeout channel to get the behavior you want now, or pass it some other kind of channel to control it in a different way.
I have some core.async code with a pipeline of two chans and three nodes :
a producer - function that puts values into chan1 with >!! (it's not in a go-block but the function is called from inside a go-loop)
a filter - another function that's not in a go-block but is called within a go-loop, which pulls items from chan1 (with <!!), does a test and if the test passes pushes them onto chan2 (with >!!)
a consumer - an ordinary loop that pulls n values of chan2 with
This code works as expected when I run it as a simple program. But when I copy and paste it to work within a unit-test, it freezes up.
My test code is roughly
(deftest a-test
(testing "blah"
(is (= (let [c1 (chan)
c2 (chan)
gen (make-generator c1)
filt (make-filter c1 c2)
result (collector c2 10) ]
result)
[0 2 4 6 8 10 12 14 16 18 20]))
))
where the generator creates a sequence of integers counting up from zero and the filter tests for evenness.
As far as I can tell, the filter is able to pull the first value from the c1, but is blocked waiting for a second value. Meanwhile, the generator is blocking while waiting to push its next value into c1.
But this doesn't happen when I run the code in a simple stand-alone program.
So, is there any reason that the unit-test framework might be interfering or causing problems with the threading management that core.async is providing? Is it possible to do unit-testing on async code like this?
I'm concerned that I'm not running the collector in any kind of go-block or go-loop so presumably it might be blocking the main thread. But equally, I presume I have to pull all the data back into the main thread eventually. And if not through that mechanism, how?
While using blocking IO within go-blocks/go-loops isn't the best solution, thread macro may be better fit here. It will execute passed body on separate thread, so you may freely use blocking operations there.
I have two unidirectional core.async channels :
channel out can only put!
channel in can only take!
And since this is ClojureScript the blocking operations are not available. I would like to make one bidirectional (in-out) channel out of those two (in and out).
(def in (async/chan))
(def out (async/chan))
(def in-out (io-chan in out)) ;; io or whatever the solution is
(async/put! in "test")
(async/take! ch (fn [x] (println x))) ;; should print "test"
(async/put! ch) ;; put into ch is equivalent to putting into `out`
I tried something like the following (not working) :
(defn io-chan [in-ch out-ch]
(let [io (chan)]
(go-loop []
(>! out-ch (<! io ))
(>! io (<! in-ch))
(recur))
io))
A schema might help :
out in-out
---------------> (unused)
<--------------- <---------------
in
----------------> ---------------->
<---------------- (unused)
Also, closing the bidirectional channel should close both underlying channels.
Is is possible ?
If I understand your use case right, I believe what you're trying to do is just a one-channel job.
On the other hand, if what you're trying to do is to present a channel-like interface for a composite of several channels (e.g some process takes data from in, processes it, and outputs the result to out), then you could always implement the right protocols (in the case of ClojureScript, cljs.core.async.impl.protocols/ReadPort and cljs.core.async.impl.protocols/WritePort).
I would personnaly not recommend it. Leaving aside the fact that you'd be relying on implementation details, I don't believe core.async channels are intended as encapsulation for processes, only as communication points between them. So in this use case, just pass the input channel to producers and the output channel to consumers.
Your example shows a flow basicly like this:
io ---> out-ch ---> worker ---> in-ch ---> io
^-------------------------------------------*
If we assume that worker reads from in-ch and writes to out-ch then perhaps these two channels are reversed in the example. if worker does the opposite then it's correct. in order to prevent loops it's important that you use non-buffered queues so you don't hear your own messages echoed back to yourself.
as a side note, there is no such thing as unidirectional and bi-directional channels. instead there are buffered and unbufferd channels. If we are talking over a buffered channel then when I have something to say to you, I park until you happen to be listening to the channel, then once you are ready to hear it I put my message into the channel and you receive it. Then to get a response I park until you are ready to send it, and once you are, you put it on the channel and I get it from the channel (all at once). This feels like a bi-directional channel though it's really just that unbuffered channels happen to coordinate this way.
If the channel if buffered then I might get my own message back from the channel, because I would finish putting it on the channel and then be ready to receive the response before you where even ready to receive the original message. If you need to use buffered channels like this then use two of them, one for each direction and they will "feel" like uni-directional channels.
I've written some simple clojure code that accesses the twitter streaming api. My code is essentially the same as the example code described in the twitter-api docs:
(def ^:dynamic *custom-streaming-callback*
(AsyncStreamingCallback. (comp println #(:text %) json/read-json #(str %2))
(comp println response-return-everything)
exception-print))
(defn start-filtering []
(statuses-filter :params {:follow 12345}
:oauth-creds *creds*
:callbacks *custom-streaming-callback*))
I'm following tweets about a specific user and using oauth for authentication (not shown). When I run the start-filtering method and a connection is opened with twitter everything works well for a spell, but if the stream is inactive for a bit (around 30 seconds), i.e. no tweets about this particular user are coming down the pike, the following error occurs:
#<EOFException java.io.EOFException: JSON error (end-of-file)>
I assumed from the twitter docs that when using a streaming connection, twitter keeps the stream open indefinitely. I must be making some incorrect assumptions. I'm currently diving into the clojure twitter-api code to see what's going on, but I thought more eyes would help me figure this out more quickly.
I had the same issue that you have. As you found, the streaming function emits an empty message if no data has been received in the last thirty seconds or so.
Trying to read this as json then causes the EOF exception that you see.
I don't know of any way to prevent these calls. In my case I worked around the issue with a simple conditional that falls back to an empty map when there is no JSON to read.
(if-not (clojure.string/blank? %)
(json/read-str % :key-fn keyword)
{})