I have a clojure application. It accepts a POST request with the following payload.
[
{
"key1": value1,
"key2": value2,
"key3": value3,
"key4": value4
},
{
"key1": value1,
"key2": value2,
"key3": value3,
"key4": value4
}
]
For each element of the array we make a SELECT call to DB using jdbc like so
(defn my-definition [db payload]
(->> payload
(map (fn [p]
(if-let [da (my-db-call-method db)]
(assoc da :key1 (:key1 p))
{:key2 (:key2 p)
:key3 (:key3 p)
})))))
DB call
(defn my-db-call-method [db json-params]
(with-sql-handling "my-db-call-method"
(jdbc/query db [my-query
(to long (:key1 json-params))
(to long (:key2 json-params))
)))
my-query.txt would look something like belowSelect * from my_table where col_name1 = ? and col_name2 =? Now, if 1 DB call takes 100 ms then 100 batch size would take 10 sec. This is too long. Is there a way to batch process the request in Clojure? Please note that code is in working condition but taking a long time due to sequential processing. How can we parallel process this? Can we use any third-party library to achieve this?
Related
I am trying usgin library clj-kafka.
Here my code
(use [clj-kafka.producer]
[clj-kafka.zk]
[clj-kafka.consumer.zk]
[clj-kafka.core]))
(brokers {"zookeeper.connect" "localhost:2181"})
(def p (producer {"metadata.broker.list" "localhost:9092"
"serializer.class" "kafka.serializer.DefaultEncoder"
"partitioner.class" "kafka.producer.DefaultPartitioner"}))
(send-message p (message "test" (.getBytes "this is my message")))
(def config {"zookeeper.connect" "localhost:2181"
"group.id" "clj-kafka.consumer"
"auto.offset.reset" "smallest"
"auto.commit.enable" "false"})
(with-resource [c (consumer config)]
shutdown
(take 2 (messages c "test"))) ;; return ()
I start zookepper-server and kafka itself with
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
The config/zookepper.properties:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
and config/server.properties:
broker.id=0
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
zocket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
The 'problem' is when I execute:
(with-resource [c (consumer config)]
shutdown
(take 2 (messages c "test"))) ;; return empty ()
Any idea here?
Thanks in advance
See this github issue. Seems the documentation isn't great. You have to force the realization of the sequence (which is lazy) with doall. try this:
(with-resource [c (consumer config)]
shutdown
(doall (take 2 (messages c "test"))))
I'm trying to found a better way of handle exceptions in clojure, I'm using https://github.com/alaisi/postgres.async which thrown an exeption when fail, but I would prefer return false (because I'm using a validator) or even better something like a Either monad (or something more simple like this http://adambard.com/blog/acceptable-error-handling-in-clojure/ )
1) I'm trying to catch the exception and if exist return false, but the code doesnt works (doesnt return false and thrown the exception).
(try
(dosql [tx (<begin! db)
ucount (<execute! tx ["UPDATE groups SET garticlescounter=garticlescounter + 1 WHERE gid=$1"
groupid])
uartc (<execute! tx ["UPDATE subtopics SET starticlescount=starticlescount + 1 WHERE stid=$1"
sid])
uartins (<insert! tx
{:table "articles"}
{:aurl url :atitle title :asuttopicid sid :acommentcount 0 :alikescount 0 :auid uid})
ok? (<commit! tx)]
ok?)
(catch Exception _ false))
2)would be possible wrap in a way where if works return ok? and if doesnt works return false, or maybe [ok? nil] and [nil error] maybe a macro?
----thanks to swinn I did this
;must receive an optional parameter for error response or
; build a [nil ok] response but just know this works for me...
(defmacro async-try-block! [block]
`(let [chn# (!/chan 1)]
(!/go
(let [response# (try
~block
(catch Throwable e#))]
(if (instance? Throwable response#) (!/put! chn# false) (!/put! chn# response#))))
chn#))
(async-try-block!
(dosql [tx (<begin! db)
ucount (<execute! tx ["UPDATE groups SET garticlescounter=garticlescounter + 1 WHERE gid=$1"
groupid])
uartc (<execute! tx ["UPDATE subtopics SET starticlescount=starticlescount + 1 WHERE stid=$1"
sid])
uartins (<insert! tx
{:table "articles"}
{:aurl url :atitle title :asuttopicid sid :acommentcount 0 :alikescount 0 :auid uid})
ok? (<commit! tx)]
ok?))
I'm not familiar with the postgres.async library, butException is not the root of all Exceptions in the JVM, Throwable is.
Changing your catch to Throwable would be the first change I would suggest, but it seems that (<execute! ...) actually catches the exception for you and returns it, so you need to check the return value using (instance? Throwable)
So I do my query on my posts for their eid and I get a list:
(map :eid (get-all-posts))
(17592186045421 17592186045438 17592186045440 17592186045540
17592186045545 17592186045550 17592186045588 17592186045590
17592186045592 17592186045594 17592186045597 17592186045608
17592186045616 17592186045721 17592186045866 17592186046045
17592186046047 17592186046075 17592186046077 17592186046079
17592186046081 17592186046083 17592186046085 17592186046088
17592186046149 17592186046158 17592186046170 17592186046174
17592186046292 17592186046352 17592186046362 17592186047146
17592186047211)
Every post can have many :ratings,
(get-all-ratings 17592186047211)
({:rating "positive",
:rid 17592186047652,
:email "vaso#ph"}
{:rating "positive",
:rid 17592186047756,
:email "sova.kua#gmcom"}
{:rating "meh",
:rid 17592186047725,
:email "badger#ton"})
Would like to combine the maps so I can `(count) the number of ratings per post.
(count (get-all-ratings 17592186047211)
3
But trying something like
(count (map get-all-ratings (map :eid (get-all-posts)))
simply returns the number of posts... it's a big list like
(({:rating "plus", :rid 7175 ..}
{:rating "plus" :rid 7374})
({:rating "whatever", :rid 7535})
()
() <some have no ratings
({:rating "green", :rid 7152}))
How can I tally up the number of times :rating shows up in a list of listed maps like above?
>(map :rating (map first (map get-all-ratings (map :eid (get-all-posts)))))
is something I've been playing with, and it returns a list, but as you can probably tell, it just returns one :rating field from each post's expanded ratings map.
Edit:
I noticed just now that something like
({:rating 5}
{:rating 3}
{:rating 7})
just becomes
{:rating 7}
...disregarding other entries in the list. How come?
I've python dictionaries within a list as follows
[{"item1": {"item2": "300", "item3" : "10"}},
{"item2": { "item4": "90", "item5": "400" }},
{"item5": {"item6": "16"}},
{"item3": {"item8": "ava", "item1" : "xxx","item5": "400"}}]
And I want to construct a dictionary as follows
{
"item1" : {
"item2": "300",
"item4": "90",
"item5": "400",
"item6": "16",
"item3" : "10",
"item8": "ava"
}
Traversing method:
1) Starting with item1 => is a dict with two keys. Add it to the new dict.
2) Then take the first key in the first dict and check if there is any dictionary with this key. item2 is again a dict with two keys and hence add those keys to the new dict.
3) Repeat the same for the keys in item2 until nothing is there to traverse down. While traversing if there is already traversed dict found skip that. For eg, in the last dict item3, we have item5 which is already traversed and added to new dict. So we can skip traversing this key.
4) Repeat step 2, for second key in item1 (Has to be repeated depends upon the number of keys in first dict.)
I know this is more complex. Is there any possibility to achieve this?
New to Clojure and wondering how to use it to make other languages I program in easier.
One thing I would like to be able to do is use Clojure for code generation.
For example, given the input from a data file (EDN format) how should I (1) walk this structure or (2) push the data into an existing templating mechanism?
Data below would be for defining simple REST API so that you could generate Clients from it. Generate clients in multiple languages by using different templates.
(:restcall "GetAccountBalance"
{:method "GET" :path "account/balance"}
{:id int})
(:restcall "GetLastTransactions"
{:method "GET" :path "account/transactions"}
{:page int})
resulting code would be something like
public void GetAccountBalance(int id)
{
var input = new { id = id };
callIntoRestLibrary("GET", "account/balance", input);
}
public void GetLastTransactions(int page)
{
var input = new { page = page };
callIntoRestLibrary("GET", "account/transactions", input);
}
Note: my end goal would having these as System.Net.Http.HttpClient calls via C#, but also be able to translate these into JavaScript/Ajax calls also
You have several choices for templating with Clojure. One place to look is the Clojure Toolbox.
Here is an example with clostache, a small library (358 loc) implementation of mustache.
(ns so.core
(:require [clostache.parser :refer (render)]))
(def template "
public void {{restcall}}({{id}} id)
{
var input = new { id = id };
callIntoRestLibrary(\"{{method}}\", \"{{path}}\", input);
}")
(def data
{:restcall "GetAccountBalance"
:method "GET" :path "account/balance" :id "int"})
(print (render template data))
Output:
public void GetAccountBalance(int id)
{
var input = new { id = id };
callIntoRestLibrary("GET", "account/balance", input);
}
To clear up confusion about what it means to read EDN.
(spit "foo.txt" (prn-str data))
Now the file foo.txt contains a textual representation of data, presumably your starting point.
(def data2 (with-open [r (java.io.PushbackReader. (java.io.FileReader. "foo.txt"))]
(clojure.edn/read r)))
(= data data2) ;=> true
So, read doesn't just pull in text, but also parses it into its data representation.