I am trying usgin library clj-kafka.
Here my code
(use [clj-kafka.producer]
[clj-kafka.zk]
[clj-kafka.consumer.zk]
[clj-kafka.core]))
(brokers {"zookeeper.connect" "localhost:2181"})
(def p (producer {"metadata.broker.list" "localhost:9092"
"serializer.class" "kafka.serializer.DefaultEncoder"
"partitioner.class" "kafka.producer.DefaultPartitioner"}))
(send-message p (message "test" (.getBytes "this is my message")))
(def config {"zookeeper.connect" "localhost:2181"
"group.id" "clj-kafka.consumer"
"auto.offset.reset" "smallest"
"auto.commit.enable" "false"})
(with-resource [c (consumer config)]
shutdown
(take 2 (messages c "test"))) ;; return ()
I start zookepper-server and kafka itself with
bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
The config/zookepper.properties:
dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=0
and config/server.properties:
broker.id=0
listeners=PLAINTEXT://:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
zocket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
The 'problem' is when I execute:
(with-resource [c (consumer config)]
shutdown
(take 2 (messages c "test"))) ;; return empty ()
Any idea here?
Thanks in advance
See this github issue. Seems the documentation isn't great. You have to force the realization of the sequence (which is lazy) with doall. try this:
(with-resource [c (consumer config)]
shutdown
(doall (take 2 (messages c "test"))))
Related
I get this error when I Deploy a Hello World contract with Kadena Chainweaver on VM (https://chainweaver-builds.s3.amazonaws.com/2.2/kadena-chainweaver-vm-2.2.0.1.ova):
Failure: Tx Failed: Keyset failure (keys-any)
This is the contract I'm trying to Deploy(It's the default hello world that exists when you create a chainweaver account):
;;
;; "Hello, world!" smart contract/module
;;
;; To try it out, click "Load into REPL" and type into the repl:
;; (free.hello-world.set-message "universe")
;; (free.hello-world.greet)
;;
;; Check lines 21 and 34 to play with Formal Verification
;;
(namespace "free")
;; Define the module.
(module hello-world MODULE_ADMIN
"A smart contract to greet the world."
; no-op module admin for example purposes.
; in a real contract this could enforce a keyset, or
; tally votes, etc.
(defcap MODULE_ADMIN () true)
(defschema message-schema
#doc "Message schema"
#model [(invariant (!= msg ""))]
msg:string)
(deftable
message:{message-schema})
(defun set-message
(
m:string
)
"Set the message that will be used next"
; uncomment the following to make the model happy!
;(enforce (!= m "") "set-message: must not be empty")
(write message "0" {"msg": m})
)
(defun greet ()
"Do the hello-world dance"
(with-default-read message "0" { "msg": "" } { "msg":= msg }
(format "Hello {}!" [msg])))
)
(create-table message)
(set-message "world")
(greet)
Failure: Tx Failed: Keyset failure (keys-any)
The problem is sovled when I changed the module name from hello-world to something else.
Change this:
;; Define the module.
(module hello-world MODULE_ADMIN
"A smart contract to greet the world."
to this:
;; Define the module.
(module hello-world-YOUR-UNIQUE-NAME MODULE_ADMIN
"A smart contract to greet the world."
When running a janet test suite with the jpm command there is no listing of test cases.
$ jpm test
running test/test.janet ...
Failed: (= "hello" "holla")
Expected: "hello"
Actual: "holla"
Test: this should faild
Tests: 1/3 passed
Duration: 0.000 seconds
What I am looking for is to have an output similar to the one that you see when you run go test -v or pytest -v.
This is possible if one uses testament:
With testament the tests are written as:
(import testament :prefix "" :exit true)
(deftest one-plus-one
(is (= 2 (+ 1 1)) "1 + 1 = 2"))
(deftest two-plus-two
(is (= 5 (+ 2 2)) "2 + 2 = 5"))
(deftest test-should-fail
(is (= "hello" "holla") "should fail"))
Each test case is defined as:
(deftest <testname> (is <assertion>) <note>)
Testament allows defining a function which will be called after each test:
(set-on-result-hook reporter)
(run-tests!)
In this case, reporter can be defined as:
(defn reporter
"pretty report"
[result]
(var symbol "\e[31m✕\e[0m")
(if (result :passed?) (set symbol "\e[32m✓\e[0m"))
(print symbol " " (result :note))
)
The result will be:
$ jpm test
running test/test.janet ...
✓ 1 + 1 = 2
✕ should fail
✕ 2 + 2 = 5
> Failed: two-plus-two
Assertion: 2 + 2 = 5
Expect: 5
Actual: 4
> Failed: test-should-fail
Assertion: should fail
Expect: "hello"
Actual: "holla"
-----------------------------------
3 tests run containing 3 assertions
1 tests passed, 2 tests failed
-----------------------------------
In a Linux terminal the x will be red, and ✓ will be green.
The complete test code:
$ cat test/testam.janet
(import testament :prefix "" :exit true)
(deftest one-plus-one
(is (= 2 (+ 1 1)) "1 + 1 = 2"))
(deftest two-plus-two
(is (= 5 (+ 2 2)) "2 + 2 = 5"))
(deftest test-should-fail
(is (= "hello" "holla") "should fail"))
(defn reporter
"pretty report"
[result]
(var symbol "\e[31m✕\e[0m")
(if (result :passed?) (set symbol "\e[32m✓\e[0m"))
(print symbol " " (result :note))
)
(set-on-result-hook reporter)
(run-tests!)
(I had a previous question here, and assumed I wouldn't have issues getting to core.async).
Given input data such as this:
(require '[clojure.core.async :as a])
(def input-data
[{:itm_na 1 :seq_no 1 :doc_img "this is a very long "}
{:itm_na 1 :seq_no 2 :doc_img "sentence from a mainframe "}
{:itm_na 1 :seq_no 3 :doc_img "system that was built before i was "}
{:itm_na 1 :seq_no 4 :doc_img "born."}
{:itm_na 2 :seq_no 1 :doc_img "this is a another very long "}
{:itm_na 2 :seq_no 2 :doc_img "sentence from the same mainframe "}
{:itm_na 3 :seq_no 1 :doc_img "Ok here we are again. "}
{:itm_na 3 :seq_no 2 :doc_img "The mainframe only had 40 char per field so"}
{:itm_na 3 :seq_no 3 :doc_img "they broke it into multiple rows "}
{:itm_na 3 :seq_no 4 :doc_img "which seems to be common"}
{:itm_na 3 :seq_no 5 :doc_img " for the time. "}
{:itm_na 3 :seq_no 6 :doc_img "thanks for your help."}])
partition-by (as expected) clumps my data into seq's (for later collapsing):
(count (partition-by :itm_na input-data ))
;;=> 3
However, when i try to do this with a core.async pipeline for some reason it
doesn't seem to do the same... How do i get the stateful transducer part of
partition-by actually preserve state when in an async pipeline?
(let
[source-chan (a/to-chan input-data)
target-chan (a/chan 100)
xf (comp (partition-by :itm_na))
]
(a/pipeline 1
target-chan
xf
source-chan)
(count (<!! (a/into [] target-chan))))
;;=>12
This should be 3?
Strangely, when I bind the xf to the channel like below I get my expected result. I'm not sure why a/pipeline behaves differently.
(let [xf (comp (partition-by :itm_na))
ch (a/chan 1 xf)]
(a/onto-chan ch input-data)
(count (<!! (a/into [] ch))))
=>3
From the doc... mentions that stateful bit:
(clojure.repl/doc partition-by)
-------------------------
clojure.core/partition-by
([f] [f coll])
Applies f to each value in coll, splitting it each time f returns a
new value. Returns a lazy seq of partitions. Returns a stateful
transducer when no collection is provided.
This particular case was briefly highlighted by Rich Hickey in his talk: you cannot use pipeline with stateful transducers, basically because of pipeline's parallel nature.
I have a function
(defn f-test [c-width] (format "%-(c-width)s" "boogey"))
This is what happens when I try to evaluate it
(f-test 10)
FormatFlagsConversionMismatchException Conversion = c, Flags = ( java.util.Formatter$FormatSpecifier.failMismatch (Formatter.java:4041)
But this is what I am trying to do
(format "%-10s" "boogey")
"boogey "
how can I substitute in c-width inside the format call?
Try this one:
(defn f-test
[c-width]
(format (str "%-" c-width "s") "boogey"))
I'm trying to found a better way of handle exceptions in clojure, I'm using https://github.com/alaisi/postgres.async which thrown an exeption when fail, but I would prefer return false (because I'm using a validator) or even better something like a Either monad (or something more simple like this http://adambard.com/blog/acceptable-error-handling-in-clojure/ )
1) I'm trying to catch the exception and if exist return false, but the code doesnt works (doesnt return false and thrown the exception).
(try
(dosql [tx (<begin! db)
ucount (<execute! tx ["UPDATE groups SET garticlescounter=garticlescounter + 1 WHERE gid=$1"
groupid])
uartc (<execute! tx ["UPDATE subtopics SET starticlescount=starticlescount + 1 WHERE stid=$1"
sid])
uartins (<insert! tx
{:table "articles"}
{:aurl url :atitle title :asuttopicid sid :acommentcount 0 :alikescount 0 :auid uid})
ok? (<commit! tx)]
ok?)
(catch Exception _ false))
2)would be possible wrap in a way where if works return ok? and if doesnt works return false, or maybe [ok? nil] and [nil error] maybe a macro?
----thanks to swinn I did this
;must receive an optional parameter for error response or
; build a [nil ok] response but just know this works for me...
(defmacro async-try-block! [block]
`(let [chn# (!/chan 1)]
(!/go
(let [response# (try
~block
(catch Throwable e#))]
(if (instance? Throwable response#) (!/put! chn# false) (!/put! chn# response#))))
chn#))
(async-try-block!
(dosql [tx (<begin! db)
ucount (<execute! tx ["UPDATE groups SET garticlescounter=garticlescounter + 1 WHERE gid=$1"
groupid])
uartc (<execute! tx ["UPDATE subtopics SET starticlescount=starticlescount + 1 WHERE stid=$1"
sid])
uartins (<insert! tx
{:table "articles"}
{:aurl url :atitle title :asuttopicid sid :acommentcount 0 :alikescount 0 :auid uid})
ok? (<commit! tx)]
ok?))
I'm not familiar with the postgres.async library, butException is not the root of all Exceptions in the JVM, Throwable is.
Changing your catch to Throwable would be the first change I would suggest, but it seems that (<execute! ...) actually catches the exception for you and returns it, so you need to check the return value using (instance? Throwable)