I want to find the string in sentence using clojure in riemann. I have wrote the code using re-matches but I am facing some error while executing it.
(smap
(fn [events]
(let [count-of-failures (count (re-matches #("POST*" (:Webservice %)) events))]
This is the error I got
java.lang.ClassCastException: riemann.config$eval96$fn__97$fn__98 cannot be cast to java.util.regex.Pattern
at clojure.core$re_matcher.invoke(core.clj:4460)
at clojure.core$re_matches.invoke(core.clj:4497)
at riemann.config$eval96$fn__97.invoke(riemann_v1.config:31)
at riemann.streams$smap$stream__3695.invoke(streams.clj:161)
at riemann.streams$fixed_time_window_fn$stream__3946$fn__3979.invoke(streams.clj:381)
at riemann.streams$fixed_time_window_fn$stream__3946.invoke(streams.clj:381)
at riemann.config$eval96$stream__145$fn__150.invoke(riemann_v1.config:25)
at riemann.config$eval96$stream__145.invoke(riemann_v1.config:25)
at riemann.core$stream_BANG_$fn__5678.invoke(core.clj:19)
at riemann.core$stream_BANG_.invoke(core.clj:18)
at riemann.transport$handle.invoke(transport.clj:159)
at riemann.transport.tcp$tcp_handler.invoke(tcp.clj:93)
at riemann.transport.tcp$gen_tcp_handler$fn__5904.invoke(tcp.clj:65)
at riemann.transport.tcp.proxy$io.netty.channel.ChannelInboundHandlerAdapter$ff19274a.channelRead(Unknown Source)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32)
at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
I often use a pattern like this for such things
(fixed-time-window 10
(smap
(fn [events]
(let [count-of-failures
(count (filter #(re-find #"POST*"
(:Webservice %))
events))]
(assoc (first events)
:service "failure-counts"
:metric count-of-failures))))
index
slack
prn
divine-intervention))
the top call to fixed-time-window takes in events one a time such as:
{:service "foo"} {:service "foo"} {:service "foo"}
and bundles them into groups by time:
[{:service "foo"} {:service "foo"}] [{:service "foo"}]
there is one child stream of that bundler which takes each group and counts things then passes them to it's child streams single events that look like this:
{:service "foo" :metric 2} ;; there where two in the first 10 second window
{:service "foo" :metris 1} ;; there was one in the second 10 second window.
the streams coming out of that are then passed to each of the child streams. In my example I called the example child streams:
index , a build in function that puts things into the index for the dash etc.
slack which is a function who's definition is not given here that forwads events to a slack channel. It's organizaion specific (channel names, tokesn etc) so there is no point in posting it.
prn: prints to the console for debugging
divine-intervention: another function, who's implementation is left as an exercise, which miraculously flips the appropriate bits in all systems to make the problem go away ;-)
Related
I'm learning core.async and have written a simple producer consumer code:
(ns webcrawler.parallel
(:require [clojure.core.async :as async
:refer [>! <! >!! <!! go chan buffer close! thread alts! alts!! timeout]]))
(defn consumer
[in out f]
(go (loop [request (<! in)]
(if (nil? request)
(close! out)
(do (print f)
(let [result (f request)]
(>! out result))
(recur (<! in)))))))
(defn make-consumer [in f]
(let [out (chan)]
(consumer in out f)
out))
(defn process
[f s no-of-consumers]
(let [in (chan (count s))
consumers (repeatedly no-of-consumers #(make-consumer in f))
out (async/merge consumers)]
(map #(>!! in %1) s)
(close! in)
(loop [result (<!! out)
results '()]
(if (nil? result)
results
(recur (<!! out)
(conj results result))))))
This code works fine when I step in through the process function in debugger supplied with Emacs' cider.
(process (partial + 1) '(1 2 3 4) 1)
(5 4 3 2)
However, if I run it by itself (or hit continue in the debugger) I get an empty result.
(process (partial + 1) '(1 2 3 4) 1)
()
My guess is that in the second case for some reason producer doesn't wait for consumers before exiting, but I'm not sure why. Thanks for help!
The problem is that your call to map is lazy, and will not run until something asks for the results. Nothing does this in your code.
There are 2 solutions:
(1) Use the eager function mapv:
(mapv #(>!! in %1) items)
(2) Use the doseq, which is intended for side-effecting operations (like putting values on a channel):
(doseq [item items]
(>!! in item))
Both will work and produce output:
(process (partial + 1) [1 2 3 4] 1) => (5 4 3 2)
P.S. You have a debug statement in (defn consumer ...)
(print f)
that produces a lot of noise in the output:
<#clojure.core$partial$fn__5561 #object[clojure.core$partial$fn__5561 0x31cced7
"clojure.core$partial$fn__5561#31cced7"]>
That is repeated 5 times back to back. You probably want to avoid that, as printing function "refs" is pretty useless to a human reader.
Also, debug printouts in general should normally use println so you can see where each one begins and ends.
I'm going to take a safe stab that this is being caused by the lazy behavior of map, and this line that's carrying out side effects:
(map #(>!! in %1) s)
Because you never explicitly use the results, it never runs. Change it to use mapv, which is strict, or more correctly, use doseq. Never use map to run side effects. It's meant to lazily transform a list, and abuse of it leads to behaviour like this.
So why is it working while debugging? I'm going to guess because the debugger forces evaluation as part of its operation, which is masking the problem.
As you can read from docstring map returns a lazy sequence. And I think the best way is to use dorun. Here is an example from clojuredocs:
;;map a function which makes database calls over a vector of values
user=> (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"])
JdbcSQLException The object is already closed [90007-170] org.h2.message.DbE
xception.getJdbcSQLException (DbException.java:329)
;;database connection was closed before we got a chance to do our transactions
;;lets wrap it in dorun
user=> (dorun (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"]))
DEBUG :db insert into person values name = 'Fred'
DEBUG :db insert into person values name = 'Ethel'
DEBUG :db insert into person values name = 'Lucy'
DEBUG :db insert into person values name = 'Ricardo'
nil
I'm currently working on a route planning robot in Clojure. The robot takes in a parcel which has a set of stops contained within it and then this robot is passed into a function which calculates the shortest route.
(defn journey [start end]
(alg/pprint-path (alg/shortest-path all-edges {:start-node start, :end-node end, :cost-attr :weight})))
(defn fullpath [& stops]
(doall (map (fn [a b] (journey a b)) stops (rest stops) )))
The two functions above calculate the shortest route between stops and print it out.
;;passed into robot
(defrecord Parcel [start
end
home])
;;passed into robotroute to plan journey of robot
(defrecord Robot [stops])
;;computes the path
(defn robotroute [robot]
(def stops (:stops robot))
(fullpath stops))
(def task1parcel (Parcel. :main-office :r131 :main-office))
(def task1robot (Robot. task1parcel))
(def task1 (robotroute task1robot))
(task1)
Above is my code for creating the robot and parcel. Robotroute is the function I am passing the robot into which is meant to strip out the stops and plan the route using fullpath.
All of the functions can be defined etc. However when trying to run task 1 I get the following error.
ClassCastException clojure.lang.LazySeq cannot be cast to clojure.lang.IFn funcprog2.core/eval13519 (form-init1291893531842170235.clj:1)
Can anyone assist with fixing this error?
Also moving forward I want a robot to hold multiple parcels so it can do more than one delivery in a row, what would be the best plan to move forward with this?
(defn fullpath [& stops]
(doall (map (fn [a b] (journey a b)) stops (rest stops) )))
function fullpath return a lazyseq . (task1) will evaluate this lazeseq again Maybe you could just get your result by putting "task1" in repl.
For example (def list '(1 2 3)) list is equal to '(1 2 3). (list) will end up getting "ClassCastException clojure.lang.PersistentList cannot be cast to clojure.lang.IFn"
Another point is that try
(defn robotroute [robot]
(let [stops (:stops robot)] (fullpath stops)))
in clojure a suggestion is that : use let rather than def to bind variable when defining a function because def means global
(def stops inside roboroute - shouldn't that be a let ?
(robotroute task1robot) returns whatever fullpath returns, which is (doall (map ... - it's a sequence - and sequences are not callable/not a function (as the error suggests). Therefor invoking it like (task1) fails.
I have this small game world state, something like the following:
(defn odds [percentage]
(< (rand-int 100) percentage))
(defn world []
{:entities []})
(defn make-bird []
{:pos [(rand-int 100) (rand-int 100)]
:age 0
:dir (vec/dir (rand (. Math PI)))})
(defn generate-entities [entities]
(if (odds 10)
(conj entities (make-bird))
entities))
(defn update-entity [entity]
(-> entity
(update :pos (partial vec/add (:dir entity)))
(update :age inc)))
(defn update-entities [entities]
(vec (map update-entity entities)))
(defn old? [{age :age}]
(> age 10))
(defn prune-entities [entities]
(vec (filter #(not (old? %)) entities)))
(defn update-world [world]
(-> world
(update :entities generate-entities)
(update :entities update-entities)
(update :entities prune-entities)))
So update-world goes through three steps. First there's a 1/10 chance of generating a new bird entity, which flies in a random direction. Then it updates all birds, updating their position and incrementing their age. Then it prunes all old birds.
I use this same technique for generating particles systems. You can do fun stuff like (iterate update-world (world)) to get a lazy list of world states which you can consume at whatever frame rate you want.
However, I now have a game world with autonomous entities which roam around and do stuff, kind of like the birds. But I want to get a textual representation of what happened when evaluating update-world. For example, update-world would ideally return a tuple of the new world state and a vector of strings - ["A bird was born at [12, 8].", "A bird died of old age at [1, 2]."].
But then I really can't use (iterate update-world (world)) anymore. I can't really see how to do this.
Is this something you'd use with-out-string for?
If you want to enhance only your top-level function (update-world) in your case you can just create a wrapper function that you can use in iterate. A simple example:
(defn increment [n]
(inc n))
(defn logging-increment [[_ n]]
(let [new-n (increment n)]
[(format "Old: %s New: %s" n new-n) new-n]))
(take 3 (iterate logging-increment [nil 0]))
;; => ([nil 0] ["Old: 0 New: 1" 1] ["Old: 1 New: 2" 2])
In case you want to do it while collecting data at multiple level and you don't want to modify the signatures of your existing functions (e.g. you want to use it only for debugging), then using dynamic scope seems like a reasonable option.
Alternatively you can consider using some tracing tools, like clojure/tools.trace. You could turn on and off logging of your function calls by simply changing defn to deftrace or using trace-ns or trace-vars.
There are two potential issues with using with-out-str
It returns a string, not a vector. If you need to use a vector, you'll need to use something else.
Only the string is returned. If you are using with-out-str to wrap a side-effect (e.g., swap!), this might be fine.
For debugging purposes, I usually just use println. You can use with-out if you want control over where the output goes. You could even implement a custom stream that collects the output into a vector of strings if you wanted. You could get similar results with a dynamically bound vector that you accumulate (via set!) the output string (or wrap the vector in an atom and use swap!).
If the accumulated vector is part of the computation per se, and you want to remain pure, you might consider using a monad.
What about using clojure.data/diff to generate the string representation of changes? You could do something like this:
(defn update-world [[world mutations]]
(let [new-world (-> world
(update :entities generate-entities)
(update :entities update-entities)
(update :entities prune-entities))]
[new-world (mutations (clojure.data/diff world new-world))]))
Then you could do something like (iterate update-world [(world) []]) to get the ball rolling.
I have the following function which gets a map with service name and threshold. It checks if the service crossed a defined threshold and then calls multiple downstream children on the event.
(defn tc
[s & children]
(where
(and (service (:service_name s)) (not (expired? event)))
(by [:host :service]
(where (> metric (:threshold s)
(with :state "critical"
(apply sdo children)))))))
I would like to build a stream dynamically using a vector of maps:
(def services [{:service "cpu/usage" :threshold 90}
{:service "memory/usage" :threshold 90}])
When trying to run it in a stream i'm getting the following warning:
(streams
(doseq [s services] (tc s prn)))
WARN [2015-01-05 14:27:07,187] Thread-15 - riemann.core - instrumentation service caught
java.lang.NullPointerException
at riemann.core$stream_BANG_$fn__11140.invoke(core.clj:19)
at riemann.core$stream_BANG_.invoke(core.clj:18)
at riemann.core$instrumentation_service$measure__11149.invoke(core.clj:57)
at riemann.service.ThreadService$thread_service_runner__8782$fn__8783.invoke(service.clj:66)
at riemann.service.ThreadService$thread_service_runner__8782.invoke(service.clj:65)
at clojure.lang.AFn.run(AFn.java:22)
at java.lang.Thread.run(Thread.java:701)
It works, if i run the streams function inside the doseq.
This one works and gives the following output:
(doseq [s services]
(streams (tc s prn)))
#riemann.codec.Event{:host "testhost", :service "memory/usage", :state "critical", :description nil, :metric 91.0, :tags nil, :time 1420460856, :ttl 60.0}
It seems to blow up if your events don't have all the required fields, here's a sample from a similar project where I build an event from a sequence of events (reducing) It's not exactly what you are doing though I'm generating events in the same way:
{:service (:service (first events))
:metric (->> events count)
:host "All-counts"
:state "OK"
:time (:time (last events))
:ttl default-interval}
I got NPE specifically when time was missing. If you can't inherit it form somewhere, just make it up (use now for instance) without a reasonable value here, event expiration will not work and you'll run out of RAM
In the following example I can see, that published messages arrive to the subscribed channel, although, they are published before subscription is made.
(let [in (async/chan)
out (async/chan)
pub (async/pub in :key)]
(async/go
(>! in {:id 1 :key :k1})
(>! in {:id 2 :key :k1})
(>! in {:id 3 :key :k1}))
(async/sub pub :k1 out)
(async/go-loop []
(println (<! out))
(recur)))
Is this expected behavior? As far as I can see in the documentation, it clearly states:
Items received when there are no matching subs get dropped.
I get same results in both Clojure and ClojureScript.
Added:
with mult/tap I see similar behavior
You don't know that the messages are published before the subscription is made. Because go is asynchronous, it's very possible that the subscription happens before the first message has been put into the channel. It's a race condition, really.
Try putting a (Thread/sleep [some value]) before the suscription and see what happens.