I started to work with Apache Storm recently. I use Storm with Clojure Storm DSL and Leiningen.
There is a very cool tool for storm topology management: Storm Flux.
My question is: How can I use flux when I am coding in storm with clojure?
I found a solution:
(ns your.namespace.boltname
(:use
[org.apache.storm clojure config])
(:gen-class :implements [org.apache.storm.topology.IRichBolt]))
(defbolt my-bolt
["data"]
[tuple collector]
(emit-bolt! collector [(f (.getString tuple 0))] :anchor tuple)
(ack! collector tuple))
(defn -execute [this tuple]
(.execute my-bolt tuple))
(defn -prepare [this conf context collector]
(.prepare my-bolt conf context collector))
(defn -cleanup [this]
(.cleanup my-bolt))
(defn -declareOutputFields [this output]
(.declareOutputFields my-bolt output))
(defn -getComponentConfiguration [this]
(.getComponentConfiguration my-bolt))
don't forget to add :aot :all to your project.clj.
And in your flux topology.yaml something like:
...
bolts:
- id: "myBolt"
className: "your.namespace.boltname"
parallelism: 1
...
That's all :)
Related
I am trying to get the namespace associated with the code actually running.
I created a template project using 'lein new app test':
(ns test.core
(:gen-class))
(defn -main
[& args]
(println (ns-name *ns*)))
When using the repl, *ns* evaluates to test.core:
echo "(-main)" | lein repl
nREPL server started on port 37435 on host 127.0.0.1 - nrepl://127.0.0.1:37435
REPL-y 0.4.4, nREPL 0.7.0
Clojure 1.10.1
OpenJDK 64-Bit Server VM 1.8.0_265-b01
Docs: (doc function-name-here)
(find-doc "part-of-name-here")
Source: (source function-name-here)
Javadoc: (javadoc java-object-or-class-here)
Exit: Control+D or (exit) or (quit)
Results: Stored in vars *1, *2, *3, an exception in *e
test.core=> (-main)
test.core
nil
test.core=> Bye for now!
But when using lein run, I get user namespace:
lein run
user
Why such a behavior ?
How can I get the namespace of a piece of code like in the repl ?
A few things to note:
*ns* is bound to the namespace wherein the expression is evaluated
user is the default namespace
when you call (test.core/-main) from the user namespace, *ns* is bound to user, not test.core.
If you would like to print the namespace in which -main was defined, you can use a top-level expression:
(ns test.core)
(def current-ns *ns*)
(defn -main [& args]
(println (ns-name current-ns)))
or use var metadata:
(ns test.core)
(defn -main [& args]
(println (ns-name (:ns (meta #'-main)))))
I'm using compojure for a basic web app, I have this code in core.clj:
(defroutes routes
(GET "/" [] (layout/application "Home" (contents/index)))
(route/resources "/"))
(def application (handler/site routes))
(defn -main []
(let [port (Integer/parseInt (or (System/getenv "PORT") "8090"))]
(jetty/run-jetty application {:port port :join? false})))
When I access the 0.0.0.0:8090 everything is loading ok, but I keep seeing this error:
java.lang.NullPointerException: Response map is nil
at ring.util.servlet$update_servlet_response.invokeStatic(servlet.clj:100)
at ring.util.servlet$update_servlet_response.invoke(servlet.clj:91)
at ring.util.servlet$update_servlet_response.invokeStatic(servlet.clj:95)
at ring.util.servlet$update_servlet_response.invoke(servlet.clj:91)
at ring.adapter.jetty$proxy_handler$fn__337.invoke(jetty.clj:27)
at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:503)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base/java.lang.Thread.run(Thread.java:844)
Any idea what's going on?
#Svante is almost certainly correct. An easy way to verify is to use the spyx function from the Tupelo library:
(ns demo.core
(:use tupelo.core))
(defroutes routes
(GET "/" [] (spyx (layout/application "Home" (contents/index))))
(route/resources "/"))
which will print something like:
(layout/application "Home" (contents/index))) => nil
when run. spyx ("spy explicit") prints the expression you give it, an arrow, and the expression value. spy, spyx, spy-pretty, etc also return the value printed (unlike println which always returns nil) so you can insert a spy printout anywhere without disrupting the processing chain. Thus, you don't need to write something like:
(defroutes routes
(GET "/" [] (let [tmp-1 (layout/application "Home" (contents/index))]
(println "layout/application result => " tmp-1)
tmp-1)))
(route/resources "/"))
in order to get a debug message printed. In order to spy & friends, add this to the :dependencies in your project.clj:
[tupelo "0.9.138"]
Update
Hmmmm.... Not sure what could be the problem. I made a simple demo app from lein new compojure demo-compojure with the following:
(ns demo-compojure.handler
(:use tupelo.core)
(:require [compojure.core :refer :all]
[compojure.route :as route]
[ring.middleware.defaults :refer [wrap-defaults site-defaults]]))
(defn index []
(spy :index--result "Hello World"))
(defroutes app-routes
(GET "/" [] (spyx (index)))
(route/not-found "Not Found"))
(def app
(wrap-defaults app-routes site-defaults))
and results:
~/expr/demo-compojure > lein ring server
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Started server on port 3000
:index--result => "Hello World"
(index) => "Hello World"
So that is working. Also, int? is a basic Clojure function, so that is puzzling. Perhaps make a clean demo project like the above and take it from there?
I am learning clojure, and I've hit a snag while trying to refactor my web app to make it more functional and less reliant on global state.
The way you make a simple auto-reloading server with ring framework is like this:
(defn handler [req]
{:status 200
:body "<h1>Hello World</h1>"
:headers {}})
(defn -main []
(jetty/run-jetty (wrap-reload #'handler)
{:port 12345}))
(source: https://practicalli.github.io/clojure-webapps/middleware-in-ring/wrap-reload.html)
So handler is a global function. It is given as a var reference to wrap-reload. On each request, wrap-reload will reload the entire namespace, and then re-resolve handler reference to a potentially different function. Or at least this is my understanding.
This is all good in this simple example. The trouble starts when I replace this hello world handler with my actual handler, which has all sorts of state bound into it (eg. database connection). This state is initialized once inside -main, and then injected into the routing stack. Something like this:
(defn init-state [args dev]
(let
[db (nth args 1 "jdbc:postgresql://localhost/mydb")
routes (make-routes dev)
app (make-app-stack routes db)
{:port (Integer. (nth args 0 3000))
:route routes
:dev dev
:db db
:app app})
(defn start [state]
(println "Running in " (if (:dev state) "DEVELOPMENT" "PRODUCTION") " mode")
(model/create-tables (:db state))
(jetty/run-jetty (:app state) {:port (:port state)}))
(defn -main [& args]
(start (init-state args false)))
make-routes generates the router based on compojure library, and make-app-stack then wraps this router into a bunch of middlewares, which will inject global state (like DB string) for the use by handlers.
So how do I add wrap-reload to this setup? #'app macro won't work on local let "variable" (or whatever it's called). Do I need to expose my app as a global variable? Or can I re-generate the entire closure on each request.
My instincts tell me to avoid having global initialization code in the module body and keep all code in the main, but maybe I am overthinking. Should I just type my init-state code as globals and call it a day, or is there a better way?
I came up with a solution I can live with.
(ns webdev.core
(:require [ring.middleware.reload :as ring-reload]))
; Defeat private defn
(def reloader #'ring-reload/reloader)
; Other stuff...
(defn load-settings [args dev]
{:port (Integer. (nth args 0 3000))
:db (nth args 1 "jdbc:postgresql://localhost/webdev")
:dev dev})
(defn make-reloading-app [settings]
(let [reload! (reloader ["src"] true)]
(fn [request]
(reload!)
((make-app settings) request))))
(defn start [settings]
(let
[app
(if (:dev settings)
(make-reloading-app settings)
(make-app settings))]
(println "Running in " (if (:dev settings) "DEVELOPMENT" "PRODUCTION") " mode")
(model/create-tables (:db settings))
(jetty/run-jetty app {:port (:port settings)})))
(defn -main [& args]
(start (load-settings args false)))
Full code is available here: https://github.com/panta82/clojure-webdev/blob/master/src/webdev/core.clj
Instead of using wrap-reload directly, I use the underlying private reload function. I have to recreate the routing stack in every request, but it seems to work fine in dev. No global state needed :)
Is it possible to grab the project information within the clojure repl?
For example, if there was a project defined:
(defproject blahproject "0.1.2" ....)
When running a repl in the project directory, is there a function like this?
> (project-version)
;=> 0.1.2
While you can parse project.clj yourself, this may be annoying. It's also a lot of work. Instead, you can just do:
(System/getProperty "projectname.version")
Leiningen project files are just Clojure data :)
(-> "/path/to/project.clj" slurp read-string (nth 2))
I use environ (https://github.com/weavejester/environ) which sucks in settings from a number of sources, including system properties. The project version appears as :<project-name>-version:
foobar.repl=> (require '[environ.core :refer [env]])
nil
foobar.repl=> (:foobar-version env)
"0.1.0-SNAPSHOT"
Add the below code to the end of project.clj:
(def project (assoc-in project [:repl-options :init]
`(~'def ~'project-version ~(project :version))))
Now you will have a var called project-version in the initial namespace for the repl.
As described in this discussion.
(ns myproject.example
(:require [clojure.java.io :as io])
(:import java.util.Properties))
(defn get-version [dep]
(let [path (str "META-INF/maven/" (or (namespace dep) (name dep))
"/" (name dep) "/pom.properties")
props (io/resource path)]
(when props
(with-open [stream (io/input-stream props)]
(let [props (doto (Properties.) (.load stream))]
(.getProperty props "version"))))))
(get-version 'myproject) ; => 0.1.0
(get-version 'org.clojure/clojure) ; => 1.3.0
As vemv said, Leiningen project files are just Clojure data. So, it's easy to access your project as an ordinary hash-map:
(->> "project.clj"
slurp
read-string
(drop 2)
(cons :version)
(apply hash-map)
(def project))
If you need this variable only in your repl, you can add it to repl-options to your project.clj:
(defproject yourproject "0.1.0"
:description ""
:url ""
:dependencies [ [org.clojure/clojure "1.4.0"]]
:repl-options { :init (->> "project.clj"
slurp
read-string
(drop 2)
(cons :version)
(apply hash-map)
(def project))})
Now, you have project variable in your repl. So, to access the version of your project you can simply type (:version project).
Of course, you can simply use native Leiningen code to parse you project file:
(defproject yourproject "0.1.0"
:description ""
:url ""
:dependencies [ [org.clojure/clojure "1.4.0"]
[leiningen-core "2.1.3"]]
:repl-options { :init (do (require 'leiningen.core.project)
(def project
(leiningen.core.project/read)))})
But, if you need only the version of your project and nothing more, then it's best to use Ankur's solution.
For a more fully-featured approach, you might want to take a look at the configleaf plugin for Leiningen (https://github.com/davidsantiago/configleaf). It will make the project map, with active profiles merged in, available to project code in a namespace of your choosing.
In case you need to do this from clojurescript you could create a macro (from another clj file) and call it from the cljs code :
;;ex: macro.clj
(defmacro get-project-version []
(System/getProperty "penelope.version"))
;;my_logic_code.cljs
(ns my-logic-code
(:require-macros [macros :as m]))
(def project-version (m/get-project-version))
I'm trying to use netty via clojure. I'm able to startup the server, however, it fails to initialize an accepted socket. Below are the error message and code respectively. Does anyone know what is/or could be wrong? I believe the issue is with (Channels/pipeline (server-handler)) Thanks.
Error Message
#<NioServerSocketChannel [id: 0x01c888d9, /0.0.0.0:843]>
Jun 6, 2012 12:15:35 PM org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink
WARNING: Failed to initialize an accepted socket.
java.lang.IllegalArgumentException: No matching method found: pipeline
project.clj
(defproject protocol "1.0.0-SNAPSHOT"
:description "Upload Protocol Server"
:dependencies [
[org.clojure/clojure "1.2.1"]
[io.netty/netty "3.4.5.Final"]])
core.clj
(ns protocol.core
(:import (java.net InetSocketAddress)
(java.util.concurrent Executors)
(org.jboss.netty.bootstrap ServerBootstrap)
(org.jboss.netty.channel Channels ChannelPipelineFactory SimpleChannelHandler)
(org.jboss.netty.channel.socket.nio NioServerSocketChannelFactory)
(org.jboss.netty.buffer ChannelBuffers)))
(def policy
"<content>Test</content>")
(defn server-handler
"Returns netty handler."
[]
(proxy [SimpleChannelHandler] []
(messageReceived [ctx e]
(let [ch (.getChannel e)]
(.write ch policy)
(.close ch)))
(channelConnected [ctx e]
(let [ch (.getChannel e)]
(.write ch policy)
(.close ch)))
(exceptionCaught [ctx e]
(let [ex (.getCause e)]
(println "Exception" ex)
(-> e .getChannel .close)))))
(defn setup-pipeline
"Returns channel pipeline."
[]
(proxy [ChannelPipelineFactory] []
(getPipeline []
(Channels/pipeline (server-handler)))))
(defn startup
"Starts netty server."
[port]
(let [channel-factory (NioServerSocketChannelFactory. (Executors/newCachedThreadPool) (Executors/newCachedThreadPool))
bootstrap (ServerBootstrap. channel-factory)]
(.setPipelineFactory bootstrap (setup-pipeline))
(.setOption bootstrap "child.tcpNoDelay" true)
(.setOption bootstrap "child.keepAlive" true)
(.bind bootstrap (InetSocketAddress. port))))
There are three problems with your code
Java interop with vararg Channels.channel() method.
you can make a vector of channel handlers and wrap it with (into-array ChannelHandler ..)
You can not write String objects directly to a Netty Channel.
you have to write the string to a ChannelBuffer first and write that buffer or use a StringCodecHandler.
Writing to Netty channel is asynchronus, so you can not close it immediately.
you have to register a future listener and close the channel when its done.
Here is the working code.
(ns clj-netty.core
(:import (java.net InetSocketAddress)
(java.util.concurrent Executors)
(org.jboss.netty.bootstrap ServerBootstrap)
(org.jboss.netty.buffer ChannelBuffers)
(org.jboss.netty.channel Channels ChannelFutureListener ChannelHandler ChannelPipelineFactory SimpleChannelHandler)
(org.jboss.netty.channel.socket.nio NioServerSocketChannelFactory)
(org.jboss.netty.buffer ChannelBuffers)))
(def policy
(ChannelBuffers/copiedBuffer
(.getBytes "<content>Test</content>")))
(defn server-handler
"Returns netty handler."
[]
(proxy [SimpleChannelHandler] []
(messageReceived [ctx e]
(let [ch (.getChannel e)]
(.addListener
(.write ch policy)
(ChannelFutureListener/CLOSE))))
(channelConnected [ctx e]
(let [ch (.getChannel e)]
(.addListener
(.write ch policy)
(ChannelFutureListener/CLOSE))))
(exceptionCaught [ctx e]
(let [ex (.getCause e)]
(println "Exception" ex)
(-> e .getChannel .close)))))
(defn setup-pipeline
"Returns channel pipeline."
[]
(proxy [ChannelPipelineFactory] []
(getPipeline []
(let [handler (server-handler)]
(Channels/pipeline (into-array ChannelHandler [handler]))))))
(defn startup
"Starts netty server."
[port]
(let [channel-factory (NioServerSocketChannelFactory. (Executors/newCachedThreadPool) (Executors/newCachedThreadPool))
bootstrap (ServerBootstrap. channel-factory)]
(.setPipelineFactory bootstrap (setup-pipeline))
(.setOption bootstrap "child.tcpNoDelay" true)
(.setOption bootstrap "child.keepAlive" true)
(.bind bootstrap (InetSocketAddress. port))))
Have a look at Aleph (also uses Netty) which can used to build clients and servers in many different protocols with nice Clojure API.
In 2021, the easiest way to build Clojure services on top of Netty would be to adopt Donkey which provides good Clojure interop with Vert.x which uses Netty as the backend. Donkey is relatively new and doesn't yet have a lot of documentation so check out this blog which covers a relevant open source project (source available) in terms of architecture, design, coding, and testing under load.