I'm trying to use netty via clojure. I'm able to startup the server, however, it fails to initialize an accepted socket. Below are the error message and code respectively. Does anyone know what is/or could be wrong? I believe the issue is with (Channels/pipeline (server-handler)) Thanks.
Error Message
#<NioServerSocketChannel [id: 0x01c888d9, /0.0.0.0:843]>
Jun 6, 2012 12:15:35 PM org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink
WARNING: Failed to initialize an accepted socket.
java.lang.IllegalArgumentException: No matching method found: pipeline
project.clj
(defproject protocol "1.0.0-SNAPSHOT"
:description "Upload Protocol Server"
:dependencies [
[org.clojure/clojure "1.2.1"]
[io.netty/netty "3.4.5.Final"]])
core.clj
(ns protocol.core
(:import (java.net InetSocketAddress)
(java.util.concurrent Executors)
(org.jboss.netty.bootstrap ServerBootstrap)
(org.jboss.netty.channel Channels ChannelPipelineFactory SimpleChannelHandler)
(org.jboss.netty.channel.socket.nio NioServerSocketChannelFactory)
(org.jboss.netty.buffer ChannelBuffers)))
(def policy
"<content>Test</content>")
(defn server-handler
"Returns netty handler."
[]
(proxy [SimpleChannelHandler] []
(messageReceived [ctx e]
(let [ch (.getChannel e)]
(.write ch policy)
(.close ch)))
(channelConnected [ctx e]
(let [ch (.getChannel e)]
(.write ch policy)
(.close ch)))
(exceptionCaught [ctx e]
(let [ex (.getCause e)]
(println "Exception" ex)
(-> e .getChannel .close)))))
(defn setup-pipeline
"Returns channel pipeline."
[]
(proxy [ChannelPipelineFactory] []
(getPipeline []
(Channels/pipeline (server-handler)))))
(defn startup
"Starts netty server."
[port]
(let [channel-factory (NioServerSocketChannelFactory. (Executors/newCachedThreadPool) (Executors/newCachedThreadPool))
bootstrap (ServerBootstrap. channel-factory)]
(.setPipelineFactory bootstrap (setup-pipeline))
(.setOption bootstrap "child.tcpNoDelay" true)
(.setOption bootstrap "child.keepAlive" true)
(.bind bootstrap (InetSocketAddress. port))))
There are three problems with your code
Java interop with vararg Channels.channel() method.
you can make a vector of channel handlers and wrap it with (into-array ChannelHandler ..)
You can not write String objects directly to a Netty Channel.
you have to write the string to a ChannelBuffer first and write that buffer or use a StringCodecHandler.
Writing to Netty channel is asynchronus, so you can not close it immediately.
you have to register a future listener and close the channel when its done.
Here is the working code.
(ns clj-netty.core
(:import (java.net InetSocketAddress)
(java.util.concurrent Executors)
(org.jboss.netty.bootstrap ServerBootstrap)
(org.jboss.netty.buffer ChannelBuffers)
(org.jboss.netty.channel Channels ChannelFutureListener ChannelHandler ChannelPipelineFactory SimpleChannelHandler)
(org.jboss.netty.channel.socket.nio NioServerSocketChannelFactory)
(org.jboss.netty.buffer ChannelBuffers)))
(def policy
(ChannelBuffers/copiedBuffer
(.getBytes "<content>Test</content>")))
(defn server-handler
"Returns netty handler."
[]
(proxy [SimpleChannelHandler] []
(messageReceived [ctx e]
(let [ch (.getChannel e)]
(.addListener
(.write ch policy)
(ChannelFutureListener/CLOSE))))
(channelConnected [ctx e]
(let [ch (.getChannel e)]
(.addListener
(.write ch policy)
(ChannelFutureListener/CLOSE))))
(exceptionCaught [ctx e]
(let [ex (.getCause e)]
(println "Exception" ex)
(-> e .getChannel .close)))))
(defn setup-pipeline
"Returns channel pipeline."
[]
(proxy [ChannelPipelineFactory] []
(getPipeline []
(let [handler (server-handler)]
(Channels/pipeline (into-array ChannelHandler [handler]))))))
(defn startup
"Starts netty server."
[port]
(let [channel-factory (NioServerSocketChannelFactory. (Executors/newCachedThreadPool) (Executors/newCachedThreadPool))
bootstrap (ServerBootstrap. channel-factory)]
(.setPipelineFactory bootstrap (setup-pipeline))
(.setOption bootstrap "child.tcpNoDelay" true)
(.setOption bootstrap "child.keepAlive" true)
(.bind bootstrap (InetSocketAddress. port))))
Have a look at Aleph (also uses Netty) which can used to build clients and servers in many different protocols with nice Clojure API.
In 2021, the easiest way to build Clojure services on top of Netty would be to adopt Donkey which provides good Clojure interop with Vert.x which uses Netty as the backend. Donkey is relatively new and doesn't yet have a lot of documentation so check out this blog which covers a relevant open source project (source available) in terms of architecture, design, coding, and testing under load.
Related
(:require [utils.base64 :as base64])
(:import [com.google.api.client.googleapis.auth.oauth2 GoogleCredential]
[com.google.api.services.indexing.v3 Indexing]
[com.google.api.services.indexing.v3.model UrlNotification]
[com.google.api.client.http HttpTransport]
[com.google.api.client.json.gson GsonFactory]
[java.io ByteArrayInputStream]
[java.util Base64]
[com.google.api.services.indexing.v3 IndexingScopes]))
;; Set the private key and client email
(def private-key "")
(def client-email "my-client-email")
(def encoded-private-key (base64/encode-string private-key))
;; Set the private key and client email
(def private-key-json '{
"type": "service_account",
"client_email": "' client-email '",
"private_key": "' encoded-private-key '"
}')
;; Create the input stream from the JSON string
(def input-stream (ByteArrayInputStream. (.getBytes private-key-json)))
;; Create the OAuth 2.0 credentials
(def credentials (doto (GoogleCredential/fromStream input-stream)
(.createScoped (java.util.Collections/singleton IndexingScopes/INDEXING))))
;; Set up the HTTP transport and JSON factory
(def http-transport (HttpTransport/newTrustedTransport))
(def json-factory (GsonFactory/getDefaultInstance))
;; Set up the Indexing API client
(def indexing-client (doto (Indexing/Builder. http-transport json-factory credentials)
(.setApplicationName "My Indexing App")
(.build)))
;; Publish a URL notification
(def url-notification (UrlNotification. "https://aaa.com" "URL_UPDATED"))
(.execute (indexing-client/urlNotifications) (publish url-notification))
I'm trying to use private-key-json but the format is not valid. What is the best way to pass data? I referred https://developers.google.com/search/apis/indexing-api/v3/prereqs#examples.
Here I'm trying not to upload json file which we get once we create a service account and make use of only mandatory fields like client_email, private_key and type fields.
Modified code:
(:require [cheshire.core :as json]
[sketches.utils.base64 :as base64]
[clojure.string :as str])
(:import [com.google.api.client.googleapis.auth.oauth2 GoogleCredential]
[com.google.api.services.indexing.v3 Indexing$Builder]
[com.google.api.services.indexing.v3.model UrlNotification]
[com.google.api.client.googleapis.javanet GoogleNetHttpTransport]
[com.google.api.client.http HttpTransport]
[com.google.api.client.json.gson GsonFactory]
[java.io ByteArrayInputStream]
[com.google.api.services.indexing.v3 IndexingScopes]))
(def creds-json {:type "service_account",
:client_email "test-indexing-api",
:client_id "117578194507835125202",
:private_key_id "964321e5fc29980944da518887116ea50dfb7803",
:private_key ""})
(defn string->stream
([s] (string->stream s "UTF-8"))
([s encoding]
(-> s
(.getBytes encoding)
(ByteArrayInputStream.))))
(defn authorize [cred]
(-> cred
(json/encode)
(string->stream)
(GoogleCredential/fromStream)
(.createScoped [(IndexingScopes/INDEXING)])))
(defn valid-private-key [private-key]
(try
(when private-key
(->> (str/trim private-key)
(re-matches #"^-----BEGIN PRIVATE KEY-----[\s|\S]*-----END PRIVATE KEY-----[\\|n]*")
(some?)))
(catch Exception e
false)))
(def encoded-private-key (base64/encode-string private-key))
(def private-creds-json-string (json/generate-string creds-json))
(def input-stream (ByteArrayInputStream. (.getBytes private-creds-json-string)))
(defn build-analytics-client [creds]
(-> (Indexing$Builder. (GoogleNetHttpTransport/newTrustedTransport) (GsonFactory/getDefaultInstance) (authorize creds))
(.build)))
(defn url-notification []
(-> (UrlNotification.)
(.setType "URL_update")
(.setUrl "http://ace.madrid.quintype.io")))
(defn realtime-data [creds]
(-> (build-analytics-client creds)
(.urlNotifications)
(.publish (url-notification))
(.execute)))
```
But I get this error `Execution error (NoSuchMethodError) at com.google.api.client.http.ConsumingInputStream/close (ConsumingInputStream.java:40).
com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J`
I am learning clojure, and I've hit a snag while trying to refactor my web app to make it more functional and less reliant on global state.
The way you make a simple auto-reloading server with ring framework is like this:
(defn handler [req]
{:status 200
:body "<h1>Hello World</h1>"
:headers {}})
(defn -main []
(jetty/run-jetty (wrap-reload #'handler)
{:port 12345}))
(source: https://practicalli.github.io/clojure-webapps/middleware-in-ring/wrap-reload.html)
So handler is a global function. It is given as a var reference to wrap-reload. On each request, wrap-reload will reload the entire namespace, and then re-resolve handler reference to a potentially different function. Or at least this is my understanding.
This is all good in this simple example. The trouble starts when I replace this hello world handler with my actual handler, which has all sorts of state bound into it (eg. database connection). This state is initialized once inside -main, and then injected into the routing stack. Something like this:
(defn init-state [args dev]
(let
[db (nth args 1 "jdbc:postgresql://localhost/mydb")
routes (make-routes dev)
app (make-app-stack routes db)
{:port (Integer. (nth args 0 3000))
:route routes
:dev dev
:db db
:app app})
(defn start [state]
(println "Running in " (if (:dev state) "DEVELOPMENT" "PRODUCTION") " mode")
(model/create-tables (:db state))
(jetty/run-jetty (:app state) {:port (:port state)}))
(defn -main [& args]
(start (init-state args false)))
make-routes generates the router based on compojure library, and make-app-stack then wraps this router into a bunch of middlewares, which will inject global state (like DB string) for the use by handlers.
So how do I add wrap-reload to this setup? #'app macro won't work on local let "variable" (or whatever it's called). Do I need to expose my app as a global variable? Or can I re-generate the entire closure on each request.
My instincts tell me to avoid having global initialization code in the module body and keep all code in the main, but maybe I am overthinking. Should I just type my init-state code as globals and call it a day, or is there a better way?
I came up with a solution I can live with.
(ns webdev.core
(:require [ring.middleware.reload :as ring-reload]))
; Defeat private defn
(def reloader #'ring-reload/reloader)
; Other stuff...
(defn load-settings [args dev]
{:port (Integer. (nth args 0 3000))
:db (nth args 1 "jdbc:postgresql://localhost/webdev")
:dev dev})
(defn make-reloading-app [settings]
(let [reload! (reloader ["src"] true)]
(fn [request]
(reload!)
((make-app settings) request))))
(defn start [settings]
(let
[app
(if (:dev settings)
(make-reloading-app settings)
(make-app settings))]
(println "Running in " (if (:dev settings) "DEVELOPMENT" "PRODUCTION") " mode")
(model/create-tables (:db settings))
(jetty/run-jetty app {:port (:port settings)})))
(defn -main [& args]
(start (load-settings args false)))
Full code is available here: https://github.com/panta82/clojure-webdev/blob/master/src/webdev/core.clj
Instead of using wrap-reload directly, I use the underlying private reload function. I have to recreate the routing stack in every request, but it seems to work fine in dev. No global state needed :)
I am having a little trouble starting my app.
Here is my core.clj
(ns myapp.core
(:require [yada.yada :as yada :refer [resource as-resource]]
[yada.resources.file-resource :refer [new-directory-resource]]
[aero.core :refer [read-config]]
[web.view :as view]
[web.routes :as routes]
[clojure.java.io :as io]
[aero.core :refer [read-config]]
[com.stuartsierra.component :as component]
[clojure.java.jdbc :as jdbc]
[clojure.tools.namespace.repl :refer (refresh)]
[ring.adapter.jetty :as jetty]
[environ.core :refer [env]]))
(defrecord Listener [listener]
component/Lifecycle
(start [component]
(assoc component :listener (yada/listener
["/"
[(view/view-route)
routes/route-handler
["public/" (new-directory-resource (io/file "target/cljsbuild/public") {})]
[true (as-resource nil)]]] )))
(stop [component]
(when-let [close (-> component :listener :close)]
(close))
(assoc component :listener nil)))
(defn new-system []
(component/system-map
:listener (map->Listener {})
))
(def system nil)
(defn init []
(alter-var-root #'system
(constantly (new-system))))
(defn start []
(alter-var-root #'system component/start))
(defn stop []
(alter-var-root #'system
(fn [s] (when s (component/stop s)))))
(defn go []
(init)
(start))
(defn reset []
(stop)
(refresh :after 'web.core/go))
(defn -main
[& [port]]
(let [port (Integer. (or port (env :port) 3300))]
(jetty/run-jetty (component/start (new-system)) {:port port :join? false})))
I am testing out Stuart Sierra's library, component.
I can start the app if I do lein repl and (go) but I am trying to start my app by running lein run (to see what the app is like if I deployed it in production). When I do lein run in the browser I get the error
HTTP ERROR: 500
Problem accessing /view. Reason:
com.stuartsierra.component.SystemMap cannot be cast to clojure.lang.IFn
I am confused because I don't know why the system-map (in new-system) is the error. I'm also not sure what the error means so I don't know how to fix it
Could someone please help. Thanks
Your -main function calls jetty/run-jetty function first argument of which must be a Ring handler - function which accepts request map and produces response map. You're passing a system instead which leads to the exception. Exception means that jetty adapter tries to call passed system as a function, but can't, because system is actually a record and doesn't implement function interface IFn.
I'm not that familiar with yada, but it looks like yada/listener starts the (Aleph) server, so there's no need to explicitly call the jetty adapter. Your main should look something like this:
(defn -main [& [port]]
(component/start (new-system)))
Port (or any other config) could be passed as an argument to the new-system and then forwarded to components requiring it (in your case port should be passed down to the Listener and then to yada/listener call in start implementation).
I have this code to get data from sumo logic and other services.
core.clj has this, which parses the arguments and routes it to the right function in route.clj
(def cli-options
[
["-a" "--app APPNAME" "set app. app can be:
sumologic or jira"]
["-?" "--help"]
])
(defn -main
[& args]
(let [{:keys [options summary errors arguments]} (parse-opts args cli-options)]
(cond
(:app options) (route/to (:app options) options arguments)
:else (print_usage summary))))
route.clj has this:
(defn to
[app options arguments]
(case app
"jira" (jira/respond options arguments)
"sumologic" (sumo/respond)))
And then sumo.clj has this. there are other functions, of course, but showing just the relevant parts.
(defn get-env-var
[var]
(let [result (System/getenv var)]
(if (nil? result)
(throw (Exception. (str "Environment variable: " var " not set. Aborting")))
result)))
(def access_key
(let [user (get-env-var "SUMO_ID")
pass (get-env-var "SUMO_KEY")]
[user pass]))
(defn respond
[]
(let [{:keys [status body error] :as response} (http/get endpoint rest-options)]
(if error
(println error)
(print-response body))))
When I run the program using leiningen as lein run -- -? or even just lein run, I get this error, even though I haven't explicitly called the sumologic function. What am I doing wrong and what are things that I can do differently?
Caused by: java.lang.Exception: Environment variable: SUMO_KEY not set. Aborting
at clarion.sumo$get_env_var.invoke(sumo.clj:14)
at clarion.sumo$fn__3765.invoke(sumo.clj:19)
at clojure.lang.AFn.applyToHelper(AFn.java:152)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:3553)
You have def'd access_key so it is being evaluated when you load the application. You probably want to make it a function instead.
I have a ClojureScript application and I want to make RPC calls to the server which would look like normal function core.async calls on the client side.
In order to do this for the moment I wrote the code below based on cljx. In the RPC definitions section I would have to add all the server-side functions which I want to expose as RPC to the client side.
Note: the send function is taken from here: https://dimagog.github.io/blog/clojure/clojurescript/2013/07/12/making-http-requests-from-clojurescript-with-core.async/
Is there a way to do this nicer without the boilerplate code?
Thinking about how to improve it the only idea that I have is to write a leiningen plugin which generates server side and client side code needed for RPC i.e. the part that I do at this moment using cljx. Is there a better way?
(ns myapp.shared.rpc
(:require
#+cljs [myapp.tools :refer [send log]]
#+cljs [cljs.reader :as reader]
#+clj [clojure.tools.logging :as log]
#+clj [noir.response :refer [edn]]
#+clj [myapp.rpc :as rpc]
))
#+cljs (defn rpc-client [function params]
#+cljs (log "RPC call: (" function params ")")
#+cljs (send "POST" "/api"
#+cljs (str "rpc=" (pr-str {:fun function :params params}))
#+cljs (fn [x]
#+cljs (log "RPC response:'" x "'")
#+cljs (:response (reader/read-string x)))))
#+clj (defmulti rpc-impl #(:fun %))
#+clj (defn rpc-server [{rpc :rpc}]
#+clj (log/info "RPC call received:" rpc)
#+clj (let [response (-> rpc read-string rpc-impl)]
#+clj (log/info "RPC response sent: '" response "'")
#+clj (edn {:response response})))
;;;;; RPC definitions
#+cljs (defn demo [ & p] (rpc-client :demo p))
#+clj (defmethod rpc-impl :demo [{p :params}] (apply rpc/demo p))
Here are three libraries I've seen that handle RPC. I don't have significant experience with any of them, so take my comments with a large grain of salt.
Castra. The most recently updated, and has a nice readme.
Fetch. Looks simple, sweet, and probably sufficient.
Shoreleave. I used this successfully a while back. It worked fine, but has not been updated in a few years.
Clojure has edn data exchange format, you can use CQRS as data exchange method