I have the following code to monitoring database table to create MQ messsage every several seconds. And I found that when I restarted the RabbitMQ server the app still runs without throwing an exception, and it still print message was created. So why it won't throw an exception? And another question is how to close the connect when I kill the app? Since it's a service, I have no place to write the code to close the RabbitMQ connection.
(require '[clojure.java.jdbc :as j])
(require '[langohr.core :as rmq])
(require '[langohr.channel :as lch])
(require '[langohr.queue :as lq])
(require '[langohr.exchange :as le])
(require '[langohr.consumers :as lc])
(require '[langohr.basic :as lb])
(require '[clojure.data.json :as json])
(require '[clojure.java.io :as io])
(defn load-props
[file-name]
(with-open [^java.io.Reader reader (io/reader file-name)]
(let [props (java.util.Properties.)]
(.load props reader)
(into {} (for [[k v] props] [(keyword k) (read-string v)])))))
(def ^{:const true}
default-exchange-name "")
(defn create-message-from-database
([channel qname db-spec]
(let [select-sql "select a.file_id from file_store_metadata
where processed=false "
results (j/query db-spec select-sql)
]
(doseq [row results]
(let [file-id (:file_id row)]
(lb/publish channel default-exchange-name qname (json/write-str row) {:content-type "text/plain" :type "greetings.hi" :persistent true})
(j/update! db-spec :file_store_metadata {:processed true} ["file_id = ?" (:file_id row)])
(println "message created for a new file id" (:file_id row))))
))
)
(defn create-message-from-database-loop
([x channel qname db] (
while true
(Thread/sleep (* x 1000))
(create-message-from-database channel qname db)
))
)
(defn -main
[& args]
(let [
properties (load-props "resource.properties")
postgres-db (:database properties)
rabbitmq-url (:rabbitmq properties)
wake-up-interval (:interval properties)
rmq-conn (rmq/connect {:uri rabbitmq-url})
ch (lch/open rmq-conn)
qname "etl scheduler"]
(println "monitoring file store meta data on " postgres-db " every " wake-up-interval " seconds")
(println "message will be created on rabbitmq server" rabbitmq-url)
(lq/declare ch qname {:exclusive false :auto-delete false :persistent true})
(create-message-from-database-loop wake-up-interval ch qname postgres-db)
)
)
Related
I want to stream large binary files (exe, jpg, ..., all types of files). It seems that the aleph client can do it.
I looked at the official sample and understood that if we pass the lazy sequence to the body, the response can be passed a stream.
(defn streaming-numbers-handler
"Returns a streamed HTTP response, consisting of newline-delimited numbers every 100
milliseconds. While this would typically be represented by a lazy sequence, instead we use
a Manifold stream. Similar to the use of the deferred above, this means we don't need
to allocate a thread per-request.
In this handler we're assuming the string value for `count` is a valid number. If not,
`Integer.parseInt()` will throw an exception, and we'll return a `500` status response
with the stack trace. If we wanted to be more precise in our status, we'd wrap the parsing
code with a try/catch that returns a `400` status when the `count` is malformed.
`manifold.stream/periodically` is similar to Clojure's `repeatedly`, except that it emits
the value returned by the function at a fixed interval."
[{:keys [params]}]
(let [cnt (Integer/parseInt (get params "count" "0"))]
{:status 200
:headers {"content-type" "text/plain"}
:body (let [sent (atom 0)]
(->> (s/periodically 100 #(str (swap! sent inc) "\n"))
(s/transform (take cnt))))}))
I have the following code:
(ns core
(:require [aleph.http :as http]
[byte-streams :as bs]
[cheshire.core :refer [parse-string generate-string]]
[clojure.core.async :as a]
[clojure.java.io :as io]
[clojure.string :as str]
[clojure.tools.logging :as log]
[clojure.walk :as w]
[compojure.core :as compojure :refer [ANY GET defroutes]]
[compojure.route :as route]
[me.raynes.fs :as fs]
[ring.util.response :refer [response redirect content-type]]
[ring.middleware.params :as params]
[ring.util.codec :as cod])
(:import java.io.File)
(:import java.io.FileInputStream)
(:import java.io.InputStream)
(:gen-class))
(def share-dir "\\\\my-pc-network-path")
(def content-types
{".png" "image/png"
".GIF" "image/gif"
".jpeg" "image/jpeg"
".svg" "image/svg+xml"
".tiff" "image/tiff"
".ico" "image/vnd.microsoft.icon"
".bmp" "image/vnd.wap.wbmp"
".css" "text/css"
".csv" "text/csv"
".html" "text/html"
".txt" "text/plain"
".xml" "text/xml"})
(defn byte-seq [^InputStream is size]
(let [ib (byte-array size)]
((fn step []
(lazy-seq
(let [n (.read is ib)]
(when (not= -1 n)
(let [cb (chunk-buffer size)]
(dotimes [i size] (chunk-append cb (aget ib i)))
(chunk-cons (chunk cb) (step))))))))))
(defn get-file [req]
(let [uri (:uri req)
file (str/replace (w/keywordize-keys (cod/form-decode uri)) #"/file/" "")
full-dir (io/file share-dir file)
_ (log/debug "FULL DIR: "full-dir)
filename (.getName (File. (.getParent full-dir)))
extension (fs/extension filename)
_ (log/debug "EXTENSION: " extension)
resp {:status 200
:headers {"Content-type" (get content-types (.toLowerCase extension) "application/octet-stream")
"Content-Disposition" (str "inline; filename=\"" filename "\"")}
:body (with-open [is (FileInputStream. full-dir)]
(let [bs (byte-seq is 4096)]
(byte-array bs)))}
]
resp))
(def handler
(params/wrap-params
(compojure/routes
(GET "/file/*" [] get-file)
(route/not-found "No such file."))))
(defn -main [& args]
(println "Starting...")
(http/start-server handler {:host "0.0.0.0" :port 5555}))
I get the uri and try to read file with chunks. I wanted to do it because files can be about 3 GB. So, what I expected is the application would not not use more memory than the size of a chunk.
But when I've set 1GB (-Xmx option) for the application, it has taken all memory (1 GB).
Why is 1GB taken? Does JVM work this way?
When I have 100 simultaneous connections (e.g., each file is 3GB), I get OutOfMemoryError.
The task is to "stream" a file with chunks to avoid OutOfMemoryError.
In the get-file function you are calling byte-array on the result of byte-seq. byte-array will realize the LazySeq returned by byte-seq meaning all of it will be in memory.
As far as I know (at least for TCP server), aleph accepts any lazy sequence of ByteBuffer as a body and will handle it optimally for you, so you could just return the result of calling byte-seq
(defn get-file [req]
(let [uri (:uri req)
file (str/replace (w/keywordize-keys (cod/form-decode uri)) #"/file/" "")
full-dir (io/file share-dir file)
_ (log/debug "FULL DIR: "full-dir)
filename (.getName (File. (.getParent full-dir)))
extension (fs/extension filename)
_ (log/debug "EXTENSION: " extension)
resp {:status 200
:headers {"Content-type" (get content-types (.toLowerCase extension) "application/octet-stream")
"Content-Disposition" (str "inline; filename=\"" filename "\"")}
:body (with-open [is (FileInputStream. full-dir)]
(byte-seq is 4096))}
]
resp))
Alternative
Aleph is in compliance with the ring specification and :body accepts File or InputStream. So there is no need to return the bytes yourself.
(defn get-file [req]
(let [uri (:uri req)
file (str/replace (w/keywordize-keys (cod/form-decode uri)) #"/file/" "")
full-dir (io/file share-dir file)
_ (log/debug "FULL DIR: "full-dir)
filename (.getName (File. (.getParent full-dir)))
extension (fs/extension filename)
_ (log/debug "EXTENSION: " extension)
resp {:status 200
:headers {"Content-type" (get content-types (.toLowerCase extension) "application/octet-stream")
"Content-Disposition" (str "inline; filename=\"" filename "\"")}
:body full-dir}
]
resp))
I can validate clojure schema inputs as string as shown below.
name :- s/Str,
place :- s/Str,
How can I write a defschema for Email to use like
email : - s/Email
which will convert/coerce it to lower case.
I happen to have exactly the one...
(ns myapp.utils.schema
(:import [org.apache.commons.validator.routines EmailValidator])
(:require [clojure.string :as str]
[schema.core :as s]
[schema.coerce :as sco]))
(def valid-email? "Checks that the value is a valid email string."
(let [ev (EmailValidator/getInstance)]
(fn [s]
(and
(string? s)
(.isValid ev, ^String s)
))))
(def Email "Email schema type, stricter than schema.core/Str."
(s/pred valid-email?))
(defn sanitize-email [s]
(if (string? s)
(-> s str/lower-case str/trim)
s))
(defn email-matcher [s]
(when (= Email s) sanitize-email))
(def matcher
"The Plumatic matcher used for Plumatic schema coercion in myapp."
(sco/first-matcher [email-matcher,,,]))
(defn coercer
[schema]
(sco/coercer schema matcher))
;; ...
(ns myapp.customer
(:require [schema.core :as s]
[myapp.utils.schema :as sc]))
(def Customer
{:customer/email sc/Email
:customer/name s/Str})
(def coerce-customer (sc/coercer Customer))
(coerce-customer {:customer/email "vAlENTin#gmaIl.com "
:customer/name "Valentin"})
=> {:customer/email "valentin#gmail.com"
:customer/name "Valentin"}
Problem: I get a null pointer exception when I call the caller function in the below code if I pass it data via json from a javascript socket client. I don't get the error when I run this same code from the repl and I pass it the same data i.e.
In repl:
(def data (json/write-str {:controller "hql" :function "something"}))
(caller data)
This works fine
From javascript: this gives me s null pointer exception when calling the function caller from the handler.
<script>
var msg = {'controller': 'hql','function':'something'};
socket = new WebSocket('ws://192.168.0.7:9090');
socket.onopen = function() {
socket.send(JSON.stringify(msg));
}
socket.onmessage = function(s) {
console.log(s);
socket.close();
}
socket.onclose = function() {
console.log('Connection Closed');
}
</script>
(ns hqlserver.core
(:use [org.httpkit.dbcp :only [use-database! close-database! insert-record update-values query delete-rows]]
[org.httpkit.server])
(:require [clojure.data.json :as json]))
(defn hql [data]
(use-database! "jdbc:mysql://localhost/xxxx" "user" "xxxxx")
(let [rows (query ("select username,firstname,lastname from users order by lastname,firstname")]
(close-database!)
(json/write-str rows)))
(defn get-controller [data]
(if data
(let [data (json/read-str data :key-fn keyword)]
(data :controller))))
(defn caller [data]
((ns-resolve *ns* (symbol (get-controller data))) data))
(defn handler [request]
(with-channel request channel
(on-close channel (fn [status] (println "channel closed: " status)))
(on-receive channel (fn [data] (send! channel (caller data))))))
(defn -main [& args]
(run-server handler {:port 9090}))
Don't know if it's the best way but this is the solution:
Thanks to noisesmith for helping and tips....
(ns hqlserver.core
(:use [org.httpkit.dbcp :only [use-database! close-database! insert- record update-values query delete-rows]]
[org.httpkit.server]
[hqlserver.s300.login :only [login]])
(:require [clojure.data.json :as json]
[hqlserver.s300.site :refer [site-index site-update]]
[clojure.string :as str]))
(defn get-site [data]
(if data
(let [data (json/read-str data :key-fn keyword)]
(-> data :site))))
(defn get-controller [data]
(if data
(let [data (json/read-str data :key-fn keyword)]
(-> data :controller))))
(defn get-function [data]
(if data
(let [data (json/read-str data :key-fn keyword)]
(-> data :function))))
(defn caller [data]
(let [site (get-site data)]
(let [controller (get-controller data)]
(let [function (get-function data)]
(cond
(and (= site "s300")(= controller "login")(= function "login")) (login data)
(and (= site "s300")(= controller "site")(= function "site-index")) (site-index data)
(and (= site "s300")(= controller "site")(= function "site-update")) (site-update data)
:else (json/write-str {:error "No records found!"}))))))
(defn async-handler [ring-request]
(with-channel ring-request channel
(if (websocket? channel)
(on-receive channel (fn [data]
(send! channel (caller data))))
(send! channel {:status 200
:headers {"Content-Type" "text/plain"}
:body "Long polling?"}))))
(defn -main [& args]
(run-server async-handler {:port 9090}));
From the immutant documentation at http://immutant.org/documentation/current/apidoc/guide-installation.html:
With the dependencies in place, you simply invoke the Immutant
services from your app’s main entry point, identified by the :main
key in your project.clj.
Immutant's web service can be invoked like:
(ns my-app.handler
...
(:require [immutant.web :as web])
... )
(def app ... )
(defn -main [& args]
(web/run app))
What's the equivalent of (web/run app) for immutant.messaging?
here your answer
to start queue
(ns my-project.name
(:require [immutant.messaging :as msg]))
(defn call-fn
[args]
(let [fun (ns-resolve "namespace where fun defined" (symbol (first args)))
params (rest args)]
(if (seq params)
(apply fun )
(fun))))
(msg/start "queue")
(msg/respond "queue" (fn [args] (call-fn args)))
to call the queue just say
#(msg/request "queue" ["fun-name" params])
I'm trying to add the following Hiccup template function to my file
(defn d3-page [title js body & {:keys [extra-js] :or {extra-js []}}]
(html5
[:head
[:title title]
(include-css "/css/nv.d3.css"))
(include-css "/css/style.css")]
[:body
(concat
[body]
[(include-js "http://d3js.org/d3.v3.min.js")
(include-js (str "https://raw.github.com"
"/novus/nvd3"
"/master/nv.d3.min.js")]
(map include-js extra-js)
[(include-js "/js/script.js")
(javascript-tag js)])]))
but keep getting an unmatched delimiter when I run lein ring server. This comes from Clojure Data Cookbook, so I am surprised to find an error and suspect the error is just on my end. Below is the rest of the code in the file:
(ns web-viz.web
(:require [compojure.route :as route]
[compojure.handler :as handler]
[clojure.string :as str])
(:use compojure.core
ring.adapter.jetty
[ring.middleware.content-type :only
(wrap-content-type)]
[ring.middleware.file :only (wrap-file)]
[ring.middleware.file-info :only
(wrap-file-info)]
[ring.middleware.stacktrace :only
(wrap-stacktrace)]
[ring.util.response :only (redirect)]
[hiccup core element page]
[hiccup.middleware :only (wrap-base-url)]))
(defn d3-page...as above
...)
(deftype Group [key values])
(deftype Point [x y size])
(defn add-label [chart axis label]
(if-not (nil? label)
(.axisLabel (aget chart axis) label)))
(defn add-axes-labels [chart x-label y-label]
(doto chart (add-label "xAxis" x-label)
(add-label "yAxis" y-label)))
(defn populate-node [selector chart groups transition continuation]
(-> (.select js/d3 selector)
(.datum groups)
(.transition)
(.duration (if transition 500 0))
(.call chart)
(.call continuation)))
(defn force-layout-plot []
(d3-page "Force-Directed Layout"
"webviz.force.force_layout();"
[:div#force.chart [:svg]]))
(defroutes site-routes
(GET "/force" [] (force-layout-plot))
(GET "/force/data.json" []
(redirect "/data/census-race.json"))
(route/resources "/")
(route/not-found "Page not found"))
(def app (-> (handler/site site-routes)))
In line 5
(include-css "/css/nv.d3.css"))
There's an extra ) there.
And in line 13,
"/master/nv.d3.min.js")]
There's one ) missing. Should be
"/master/nv.d3.min.js"))]
You should use an editor which can do the matching of braces, parentheses, and brackets, etc. automatically.