I'm trying to log to two different files from the same namespace with Timbre. Or if that's not possible, at least to different files from the two different namespaces.
Inspecting timbre/*config* I get the impression that I'd need two configuration maps to configure something like that. I can create another config map and use it with timbre/log* in place of the standard config map but I can't shake off the feeling that it's not how this is supposed to be used...?
(timbre/log* timbre/*config* :info "Test with standard config")
AFAIK, the easiest way is indeed to create two config maps:
(def config1
{:level :debug
:appenders {:spit1 (appenders/spit-appender {:fname "file1.log"})}})
(def config2
{:level :debug
:appenders {:spit2 (appenders/spit-appender {:fname "file2.log"})}})
(timbre/with-config config1
(info "This will print in file1") )
(timbre/with-config config2
(info "This will print in file2") )
A second way would be to write your own appender from the spit-appender:
https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre/appenders/core.cljx
(defn my-spit-appender
"Returns a simple `spit` file appender for Clojure."
[& [{:keys [fname] :or {fname "./timbre-spit.log"}}]]
{:enabled? true
:async? false
:min-level nil
:rate-limit nil
:output-fn :inherit
:fn
(fn self [data]
(let [{:keys [output_]} data]
(try
;; SOME LOGIC HERE TO CHOOSE THE FILE TO OUTPUT TO ...
(spit fname (str (force output_) "\n") :append true)
(catch java.io.IOException e
(if (:__spit-appender/retry? data)
(throw e) ; Unexpected error
(let [_ (have? enc/nblank-str? fname)
file (java.io.File. ^String fname)
dir (.getParentFile (.getCanonicalFile file))]
(when-not (.exists dir) (.mkdirs dir))
(self (assoc data :__spit-appender/retry? true))))))))})
Related
I have a Ring handler that needs to:
Zip a few files
Stream the Zip to the client.
Now I have it sort of working, but only the first zipped entry gets streamed, and after that it stalls/stops. I feel it has something to do with flushing/streaming that is wrong.
Here is my (compojure) handler:
(GET "/zip" {:as request}
:query-params [order-id :- s/Any]
(stream-lessons-zip (read-string order-id) (:db request) (:auth-user request)))
Here is the stream-lessons-zip function:
(defn stream-lessons-zip
[]
(let [lessons ...];... not shown
{:status 200
:headers {"Content-Type" "application/zip, application/octet-stream"
"Content-Disposition" (str "attachment; filename=\"files.zip\"")
:body (futil/zip-lessons lessons)}))
And i use a piped-input-stream to do the streaming like so:
(defn zip-lessons
"Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
[lessons]
(let [paths (map #(select-keys % [:file_path :file_name]) lessons)]
(ring-io/piped-input-stream
(fn [output-stream]
; build a zip-output-stream from a normal output-stream
(with-open [zip-output-stream (ZipOutputStream. output-stream)]
(doseq [{:keys [file_path file_name] :as p} paths]
(let [f (cio/file file_path)]
(.putNextEntry zip-output-stream (ZipEntry. file_name))
(cio/copy f zip-output-stream)
(.closeEntry zip-output-stream))))))))
So I have confirmed that the 'lessons' vector contains like 4 entries, but the zip file only contains 1 entry. Furthermore, Chrome doesn't seem to 'finalize' the download, ie. it thinks it is still downloading.
How can I fix this?
It sounds like producing a stateful stream using blocking IO is not supported by http-kit. Non-stateful streams can be done this way:
http://www.http-kit.org/server.html#async
A PR to introduce stateful streams using blocking IO was not accepted:
https://github.com/http-kit/http-kit/pull/181
It sounds like the option to explore is to use a ByteArrayOutputStream to fully render the zip file to memory, and then return the buffer that produces. If this endpoint isn't highly trafficked and the zip file it produces is not large (< 1 gb) then this might work.
So, it's been a few years, but that code still runs in production (ie. it works). So I made it work back then, but forgot to mention it here (and forgot WHY it works, to be honest,.. it was very much trial/error).
This is the code now:
(defn zip-lessons
"Returns an inputstream (piped-input-stream) to be used directly in Ring HTTP responses"
[lessons {:keys [firstname surname order_favorite_name company_name] :as annotation
:or {order_favorite_name ""
company_name ""
firstname ""
surname ""}}]
(debug "zipping lessons" (count lessons))
(let [paths (map #(select-keys % [:file_path :file_name :folder_number]) lessons)]
(ring-io/piped-input-stream
(fn [output-stream]
; build a zip-output-stream from a normal output-stream
(with-open [zip-output-stream (ZipOutputStream. output-stream)]
(doseq [{:keys [file_path file_name folder_number] :as p} paths]
(let [f (cio/as-file file_path)
baos (ByteArrayOutputStream.)]
(if (.exists f)
(do
(debug "Adding entry to zip:" file_name "at" file_path)
(let [zip-entry (ZipEntry. (str (if folder_number (str folder_number "/") "") file_name))]
(.putNextEntry zip-output-stream zip-entry)
(.close baos)
(.writeTo baos zip-output-stream)
(.closeEntry zip-output-stream)
(.flush zip-output-stream)
(debug "flushed")))
(warn "File '" file_name "' at '" file_path "' does not exist, not adding to zip file!"))))
(.flush zip-output-stream)
(.flush output-stream)
(.finish zip-output-stream)
(.close zip-output-stream))))))
This piece of code runs on the server and it detects the changes to a file and sends it to the client. This is working for the first time and after that the file length is not getting updated even the I changed the file and saved it. I guess the clojure immutability is the reason here. How can I make this work?
(def clients (atom {}))
(def rfiles (atom {}))
(def file-pointers (atom {}))
(defn get-rfile [filename]
(let [rdr ((keyword filename) #rfiles)]
(if rdr
rdr
(let [rfile (RandomAccessFile. filename "rw")]
(swap! rfiles assoc (keyword filename) rfile)
rfile))))
(defn send-changes [changes]
(go (while true
(let [[op filename] (<! changes)
rfile (get-rfile filename)
ignore (println (.. rfile getChannel size))
prev ((keyword filename) #file-pointers)
start (if prev prev 0)
end (.length rfile) // file length is not getting updated even if I changed the file externally
array (byte-array (- end start))]
(do
(println (str "str" start " end" end))
(.seek rfile start)
(.readFully rfile array)
(swap! file-pointers assoc (keyword filename) end)
(doseq [client #clients]
(send! (key client) (json/write-str
{:changes (apply str (map char array))
:fileName filename}))
false))))))
There is no immutability here. In the rfiles atom, you store standard Java objects that are mutable.
This code works well only if data are appended to the end of the file, and the size is always increasing.
If there is an update/addition (of length +N) in the file other than at the end, the pointers start and end won't point to the modified data, but just to the last N characters and you will send dummy stuff to the clients.
If there is a delete or any change that decrease the length,
array (byte-array (- end start))
will throw a NegativeArraySizeException you don't see (eaten by the go bloc?). You can add some (try (...) catch (...)) or test that (- end start) is alway positive or null, to manage it and do the appropriate behaviour: resetting the pointers?,...
Are you sure the files you scan for changes are only changed by appending data? If not, you need to handle this case by resetting or updating the pointers accordingly.
I hope it will help.
EDIT test environment.
I defined the following. There is no change to the code you provided.
;; define the changes channel
(def notif-chan (chan))
;; define some clients
(def clients (atom {:foo "foo" :bar "bar"}))
;; helper function to post a notif of change in the channel
(defn notify-a-change [op filename]
(go (>! notif-chan [op filename])))
;; mock of the send! function used in send-changes
(defn send! [client message]
(println client message))
;; main loop
(defn -main [& args]
(send-changes notif-chan))
in a repl, I ran:
repl> (-main)
in a shell (I tested with an editor too):
sh> echo 'hello there' >> ./foo.txt
in the repl:
repl> (notify-a-change "x" "./foo.txt")
str0 end12
:bar {"changes":"hello there\n","fileName":".\/foo.txt"}
:foo {"changes":"hello there\n","fileName":".\/foo.txt"}
repl> (notify-a-change "x" "./foo.txt")
str12 end12
:bar {"changes":"","fileName":".\/foo.txt"}
:foo {"changes":"","fileName":".\/foo.txt"}
in a shell:
sh> echo 'bye bye' >> ./foo.txt
in a repl:
repl> (notify-a-change "x" "./foo.txt")
str12 end20
:bar {"changes":"bye bye\n","fileName":".\/foo.txt"}
:foo {"changes":"bye bye\n","fileName":".\/foo.txt"}
I use a dynamic var to make an easy way to use ssh. However, it suddenly stops working in a multi-parameter for!
So, here is my core.clj (which is kind of sketchy now):
(use 'clj-ssh.ssh)
(def the-agent (ssh-agent {}))
(def ^:dynamic *session* nil)
(defmacro on-host [host & body]
`(binding [*session* (clj-ssh.ssh/session the-agent ~host {})]
~#body))
(defn cmd [& args]
(split (:out (ssh *session* {:cmd (join " " args)})) #"\n"))
(defn attempt-1 []
(cmd "ls -a"))
(defn attempt-2 []
(for [f (cmd "ls -a")]
f))
(defn attempt-3 []
(for [r (range 3)
f (cmd "ls -a")]
[r f]))
For some reason, first two trial functions work, and the third doesn't (the hosts and files are censured):
user=> (on-host "(some host)" (attempt-1))
["." ".." ".ackrc" ...]
user=> (on-host "(some host)" (attempt-2))
("." ".." ".ackrc" ...)
user=> (on-host "(some host)" (attempt-3))
IllegalArgumentException No implementation of method: :connected? of protocol: #'clj-ssh.ssh.protocols/Session found for class: nil clojure.core/-cache-protocol-fn (core_deftype.clj:544)
Just in case you need a stacktrace:
user=> (use 'clojure.stacktrace)
nil
user=> (print-stack-trace *e 7)
java.lang.IllegalArgumentException: No implementation of method: :connected? of protocol: #'clj-ssh.ssh.protocols/Session found for class: nil
at clojure.core$_cache_protocol_fn.invoke (core_deftype.clj:544)
clj_ssh.ssh.protocols$eval1554$fn__1566$G__1543__1571.invoke (protocols.clj:4)
clj_ssh.ssh$connected_QMARK_.invoke (ssh.clj:411)
clj_ssh.ssh$ssh.invoke (ssh.clj:712)
census.core$cmd.doInvoke (core.clj:15)
clojure.lang.RestFn.invoke (RestFn.java:408)
census.core$attempt_3$iter__1949__1955$fn__1956.invoke (core.clj:29)
nil
I'm really not sure what it is all about. Can you help me? Thank you!
Use doseq, not for
What you have there is a lazy sequence that will be evaluated after the binding form has returned. doseq forces evaluation.
For further reading:
Difference between doseq and for in Clojure
http://cemerick.com/2009/11/03/be-mindful-of-clojures-binding/
In clojure.java.io, there is a io/resource function but I think it just loads the resource of the current jar that is running. Is there a way to specify the .jar file that the resource is in?
For example:
I have a jar file: /path/to/abc.jar
abc.jar when unzipped contains some/text/output.txt in the root of the unzipped directory
output.txt contains the string "The required text that I want."
I need functions that can do these operations:
(list-jar "/path/to/abc.jar" "some/text/")
;; => "output.txt"
(read-from-jar "/path/to/abc.jar" "some/text/output.txt")
;; => "The required text that I want"
Thanks in advance!
From Ankur's comments, I managed to piece together the functions that I needed:
The java.util.jar.JarFile object does the job.
you can call the method (.entries (Jarfile. a-path)) to give the list of files but instead of returning a tree structure:
i.e:
/dir-1
/file-1
/file-2
/dir-2
/file-3
/dir-3
/file-4
it returns an enumeration of filenames:
/dir-1/file-1, /dir-1/file-2, /dir-1/dir-2/file-3, /dir-1/dir-3/file-4
The following functions I needed are defined below:
(import java.util.jar.JarFile)
(defn list-jar [jar-path inner-dir]
(if-let [jar (JarFile. jar-path)]
(let [inner-dir (if (and (not= "" inner-dir) (not= "/" (last inner-dir)))
(str inner-dir "/")
inner-dir)
entries (enumeration-seq (.entries jar))
names (map (fn [x] (.getName x)) entries)
snames (filter (fn [x] (= 0 (.indexOf x inner-dir))) names)
fsnames (map #(subs % (count inner-dir)) snames)]
fsnames)))
(defn read-from-jar [jar-path inner-path]
(if-let [jar (JarFile. jar-path)]
(if-let [entry (.getJarEntry jar inner-path)]
(slurp (.getInputStream jar entry)))))
Usage:
(read-from-jar "/Users/Chris/.m2/repository/lein-newnew/lein-newnew/0.3.5/lein-newnew-0.3.5.jar"
"leiningen/new.clj")
;=> "The list of built-in templates can be shown with `lein help new`....."
(list-jar "/Users/Chris/.m2/repository/lein-newnew/lein-newnew/0.3.5/lein-newnew-0.3.5.jar" "leiningen")
;; => (new/app/core.clj new/app/project.clj .....)
I want to send var-args of a function to a macro, still as var-args.
Here is my code:
(defmacro test-macro
[& args]
`(println (str "count=" ~(count args) "; args=" ~#args)))
(defn test-fn-calling-macro
[& args]
(test-macro args))
The output of (test-macro "a" "b" "c") is what I want: count=3; args=abc
The output of (test-fn-calling-macro "a" "b" "c") is : count=1; args=("a" "b" "c") because args is sent as a single argument to the macro. How can I expand this args in my function in order to call the macro with the 3 arguments?
I guess I'm just missing a simple core function but I'm not able to find it. Thanks
EDIT 2 - My "real" code, shown in EDIT section below is not a valid situation to use this technique.
As pointed out by #Brian, the macro xml-to-cass can be replaced with a function like this:
(defn xml-to-cass
[zipper table key attr & path]
(doseq [v (apply zf/xml-> zipper path)] (cass/set-attr! table key attr v)))
EDIT - the following section goes beyond my original question but any insight is welcome
The code above is just the most simple I could come with to pinpoint my problem. My real code deals with clj-cassandra and zip-filter. It may also look over-engineering but it's just a toy project and I'm trying to learn the language at the same time.
I want to parse some XML found on mlb.com and insert values found into a cassandra database. Here is my code and the thinking behind it.
Step 1 - Function which works fine but contains code duplication
(ns stats.importer
(:require
[clojure.xml :as xml]
[clojure.zip :as zip]
[clojure.contrib.zip-filter.xml :as zf]
[cassandra.client :as cass]))
(def root-url "http://gd2.mlb.com/components/game/mlb/year_2010/month_05/day_01/")
(def games-table (cass/mk-cf-spec "localhost" 9160 "mlb-stats" "games"))
(defn import-game-xml-1
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))]
(doseq [v (zf/xml-> zipper (zf/attr :type))] (cass/set-attr! games-table game-id :type v))
(doseq [v (zf/xml-> zipper (zf/attr :local_game_time))] (cass/set-attr! games-table game-id :local_game_time v))
(doseq [v (zf/xml-> zipper :team [(zf/attr= :type "home")] (zf/attr :name_full))] (cass/set-attr! games-table game-id :home_team v))))
The parameter to import-game-xml-1 can be for example "gid_2010_05_01_colmlb_sfnmlb_1/". I remove the "gid_" and the trailing slash to make it the key of the ColumnFamily games in my database.
I found that the 3 doseq were a lot of duplication (and there should be more than 3 in the final version). So code templating using a macro seemed appropriate here (correct me if I'm wrong).
Step 2 - Introducing a macro for code templating (still works)
(defmacro xml-to-cass
[zipper table key attr & path]
`(doseq [v# (zf/xml-> ~zipper ~#path)] (cass/set-attr! ~table ~key ~attr v#)))
(defn import-game-xml-2
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))]
(xml-to-cass zipper games-table game-id :type (zf/attr :type))
(xml-to-cass zipper games-table game-id :local_game_time (zf/attr :local_game_time))
(xml-to-cass zipper games-table game-id :home_team :team [(zf/attr= :type "home")] (zf/attr :name_full))))
I believe that's an improvement but I still see some duplication in always reusing the same 3 parameters in my calls to xml-to-cass. That's were I introduced an intermediate function to take care of those.
Step 3 - Adding a function to call the macro (the problem is here)
(defn import-game-xml-3
"Import the content of xml into cassandra"
[game-dir]
(let [url (str root-url game-dir "game.xml")
zipper (zip/xml-zip (xml/parse url))
game-id (.substring game-dir 4 (- (.length game-dir) 1))
save-game-attr (fn[key path] (xml-to-cass zipper games-table game-id key path))]
(save-game-attr :type (zf/attr :type)) ; works well because path has only one element
(save-game-attr :local_game_time (zf/attr :local_game_time))
(save-game-attr :home :team [(zf/attr= :type "home"] (zf/attr :name_full))))) ; FIXME this final line doesn't work
Here's a some simple code which may be illuminating.
Macros are about code generation. If you want that to happen at runtime, for some reason, then you have to build and evaluate the code at runtime. This can be a powerful technique.
(defmacro test-macro
[& args]
`(println (str "count=" ~(count args) "; args=" ~#args)))
(defn test-fn-calling-macro
[& args]
(test-macro args))
(defn test-fn-expanding-macro-at-runtime
[& args]
(eval (cons `test-macro args)))
(defmacro test-macro-expanding-macro-at-compile-time
[& args]
(cons `test-macro args))
;; using the splicing notation
(defmacro test-macro-expanding-macro-at-compile-time-2
[& args]
`(test-macro ~#args))
(defn test-fn-expanding-macro-at-runtime-2
[& args]
(eval `(test-macro ~#args)))
(test-macro "a" "b" "c") ;; count=3; args=abc nil
(test-fn-calling-macro "a" "b" "c") ;; count=1; args=("a" "b" "c") nil
(test-fn-expanding-macro-at-runtime "a" "b" "c") ; count=3; args=abc nil
(test-macro-expanding-macro-at-compile-time "a" "b" "c") ; count=3; args=abc nil
(test-macro-expanding-macro-at-compile-time-2 "a" "b" "c") ; count=3; args=abc nil
(test-fn-expanding-macro-at-runtime "a" "b" "c") ; count=3; args=abc nil
If contemplation of the above doesn't prove enlightening, might I suggest a couple of my own blog articles?
In this one I go through macros from scratch, and how clojure's work in particular:
http://www.learningclojure.com/2010/09/clojure-macro-tutorial-part-i-getting.html
And in this one I show why run-time code generation might be useful:
http://www.learningclojure.com/2010/09/clojure-faster-than-machine-code.html
The typical way to use a collection as individual arguments to a function is to use (apply function my-list-o-args)
(defn test-not-a-macro [& args]
(print args))
(defn calls-the-not-a-macro [& args]
(apply test-not-a-macro args))
though you wont be able to use apply because test-macro is a macro. to solve this problem you will need to wrap test macro in a function call so you can apply on it.
(defmacro test-macro [& args]
`(println ~#args))
(defn calls-test-macro [& args]
(eval (concat '(test-macro) (args)))) ;you almost never need eval.
(defn calls-calls-test-macro [& args]
(calls-test-macro args))
This is actually a really good example of one of the ways macros are hard to compose. (some would say they cant be composed cleanly, though i think thats an exageration)
Macros are not magic. They are a mechanism to convert code at compile-time to equivalent code; they are not used at run-time. The pain you are feeling is because you are trying to do something you should not be trying to do.
I don't know the library in question, but if cass/set-attr! is a function, I see no reason why the macro you defined has to be a macro; it could be a function instead. You can do what you want to do if you can rewrite your macro as a function instead.
Your requirements aren't clear. I don't see why a macro is necessary here for test-macro, unless you're trying to print the unevaluated forms supplied to your macro.
These functions provide your expected results, but that's because your sample data was self-evaluating.
(defn test-args
[& args]
(println (format "count=%d; args=%s"
(count args)
(apply str args))))
or
(defn test-args
[& args]
(print (format "count=%d; args=" (count args)))
(doseq [a args]
(pr a))
(newline))
You can imagine other variations to get to the same result.
Try calling that function with something that doesn't evaluate to itself, and note the result:
(test-args (+ 1 2) (+ 3 4))
Were you looking to see the arguments printed as "37" or "(+ 1 2)(+ 3 4)"?
If you were instead trying to learn about macros and their expansion in general, as opposed to solving this particular problem, please tune your question to probe further.