I think this is a bug in clojure/tools.logging. I have the following db.clj file. What it does is unimportant. The important part is that for safety I have disabled the *read-eval*. I invoke db/start with no problems. However, if I uncomment the #_(log/info "Failed to bootstrap") form, it throws an EvalReader not allowed error. I've tried all sorts of combinations for the log/info call. If it's outside of the try block, it's fine. Inside the try block anywhere, whether it's in the body, catch, or finally, it raises this exception. However, when I wrap a try around log/info elsewhere, it's fine.
What gives?
(ns extenium.db
(:require [clojure.tools.logging :as log]
[clojure.java.io :as io])
(:import com.thinkaurelius.titan.core.TitanGraph
com.thinkaurelius.titan.core.TitanFactory))
(def ^:private
sentinel- (Object.))
(def ^:private
db- (atom nil))
...
(defn start [^String path]
(locking sentinel-
(log/info "Starting database at path" path)
(let [exists (.exists (io/file path))
^TitanGraph db_ (TitanFactory/open path)]
(if exists
(log/info "Path" path "exists")
(log/info "Path" path "does not exist"))
(log/info "Starting database engine")
(swap! db- (constantly db_))
(log/info "Started database engine")
(if (not exists)
(try
(bootstrap-)
(catch Throwable t
#_(log/info "Failed to bootstrap")
(stop)
(.delete (io/file path))
(throw t)))))
(log/info "Started database")
true))
EDIT: Trimmed down code per #alex-taggart. bootstrap- implementation not shown. I had originally included everything because this seemed like a context-specific bug and I felt it was safer to provide as much context as possible.
EDIT: Per #chouser, added how I'm disabling *read-eval*. This is the template that is generated by lein new app.
(defn -main
"The main entry point into Extenium."
[& args]
;; Prevent arbitrary eval injection
(alter-var-root #'*read-eval* (constantly false))
;; Initialize system settings from the command line and configuration file
(init!- args)
;; Start the service
(start!-))
It's not a bug, really. The clojure.tools.logging library is just an abstraction upon other Java logging facilities. To discover which one is available, it uses eval expression. You're welcome to check for your own: here is a quick search result and a certain file where it gets used.
In your case, I believe it's not necessary to care about the global read-eval. This is an internal feature and who knows what other libraries depend on it. If you validate the user input and prevent it from being evaluated, it's fine to leave the flag as is. I would say, SQL injections and XSS are the things you should be worried about first.
Related
I want to have a *flag* variable in a given namespace that is set to true only when :aot compiling.
Is there a way to do that?
Your issue is kind of complicated because the definition of Clojure's own dynamic nature, so there's no rough equivalent of C's #ifdef or some other mechanism that happens at compile time, but here's a workaround:
I created a Leiningen project with lein new app flagdemo. This trick detects when AOT is performed, as #Biped Phill mentioned above, using the dynamic var *compile-files*, and saves a resource in the classpath of the compiled code:
The code looks like this:
(ns flagdemo.core
(:gen-class))
(def flag-path "target/uberjar/classes/flag.edn")
(defn write-flag [val]
(try (spit flag-path (str val)) (catch Exception _)))
(defn read-flag []
(some-> (clojure.java.io/resource "flag.edn") slurp clojure.edn/read-string))
(write-flag false)
(when clojure.core/*compile-files*
(write-flag true))
(defn -main
[& args]
(println "Flag is" (read-flag)))
So, when the file loads using, say, lein run or when you load it in the REPL, it will try to write an EDN file with the value false.
When you compile the package using lein uberjar, it loads the namespace and finds that *compile-files* is defined, thus it saves an EDN file that is packaged with the JAR as a resource.
The function read-flag just tries to load the EDN file from the classpath.
It works like this:
$ lein clean
$ lein run
Flag is nil
$ lein uberjar
Compiling flagdemo.core
Created /tmp/flagdemo/target/uberjar/flagdemo-0.1.0-SNAPSHOT.jar
Created /tmp/flagdemo/target/uberjar/flagdemo-0.1.0-SNAPSHOT-standalone.jar
$ java -jar target/uberjar/flagdemo-0.1.0-SNAPSHOT-standalone.jar
Flag is true
Credit to #Biped Phill and #Denis Fuenzalida for explaining it to me.
I think this works as well:
(def ^:dynamic *static* false)
(when (and *compile-files* boot/*static-flag*)
(alter-var-root #'*static* true))
then in profiles.clj:
:injections [(require 'boot)
(intern 'boot '*static-flag* true)]
First you need to indicate that a variable will change over time:
(def ^:dynamic *the-answer* nil)
*the-answer*
;=> nil
The ^:dynamic bit will allow that variable to be rebound later:
(binding [*the-answer* 42] *the-answer*)
;=> 42
ADDENDUM 1
Here's what would happen if you didn't use ^:dynamic:
(def *the-answer* nil)
(binding [*the-answer* 42] *the-answer*)
;=> [...] Can't dynamically bind non-dynamic var [...]
ADDENDUM 2
These kind of variables are commonly referred to as "earmuffs". The naming convention is unenforced but strongly encouraged.
Here's what happen in the REPL when you use this naming without declaring a dynamic variable:
(def *the-answer* nil)
; Warning: *the-answer* not declared dynamic and thus is not dynamically rebindable, but its name suggests otherwise. Please either indicate ^:dynamic *the-answer* or change the name. (/tmp/form-init7760459636905875407.clj:1)
I'd like to mock out a macro in a namespace.
For instance, clojure.tools.logging/error.
I tried with-redefs with no luck
(def logged false)
(defmacro testerror
{:arglists '([message & more] [throwable message & more])}
[& args]
`(def logged true))
(deftest foo
...
(with-redefs
[log/error testerror]
...
That gave this error:
CompilerException java.lang.RuntimeException: Can't take value of a macro
Amalloy provided you the answer for your direct question on how to mock a macro - you cannot.
However, you can solve your problem with other solutions (simpler than moving your whole application to component dependency injection). Let me suggest two alternative implementations (unfortunately, not very straightforward but still simpler than using component).
Mock the function called by logging macro
You cannot mock a macro but you can mock a function that will be used when the logging macro get expanded.
(require '[clojure.tools.logging :as log])
(require '[clojure.pprint :refer [pprint]])
(pprint (macroexpand `(log/error (Exception. "Boom") "There was a failure")))
Gives:
(let*
[logger__739__auto__
(clojure.tools.logging.impl/get-logger
clojure.tools.logging/*logger-factory*
#object[clojure.lang.Namespace 0x2c50fafc "boot.user"])]
(if
(clojure.tools.logging.impl/enabled? logger__739__auto__ :error)
(clojure.core/let
[x__740__auto__ (java.lang.Exception. "Boom")]
(if
(clojure.core/instance? java.lang.Throwable x__740__auto__)
(clojure.tools.logging/log*
logger__739__auto__
:error
x__740__auto__
(clojure.core/print-str "There was a failure"))
(clojure.tools.logging/log*
logger__739__auto__
:error
nil
(clojure.core/print-str x__740__auto__ "There was a failure"))))))
As you can see, the function that does actual logging (if a given level is enabled) is done with clojure.tools.logging/log* function.
We can mock it and write our test:
(require '[clojure.test :refer :all])
(def log-messages (atom []))
(defn log*-mock [logger level throwable message]
(swap! log-messages conj {:logger logger :level level :throwable throwable :message message}))
(with-redefs [clojure.tools.logging/log* log*-mock]
(let [ex (Exception. "Boom")]
(log/error ex "There was a failure")
(let [logged (first #log-messages)]
(is (= :error (:level logged)))
(is (= "There was a failure!" (:message logged)))
(is (= ex (:throwable logged))))))
Use your logging library API to collect and inspect log messages
Your logging library API might provide features that would allow you to plug into in your test to collect and assert logging events. For example with java.util.logging you can write your own implementation of Handler that would collect all logged log records and add it to a specific (or root) logger.
You cannot do this. The point of macros is that they are expanded when the code is compiled, and after that they are gone. The original code that included a call to the macro is unrecoverable. You cannot retroactively redefine a macro at runtime: you're too late already.
An alternative approach, if you want to have swappable logging implementations, would be to use something like Component for dependency injection, and use a different logging component depending on whether you are running tests or running your real program. Arguably that's a bit heavy-handed, and maybe there is a simpler approach, but I don't know it.
I'm learning about Hystrix and Clojure and don't understand how to (properly) set a timeout on a Hystrix command in Clojure.
I searched StackOverflow and the web more generally. I looked at Hystrix's Clojure wrapper source code (https://github.com/Netflix/Hystrix/blob/master/hystrix-contrib/hystrix-clj/src/main/clojure/com/netflix/hystrix/core.clj). There is a init-fn function parameter that looked promising, but the comments seem to suggest that this won't be a sustainable solution. But would this be a simple start?
I have a ridiculously simple Hystrix command running in Clojure and would appreciate help in extending this to set, say, a 200ms timeout:
(ns hystrix-timeout.core
(:require [clojure.string :as str])
(:require [com.netflix.hystrix.core :as hystrix])
(:gen-class))
(defn my-primary [a]
(if (= a true) (throw (Exception. "Primary failed")) (str "primary: " a)))
(defn my-fallback [a]
(str "fallback: " a))
(hystrix/defcommand my-command
{:hystrix/fallback-fn my-fallback}
[a]
(my-primary a))
(defn -main
"Executes a simple Hystrix command. Will use a timeout when I know how to do this in Clojure."
[& args]
(println (my-command false))
(println (my-command true))
(System/exit 0) ; Exit explicitly as Hystrix threads are still running.
)
I've put my lein project up at https://github.com/oliverbaier/learning/tree/master/hystrix-timeout in case this makes answering easier.
Thanks a lot,
Oliver
The simplest route is to just use System properties. You can set either a global default:
(System/setProperty "hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds" "200")
or a command specific value:
(System/setProperty "hystrix.command.my-command.execution.isolation.thread.timeoutInMilliseconds" "200")
You can perform this in your -main method. If you are running a real web-app sans -main, you can add a ring init in project.clj :ring {:handler your-handler :init your-config-method} your-config-method will be invoked on startup.
Now, I need some kind of code reloading in the clojure/java mixing system.
I know tools.namespace lib can be used to refresh clj code. But when I try to reload code which is used in a thread, it is not updated.
This is the example clojure code I need to update: (just switch between upper case and lower case.)
(ns com.test.test-util)
(defn get-word [sentence]
(let [a-word (rand-nth (clojure.string/split sentence #"\s"))
;;a-word (clojure.string/upper-case a-word)
]
(println a-word)
a-word))
I used a future to start a thread and call the code above.
In main method, in my first test, I just create a future and refresh the code after a while. But it is not updated. So, I also tested to create another thread after refresh. But the code is not updated.
(defn start-job []
(future (
(println "start worker job.")
(while #job-cond
(do (get-word (rand-nth sentences))
(Thread/sleep 2000)) )
(println "return!")
)))
(defn -main []
(let [work-job (start-job)]
(println "modify clj file...")
(Thread/sleep 10000)
(swap! job-cond not)
(future-cancel work-job)
(swap! job-cond not)
(refresh)
(Thread/sleep 3000)
(let [new-work-job (start-job)]
(Thread/sleep 10000)
(swap! job-cond not)
(future-cancel new-work-job))))
Thanks.
In general you don't want code-reloading to affect runtime. Rather you should try to seek for clean ways to boot and shutdown the running application code using e. g. Stuart Sierras library component or juxt/jig by Malcolm Sparks. Then reload namespaces only when your application is not up and running (or at least not the parts of running code that depend on namespaces being reloaded).
That being said, here is why your example does not work:
start-job can not automatically refer to a reloaded version of get-word in its lifetime because the version of get-work it uses is resolved when it is compiled. So it will still use the old version.
Explicitly resolve the var of get-word using
(resolve `get-word)
and invoke the returned var as a function instead of using get-word directly in the future.
What will happen then?
After you made the changes and refresh has been invoked, -main will still refer to the old-version of start-job (so if you change that, it will not show effect in runtime). However within that get-word will be resolved as described above and the new version of get-word will be used.
Short explanation: You can't expect a running compiled function to replace itself with reloaded code on the fly (no matter whether it runs on a separate thread or not). That would (unless you are using resolve in it) be a necessity though for it to invoke reloaded functions.
The following program, when run from an überjar, exits at the end only when using the in-memory Datomic database; when connecting to the Datomic server, it hangs indefinitely rather than exiting the JVM:
(ns myns.example
(:use [datomic.api :only [db q] :as d])
(:gen-class))
;; WORKS: (def uri "datomic:mem://testdb")
(def uri "datomic:free://localhost:4334/testdb2")
(defn -main []
(println 1)
(when (d/create-database uri)
(d/connect uri))
(shutdown-agents)
(println 2))
Run as:
lein uberjar && java -cp target/myns-0.1.0-SNAPSHOT-standalone.jar myns.example
Outputs:
1
2
and hangs. It only hangs if the DB doesn't exist when the program starts.
Anyone know why, or how to fix? This is with both datomic-free-0.8.4020.26 and datomic-free-0.8.3941.
UPDATE -- the above program does actually terminate, but it takes a very long time (> 1 minute). I'd like to know why.
shutdown-agents takes up to one minute to complete (assuming no agents are running an action).
This is due to the way java.util.concurrent cached thread pools work.
Use datomic.api/shutdown
shutdown
function
Usage: (shutdown shutdown-clojure)
Shut down all peer
resources. This method should be called as part of clean shutdown of
a JVM process. Will release all Connections, and, if shutdown-clojure
is true, will release Clojure resources. Programs written in Clojure
can set shutdown-clojure to false if they manage Clojure resources
(e.g. agents) outside of Datomic; programs written in other JVM
languages should typically set shutdown-clojure to true.
Added in Datomic Clojure version 0.8.3861
(ns myns.example
(:require [datomic.api :as d])
(:gen-class))
(def uri "datomic:free://localhost:4334/testdb2")
(defn -main []
(d/create-database uri)
(let [conn (d/connect uri)]
(try
;; do something
(finally (d/shutdown true)))