I have a rabbitMQ connection that seems to be started at compile time (when I type lein compile) and then blocks the building of my project. Here are more details on the problem. Let us say this is the clojure file bla_test.clj
(import (com.rabbitmq.client ConnectionFactory Connection Channel QueueingConsumer))
;; And then we have to translate the equivalent java hello world program using
;; Clojure's excellent interop.
;; It feels very strange writing this sort of ceremony-oriented imperative code
;; in Clojure:
;; Make a connection factory on the local host
(def connection-factory
(doto (ConnectionFactory.)
(.setHost "localhost")))
;; and get it to make you a connection
(def connection (.newConnection connection-factory))
;; get that to make you a channel
(def channel (. connection createChannel))
;;HERE I WOULD LIKE TO USE THE SAME CONNECTION AND THE SAME CHANNEL INSTANCE AS OFTEN AS
;; I LIKE
(dotimes [ i 10 ]
(. channel basicPublish "" "hello" nil (. (format "Hello World! (%d)" i) getBytes)))
The clojure file above is part of a bigger clojure program that I build using lein. My problem is that when I compile with "lein compile", a connection is done because of the line (def connection (.newConnection connection-factory)) and then the compilation is stopped! How can I avoid this? Is there a way to compile without building connection? How can I manage to use the same instance of channel over several calls coming from external components?
Any help would be appreciated.
Regards,
Horace
The Clojure compiler must evaluate all top-level forms, because it can be required to run arbitrary code when expanding calls to macros.
The usual solution to issues like the one you describe is to define a top-level Var holding an object of a dereferenceable type, for example an atom or a promise, and have an initialization function provide the value at runtime. (You could also use a delay and specify the value inline; this is less flexible, since it makes it more difficult to use a different value for testing etc.)
Related
I was exploring Clojure java.jmx API Reference and trying the examples mentioned there e.g.
;; Can I serve my own beans? Sure, just drop a Clojure ref
;; into an instance of clojure.java.jmx.Bean, and the bean
;; will expose read-only attributes for every key/value pair
;; in the ref:
(jmx/register-mbean
(create-bean
(ref {:string-attribute "a-string"}))
"my.namespace:name=Value")
It works great, the bean's attribute value is visible in the console but it is read-only.
Is there a way to create a writeable bean(so that it is listed in "Operations" folder and manageable from console)?
It looks like the clojure.java.jmx code supports setAttribute.
(see (deftype Bean ...) in https://github.com/clojure/java.jmx/blob/master/src/main/clojure/clojure/java/jmx.clj
The easiest way seems to be just using an atom instead of a ref.
You could then have atom watchers to run some code if JMX changes it.
Maybe give that a try. I've forgotten most of JMX ;)
EDIT: Quickly tried it. It looks like the attribute is still read-only :(
I gotta look deeper. Nevertheless, the source code is pretty nice and hopefully easy to understand.
EDIT2:
The issue is with build-attribute-info which passes false to the writeable? flag in the `MBeanAttributeInfo.
You can monkey patch that:
(import 'java.jmx.MBeanAttributeInfo)
(require '[clojure.java.jmx :as jmx])
(defn build-my-attribute-info
"Construct an MBeanAttributeInfo. Normally called with a key/value pair from a Clojure map."
([attr-name attr-value]
(build-my-attribute-info
(name attr-name)
(.getName (class attr-value)) ;; you might have to map primitive types here
(name attr-name) true true false)) ;; false -> true compared to orig code
([name type desc readable? writable? is?] (println "Build info:" name type readable? writable?) (MBeanAttributeInfo. name type desc readable? writable? is? )))
;; the evil trick one should only use in tests, maybe
;; overwrite the original definition of build-attribute-info with yours
(with-redefs [jmx/build-attribute-info build-my-attribute-info]
(jmx/register-mbean (jmx/create-bean (atom {:foo "bar"})) "my.bean:name=Foo"))
;; write to it
(jmx/write! "my.bean:name=Foo" "foo" "hello world")
;; read it
(jmx/read "my.bean:name=Foo" "foo")
=> "hello world"
Now, unfortunately, Java Mission Control still can't change the value, but I'm not sure why. The MBean info is correct.
Must be a permission thing.
(defn my-func [opts]
(assoc opts :something :else))
What i want to be able to do, is serialize a reference to the function (maybe via #'my-func ?) to a string in such a way that i can upon deserializing it, invoke it with args.
How does this work?
Edit-- Why This is Not a Duplicate
The other question asked how to serialize a function body-- the entire function code. I am not asking how to do that. I am asking how to serialize a reference.
Imagine a cluster of servers all running the same jar, attached to a MQ. The MQ pubs in fn-reference and fn-args for functions in the jar, and the server in the cluster runs it and acks it. That's what i'm trying to do-- not pass function bodies around.
In some ways, this is like building a "serverless" engine in clojure.
Weirdly, a commit for serializing var identity was just added to Clojure yesterday: https://github.com/clojure/clojure/commit/a26dfc1390c53ca10dba750b8d5e6b93e846c067
So as of the latest master snapshot version, you can serialize a Var (like #'clojure.core/conj) and deserialize it on another JVM with access to the same loaded code, and invoke it.
(import [java.io File FileOutputStream FileInputStream ObjectOutputStream ObjectInputStream])
(defn write-obj [o f]
(let [oos (ObjectOutputStream. (FileOutputStream. (File. f)))]
(.writeObject oos o)
(.close oos)))
(defn read-obj [f]
(let [ois (ObjectInputStream. (FileInputStream. (File. f)))
o (.readObject ois)]
(.close ois)
o))
;; in one JVM
(write-obj #'clojure.core/conj "var.ser")
;; in another JVM
(read-obj "var.ser")
As suggested on the comments, if you can just serialize a keyword label for the function and store/retrieve that, you are finished.
If you need to transmit the function from one place to another, you essentially need to send the function source code as a string and then have it compiled via eval on the other end. This is what Datomic does when a Database Function is stored in the DB and automatically run by Datomic for any new additions/changes to the DB (these can perform automatic data validation, for example). See:
http://docs.datomic.com/database-functions.html
http://docs.datomic.com/clojure/index.html#datomic.api/function
As similar technique is used in the book Clojure in Action (1st Edition) for the distributed compute engine example using RabbitMQ.
While my lein new app project runs merrily inside the Light Table, lein uberjar won't work. Curiously, it behaves exactly like a classic Pascal compiler: it can't resolve references ahead of definitions. Another curiosity: yesterday it worked. I am not aware of fooling with anything sensitive.
Google says that the subj symptoms are quite commonplace; I tried whatever helped other people in the same (?) plight, but to no avail. By the way, usually they blame it on software bugs: "get the latest version of Leiningen and Clojure". I've got 2.5.0 and 1.6.
The project (main file) is here: https://github.com/Tyrn/pcc/blob/master/src/pcc/core.clj
As it is, parsed-args can't be resolved inside build-album; if I move the -main function to the top of the file, 'No such var' happens to cli-options inside -main. No amount of fiddling with explicit use of namespaces makes any difference.
Again, inside the Light Table everything runs fine.
Using def inside of a function is not idiomatic, especially if there is no reason to have it as a global variable. Just pass it as a function parameter:
(let [parsed-args (parse-opts ...)]
...
(build-album parsed-args))
If you really need global state, you can use e.g. a promise (alternatively, an atom):
(defonce parsed-args (promise))
...
(deliver parsed-args (parse-opts ...))
However, Clojure files are read from top to bottom, and yes, functions not having access to bindings introduced later in the file is by design. You can use declare to tell the parser what to expect:
(declare ^:dynamic *parsed-args*)
(defn build-album ...)
(def ^:dynamic *parsed-args* ...)
...
(binding [*parsed-args* (parse-opts ...)]
(build-album))
TL;DR: If not necessary, avoid global state; if necessary, minimize it.
I'm confused by how calls with carmine should be done. I found the wcar macro described in carmine's docs:
(defmacro wcar [& body] `(car/with-conn pool spec-server1 ~#body))
Do I really have to call wcar every time I want to talk to redis in addition to the redis command? Or can I just call it once at the beginning? If so how?
This is what some code with tavisrudd's redis library looked like (from my toy url shortener project's testsuite):
(deftest test_shorten_doesnt_exist_create_new_next
(redis/with-server test-server
(redis/set "url_counter" 51)
(shorten test-url)
(is (= "1g" (redis/get (str "urls|" test-url))))
(is (= test-url (redis/get "shorts|1g")))))
And now I can only get it working with carmine by writing it like this:
(deftest test_shorten_doesnt_exist_create_new_next
(wcar (car/set "url_counter" 51))
(shorten test-url)
(is (= "1g" (wcar (car/get (str "urls|" test-url)))))
(is (= test-url (wcar (car/get "shorts|1g")))))
So what's the right way of using it and what underlying concept am I not getting?
Dan's explanation is correct.
Carmine uses response pipelining by default, whereas redis-clojure requires you to ask for pipelining when you want it (using the pipeline macro).
The main reason you'd want pipelining is for performance. Redis is so fast that the bottleneck in using it is often the time it takes for the request+response to travel over the network.
Clojure destructuring provides a convenient way of dealing with the pipelined response, but it does require writing your code differently to redis-clojure. The way I'd write your example is something like this (I'm assuming your shorten fn has side effects and needs to be called before the GETs):
(deftest test_shorten_doesnt_exist_create_new_next
(wcar (car/set "url_counter" 51))
(shorten test-url)
(let [[response1 response2] (wcar (car/get (str "urls|" test-url))
(car/get "shorts|1g"))]
(is (= "1g" response1))
(is (= test-url response2))))
So we're sending the first (SET) request to Redis and waiting for the reply (I'm not certain if that's actually necessary here). We then send the next two (GET) requests at once, allow Redis to queue the responses, then receive them all back at once as a vector that we'll destructure.
At first this may seem like unnecessary extra effort because it requires you to be explicit about when to receive queued responses, but it brings a lot of benefits including performance, clarity, and composable commands.
I'd check out Touchstone on GitHub if you're looking for an example of what I'd consider idiomatic Carmine use (just search for the wcar calls). (Sorry, SO is preventing me from including another link).
Otherwise just pop me an email (or file a GitHub issue) if you have any other questions.
Don't worry, you're using it the correct way already.
The Redis request functions (such as the get and set that you're using above) are all routed through another function send-request! that relies on a dynamically bound *context* to provide the connection. Attempting to call any of these Redis commands without that context will fail with a "no context" error. The with-conn macro (used in wcar) sets that context and provides the connection.
The wcar macro is then just a thin wrapper around with-conn making the assumption that you will be using the same connection details for all Redis requests.
So far this is all very similar to how Tavis Rudd's redis-clojure works.
So, the question now is why does Carmine need multiple wcar's when Redis-Clojure only required a single with-server?
And the answer is, it doesn't. Apart from sometimes, when it does. Carmine's with-conn uses Redis's "Pipelining" to send multiple requests with the same connection and then package the responses together in a vector. The example from the README shows this in action.
(wcar (car/ping)
(car/set "foo" "bar")
(car/get "foo"))
=> ["PONG" "OK" "bar"]
Here you will see that ping, set and get are only concerned with sending the request, leaving the receiving of response up to wcar. This precludes asserts (or any result access) from inside of wcar and leads to the separation of requests and multiple wcar calls that you have.
I have a Clojure program that I build as a JAR file using Maven. Embedded in the JAR Manifest is a build-version number, including the build timestamp.
I can easily read this at runtime from the JAR Manifest using the following code:
(defn set-version
"Set the version variable to the build number."
[]
(def version
(-> (str "jar:" (-> my.ns.name (.getProtectionDomain)
(.getCodeSource)
(.getLocation))
"!/META-INF/MANIFEST.MF")
(URL.)
(.openStream)
(Manifest.)
(.. getMainAttributes)
(.getValue "Build-number"))))
but I've been told that it is bad karma to use def inside defn.
What is the Clojure-idiomatic way to set a constant at runtime? I obviously do not have the build-version information to embed in my code as a def, but I would like it set once (and for all) from the main function when the program starts. It should then be available as a def to the rest of the running code.
UPDATE: BTW, Clojure has to be one of the coolest languages I have come across in quite a while. Kudos to Rich Hickey!
I still think the cleanest way is to use alter-var-root in the main method of your application.
(declare version)
(defn -main
[& args]
(alter-var-root #'version (constantly (-> ...)))
(do-stuff))
It declares the Var at compile time, sets its root value at runtime once, doesn't require deref and is not bound to the main thread. You didn't respond to this suggestion in your previous question. Did you try this approach?
You could use dynamic binding.
(declare *version*)
(defn start-my-program []
(binding [*version* (read-version-from-file)]
(main))
Now main and every function it calls will see the value of *version*.
While kotarak's solution works very well, here is an alternative approach: turn your code into a memoized function that returns the version. Like so:
(def get-version
(memoize
(fn []
(-> (str "jar:" (-> my.ns.name (.getProtectionDomain)
(.getCodeSource)
(.getLocation))
"!/META-INF/MANIFEST.MF")
(URL.)
(.openStream)
(Manifest.)
(.. getMainAttributes)
(.getValue "Build-number")))))
I hope i dont miss something this time.
If version is a constant, it's going to be defined one time and is not going to be changed you can simple remove the defn and keep the (def version ... ) alone. I suppose you dont want this for some reason.
If you want to change global variables in a fn i think the more idiomatic way is to use some of concurrency constructions to store the data and access and change it in a secure way
For example:
(def *version* (atom ""))
(defn set-version! [] (swap! *version* ...))