A Clojure-based project of mine uses the netty (required by aleph) web server. I start the server, along with other components, in a web.clj file like this:
(ns myproject.web)
(def server (atom nil))
(defn initialize []
(if #server
(println "Warning: already initialized")
(let [port 8001]
(println (format "Starting http://localhost:%s/" port))
(swap! server (fn [_] (start-http-server
(wrap-ring-handler app-routes)
{:port port}))))))
(defn shutdown []
(when #server
(do
(println "Shutting down web server")
(#server)
(swap! server (fn [_] nil)))))
(defn reinitialize []
"Run this on the REPL to reload web.clj and restart the web server"
(myproject.web/shutdown)
(use :reload-all 'myproject.web)
(myproject.web/initialize))
The server instance is stored in a Clojure atom, so that it can be stopped later.
I use Emacs and Swank to directly launch the server on the REPL like this (after compiling web.clj with C-c C-k):
user> (myproject.web/initialize)
Whenever web.clj or other dependent modules are edited, I must
remember NOT to recompile web.clj using C-c C-k because the atom holding the running instance would vanish (due to the atom from a newly compiled module) from the REPL.
Run (myproject.web/reinitialize) which stops the server and then reloads the module before starting it again.
There are two problems with this:
Often I forget point #1 and press C-c C-k anyway. This causes the loss of server atom in REPL, leading to having to kill swank (or restart emacs) so that I can start the server at the same port number.
:reload-all doesn't report compile errors as friendly as C-c C-k (ugly traceback vs concise clickable errors).
How can I best address these two problems in this edit-compile-restart workflow?
You can replace
(def server (atom nil))
with
(defonce server (atom nil))
that way when you evaluate the buffer it will not redefine server.
for your first problem you can store the atom in a different namespace and on load only overwrite it if it is not yet defined. putting it in it's own namepspace will prevent it from being erased by the reload-all
Related
How does one send a message to a running clojure application? For example, if I have a particular variable or flag I want to set from the terminal when the uberjar is running - is this possible?
One way is to read a file in the application that you can change, but that sounds clunky.
Thanks in advance!
One way to do this is by having your application host a nREPL (network REPL). You can then connect to your running app's nREPL and mess around.
For example, your app may look like this:
(ns sandbox.main
(:require [clojure.tools.nrepl.server :as serv]))
(def value (atom "Hey!"))
(defn -main []
(serv/start-server :port 7888)
(while true
(Thread/sleep 1000)
(prn #value)))
While that's running you can lein repl :connect localhost:7888 from elsewhere and change that value atom:
user=> (in-ns 'sandbox.main)
#object[clojure.lang.Namespace 0x12b094cf "sandbox.main"]
sandbox.main=> (reset! value "Bye!")
"Bye!"
By this time the console output of your app should look like this:
$ lein run
"Hey!"
"Hey!"
<...truncated...>
"Bye!"
"Bye!"
There are many options for interprocess communication on the JVM, but this approach is unique to Clojure.
With the mount library, how do I reload (stop and start) an http-kit "mount state" on a -main function?
My current code is this:
(defstate server-config :start {:port 7890 :join? false})
(defn start-server [server-config]
(when-let [server (run-server myserv-ring-handler server-config)]
(println "Server has started!")
server))
(defstate myserv-server :start (start-server server-config)
:stop (myserv-server :timeout 100))
(defn system-port [args]
(Integer/parseInt
(or (System/getenv "PORT")
(first args)
"7890")))
(defn -main [& args]
(mount/start-with-states
{#'myserv/server-config
{:start #(array-map :port (system-port args)
:join? false)}}))
So when I "lein run" everything works, but whenever I change a file, and the http-kit server is stopped, the command stops. For the moment I'm doing "while true; do lein run; done" to work, so I've thought about adding an infinite loop to the -main function, but it doesn't feel like this is the right way.
How should I do this?
I would suggest adding some metadata to your http server defstate.
From the mount readme:
In case nothing needs to be done to a running state on reload /
recompile / redef, set :on-reload to :noop:.
So try something like this:
(defstate ^{:on-reload :noop}
myserv-server
:start (start-server server-config)
:stop (my-stop-func myserv-server))
This means that when you change a file, the affected code will be reloaded, but the http server will continue to run.
I hope that I've correctly understood your question and that this is what you wanted.
Could I also suggest that if you want to get up and running quickly, then there are various templated web app projects for Leiningen. For example, the Luminus project. You can pass an +http-kit parameter to the lein new luminus myapp command and that will wire up an app correctly for you. You can then go and read the generated code and learn how it all fits together.
So I had a couple of separate problems:
I didn't understand that now I didn't need to use lein run, but instead I could just do lein repl and start the server from there. The restarting problem is avoided that way.
The other one was that I was misusing start-with-states instead of a config state.
You can see a discussion about this with the author of the library here.
I am creating an agent for writing changes back to a database (as discussed in the agent-based write-behind log in Clojure Programming).
This is working fine, but I am struggling to create the agent late. I don't want to create it as a def, as I don't want it created when my tests are running (I see the pool starting up when the tests load the forms even though I use with-redefs to set a test value).
The code I started with is (using c3p0 pooling):
(def dba (agent (pool/make-datasource-spec (u/load-config "db.edn"))))
I tried making the agent nil, and investigated how I could set it in the main of my application, when it's really needed. But there doesn't seem to be an equivalent reset! function as there is with an atom. And the following code also failed saying the agent wasn't in error so didn't need restarting:
(when (not #dba)
(restart-agent dba (create-db-pool)))
So at the moment, I have an atom containing the agent, where I then do:
(def dba (atom nil))
;; ...
(defn init-db! []
(when (not #dba)
(log/info "Creating agent for pooled connection")
(reset! dba (agent (create-db-pool))))
But the very fact I'm having to do ##dba to reference the contents of the agent (i.e. the pool) makes me think this is insane.
Is there a more obvious way of creating the pool agent lazily?
delay is useful for cases like this. It causes the item to be created the first time it is read. so if your tests don't read it it will not be created.
user=> (def my-agent (delay (do (println "im making the agent now") (agent 0))))
#'user/my-agent
user=>
user=> #my-agent
im making the agent now
#object[clojure.lang.Agent 0x2cd73ca1 {:status :ready, :val 0}]
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm struggling to get my head around how to use Stuart Sierra's component library within a Clojure app. From watching his Youtube video, I think I've got an OK grasp of the problems that led to him creating the library; however I'm struggling to work out how to actually use it on a new, reasonably complex project.
I realise this sounds very vague, but it feels like there's some key concept that I'm missing, and once I understand it, I'll have a good grasp on how to use components. To put it another way, Stuart's docs and video go into the WHAT and WHY of components in considerable detail, but I'm missing the HOW.
Is there any sort of detailed tutorial/walkthrough out there that goes into:
why you'd use components at all for a non-trivial Clojure app
a methodology for how you'd break down the functionality in a non-trivial Clojure app, such that components can be implemented in a reasonably optimal fashion. It's reasonably simple when all you've got is e.g. a database, an app server and a web server tier, but I'm struggling to grasp how you'd use it for a system that has many different layers that all need to work together coherently
ways to approach development/testing/failover/etc. in a non-trivial Clojure app that's been built using components
Thanks in advance
In short, Component is a specialized DI framework. It can set up an injected system given two maps: the system map and the dependency map.
Let's look at a made-up web app (disclaimer, I typed this in a form without actually running it):
(ns myapp.system
(:require [com.stuartsierra.component :as component]
;; we'll talk about myapp.components later
[myapp.components :as app-components]))
(defn system-map [config] ;; it's conventional to have a config map, but it's optional
(component/system-map
;; construct all components + static config
{:db (app-components/map->Db (:db config))
:handler (app-components/map->AppHandler (:handler config))
:server (app-components/map->Server (:web-server config))}))
(defn dependency-map
;; list inter-dependencies in either:
;; {:key [:dependency1 :dependency2]} form or
;; {:key {:name-arg1 :dependency1
;; :name-arg2 :dependency2}} form
{:handler [:db]
:server {:app :handler})
;; calling this creates our system
(def create-system [& [config]]
(component/system-using
(system-map (or config {})
(dependency-map)))
This allows us to call (create-system) to create a new instance of our entire application when we need one.
Using (component/start created-system), we can run a system's services it provides. In this case, it's the webserver that's listening on a port and an open db connection.
Finally, we can stop it with (component/stop created-system) to stop the system from running (eg - stop the web server, disconnect from db).
Now let's look at our components.clj for our app:
(ns myapp.components
(:require [com.stuartsierra.component :as component]
;; lots of app requires would go here
;; I'm generalizing app-specific code to
;; this namespace
[myapp.stuff :as app]))
(defrecord Db [host port]
component/Lifecycle
(start [c]
(let [conn (app/db-connect host port)]
(app/db-migrate conn)
(assoc c :connection conn)))
(stop [c]
(when-let [conn (:connection c)]
(app/db-disconnect conn))
(dissoc c :connection)))
(defrecord AppHandler [db cookie-config]
component/Lifecycle
(start [c]
(assoc c :handler (app/create-handler cookie-config db)))
(stop [c] c))
;; you should probably use the jetty-component instead
;; https://github.com/weavejester/ring-jetty-component
(defrecord Server [app host port]
component/Lifecycle
(start [c]
(assoc c :server (app/create-and-start-jetty-server
{:app (:handler app)
:host host
:port port})))
(stop [c]
(when-let [server (:server c)]
(app/stop-jetty-server server)
(dissoc c :server)))
So what did we just do? We got ourselves a reloadable system. I think some clojurescript developers using figwheel start seeing similarities.
This means we can easily restart our system after we reload code. On to the user.clj!
(ns user
(:require [myapp.system :as system]
[com.stuartsierra.component :as component]
[clojure.tools.namespace.repl :refer (refresh refresh-all)]
;; dev-system.clj only contains: (def the-system)
[dev-system :refer [the-system]])
(def system-config
{:web-server {:port 3000
:host "localhost"}
:db {:host 3456
:host "localhost"}
:handler {cookie-config {}}}
(def the-system nil)
(defn init []
(alter-var-root #'the-system
(constantly system/create-system system-config)))
(defn start []
(alter-var-root #'the-system component/start))
(defn stop []
(alter-var-root #'the-system
#(when % (component/stop %))))
(defn go []
(init)
(start))
(defn reset []
(stop)
(refresh :after 'user/go))
To run a system, we can type this in our repl:
(user)> (reset)
Which will reload our code, and restart the entire system. It will shutdown the exiting system that is running if its up.
We get other benefits:
End to end testing is easy, just edit the config or replace a component to point to in-process services (I've used it to point to an in-process kafka server for tests).
You can theoretically spawn your application multiple times for the same JVM (not really as practical as the first point).
You don't need to restart the REPL when you make code changes and have to restart your server
Unlike ring reload, we get a uniform way to restart our application regardless of its purpose: a background worker, microservice, or machine learning system can all be architected in the same way.
It's worth noting that, since everything is in-process, Component does not handle anything related to fail-over, distributed systems, or faulty code ;)
There are plenty of "resources" (aka stateful objects) that Component can help you manage within a server:
Connections to services (queues, dbs, etc.)
Passage of Time (scheduler, cron, etc.)
Logging (app logging, exception logging, metrics, etc.)
File IO (blob store, local file system, etc.)
Incoming client connections (web, sockets, etc.)
OS Resources (devices, thread pools, etc.)
Component can seem like overkill if you only have a web server + db. But few web apps are just that these days.
Side Note: Moving the-system into another namespace reduces the likelihood of refreshing the the-system var when developing (eg - calling refresh instead of reset).
The following program, when run from an überjar, exits at the end only when using the in-memory Datomic database; when connecting to the Datomic server, it hangs indefinitely rather than exiting the JVM:
(ns myns.example
(:use [datomic.api :only [db q] :as d])
(:gen-class))
;; WORKS: (def uri "datomic:mem://testdb")
(def uri "datomic:free://localhost:4334/testdb2")
(defn -main []
(println 1)
(when (d/create-database uri)
(d/connect uri))
(shutdown-agents)
(println 2))
Run as:
lein uberjar && java -cp target/myns-0.1.0-SNAPSHOT-standalone.jar myns.example
Outputs:
1
2
and hangs. It only hangs if the DB doesn't exist when the program starts.
Anyone know why, or how to fix? This is with both datomic-free-0.8.4020.26 and datomic-free-0.8.3941.
UPDATE -- the above program does actually terminate, but it takes a very long time (> 1 minute). I'd like to know why.
shutdown-agents takes up to one minute to complete (assuming no agents are running an action).
This is due to the way java.util.concurrent cached thread pools work.
Use datomic.api/shutdown
shutdown
function
Usage: (shutdown shutdown-clojure)
Shut down all peer
resources. This method should be called as part of clean shutdown of
a JVM process. Will release all Connections, and, if shutdown-clojure
is true, will release Clojure resources. Programs written in Clojure
can set shutdown-clojure to false if they manage Clojure resources
(e.g. agents) outside of Datomic; programs written in other JVM
languages should typically set shutdown-clojure to true.
Added in Datomic Clojure version 0.8.3861
(ns myns.example
(:require [datomic.api :as d])
(:gen-class))
(def uri "datomic:free://localhost:4334/testdb2")
(defn -main []
(d/create-database uri)
(let [conn (d/connect uri)]
(try
;; do something
(finally (d/shutdown true)))