When running a clojure core.clj as Clojure Application will the following method gets executed and "Test" gets printed?
(defn -main
[& args]
(println "Test")
I know a few of my friends who use Counterclockwise plugin on Eclipse.
Here is a link to their docs. It also has a step by step way to Create your first project
I am wondering how one goes about deploying a Common Lisp web application written in, say, Hunchentoot, Wookie, Woo, or even Clack.
That is, suppose I write an app that contains some files, packages, etc. Typically when I am working locally, I simply run a command in REPL that starts the server and then visit it using localhost:8000 or something like that.
However, I am a bit puzzled as to what the process is for deploying an app to a production server like AWS EC2. In what form should I deploy the Lisp code? Are there different options? What happens if the server needs to be restarted or is experiencing problems?
I've figured something out lately by building self-contained executables for web apps and I wrote about it on lisp-journey/web-dev (shipping and deployment sections), as well as for the building part on the Common Lisp Cookbook/scripting#for-web-apps.
I copy the interesting parts here, there's a bit more on each resource. Edits are welcome, primarily on those resources thanks !
edit july 2019: I contributed a page on the Cookbook: https://lispcookbook.github.io/cl-cookbook/web.html
edit: see also a list of tools and platforms that provide professional CL support: https://github.com/CodyReichert/awesome-cl#deployment
(edited) How to run the web app as a script
I explain below how to build and run executables, but we can of course run the application as a script. In a lisp file, say run.lisp, ensure:
to load your project's asd file: (load "my-project.asd")
to load its dependencies: (ql:quickload :my-project)
to call its main function: (my-project:start) (given start is an exported symbol, otherwise ::start).
In doing so, the application starts and gives you back a Lisp REPL. You can interact with the running application. You can update it and even install new Quicklisp libraries as it runs.
How to build a self-contained executable
See also https://github.com/CodyReichert/awesome-cl#interfaces-to-other-package-managers for bindings to Homebrew and Debian packages.
With SBCL
How to build (self-contained) executables is implementation-specific (see
below Buildapp and Rowsell). With SBCL, as says
its documentation,
it is a matter of:
(sb-ext:save-lisp-and-die #P"path/name-of-executable" :toplevel #'my-app:main-function :executable t)
sb-ext is an SBCL extension to run external processes. See other
SBCL extensions
(many of them are made implementation-portable in other libraries).
:executable t tells to build an executable instead of an
image. We could build an image to save the state of our current
Lisp image, to come back working with it later. Specially useful if
we made a lot of work that is computing intensive.
If you try to run this in Slime, you'll get an error about threads running:
Cannot save core with multiple threads running.
Run the command from a simple SBCL repl.
I suppose your project has Quicklisp dependencies. You must then:
ensure Quicklisp is installed and loaded at Lisp startup (you
completed Quicklisp installation)
load the project's .asd
install dependencies
build the executable.
That gives:
(load "my-app.asd")
(ql:quickload :my-app)
(sb-ext:save-lisp-and-die #p"my-app-binary" :toplevel #'my-app:main :executable t)
From the command line, or from a Makefile, use --load and --eval:
build:
sbcl --non-interactive \
--load my-app.asd \
--eval '(ql:quickload :my-app)' \
--eval "(sb-ext:save-lisp-and-die #p\"my-app\" :toplevel #my-app:main :executable t)"
With ASDF
Now that we'seen the basics, we need a portable method. Since its
version 3.1, ASDF allows to do that. It introduces the make command,
that reads parameters from the .asd. Add this to your .asd declaration:
:build-operation "program-op" ;; leave as is
:build-pathname "<binary-name>"
:entry-point "<my-system:main-function>"
and call asdf:make :my-system.
So, in a Makefile:
LISP ?= sbcl
build:
$(LISP) --non-interactive \
--load my-app.asd \
--eval '(ql:quickload :my-app)' \
--eval '(asdf:make :my-system)'
With Roswell or Buildapp
Roswell, an implementation manager and much
more, also has the ros build command, that should work for many
implementations.
We can also make our app installable with Roswell by a ros install my-app. See its documentation.
We'll finish with a word on
Buildapp, a battle-tested and
still popular "application for SBCL or CCL that configures and saves
an executable Common Lisp image".
Many applications use it (for example,
pgloader), it is available on
Debian: apt install buildapp, but you shouldn't need it now with asdf:make or Roswell.
For web apps
We can similarly build a self-contained executable for our web-app. It
would thus contain a web server and would be able to run on the
command line:
$ ./my-web-app
Hunchentoot server is started.
Listening on localhost:9003.
Note that this runs the production webserver, not a development one,
so we can run the binary on our VPS right away and access the app from
outside.
We have one thing to take care of, it is to find and put the thread of
the running web server on the foreground. In our main function, we
can do something like this:
(defun main ()
(start-app :port 9003) ;; our start-app, for example clack:clack-up
;; let the webserver run.
;; warning: hardcoded "hunchentoot".
(handler-case (bt:join-thread (find-if (lambda (th)
(search "hunchentoot" (bt:thread-name th)))
(bt:all-threads)))
;; Catch a user's C-c
(#+sbcl sb-sys:interactive-interrupt
#+ccl ccl:interrupt-signal-condition
#+clisp system::simple-interrupt-condition
#+ecl ext:interactive-interrupt
#+allegro excl:interrupt-signal
() (progn
(format *error-output* "Aborting.~&")
(clack:stop *server*)
(uiop:quit)))
(error (c) (format t "Woops, an unknown error occured:~&~a~&" c))))
We used the bordeaux-threads library ((ql:quickload "bordeaux-threads"), alias bt) and uiop, which is part of ASDF so
already loaded, in order to exit in a portable way (uiop:quit, with
an optional return code, instead of sb-ext:quit).
Parsing command line arguments
see the Cookbook here. TLDR; use uiop:command-line-arguments to get a list of the arguments. To parse them for real, there are libraries.
Deployment
Straightforward with an executable. The web app is visible from the outside right away.
On Heroku
See this buildpack.
Daemonizing, restarting in case of crashes, handling logs
See how to do that on your system.
Most GNU/Linux distros now come with Systemd.
Examples search result:
https://seanmcgary.com/posts/deploying-nodejs-applications-with-systemd/
It is as simple as writing a configuration file:
# /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example
[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/sthg sthg
Type=simple
Restart=always
RestartSec=10
running a command to start it:
sudo systemctl start my-app.service
a command to check its status:
systemctl status my-app.service
and Systemd can handle logging (we write to stdout or stderr, it writes logs):
journalctl -f -u my-app.service
and it handles crashes and restarts the app:
Restart=always
and it can start the app after a reboot:
[Install]
WantedBy=basic.target
to enable it:
sudo systemctl enable my-app.service
Debugging SBCL error: ensure_space: failed to allocate n bytes
If you get this error with SBCL on your server:
mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000
ensure_space: failed to allocate 1040384 bytes at 0x20000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)
then disable ASLR:
sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"
Connecting to a remote Swank server
Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.
Demo project here: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/
It defines a simple function that prints forever:
;; a little common lisp swank demo
;; while this program is running, you can connect to it from another terminal or machine
;; and change the definition of doprint to print something else out!
;; (ql:quickload :swank)
;; (ql:quickload :bordeaux-threads)
(require :swank)
(require :bordeaux-threads)
(defparameter *counter* 0)
(defun dostuff ()
(format t "hello world ~a!~%" *counter*))
(defun runner ()
(bt:make-thread (lambda ()
(swank:create-server :port 4006)))
(format t "we are past go!~%")
(loop while t do
(sleep 5)
(dostuff)
(incf *counter*)
))
(runner)
On our server, we run it with
sbcl --load demo.lisp
we do port forwarding on our development machine:
ssh -L4006:127.0.0.1:4006 username#example.com
this will securely forward port 4006 on the server at example.com to
our local computer's port 4006 (swanks accepts connections from
localhost).
We connect to the running swank with M-x slime-connect, typing in
port 4006.
We can write new code:
(defun dostuff ()
(format t "goodbye world ~a!~%" *counter*))
(setf *counter* 0)
and eval it as usual with M-x slime-eval-region for instance. The output should change.
There are more pointers on CV Berry's page.
Hot reload
Example with Quickutil. See notes on lisp-journey.
It has to be run on the server (a simple fabfile command can call this
through ssh). Beforehand, a fab update has run git pull on the
server, so new code is present but not running. It connects to the
local swank server, loads the new code, stops and starts the app in a
row.
Continuous Integration, continuous delivery of executables, Docker
See https://lispcookbook.github.io/cl-cookbook/testing.html#continuous-integration
To run a lisp image in production, you can generate a fasl file from your lisp code with:
(compile-file "app.lisp")
run your generated .fas file by invoking sbcl.
sbcl --noinform \
--load app.fas \
--eval "(defun main (argv) (declare (ignore argv)) (hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4242)))"
I found a blog with a solution that I have adapted to my needs for a production system on a linux box. Unfortunately I cannot find the reference to that blog anymore, so that I can just show you my solution, which is for CCL (while the original solution was for SBCL), which with I am more familiar. Here is the program that starts the system:
(require 'swank)
(require 'hunchentoot)
(defparameter *httpd-port* 9090) ; The port Hunchentoot will be listening on
(defparameter *shutdown-port* 6700) ; The port CCL will be listening for shutdown
; this port is the same used in /etc/init.d/hunchentoot
(defparameter *swank-port* 5016) ; The port used for remote interaction with slime
;; Start the Swank server
(defparameter *swank-server*
(swank:create-server :port *swank-port* :dont-close t))
(require 'YOUR-PACKAGE)
(YOUR-PACKAGE:YOUR-STARTING-FUNCTION)
(princ "Hunchentoot started on port ")
(princ *httpd-port*)
(terpri)
(let* ((socket (make-socket :connect :passive :local-host "127.0.0.1" :local-port *shutdown-port* :reuse-address t))
(stream (accept-connection socket)))
(close stream)
(close socket))
(print "Stopping Hunchentoot...")
(YOUR-PACKAGE:YOUR-STOPPING-FUNCTION)
(dolist (proc (all-processes))
(unless (equal proc *current-process*)
(process-kill proc)))
(sleep 1)
(quit)
The idea is that you can connect to the running system with slime, by specifying the port used by swank. I used it a couple of times, for instance to change a database link on the fly, and was quite impressed by the power of such possibility.
The running system can be terminated by:
telnet 127.0.0.1 6700
and initiated by something like:
nohup ccl -l initcclserver.lisp >& server.out &
In a previous version of the script I found the SBCL-specific parts, so if you use it you could modify the script.
For accepting the terminating connections:
(sb-bsd-sockets:socket-bind socket #(127 0 0 1) *shutdown-port*)
(sb-bsd-sockets:socket-listen socket 1)
(multiple-value-bind (client-socket addr port)
(sb-bsd-sockets:socket-accept socket)
(sb-bsd-sockets:socket-close client-socket)
(sb-bsd-sockets:socket-close socket)))
To close the system:
(dolist (thread (sb-thread:list-all-threads))
(unless (equal sb-thread:*current-thread* thread)
(sb-thread:terminate-thread thread)))
(sleep 1)
(sb-ext:quit)
Hope this can help.
I am currently trying to learn Clojure and as part of my practical training, I am
implementing the very basic behavior of some of the well known Unix tools like grep,
cat, ls and so on.
While implementing cat, I stumbled upon some seemingly strange behavior of
slurp. When I run the following code with lein run some-file.txt while some-file.txt
lies within the current directory, the content is printed to STDOUT as expected.
(ns cat.core
(:gen-class))
(defn -main
"Reads the content of its arguments representing filenames and outputs the
content in succession."
[& filenames]
(doseq [filename filenames]
(println "Reading" filename) ; Just for debugging purposes
(print (slurp filename))))
However, if I uberjar the project with
lein compile
lein uberjar
and then cd to target/uberjar to run the standalone JAR with
java -jar cat-0.1.0-SNAPSHOT-standalone.jar some-file.txt (with the text file present in that directory),
nothing but my debug message gets printed. What I find so strange is that there is no error
message being shown, so it seems to me that the file can be found. If I run the
JAR with a file that does not exist as parameter, I get an exeption that the specified
file cannot be found (as expected).
Because I am working on a Windows machine, a colleague suggested that perhaps Windows
shadow files might be a problem. So I tested the program under Linux again and the
same behavior occured. So this seems to be a "problem" with my Clojure understanding / my project
settings.
My question is: Why is slurp's (or the program's) behavior different when running
with lein run and when running the standalone JAR with java -jar cat-0.1.0-SNAPSHOT-standalone.jar and what can I do to solve this?
As the file parameter is not a resource that is compiled into the JAR, there is no
need for (slurp (clojure.java.io/resource filename)) if I am not mistaken.
print does not flush the output buffer. You need to use flush after your print statement, or use another println, which flushes on newline:
(defn -main
"Reads the content of its arguments representing filenames and outputs the
content in succession."
[& filenames]
(doseq [filename filenames]
(print (slurp filename))
(flush)))
I want to specify the Clojure version for my project that uses Boot. According to the Boot Wiki, the way to do this is by providing a value for BOOT_CLOJURE_VERSION in a boot.properties file in my project root. So I did that:
$ cat boot.properties
BOOT_CLOJURE_VERSION=1.7.0
It seems to work just fine:
$ tail -2 ~/.boot/boot.properties
BOOT_VERSION=2.5.5
BOOT_CLOJURE_VERSION=1.8.0
$ cat build.boot
(deftask version []
(println "Clojure" (clojure-version))
(println "Boot core" *boot-version*)
(println "Boot app" *app-version*))
$ boot version
Clojure 1.7.0
Boot core 2.5.5
Boot app 2.5.5
However, that same wiki page specifically says to create the boot.properties file like this:
$ boot -V > boot.properties
This adds two lines at the beginning, which look like comments to me, and one at the end that specifies the Boot version. I have no problem with specifying the Boot version for my project, but the wiki page makes it sound as if it's required:
Note: When using boot.properties file you have to also pin the project to specific Boot version as the file must specify both variables.
I'm a bit confused about why the page specifically says to add these three lines to boot.properties when omitting them doesn't seem to cause any problems. Additionally, if I'm using revision control, I see no need to put a timestamp in boot.properties. Is it OK to omit these lines? If not, why are they needed?
This is most likely a case of outdated wiki information. From the source:
// BOOT_VERSION is decided by the loader; it will respect the
// boot.properties files, env vars, system properties, etc.
// or it will use the latest installed version.
I guess you could consider it good practice to lock both the Clojure and Boot version per project as this will prevent any future problems that may arise from incompatible versions.
The 2 comment lines added by Boot are just for information purposes and can be safely omitted.
I found while adding warnings to code that writes macros that the body of a file was being executed twice during compilation. Is there a reason for this? Is this specific to leiningen? I can not reproduce this with (compile ...).
Simplified version:
(ns foo.core
(:require foo.bar))
;; empty
(ns foo.bar)
(println "hello")
$ lein compile :all
Compiling foo.core
hello
Compiling foo.bar
hello
Further testing shows that the namespace is reloaded on top of itself during compile:
(ns foo.bar)
(declare wasd)
(println wasd)
(def wasd 2)
$ lein clean
$ lein compile :all
Compiling foo.core
#<Unbound Unbound: #'foo.bar/wasd>
Compiling foo.bar
2
In a more complicated case I have this happening during compile, and then once every time when run or started a repl from lein. I am not sure why. This is all with clojure 1.6 and leiningen 2.5.0.
Leiningen knows nothing about the structure of your project in terms of how namespaces relate to each-other. Consequently, when compiling a project, lein simply boots a JVM and forces each namespace to be loaded one at a time. This means that, as you noticed, namespaces will be reloaded causing the double evaluation behaviour you are noticing.
In comparison, (clojure.core/compile) simply loads the targeted resource with clojure.core/*compile-files* bound. This will cause the targeted resource and all the resources it requires to be loaded and compiled to class-files. However, it will not traverse your entire project structure compiling all resources as Leiningen's compile operation does.
The reason why you're seeing println output during compile time is because println is called during namespace evaluation. You should have a -main fn or some other entry point for your program that calls on println.
(defn -main [& _]
(println "Won't see on compile.")
(println "lein run -- is printing from -main"))
I think your project is throwing an Unbound Exception because you're trying to reference the wasd variable with println before a value is assigned to it. The variable was declared, but nothing was assigned to it before the println fn tried to get the value.