Deploying Common Lisp Web Applications - amazon-web-services

Question

I am wondering how one goes about deploying a Common Lisp web application written in, say, Hunchentoot, Wookie, Woo, or even Clack.
That is, suppose I write an app that contains some files, packages, etc. Typically when I am working locally, I simply run a command in REPL that starts the server and then visit it using localhost:8000 or something like that.
However, I am a bit puzzled as to what the process is for deploying an app to a production server like AWS EC2. In what form should I deploy the Lisp code? Are there different options? What happens if the server needs to be restarted or is experiencing problems?

Answer 1

I've figured something out lately by building self-contained executables for web apps and I wrote about it on lisp-journey/web-dev (shipping and deployment sections), as well as for the building part on the Common Lisp Cookbook/scripting#for-web-apps.
I copy the interesting parts here, there's a bit more on each resource. Edits are welcome, primarily on those resources thanks !
edit july 2019: I contributed a page on the Cookbook: https://lispcookbook.github.io/cl-cookbook/web.html
edit: see also a list of tools and platforms that provide professional CL support: https://github.com/CodyReichert/awesome-cl#deployment
(edited) How to run the web app as a script
I explain below how to build and run executables, but we can of course run the application as a script. In a lisp file, say run.lisp, ensure:
to load your project's asd file: (load "my-project.asd")
to load its dependencies: (ql:quickload :my-project)
to call its main function: (my-project:start) (given start is an exported symbol, otherwise ::start).
In doing so, the application starts and gives you back a Lisp REPL. You can interact with the running application. You can update it and even install new Quicklisp libraries as it runs.
How to build a self-contained executable
See also https://github.com/CodyReichert/awesome-cl#interfaces-to-other-package-managers for bindings to Homebrew and Debian packages.
With SBCL
How to build (self-contained) executables is implementation-specific (see
below Buildapp and Rowsell). With SBCL, as says
its documentation,
it is a matter of:
(sb-ext:save-lisp-and-die #P"path/name-of-executable" :toplevel #'my-app:main-function :executable t)
sb-ext is an SBCL extension to run external processes. See other
SBCL extensions
(many of them are made implementation-portable in other libraries).
:executable t tells to build an executable instead of an
image. We could build an image to save the state of our current
Lisp image, to come back working with it later. Specially useful if
we made a lot of work that is computing intensive.
If you try to run this in Slime, you'll get an error about threads running:
Cannot save core with multiple threads running.
Run the command from a simple SBCL repl.
I suppose your project has Quicklisp dependencies. You must then:
ensure Quicklisp is installed and loaded at Lisp startup (you
completed Quicklisp installation)
load the project's .asd
install dependencies
build the executable.
That gives:
(load "my-app.asd")
(ql:quickload :my-app)
(sb-ext:save-lisp-and-die #p"my-app-binary" :toplevel #'my-app:main :executable t)
From the command line, or from a Makefile, use --load and --eval:
build:
sbcl --non-interactive \
--load my-app.asd \
--eval '(ql:quickload :my-app)' \
--eval "(sb-ext:save-lisp-and-die #p\"my-app\" :toplevel #my-app:main :executable t)"
With ASDF
Now that we'seen the basics, we need a portable method. Since its
version 3.1, ASDF allows to do that. It introduces the make command,
that reads parameters from the .asd. Add this to your .asd declaration:
:build-operation "program-op" ;; leave as is
:build-pathname "<binary-name>"
:entry-point "<my-system:main-function>"
and call asdf:make :my-system.
So, in a Makefile:
LISP ?= sbcl
build:
$(LISP) --non-interactive \
--load my-app.asd \
--eval '(ql:quickload :my-app)' \
--eval '(asdf:make :my-system)'
With Roswell or Buildapp
Roswell, an implementation manager and much
more, also has the ros build command, that should work for many
implementations.
We can also make our app installable with Roswell by a ros install my-app. See its documentation.
We'll finish with a word on
Buildapp, a battle-tested and
still popular "application for SBCL or CCL that configures and saves
an executable Common Lisp image".
Many applications use it (for example,
pgloader), it is available on
Debian: apt install buildapp, but you shouldn't need it now with asdf:make or Roswell.
For web apps
We can similarly build a self-contained executable for our web-app. It
would thus contain a web server and would be able to run on the
command line:
$ ./my-web-app
Hunchentoot server is started.
Listening on localhost:9003.
Note that this runs the production webserver, not a development one,
so we can run the binary on our VPS right away and access the app from
outside.
We have one thing to take care of, it is to find and put the thread of
the running web server on the foreground. In our main function, we
can do something like this:
(defun main ()
(start-app :port 9003) ;; our start-app, for example clack:clack-up
;; let the webserver run.
;; warning: hardcoded "hunchentoot".
(handler-case (bt:join-thread (find-if (lambda (th)
(search "hunchentoot" (bt:thread-name th)))
(bt:all-threads)))
;; Catch a user's C-c
(#+sbcl sb-sys:interactive-interrupt
#+ccl ccl:interrupt-signal-condition
#+clisp system::simple-interrupt-condition
#+ecl ext:interactive-interrupt
#+allegro excl:interrupt-signal
() (progn
(format *error-output* "Aborting.~&")
(clack:stop *server*)
(uiop:quit)))
(error (c) (format t "Woops, an unknown error occured:~&~a~&" c))))
We used the bordeaux-threads library ((ql:quickload "bordeaux-threads"), alias bt) and uiop, which is part of ASDF so
already loaded, in order to exit in a portable way (uiop:quit, with
an optional return code, instead of sb-ext:quit).
Parsing command line arguments
see the Cookbook here. TLDR; use uiop:command-line-arguments to get a list of the arguments. To parse them for real, there are libraries.
Deployment
Straightforward with an executable. The web app is visible from the outside right away.
On Heroku
See this buildpack.
Daemonizing, restarting in case of crashes, handling logs
See how to do that on your system.
Most GNU/Linux distros now come with Systemd.
Examples search result:
https://seanmcgary.com/posts/deploying-nodejs-applications-with-systemd/
It is as simple as writing a configuration file:
# /etc/systemd/system/my-app.service
[Unit]
Description=stupid simple example
[Service]
WorkingDirectory=/path/to/your/app
ExecStart=/usr/local/bin/sthg sthg
Type=simple
Restart=always
RestartSec=10
running a command to start it:
sudo systemctl start my-app.service
a command to check its status:
systemctl status my-app.service
and Systemd can handle logging (we write to stdout or stderr, it writes logs):
journalctl -f -u my-app.service
and it handles crashes and restarts the app:
Restart=always
and it can start the app after a reboot:
[Install]
WantedBy=basic.target
to enable it:
sudo systemctl enable my-app.service
Debugging SBCL error: ensure_space: failed to allocate n bytes
If you get this error with SBCL on your server:
mmap: wanted 1040384 bytes at 0x20000000, actually mapped at 0x715fa2145000
ensure_space: failed to allocate 1040384 bytes at 0x20000000
(hint: Try "ulimit -a"; maybe you should increase memory limits.)
then disable ASLR:
sudo bash -c "echo 0 > /proc/sys/kernel/randomize_va_space"
Connecting to a remote Swank server
Little example here: http://cvberry.com/tech_writings/howtos/remotely_modifying_a_running_program_using_swank.html.
Demo project here: https://lisp-journey.gitlab.io/blog/i-realized-that-to-live-reload-my-web-app-is-easy-and-convenient/
It defines a simple function that prints forever:
;; a little common lisp swank demo
;; while this program is running, you can connect to it from another terminal or machine
;; and change the definition of doprint to print something else out!
;; (ql:quickload :swank)
;; (ql:quickload :bordeaux-threads)
(require :swank)
(require :bordeaux-threads)
(defparameter *counter* 0)
(defun dostuff ()
(format t "hello world ~a!~%" *counter*))
(defun runner ()
(bt:make-thread (lambda ()
(swank:create-server :port 4006)))
(format t "we are past go!~%")
(loop while t do
(sleep 5)
(dostuff)
(incf *counter*)
))
(runner)
On our server, we run it with
sbcl --load demo.lisp
we do port forwarding on our development machine:
ssh -L4006:127.0.0.1:4006 username#example.com
this will securely forward port 4006 on the server at example.com to
our local computer's port 4006 (swanks accepts connections from
localhost).
We connect to the running swank with M-x slime-connect, typing in
port 4006.
We can write new code:
(defun dostuff ()
(format t "goodbye world ~a!~%" *counter*))
(setf *counter* 0)
and eval it as usual with M-x slime-eval-region for instance. The output should change.
There are more pointers on CV Berry's page.
Hot reload
Example with Quickutil. See notes on lisp-journey.
It has to be run on the server (a simple fabfile command can call this
through ssh). Beforehand, a fab update has run git pull on the
server, so new code is present but not running. It connects to the
local swank server, loads the new code, stops and starts the app in a
row.
Continuous Integration, continuous delivery of executables, Docker
See https://lispcookbook.github.io/cl-cookbook/testing.html#continuous-integration

Answer 2

To run a lisp image in production, you can generate a fasl file from your lisp code with:
(compile-file "app.lisp")
run your generated .fas file by invoking sbcl.
sbcl --noinform \
--load app.fas \
--eval "(defun main (argv) (declare (ignore argv)) (hunchentoot:start (make-instance 'hunchentoot:easy-acceptor :port 4242)))"

Answer 3

I found a blog with a solution that I have adapted to my needs for a production system on a linux box. Unfortunately I cannot find the reference to that blog anymore, so that I can just show you my solution, which is for CCL (while the original solution was for SBCL), which with I am more familiar. Here is the program that starts the system:
(require 'swank)
(require 'hunchentoot)
(defparameter *httpd-port* 9090) ; The port Hunchentoot will be listening on
(defparameter *shutdown-port* 6700) ; The port CCL will be listening for shutdown
; this port is the same used in /etc/init.d/hunchentoot
(defparameter *swank-port* 5016) ; The port used for remote interaction with slime
;; Start the Swank server
(defparameter *swank-server*
(swank:create-server :port *swank-port* :dont-close t))
(require 'YOUR-PACKAGE)
(YOUR-PACKAGE:YOUR-STARTING-FUNCTION)
(princ "Hunchentoot started on port ")
(princ *httpd-port*)
(terpri)
(let* ((socket (make-socket :connect :passive :local-host "127.0.0.1" :local-port *shutdown-port* :reuse-address t))
(stream (accept-connection socket)))
(close stream)
(close socket))
(print "Stopping Hunchentoot...")
(YOUR-PACKAGE:YOUR-STOPPING-FUNCTION)
(dolist (proc (all-processes))
(unless (equal proc *current-process*)
(process-kill proc)))
(sleep 1)
(quit)
The idea is that you can connect to the running system with slime, by specifying the port used by swank. I used it a couple of times, for instance to change a database link on the fly, and was quite impressed by the power of such possibility.
The running system can be terminated by:
telnet 127.0.0.1 6700
and initiated by something like:
nohup ccl -l initcclserver.lisp >& server.out &
In a previous version of the script I found the SBCL-specific parts, so if you use it you could modify the script.
For accepting the terminating connections:
(sb-bsd-sockets:socket-bind socket #(127 0 0 1) *shutdown-port*)
(sb-bsd-sockets:socket-listen socket 1)
(multiple-value-bind (client-socket addr port)
(sb-bsd-sockets:socket-accept socket)
(sb-bsd-sockets:socket-close client-socket)
(sb-bsd-sockets:socket-close socket)))
To close the system:
(dolist (thread (sb-thread:list-all-threads))
(unless (equal sb-thread:*current-thread* thread)
(sb-thread:terminate-thread thread)))
(sleep 1)
(sb-ext:quit)
Hope this can help.