Is there a list of available Jetty 9 modules somewhere?
Just a simple table "this is the name, this is what it does, and here are links" type.
I have searched the Eclipse site and used search engines for some time now, without any usable result. Is it really that much of a secret what jetty modules exist, and what they do?
Use the command line.
$ cd /path/to/mybase
$ java -jar /path/to/jetty-home/start.jar --list-modules
Some modules are dynamic/virtual (dependent on your environment).
Some are 3rd party (jsp, jolokia, gcloud, etc).
Of the remaining few, you have the module information itself.
IE: rewrite is the rewrite behaviors in doc, http is the http server connector, etc.
Going from module to doc is a 1::n scenario, while going from doc to module is a 1::1 scenario.
If you want to know what they do, look at the module definition - (aka ${jetty.home}/modules/${name}.mod
They might have properties (documented in module)
They might have libs (obvious in module)
They might have xml (see standard XML configuration behaviors in Jetty doc)
They might have a non-Eclipse license (documented in module)
They might have a dependent module (documented in module)
The result of enabling a module is simply a command line along the lines of --module=http.
The combination of enabled modules (via the combination of ini files) is a longer command line + server classpath + xml load order.
You can see this via ...
$ cd /path/to/mybase
$ java -jar /path/to/jetty-home/start.jar --list-config
Related
My goal is to set jetty to run as a service and figured I should have it run in command line first. My current aim is to be able to run jetty from root path. Once that's done, I can then set the same command to run as a service via https://nssm.cc/.
The issue
The webapp I am deploying in this Jetty instance uses logback to manage it's logging <file>logs/app.log</file>
This works as expected.
C:\jetty>java -jar start.jar
This does not C:\>java -jar C:\jetty\start.jar. (Note the different cmd path). Jetty can't figure out correct paths for log files. After some testing, I was able to rectify the path issue but still jetty is complaining about not being able to find the log file, even though I see it exists. So, permissions issue perhaps? I go ahead and give everyone full read and write access to the entire folder. No dice, still the same error.
Command I run now
C:\>java -Djetty.base=C:\jetty -Duser.dir=C:\jetty -jar C:\jetty\start.jar but the error persists.
....
12:11:51,095 |-INFO in ch.qos.logback.core.FileAppender[FILE] - File property is set to [logs/app.log]
12:11:51,111 |-ERROR in ch.qos.logback.core.FileAppender[FILE] - Failed to create parent directories for [C:\jetty\logs\app.log]
12:11:51,111 |-ERROR in ch.qos.logback.core.FileAppender[FILE] - openFile(logs/app.log,true) call failed. java.io.FileNotFoundException: logs\app.log (The system cannot find the path specified)
at java.io.FileNotFoundException: logs\app.log (The system cannot find the path specified)
....
Jetty version is 9.4.9.v20180320
Create a proper ${jetty.base} directory, outside of of the ${jetty.home} directory.
Next, get yourself a fresh copy of jetty-home (or the jetty-distribution which really only adds documentation, demos, and sample bases to jetty-home).
http://search.maven.org/#search|gav|1|g:"org.eclipse.jetty" AND a:"jetty-home"
Unpack that jetty-home artifact into a fresh directory.
Important:
Don't touch any content in jetty-home
Don't edit anything in jetty-home
Don't remove anything from jetty-home
Don't add anything into jetty-home
Leave jetty-home alone
Treat jetty-home as a read-only directory
All above bullets apply even if using jetty-distribution
Lastly, start jetty per the recommendations in the Jetty Documentation.
> cd \path\to\myjettybase
> java -jar \path\to\jetty-home\start.jar
Note the lack of declared system properties for -Djetty.home and -Djetty.base, this is important!
This will set up the following important directory properties.
jetty.base will be your CWD, or \path\to\myjettybase
jetty.home will be where your start.jar was found, aka \path\to\jetty-home\
user.dir will be your CWD and point to \path\to\myjettybase
What's important, for logging configurations like yours, is the user.dir system property. (aka the current working directory)
The technique you are using where you don't pay attention to the working directory and just attempt to force values into the JVM with system properties is only viable in the simplest of projects and usecases. (your requirements have outgrown that kind of setup)
What are the common/standard ways to build CLI scripts in Clojure?
In my view such a method should include the following characteristics:
A way of easily dealing with arguments, stdin/out/err.
Without taking too much to boot (ideally having some sort of JIT), otherwise one loses the purpose of hacking things together in one's shell.
Also it is reasonable to expect a easy way of including one time dependencies without setting up a project (maybe installing them globally).
Ideally, providing a simple example of the solution usage would be much appreciated. Somewhat equivalent to:
#!/bin/bash
echo "$#"
cat /dev/stdin
Note: I'm aware that this question was somewhat questioned previously here. But the question is incomplete and the answers don't reach a consensus neither a significant proportion of the solutions that seems to exist.
Now that there is new CLI tooling it is possible to create a standalone Clojure script without using third party tools. Once you've got the clj command line tool installed, a script like the one below should just work.
In terms of the original question, this can be as good as any Clojure/JVM CLI program at dealing with command line arguments and system input/output depending on what libraries you :require. I've haven't benchmarked it, so I won't comment on performance but if it worries you then please experiment yourself to see if startup time is acceptable to you. I would say this scores highly on dependency management though, as the script is entirely standalone (apart from the clj tool which is now the recommended way to run Clojure anyway).
File: ~/bin/script.sh
#!/bin/sh
"exec" "clj" "-Sdeps" "{:deps,{hiccup,{:mvn/version,\"1.0.5\"}}}" "$0" "$#"
(ns my-script
(:require
[hiccup.core :as hiccup]))
(println
(hiccup/html
[:div
[:span "Command line args: " (clojure.string/join ", " *command-line-args*)]
[:span "Stdin: " (read-line)]]))
Then ensure it is executable:
$ chmod +x ~/bin/script.sh
And run it:
$ echo "stdin" | script.sh command line args
<div><span>Command line args: command, line, args</span><span>Stdin: stdin</span></div>
NB. This is primarily a shell script which treats the strings on line three as commands to execute. That subsequent execution will run the clj command line tool with the given arguments, which will evaluate those strings as strings (without side effects) and then proceed to evaluate the Clojure code below.
Note also that dependencies are specified as a map passed to clj on line three. You can read more about how that works on the Clojure website. The tokens in the dependency map are separated by commas, which Clojure treats as whitespace but which most shells do not.
Thanks to the good folk on the #tools-deps channel of the "clojurians" Slack group whence this solution came.
An option would be Planck which runs on MacOS and Linux. It uses self-hosted ClojureScript, has fast startup and targets JavaScriptCore.
It has a nice SDK and mimics some things from Clojure which you do not have in ClojureScript, e.g. planck.io resembles clojure.java.io. It supports loading dependencies via tools.deps.alpha/deps.edn.
Echoing stdin is as easy as:
(require '[planck.core :refer [*in* slurp]])
(print (slurp *in*))
and printing the command line arguments:
(println *command-line-args*)
...
$ echo "foo" | planck stdin.cljs 1 2 3
foo
(1 2 3)
An example of a standalone script, i.e. not a project, with dependencies: the tree command line tool in Planck.
One caveat is that Planck doesn't support using npm dependencies. So if you need those, go for Lumo which targets NodeJS.
A third option would be joker which is a Clojure interpreter written in Go.
I know you asked for non project creating methods to accomplish this but as this specific issue has been on my mind for quite some time I figured I would throw in another alternative.
TLDR: jump to the "Creating an Executable CLI Command" section below
Background
I had pretty much the same list of requirements as you do a while back and landed on creating executable jar files. I'm not talking about executable via java -jar myfile.jar, but rather self-contained uber-jars which you can execute directly as you would with any other binary file.
If you read the zip file specification (which jar files adher to as a jar file is a zip file), it turns out this is actually possible. The short version is that you need to:
build a fat jar with the stuff you need
insert a bash / bat / shell script into the binary jar content at the beginning of your file
chmod +x the uber jar file (or if on windows, check the executable box)
rewrite the jar file meta data records so that the inserted script text does not invalidate the zip file internal offsets
It should be noted that this is actually supported by the zip file specification. This is how self extracting zip files etc work and the resulting fat jar (after the above process) is still a valid jar file and a valid zip archive. All relevant commands such as java -jar still work and the file is now also executable directly from the command line.
In addition, following the above pattern it is also possible to add support for things like the drip jvm launcher which greatly accelerates the startup times of your cli scripts.
As it turns out when I started looking into this about a year ago, a library for the last point of rewriting the jar file meta data did not exist. Not just in clojure but on the JVM as a whole. This still blows my mind: the central deployment unit of all languages on the jvm is the jar file and there was no library out there that actually read the internals of jar files. Internals as in the actual zip file structure, not just what java's ZipFile and friends does.
Furthermore, I could not find a library for clojure which dealt with the kind of binary structure the zip file specification required in a clean way.
Solution:
octet has what I consider the cleanest interface of the available binary libraries for clojure, so I wrote a pull request for octet adding support for the features required by the zip file specification.
I then created a new library clj-zip-meta which reads and interprets the zip file meta data and is capable of the offset rewriting described in the last point above.
I then created a pull request to an existing clojure lib lein-binplus to add support for the zip meta rewriting implemented by clj-zip-meta and also add support for custom preamble scripts to be able to create real executable jars without the need for java -jar.
After all this I created a leiningen template cli-cmd to support creating cli command projects which support all the above bells and whistles and has a well structured command line parsing setup...or what I considered well structured : ). Comments welcomed.
Creating an Executable CLI Command
So with all that, you can create a new command line clojure app with leiningen and run it using:
~> lein new cli-cmd mycmd
~> cd mycmd
~> lein bin
Compiling mycmd.core
Compiling mycmd.core
Created /home/mbjarland/tmp/clj-cmd/mycmd/target/mycmd-0.1.0-SNAPSHOT.jar
Created /home/mbjarland/tmp/clj-cmd/mycmd/target/mycmd-0.1.0-SNAPSHOT-standalone.jar
Creating standalone executable: /home/mbjarland/tmp/clj-cmd/mycmd/target/mycmd
Re-aligning zip offsets
~> target/mycmd
---- debug output, remove for production code ----
options {:port 80, :hostname "localhost", :verbosity 0}
arguments []
errors nil
summary
-p, --port PORT 80 Port number
-H, --hostname HOST localhost Remote host
--detach Detach from controlling process
-v Verbosity level; may be specified multiple times to increase value
-h, --help
--------------------------------------------------
This is my program. There are many like it, but this one is mine.
Usage: mycmd [options] action
Options:
-p, --port PORT 80 Port number
-H, --hostname HOST localhost Remote host
--detach Detach from controlling process
-v Verbosity level; may be specified multiple times to increase value
-h, --help
Actions:
start Start a new server
stop Stop an existing server
status Print a server's status
Please refer to the manual page for more information.
Error: invalid action '' specified!
Where the output from the command is just the boilerplate sample command line parsing I've added to the leiningen template.
The custom preamble script is located at boot/jar-preamble.sh and it has support for drip. In other words, if you have drip on your path, the generated executable will use it, otherwise it will fall back to standard java -jar way of launching the uber jar internally.
The source for the command line parsing and the code for the cli app live under the src directory as per normal.
If you feel like hacking, it is possible to change the preamble script and re-run lein bin and the new preamble will be inserted into your executable by the build process.
Also it should be noted that this method still does java -jar under the covers so you do need java on your path.
Ayway, long-winded explanation, but hopefully it will be of some use for somebody with this problem.
Consider Lumo, a ClojureScript environment which was specially designed for scripting.
Note that while it supports both ClojureScript (JAR) and NPM dependencies, the dependency support is still under development.
I write a number of Clojure (JVM) scripts, and use a the CLI-matic library https://github.com/l3nz/cli-matic/ to abstract most of the boilerplate that goes with command-line parsing, creation and maintenance of help, errors, etc.
I have an OCaml project that is currently built using OCamlMake. I am not happy with the current build system since it leaves all build artefacts in the same directory as source files, plus it also requires to manually specify order of dependencies between modules. I would like to switch to a build system that does not suffer from these issues. I decided to give Oasis a try but have run into issues.
The problems arise from the fact that the project is built in a very specific way. It supports several different database backends (PostgreSQL, MySQL, SQLite). Currently, to compile a database backend user must install extra libraries required by that backend and enable it by setting an environment variable. This is how it looks in the Makefile:
ifdef MYSQL_LIBDIR
DB_CODE += mysql_database.ml
DB_AUXLIBS += $(MYSQL_LIBDIR)
DB_LIBS += mysql
endif
Notice that this also adds extra module to the list of compiled modules. The important bit is that there is no dependency (in a sense of module import) between any module reachable from the applications entry point and database backend module. What happens rather is that each database backend module contains top-level code that runs when a module is initiated and registers itself, using side-effects, with the main application.
I cannot get this setup to work with Oasis. I declared each of the database backend modules as a separate library that can be enabled for compilation with a flag:
Library mysql-backend
Path : .
Build $: flag(mysql)
Install : false
BuildTools : ocamlbuild
BuildDepends : main, mysql
Modules : Mysql_backend
However, I cannot figure out a way to tell Oasis to link the optional modules into the executable. I tried to figure out a way of doing this by modifying myocamlbuild.ml file but failed. Can I achieve this with the rule function described here?
If what I describe cannot be achieved with ocamlbuild, is there any ither tool that would do the job and avoid problems of OCamlMake?
Well, I guess that answers it: https://github.com/links-lang/links/pull/77 :)
I saw the question and started working on it before I noticed Drup's answer above. Below is a self-contained ocamlbuild solution that is essentially the same as Drup's.
open Ocamlbuild_plugin
let enable_plugin () =
let plugins = try string_list_of_file "plugin.config" with _ -> [] in
dep ["ocaml"; "link_with_plugin"; "byte"]
(List.map (fun p -> p^".cmo") plugins);
dep ["ocaml"; "link_with_plugin"; "native"]
(List.map (fun p -> p^".cmx") plugins);
()
let () = dispatch begin
function
| Before_rules -> enable_plugin ()
| _ -> ()
end
Using the tag link_with_plugin on an ocamlbuild target will make it depend on any module whose path (without extension) is listed in the file plugin.config. For example if you have plugins pluga.ml, plugb.ml and a file main.ml, then writing pluga plugb in plugin.config and having <main.{cmo,cmx}>: link_with_plugin will link both plugin modules in the main executable.
Unfortunately, this is beyond oasis capabilities. This limitation has nothing to do with ocamlbuild, it just because oasis author tried to keep it simple, and didn't provide optional dependencies as a feature.
As always, an extra level of indirection may solve your problem. What you need, is a configuration script (configure) that will generate _oasis file for you, depending on parameters, provided by a user.
For example, in our project we have a similar setup, i.e., multiple different backends, that might be chosen by a user during the configuration phase, with --{enable,disable}-<feature>. We achieved this by writing our own ./configure script that generate _oasis file, depending on configuration. The configuration script just concatenates the resulting _oasis files from pieces, described in oasis folder.
An alternative solution would be to use m4 or just cpp, and have an _oasis.in file, that is preprocessed.
Is there a way to fix the file locking issue caused by jetty entirely from gradle?
Some clarification:
When using the Gradle Jetty plugin by running gradle jettyRun, jetty causes the static resource files (html, css, js, etc.) to be locked when using Windows.
You can see a description of the problem in Files locked on Windows.
The same article also describes how you can fix that. Basically you have to either:
Disable the use of file mapped buffer
Not use NIO at all.
Both things require to add some jetty specific configuration files to the project, which I do not want to do - the jetty plugin is used only for convenience, and maintaining configuration for it does not feel right.
I do not need NIO for testing on the local machine, so any solution works.
Edit:
For now, I picked the option at which you set useFileMappedBuffer to false. This is how to do it:
Specify a path to your webdefault.xml like
[jettyRun, jettyRunWar,jettyStop]*.with {
//other configs
webDefaultXml = file("${project.webAppDir}/WEB-INF/jetty-webdefault.xml")
}
Get file from the latest 6.1.x distribution of jetty. The plugin seems to support only jetty 6. You can localte it at jetty-6.1.26\etc\webdefault.xml. Obviously, you have to place it at the path specified at the previous step.
Change the default servlet init parameter useFileMappedBuffer to false
I will research the option of using embeded jetty insted of the plugin.
I found a plugin that seem to be a better alternative:
https://github.com/akhikhl/gretty
Positives
Does not lock your files and support hot deployment (even something Gretty call "fast reload")
Gretty 1.2.0 uses Jetty 9.2.9.v20150224. Jetty plugin provided by Gradle 2.2.1 uses Jetty 6.1.25.
same task is used jettyRun (or more simply run).
"Press any key to stop the server". Jetty plugin required CTRL+C then Y.
From what I can tell, the documentation seem to be awesome (Gradle's not so much)
Negatives
A bit more bloated code to setup the buildscript's classpath dependency or apply plugin directly from URL (see doc)
Gretty crash unless you explicitly apply plugin: 'war' (Jetty plugin extends the War plugin)
Kiril answered his own question, many thanks. You should follow Kiril's instructions and this will help you find the appropriate webdefault.xml.
To find out what version of Jetty is started by Gradle, execute
gradle jettyRun -i
And you'll see something like this:
...
Tmp directory = determined at runtime
Web defaults = org/mortbay/jetty/webapp/webdefault.xml
Web overrides = none
Webapp directory = C:\dev\my-project\src\main\webapp
Starting jetty 6.1.25 ...
jetty-6.1.25
...
It took me a while to find a copy of Jetty 6.1.25 as it is no longer listed on the Jetty download page (not even in the archive section!).
You can then grab the appropriate copy of webdefault.xml from here, adjusting the version number as appropriate for your needs:
http://grepcode.com/file/repo1.maven.org/maven2/org.mortbay.jetty/jetty/6.1.25/org/mortbay/jetty/webapp/webdefault.xml
I need to add http://us3.php.net/xsl this module to my PHP but I'm really confused how I actually do this? It says in install from that link 'PHP 5 includes the XSL extension by default and can be enabled by adding the argument --with-xsl[=DIR] to your configure line. DIR is the libxslt installation directory.' But unsure where these files would be? I have a VPS and access to WHM Cpanel, can I do this from there? I came across the software and install module bit in WHM but get the feeling that's not what I want...
Go to WHM -> Software -> EasyApache (Apache Update)
On the EasyApache page, choose Start customizing based on profile now, and after a couple screens, choose Exhaustive Options List. Select the modules you want to install, XSL in your case, and proceed to recompile.
You need to compile PHP manually, if you want any modules that you don't currently have. There are plenty of tutorials how to do it, for example if you have Apache 2 as a webserver, then follow this page on php.net: http://www.php.net/manual/en/install.unix.apache2.php
When you compile PHP, one of the commands to do is
./configure
in which you can (and most webhostings do) supply many many arguments, which say which modules you want to have installed and what features PHP can use. It may look like this:
./configure '--prefix=/usr/local/php' '--with-config-file-path=/usr/local/php/lib' '--with-apxs' '--with-iconv=/usr/local/php' '--with-openssl=/usr' '--with-zlib=/usr' '--with-mysql' '--with-pgsql=/Users/marc/cvs/entropy-php4/php-module/build/postgresql-build' '--enable-cgi' '--with-gd' '--with-png-dir=/usr/local/php' '--with-freetype-dir=/usr/local/php' '--with-t1lib=/usr/local/php' '--with-jpeg-dir=/usr/local/php' '--with-tiff-dir=/usr/local/php' '--with-curl=/usr/local/php' '--with-mcal=/usr/local/php' '--with-mcrypt=/usr/local/php' '--with-mhash=/usr/local/php' '--with-pdflib=/usr/local/php' '--with-imap=../imap-2002d' '--with-imap-ssl=/usr' '--with-expat-dir=/usr/local/php' '--with-gettext=/usr/local/php' '--with-dom=/usr/local/php' '--with-dom-xslt=/usr/local/php' '--with-dom-exslt=/usr/local/php' '--with-xslt-sablot=/usr/local/php' '--with-mime-magic=/usr/local/php/etc/magic.mime' '--with-mssql=/usr/local/php' '--with-fbsql=/Users/marc/cvs/entropy-php4/php-module/build/frontbase-build/Library/FrontBase' '--with-ming=/Users/marc/cvs/entropy-php4/php-module/build/ming-build' '--enable-xslt' '--with-ldap' '--with-xml' '--with-xmlrpc' '--with-iodbc=/usr' '--enable-sockets' '--enable-dbx' '--enable-dbase' '--enable-trans-sid' '--enable-exif' '--enable-wddx' '--enable-ftp' '--enable-mbstring'
it this example there are a lot of enabled modules. You can also see what "configure line" was used to compile your current version of PHP in phpinfo (it's right at the top, third line).