Negative response time in Jetty server access log - jetty

I'm running on Jetty and my access log is configured to show the response time.
today I noticed that a very minimal amount of the requests (about 30 out of 600K) have a negative response time and I was wondering if anyone ever encountered such a behavior.
This is a sample of my response:
<[IP]> - - <[date]> "POST <[url]> HTTP/1.0" 201 461 -18096
In case you want to identify this in the access log - this is the grep command i used:
grep --color "-[0-9][0-9]*" server-access.2013_12_09.log
Jetty Version: 8.1.8
Setup in jetty.xml:
<New id="request-log-handler" class="org.eclipse.jetty.server.handler.RequestLogHandler">
<Set name="requestLog">
<New class="org.eclipse.jetty.server.NCSARequestLog">
<Arg>
<Property name="logging.httpAccessLog" default="logs/app-access.yyyy_mm_dd.log" />
</Arg>
<Set name="retainDays">
<Property name="logging.accessLogRetentionInDays" default="10" />
</Set>
<Set name="append">
<Property name="logging.httpAccessLogAppend" default="true" />
</Set>
<!-- logs referer and user agent -->
<Set name="extended">
<Property name="logging.httpAccessLogExtended" default="false" />
</Set>
<!-- response time -->
<Set name="logLatency">
<Property name="logging.httpAccessLogLatency" default="true" />
</Set>
</New>
</Set>

Using Jetty 9.1.0.
Key:
<name> = optional / configurable entry on line
{name} = mandatory entry
<servername> {X-Forwarded-For||remote-addr} - {authentication/principal/name} [{request-timestamp}] "{method} {uri} {protocol}" {response-status-code} {response-content-length} <extended-log> <cookies> <latency>
Where:
<servername> can be the request/Host header
<extended-log> can be "{referer} {user-agent}"
<cookies> can be " -" or "{cookie.name}={cookie-value};"
<latency> can be (now - request.getTimeStamp())
Interestingly, it appears that your system clock adjusted during a request. A full 18 seconds!
Here's why this is the leading theory ...
The request.timestamp is set when the request starts, then the latency is computed later during the access/request logging. For there to be a negative value, either something outside of Jetty itself reset the request.timeout, or the system clock changed.

Related

defining webapp load order in jetty

I have two webapps that might be deployed separately or in the same jetty server. In my situation, one webapp must deploy before the other can complete its deployment, so I'm looking for a solution in the dual deployment case where I can define the deployment order.
I've defined a context xml file with the following:
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN"
"http://www.eclipse.org/jetty/configure.dtd">
<Configure class="org.eclipse.jetty.server.handler.HandlerList">
<New class="org.eclipse.jetty.server.handler.HandlerList">
<Set name="handlers">
<Array type="org.eclipse.jetty.server.Handler">
<Item>
<New class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/security</Set>
<Set name="war"><SystemProperty name="jetty.home"/>/security_app/war/</Set>
<Call name="addAliasCheck">
<Arg>
<New class="org.eclipse.jetty.server.handler.ContextHandler$ApprovePathPrefixAliases"/>
</Arg>
</Call>
</New>
</Item>
<Item>
<New class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="war"><SystemProperty name="jetty.home"/>/main_app/war/</Set>
<Call name="addAliasCheck">
<Arg>
<New class="org.eclipse.jetty.server.handler.ContextHandler$ApprovePathPrefixAliases"/>
</Arg>
</Call>
</New>
</Item>
</Array>
</Set>
</New>
</Configure>
However, with this solution, I get the following error:
java.lang.ClassCastException: org.eclipse.jetty.server.handler.HandlerList cannot be cast to org.eclipse.jetty.server.handler.ContextHandler
at org.eclipse.jetty.deploy.providers.WebAppProvider.createContextHandler(WebAppProvider.java:292)
at org.eclipse.jetty.deploy.App.getContextHandler(App.java:101)
at org.eclipse.jetty.deploy.bindings.StandardDeployer.processBinding(StandardDeployer.java:36)
at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186)
at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:498)
at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:146)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:605)
at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:528)
at org.eclipse.jetty.util.Scanner.scan(Scanner.java:391)
at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
which of course, is true, though they both are AbstractHandlers in the end. What might I be doing wrong or is there another way to pull this off?
Using Jetty 9.2.1
Thanks
You have hardcoded deployments, don't use the deploy module.
The exception you are getting is the deploy module attempting to find and deploy webapps itself, and is failing to find the org.eclipse.jetty.server.handler.ContextHandler entry it is expecting.
The best way to fix this is to not have separate webapps that depend on each other during deploy/init (that is out of scope for the servlet spec). Make the /main_app/war/ webapp init on demand, not on startup.
Now back to your specific problem ...
Important: do not work / edit / run / configure the ${jetty.home} directory directly.
The ${jetty.base} directory exists for a reason, use it, you'll be much happier when you do.
http://www.eclipse.org/jetty/documentation/current/startup.html
For those things that are load order sensitive, do not use the deploy module with those artifacts. (its safe to use the deploy module for other webapps you have that don't have load order dependencies)
To accomplish this, we'll configure a ${jetty.base} directory to use a custom xml file on startup, along with a /special/ directory for these load order dependent WebAppContexts, making sure that these contexts are not loaded, and managed by the deploy module.
$ cd my-jetty-base
$ mkdir etc
$ gvim etc/special-deploy.xml
$ echo "etc/special-deploy.xml" >> start.ini
$ ls -F
etc/ special/ start.ini webapps
$ ls -F special/
main.war security.war
$ cat start.ini
--module=http
jetty.port=8080
--module=deploy
etc/special-deploy.xml
The special-deploy.xml looks like this ...
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">
<!-- =============================================================== -->
<!-- Add Load Order Dependant Webapps -->
<!-- =============================================================== -->
<Configure id="Contexts" class="org.eclipse.jetty.server.handler.ContextHandlerCollection">
<Call name="addHandler">
<Arg>
<New class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/security</Set>
<Set name="war"><SystemProperty name="jetty.base"/>/special/security.war</Set>
<Call name="addAliasCheck">
<Arg>
<New class="org.eclipse.jetty.server.handler.ContextHandler$ApprovePathPrefixAliases"/>
</Arg>
</Call>
</New>
</Arg>
</Call>
<Call name="addHandler">
<Arg>
<New class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="contextPath">/</Set>
<Set name="war"><SystemProperty name="jetty.base"/>/special/main.war</Set>
<Call name="addAliasCheck">
<Arg>
<New class="org.eclipse.jetty.server.handler.ContextHandler$ApprovePathPrefixAliases"/>
</Arg>
</Call>
</New>
</Arg>
</Call>
</Configure>
When you run you'll see the following ...
$ java -jar /path/to/jetty-distribution-9.2.10.v20150310/start.jar
2015-04-29 12:18:55.929:INFO::main: Logging initialized #275ms
2015-04-29 12:18:56.093:WARN:oejsh.ContextHandler:main: ApprovePathPrefixAliases is not safe for production
2015-04-29 12:18:56.094:WARN:oejsh.ContextHandler:main: ApprovePathPrefixAliases is not safe for production
2015-04-29 12:18:56.097:INFO:oejs.Server:main: jetty-9.2.10.v20150310
2015-04-29 12:18:56.150:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /security, did not find org.eclipse.jetty.jsp.JettyJspServlet
2015-04-29 12:18:56.166:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext#751d30f6{/security,file:/tmp/jetty-0.0.0.0-8080-security.war-_security-any-3863723758166154575.dir/webapp/,AVAILABLE}{/home/joakim/examples/load-order-example/special/security.war}
2015-04-29 12:18:56.181:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2015-04-29 12:18:56.183:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext#4f79a28b{/,file:/tmp/jetty-0.0.0.0-8080-main.war-_-any-2704851460101920295.dir/webapp/,AVAILABLE}{/home/joakim/examples/load-order-example/special/main.war}
2015-04-29 12:18:56.184:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:/home/joakim/examples/load-order-example/webapps/] at interval 1
2015-04-29 12:18:56.195:INFO:oejs.ServerConnector:main: Started ServerConnector#4ef6d773{HTTP/1.1}{0.0.0.0:8080}
2015-04-29 12:18:56.196:INFO:oejs.Server:main: Started #542ms
^C2015-04-29 12:19:09.594:INFO:oejs.ServerConnector:Thread-0: Stopped ServerConnector#4ef6d773{HTTP/1.1}{0.0.0.0:8080}
2015-04-29 12:19:09.599:INFO:oejsh.ContextHandler:Thread-0: Stopped o.e.j.w.WebAppContext#4f79a28b{/,file:/tmp/jetty-0.0.0.0-8080-main.war-_-any-2704851460101920295.dir/webapp/,UNAVAILABLE}{/home/joakim/examples/load-order-example/special/main.war}
2015-04-29 12:19:09.602:INFO:oejsh.ContextHandler:Thread-0: Stopped o.e.j.w.WebAppContext#751d30f6{/security,file:/tmp/jetty-0.0.0.0-8080-security.war-_security-any-3863723758166154575.dir/webapp/,UNAVAILABLE}{/home/joakim/examples/load-order-example/special/security.war}

Jetty does not want to listen on port 80

I want to change the default port the Jetty 9 is listening on from 8080 to 80. I've edited the jetty-http.xml file to look like this:
<Call name="addConnector">
<Arg>
<New class="org.eclipse.jetty.server.ServerConnector">
<Arg name="server"><Ref refid="Server" /></Arg>
<Arg name="factories">
<Array type="org.eclipse.jetty.server.ConnectionFactory">
<Item>
<New class="org.eclipse.jetty.server.HttpConnectionFactory">
<Arg name="config"><Ref refid="httpConfig" /></Arg>
</New>
</Item>
</Array>
</Arg>
<Set name="host"><Property name="jetty.host" /></Set>
<Set name="port"><Property name="jetty.port" default="80" /></Set>
<Set name="idleTimeout"><Property name="http.timeout" default="30000"/></Set>
</New>
</Arg>
However, for some strange reason when I startup the server it still listens on 8080.
The way you have your XML defined, it will use any jetty.port property first, then if not found, use your hardcoded port 80.
Check your ${jetty.home}/start.ini and also your ${jetty.home}/start.d/* files for that property.
Or alternatively, run $ java -jar start.jar --help and see if it shows up as defined at the end of the output of that command)
$ java -jar start.jar --help
(...snip...)
The current start.ini arguments are:
OPTIONS=Server,websocket,resources,ext
threads.min=10
threads.max=200
threads.timeout=60000
jetty.dump.start=false
jetty.dump.stop=false
etc/jetty.xml
OPTIONS=jmx
etc/jetty-jmx.xml
OPTIONS=jsp
jetty.port=8080
http.timeout=30000
etc/jetty-http.xml
At this point you have 2 options.
Don't edit the XML file and simply configure the start.ini entry for jetty.port
Change the XML file entry to not look for the property.
<Set name="port">80</Set>

Empty request logs

I've added this to jetty.xml:
<Get name="handler">
<Call name="addHandler">
<Arg>
<New id="RequestLog" class="org.eclipse.jetty.server.handler.RequestLogHandler" />
</Arg>
</Call>
</Get>
<Ref id="RequestLog">
<Set name="requestLog">
<New id="RequestLogImpl" class="org.eclipse.jetty.server.NCSARequestLog">
<Arg>
/home/aromanov/workspaces/odp-server/ru.focusmedia.odp.server.karaf.launcher/data/jettylog/yyyy_mm_dd.request.log
</Arg>
<Set name="retainDays">90</Set>
<Set name="append">true</Set>
<Set name="extended">true</Set>
<Set name="logServer">true</Set>
<Set name="logDispatch">true</Set>
<Set name="LogTimeZone">GMT</Set>
</New>
</Set>
</Ref>
The log file is created, but after accessing the page in the browser (and getting "404, powered by Jetty"), nothing appears in the log. Why could this be?
I suspect your handler chain is not configured correctly. Think of the request getting processed by a chain of handlers, and this logging handler is one that just makes a reference to the log file as the request processes through. So if your not getting content in the log then your handler is not in the chain of execution that is occurring. You likely just need to add it to the handler list, before something else that might handle it.

How to get Jetty to use JVM scoped JNDI?

I'm trying to setup a JNDI resource for a Jackrabbit repository factory in Jetty. The problem is that I seem to be getting thhe JNDI as webapp scoped. I need it to be JVM scoped. As far as I understood from the docs, you need to specify a null arg as shown below (<Arg></Arg>) in order to do so (explained here).
I have two webapps deployed to Jetty and I need them to be sharing the same JNDI. If they're not sharing it, Jackrabbit tries to get initialized twice and fails which breaks my whole app.
I ran the code through a debugger and I can see that the first webapp that gets accessed and needs a JNDI lookup for the Jackrabbit repository, gets an instance of the BindableRepositoryFactory and correctly adds records to the cache object. However, a completely different object is created for the cache upon JNDI lookup. That is obviously still empty and thus a new instance gets created, which screws things up.
I am using Jetty 7.6.2.v20120308.
Here's my jetty-jndi.xml:
<New class="org.eclipse.jetty.plus.jndi.Resource">
<Arg></Arg>
<Arg>jndi:comp/env/jcr/repository</Arg>
<Arg>
<New class="javax.naming.Reference" id="reference">
<Arg>javax.jcr.Repository</Arg>
<Arg>org.apache.jackrabbit.core.jndi.BindableRepositoryFactory</Arg>
<Arg>null</Arg>
<Call name="add" id="reference">
<Arg>
<New class="javax.naming.StringRefAddr">
<Arg>configFilePath</Arg>
<Arg><SystemProperty name="jetty.home" default="."/>/jackrabbit/repository.xml</Arg>
</New>
</Arg>
</Call>
<Call name="add" id="reference">
<Arg>
<New class="javax.naming.StringRefAddr">
<Arg>repHomeDir</Arg>
<Arg><SystemProperty name="jetty.home" default="."/>/jackrabbit</Arg>
</New>
</Arg>
</Call>
</New>
</Arg>
</New>
In the pom file where I'm invoking jetty from, I have the following relevant settings that tell Jetty to use the JNDI and Plus settings xml-s:
<jettyConfig>${project.build.directory}/jetty/etc/jetty-plus.xml,${project.build.directory}/jetty/etc/jetty-jndi.xml</jettyConfig>
In the jetty-plus.xml I have:
<!-- =========================================================== -->
<!-- Sequence of configurations to defining Plus features. -->
<!-- =========================================================== -->
<Array id="plusConfig" type="java.lang.String">
<Item>org.eclipse.jetty.webapp.WebInfConfiguration</Item>
<Item>org.eclipse.jetty.webapp.WebXmlConfiguration</Item>
<Item>org.eclipse.jetty.webapp.MetaInfConfiguration</Item>
<Item>org.eclipse.jetty.webapp.FragmentConfiguration</Item>
<Item>org.eclipse.jetty.plus.webapp.EnvConfiguration</Item> <!-- Add for JNDI -->
<Item>org.eclipse.jetty.plus.webapp.PlusConfiguration</Item> <!-- Add for JNDI -->
<Item>org.eclipse.jetty.annotations.AnnotationConfiguration</Item>
<Item>org.eclipse.jetty.webapp.JettyWebXmlConfiguration</Item>
<Item>org.eclipse.jetty.webapp.TagLibConfiguration</Item> <!-- Not needed for Jetty-8 -->
</Array>
<!-- =========================================================== -->
<!-- Apply plusConfig to all webapps for this Server -->
<!-- =========================================================== -->
<Call name="setAttribute">
<Arg>org.eclipse.jetty.webapp.configuration</Arg>
<Arg>
<Ref id="plusConfig"/>
</Arg>
</Call>
Any ideas what I might be doing wrong?
Many thanks in advance! :)
This appears to be a bug in Jetty's global JNDI resource configuration.
I have filed a bug in Jetty's bugzilla and will accept this answer as correct, despite the fact it is no solution.

Map external directory to web.xml

Is there an easy way to map a directory in the web.xml or other deployment descriptor (jetty.xml, etc) files?
For example, if I have a directory /opt/files/ is there a way that I can access its files and sub-directories by visiting http://localhost/some-mapping/? It strikes me that there should be some simple way of doing this, but I haven't been able to find out how (via google, stackoverflow, etc). All I've found are servlets that mimic fileservers, which is not what I would like.
For reference I am using jetty on an AIX box.
No idea how to do it with Jetty, but in Tomcat you can just add a new <Context> to server.xml:
<Context docBase="/opt/files" path="/files" />
This way it's accessible by http://example.com/files/.... See if something similar exist for Jetty.
Update: after Googling, the "normal Java code" equivalent would be something like:
WebAppContext files = new WebAppContext("/opt/files", "/files");
Server server = new Server(8080);
server.setHandler(files);
server.start();
Now yet to translate that into jetty.xml flavor. I am a bit guessing based on the documentation and examples found on the web, so don't pin me on it:
<Configure class="org.mortbay.jetty.webapp.WebAppContext">
<Set name="webApp">/opt/files</Set>
<Set name="contextPath">/files</Set>
</Configure>
Another possibility may be this:
<Configure class="org.mortbay.jetty.Server">
<Call name="addHandler">
<Arg>
<New class="org.mortbay.jetty.webapp.WebAppContext">
<Arg name="webApp">/opt/files</Arg>
<Arg name="contextPath">/files</Arg>
</New>
</Arg>
</Call>
</Configure>
After some more fiddling around, the best way to do this (for jetty) is to deploy a descriptor in the context directory that looks like the following...
<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">
<!-- Configuration of a custom context. -->
<Configure class="org.eclipse.jetty.server.handler.ContextHandler">
<Call class="org.eclipse.jetty.util.log.Log" name="debug">
<!-- The default message to show if our context breaks. -->
<Arg>Configure context.xml</Arg>
</Call>
<!--
The context path is the web location of the context in relation to the
server address.
-->
<Set name="contextPath">/context</Set>
<!--
The resource base is the server directory to use for fetching files.
-->
<Set name="resourceBase">/path/to/files/on/server</Set>
<Set name="handler">
<New class="org.eclipse.jetty.server.handler.ResourceHandler">
<Set name="directoriesListed">true</Set>
<!-- For now we don't need any welcome files -->
<!--
<Set name="welcomeFiles"> <Array type="String">
<Item>index.html</Item> </Array> </Set>
-->
<!--
The cache time limit in seconds (ie max-age=3600 means that if the
document is older than 1 hour a fresh copy will be fetched).
-->
<Set name="cacheControl">max-age=3600,public</Set>
</New>
</Set>
</Configure>
I hope that this helps someone else!
I don't know that you can do that as a URL mapping, however files and folders in the same directory that your web.xml file is in will be accessible in the way you describe, although you can't control the mapping under your webapp.
Does that suit your needs?