Embedded Jetty (Access) Request Logging - jetty

I am working on creating some rest API using JAX-RS and embedded Jetty.I Enable Server logging by adding LogRequestHandler in Server.java file.
Question is why jetty is writing 200 for every request 0:0:0:0:0:0:0:1 - - [03/Nov/2016:16:59:57 +0500] "GET /app/check HTTP/1.1" 200 - 4
though check end point is not implemented not exists in the app.
source code:
ServletContextHandler context = new ServletContextHandler(ServletContextHandler.SESSIONS);
//context.setContextPath("/");
ResourceConfig config = new ResourceConfig();
config.packages("server");
ServletHolder servlet = new ServletHolder(new ServletContainer(config));
context.addServlet(servlet,"/*");
NCSARequestLog requestLog = new NCSARequestLog("/var/logs/jetty/log-yyyy_mm_dd.request.log");
requestLog.setAppend(true);
requestLog.setExtended(false);
requestLog.setLogTimeZone("GMT+5");
requestLog.setLogLatency(true);
requestLog.setRetainDays(90);
RequestLogHandler requestLogHandler = new RequestLogHandler();
requestLogHandler.setRequestLog(requestLog);
HandlerList topLevelHandlers = new HandlerList();
topLevelHandlers.addHandler(context);
topLevelHandlers.addHandler(requestLogHandler);
try {
jettyServer.setHandler(topLevelHandlers);
jettyServer.dumpStdErr();
jettyServer.start();
}

Updated Answer
Do not use RequestLogHandler it is now deprecated.
Any RequestLog interface implementation should be set on the Server.setRequestLog(RequestLog) method now.
This will catch more requests, even 400 (Bad Request), requests that fail parsing, requests that don't belong to a context, and even requests that are not dispatched to the handler tree.
Original Answer
The RequestLogHandler should be executed before any context its tracking.
The ideal way to use RequestLog is to set it on the server and not use the RequestLogHandler ...
jettyServer.setRequestLog(requestLog);
But if you have a need to use it as a Handler, then you should make sure it executes before any contexts you want to track ...
HandlerList topLevelHandlers = new HandlerList();
topLevelHandlers.addHandler(requestLogHandler);
topLevelHandlers.addHandler(context);
jettyServer.setHandler(topLevelHandlers);
or wrap the context with the request log handler ...
requestLogHandler.setHandler(context);
jettyServer.setHandler(requestLogHandler);

Fix this by re-arranging the handler in the following order
I set resource handler as handlers for the request log (ie it is the outer handler) then the requestlog handler is added to the main jetty handlers. Then requests would go to request log -> handlers -> request log and back to jetty

Related

Jetty HTTP 2 Clients sharing same thread pool

Using Jetty 9.4.7
We are creating 2 clients using this code:
client = new HTTP2Client();
client.setIdleTimeout(-1);//disable client session timeout
client.setExecutor(httpThreadPool);
client.setConnectTimeout(connectionTimeoutMs);
try {
client.addBean(this.sslContextFactory);
client.start();
FuturePromise<Session> sessionPromise = new FuturePromise<>();
client.connect(sslContextFactory, new InetSocketAddress(this.host, this.port), new CustomSessionListener(clientInstanceName, this), sessionPromise);
this.session = sessionPromise.get(this.connectionTimeoutMs, TimeUnit.MILLISECONDS);
} catch(...
The httpThreadPool is common between the two clients. It is a ThreadPoolExecutor with core pool size 4 and max pool size 128.
The first client is created successfully. The second client fails with TimeoutException (regardless of the target servers, we even pointed them at the same server).
If we assign separate thread pools (or let the client construct its own default QueuedThreadPool) everything works fine.
Aside for an advice on the issue itself, is there any way to unwrap whatever exception is thrown when connecting the Http2 client? We tried overriding onFailure(Session,Throwable) in SessionListener, but it doesn't get there.
Thanks.
EDIT: Log excerpt on DEBUG: https://pastebin.com/MUKrw4JP

Wildfly fail to respond to some of the requests when there are too many requests

I am using Wildfly 9.0. I am developing a web program with two main parts.
One is submitting the request the server itself's webservice, and another one is a web service which receive request and work on it with the database.
The whole flow is like
1.user open a webpage rendered from the server and submit some information
2.the information is submitted to the server
3.the server then submit a HTTP request to the web service of the same server
4.the web service handle the request and reply.
5.the resulting information is then rendered to the user
For example, some of the requests are generated like this in the server to the web service...
String url = "http://127.0.0.1:8080/myPathOfRFWS/someWS";
HttpUriRequest httpReq = new HttpPost(url);
httpReq.addHeader("userKey", "someString");
try (DefaultHttpClient client = new DefaultHttpClient();) {
HttpResponse resp = client.execute(httpReq);
return resp;
} catch (Exception e) {
logException(e);
}
The code work perfectly fine under normal situation.
However, when I am performing Stress Test on the server with JMeter (maybe like submitting 60 requests in 60seconds, each with some steps of processing, so probably there may be around 8-10 on-going requests running at the same time in JMeter), I find that there are often a few requests failing to be received on the web service side, and hence no response to the client.
I have checked with the thread dump, and find that the server has submitted the request to web service, but the web service seems doesn't receive the request nor do any action after the action is passed through the filter.
Here are some part of the thread dump.
"default task-26" #298 prio=5 os_prio=0 tid=0x000000001a497000 nid=0xd3c runnable [0x000000001fa7d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000000fde172b0> (a sun.nio.ch.Util$2)
- locked <0x00000000fde172a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000fde17040> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.SelectorUtils.await(SelectorUtils.java:46)
at org.xnio.nio.NioSocketConduit.awaitReadable(NioSocketConduit.java:345)
at org.xnio.conduits.AbstractSourceConduit.awaitReadable(AbstractSourceConduit.java:66)
at io.undertow.conduits.ReadDataStreamSourceConduit.awaitReadable(ReadDataStreamSourceConduit.java:101)
at io.undertow.conduits.FixedLengthStreamSourceConduit.awaitReadable(FixedLengthStreamSourceConduit.java:272)
at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:151)
at io.undertow.channels.DetachableStreamSourceChannel.awaitReadable(DetachableStreamSourceChannel.java:77)
at io.undertow.server.HttpServerExchange$ReadDispatchChannel.awaitReadable(HttpServerExchange.java:1997)
at org.xnio.channels.Channels.readBlocking(Channels.java:295)
at io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:170)
at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:146)
at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:133)
at org.jboss.resteasy.plugins.providers.ProviderHelper.writeTo(ProviderHelper.java:124)
at org.jboss.resteasy.plugins.providers.FileProvider.readFrom(FileProvider.java:88)
at org.jboss.resteasy.plugins.providers.FileProvider.readFrom(FileProvider.java:34)
at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.readFrom(AbstractReaderInterceptorContext.java:59)
at org.jboss.resteasy.core.interception.ServerReaderInterceptorContext.readFrom(ServerReaderInterceptorContext.java:62)
at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:51)
at org.jboss.resteasy.security.doseta.DigitalVerificationInterceptor.aroundReadFrom(DigitalVerificationInterceptor.java:32)
at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:53)
at org.jboss.resteasy.plugins.interceptors.encoding.GZIPDecodingInterceptor.aroundReadFrom(GZIPDecodingInterceptor.java:59)
at org.jboss.resteasy.core.interception.AbstractReaderInterceptorContext.proceed(AbstractReaderInterceptorContext.java:53)
at org.jboss.resteasy.core.MessageBodyParameterInjector.inject(MessageBodyParameterInjector.java:150)
at org.jboss.resteasy.core.MethodInjectorImpl.injectArguments(MethodInjectorImpl.java:89)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:112)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)
at io.undertow.websockets.jsr.JsrWebSocketFilter.doFilter(JsrWebSocketFilter.java:151)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)
at com.mypackage.web.filter.AccessFilter.doFilter(AccessFilter.java:54)
Locked ownable synchronizers:
- <0x0000000084370a18> (a java.util.concurrent.ThreadPoolExecutor$Worker)
"default task-23" #295 prio=5 os_prio=0 tid=0x000000001a494800 nid=0x3d8 runnable [0x000000001dfac000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at com.mypackage.WsLib.callUriWs(WsLib.java:241)
Locked ownable synchronizers:
- <0x0000000084363dd8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
Please advise if there is anything I can do to solve the issue or to check with the issue.

How to define timeout for webservice client in websphere

Our application is hosted in websphere, my webservice client (jax-ws) is making webservice call to remote server. I will need to define timeout for this webservice call. I tried different way to set timeout up with no luck. here is what i tried:
Map<String, Object> requestContext = ((BindingProvider) binding).getRequestContext();
requestContext.put("com.ibm.websphere.webservices.jaxws.asynctimeout", 15000);
or
Map<String, Object> requestContext = ((BindingProvider) binding).getRequestContext();
requestContext.put(BindingProviderProperties.REQUEST_TIMEOUT, 15000);
requestContext.put(BindingProviderProperties.CONNECT_TIMEOUT, 15000);
None of them works
Any one can give hint, how to setup timeout for webservice client in websphere?
Thx
because Jax-WS in WAS relies on Axis 2 I believe you could use standard Axis 2 approach to do that, try(From axis 2 docs):
Timeout Configuration
Two timeout instances exist in the transport level, Socket timeout and Connection timeout. These can be configured either at deployment or run time. If configuring at deployment time, the user has to add the following lines in axis2.xml.
For Socket timeout:
<parameter name="SO_TIMEOUT">some_integer_value</parameter>
For Connection timeout:
<parameter name="CONNECTION_TIMEOUT">some_integer_value</parameter>
For runtime configuration, it can be set as follows within the client stub:
...
Options options = new Options();
options.setProperty(HTTPConstants.SO_TIMEOUT, new Integer(timeOutInMilliSeconds));
options.setProperty(HTTPConstants.CONNECTION_TIMEOUT, new Integer(timeOutInMilliSeconds));
// or
options.setTimeOutInMilliSeconds(timeOutInMilliSeconds);
...
If you want more information check: http://axis.apache.org/axis2/java/core/docs/http-transport.html
Also:
http://wso2.org/library/209
http://singztechmusings.wordpress.com/2011/05/07/how-to-configure-timeout-duration-at-client-side-for-axis2-web-services/
If you're using ServiceClient check this thread please: Axis2 ServiceClient options ignore timeout
Please let me know if it worked ;)
You should set JVM properties mentioned in following link -
https://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/rwbs_jaxwstimeouts.html
bindingProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.CONNECTION_TIMEOUT_PROPERTY, timeout);
bindingProvider.getRequestContext().put(com.ibm.wsspi.webservices.Constants.RESPONSE_TIMEOUT_PROPERTY, timeout);
Set timeout in seconds as String.

Embedded Jetty app with 2 ContextHandlers listening at same port - Loading issue

My embedded jetty app (using 6.1.26 jetty) has 2 context handlers registered to it. Both are listening at same port. Below is the sample.
Server s = new Server();
Connector c = new SelectChannelConnector();
((SelectChannelConnector)connector).setAcceptors(2);
connector.setHost(IP);
connector.setPort(port);
server.addConnector(connector);
ContextHandler context1 = new ContextHandler();
context.setContextPath("/abc");
context.setHandler(handler1);
context.setAllowNullPathInfo(true);
ContextHandler context2 = new ContextHandler();
context2.setContextPath("/xyz");
context2.setHandler(handler2);
context2.setAllowNullPathInfo(true);
ContextHandlerCollection hc = new ContextHandlerCollection();
hc.addHandler(context1);
hc.addHandler(context2);
server.setHandler(hc);
server.start();
I am also using a thread pool which is set at server level.
When I send requests to one context and put load on that so that all threads are used, At that time when I send a request to the second context its taking time to process the request to 2nd context.
I also tried by setting thread pool at SelectChannelConnector level and tried.
Also tried by adding more connectors using same host/port so that each will have its own thread pool.
My requirement is that other context (but port is same) should not delay processing when one context is under load.
Can I have dedicated thread pool for each context. Is there any other work around.
Appreciate reply to this.
Thanks
Sarath
With Jetty, the ThreadPool is at the connector level you can't have 2 different ThreadPools for handling different contexts. As the connector accepts the request, it pulls a Thread from the ThreadPool, and hands it off to the Server.getHandler() chain. at which point it goes through the hierarchy of handlers until one of your Contexts is used.
This means that the knowledge of the context comes in far too late to split up the ThreadPools.
Have you tried upgrading to Jetty 8 or Jetty 9 and using Async processing instead?
Or have you tried using the QoSFilter in Jetty 7, 8, or 9 to prioritize the handling better?

how to set connection/request timeout for jetty server?

I'm running an embedded jetty server (jetty 6.1.24) inside my application like this:
Handler handler=new AbstractHandler()
{
#Override
public void handle(String target, HttpServletRequest request,
HttpServletResponse response, int dispatch)
throws IOException, ServletException {
//this can take a long time
doSomething();
}
};
Server server = new Server(8080);
Connector connector = new org.mortbay.jetty.nio.SelectChannelConnector();
server.addConnector(connector);
server.setHandler(handler);
server.start();
I would like to set a timeout value (2 seconds) so that if handler.handle() method takes more than 2 seconds, jetty server will timeout and response to the client with 408 http code (request timeout).
This is to guarantee that my application will not hold the client request for a long time and always response within 2 seconds.
I did some research and tested it with "connector.setMaxIdleTime(2000);" but it doesn't work.
Take a look at the API for SelectChannelConnector (Jetty):
http://download.eclipse.org/jetty/7.6.17.v20150415/apidocs/org/eclipse/jetty/server/nio/SelectChannelConnector.html
I've tried to locate any timeout features of the channel (which controls incoming connections): setMaxIdleTime(), setLowResourceMaxIdleTime() and setSoLingerTime() are available it appears.
NOTE: the reason for your timeout feature not to work has to do with the nature of the socket on your operating system. Perhaps even the nature of Jetty (i've read about it somewhere, but cannot remember where it was).
NOTE2: i'm not sure why you try to limit the timeout, perhaps a better approach is limiting the buffer sizes? If you're trying to prevent denial of service...
Yes, this is possible. You could do this using DosFilter of Jetty. This filter is generally used to configure a DOS attack prevention mechanism for your Jetty web server. A property of this filter called 'MaxRequestMs' provides what you are looking for.
For more details, check this.
https://www.eclipse.org/jetty/javadoc/jetty-9/org/eclipse/jetty/servlets/DoSFilter.html