Jetty Idle timeout exception - jetty

Jetty version - 9.4.8.v20171121
I am making a request to jetty server and it takes a long time before it sends a response. That is okay for us.
I made the idle timeout as 9 seconds for jetty.
Jetty is continuously giving these exceptions
[s3proxy] DEBUG 07-19 11:33:55.362 Scheduler-1321115948 org.eclipse.jetty.io.IdleTimeout:156 |:: ] SocketChannelEndPoint#1bccc548{/127.0.0.1:63602<->/127.0.0.1:8090,OPEN,fill=-,flush=-,to=9001/9000}{io=0/0,kio=0,kro=1}->HttpConnection#2b389aa6[p=HttpParser{s=END,0 of 0},g=HttpGenerator#562a5b2a{s=START}]=>HttpChannelOverHttp#2b312222{r=1,c=false,a=ASYNC_WAIT,uri=//127.0.0.1:8090/holani/testfilec2e0d109-0dc8-4b72-aaa0-b3b7c7bf57e9} idle timeout check, elapsed: 9001 ms, remaining: -1 ms
[s3proxy] DEBUG 07-19 11:33:55.362 Scheduler-1321115948 org.eclipse.jetty.io.IdleTimeout:163 |:: ] SocketChannelEndPoint#1bccc548{/127.0.0.1:63602<->/127.0.0.1:8090,OPEN,fill=-,flush=-,to=9001/9000}{io=0/0,kio=0,kro=1}->HttpConnection#2b389aa6[p=HttpParser{s=END,0 of 0},g=HttpGenerator#562a5b2a{s=START}]=>HttpChannelOverHttp#2b312222{r=1,c=false,a=ASYNC_WAIT,uri=//127.0.0.1:8090/holani/testfilec2e0d109-0dc8-4b72-aaa0-b3b7c7bf57e9} idle timeout expired
[s3proxy] DEBUG 07-19 11:33:55.362 Scheduler-1321115948 org.eclipse.jetty.io.FillInterest:130 |:: ] onFail FillInterest#6b7ab09f{null}
java.util.concurrent.TimeoutException: Idle timeout expired: 9001/9000 ms
at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) [s3-1.5.1.jar:1.5.1]
at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) [s3-1.5.1.jar:1.5.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_172]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_172]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
But once my server has the response ready, it sends it correctly and my client also gets the correct response.
How is this working ?
If the connection has been timed out, it should not be able to send the response back right ?

Related

email sending via wso2 identity server fails

I am using wso2 identity server 5.9.0 on my linux machine.
I am getting the error below while doing account creation.
I get message Email sent successfully, but at backend it throws error after some time and the mail is never received.
[2020-02-15 13:09:22,241] [725c9824-a414-44f4-9b4f-e5f8da8db48c] ERROR
{org.wso2.carbon.event.output.adapter.email.EmailEventAdapter} - Event
dropped at Output Adapter 'EmailPublisher' for tenant id '-1234',
Error in message format, Exception reading response
javax.mail.MessagingException: Exception reading response; nested
exception is:
java.net.SocketException: Connection timed out (Read failed)
at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2460)
at com.sun.mail.smtp.SMTPTransport.ehlo(SMTPTransport.java:1699)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:744)
at javax.mail.Service.connect(Service.java:366)
at javax.mail.Service.connect(Service.java:246)
at javax.mail.Service.connect(Service.java:195)
at javax.mail.Transport.send0(Transport.java:254)
at javax.mail.Transport.send(Transport.java:124)
at org.wso2.carbon.event.output.adapter.email.EmailEventAdapter$EmailSender.run(EmailEventAdapter.java:306)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by: java.net.SocketException: Connection timed out (Read failed)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.sun.mail.util.TraceInputStream.read(TraceInputStream.java:126)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at com.sun.mail.util.LineInputStream.readLine(LineInputStream.java:106)
at com.sun.mail.smtp.SMTPTransport.readServerResponse(SMTPTransport.java:2440)
... 13 more
I already made changes in output-event-adapters.xml and axis2.xml for smtp details.
Any help on this?
output-event-adapters.xml configuration is as below:
noreply#xyz.com
noreply#xyz.com
ABC.XXX.XXX.XXX
25
true
false
8
100
20000
10000
It seems like it is blocked by google. Refer to this doc
If you are using Gmail account you have to on "Allow less secure apps"
in your account.
For more info refer this

Cannot borrow client for ssl://wso2apim-analytics-service.domain.com:7712

I am setting up a multi-region WSO2 APIM environment. The gateway component in the slave region is failing to connect with the API Analytics component in the Master region with the following error:
[2019-09-02 09:24:02,563] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://wso2apim-analytics-service.domain.com:7712.
org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: Cannot borrow client for ssl://wso2apim-analytics-service.domain.com:7712.
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:134)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: Error while trying to login to the data receiver.
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:54)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:128)
... 6 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.send_connect(ThriftSecureEventTransmissionService.java:104)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:95)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:47)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
Analytics is already enabled in the slave region. Also, we are able to telnet on port 7712 to the analytics service in the master region.
Why is the "java.net.SocketTimeoutException: Read timed out" occurring?
There are a lot of points that you should check.
Domain resolution on the slave region.
Check the connection from the slave to the master , firewall rules , etc.
Check the certificate on both sides and that it match with the domain name.
Use openssl s_client -connect wso2apim-analytics-service.domain.com:7712
Good luck.
Once the debug logs are enabled for below class, when graphs are selected for analytics, it will log the Siddhi queries it sends to the analytics server.
This would help us to capture exact queries publisher portal sends to the analytics server and try to identify any culprit points available.
Shutdown the server.
Add the following property to /repository/conf/log4j.properties
log4j.logger.org.wso2.carbon.apimgt.impl.utils.APIUtil=DEBUG
Restart the server
Please refer to [1] for more details on logging.
[1] https://docs.wso2.com/display/AM260/Setting+Up+Logging

Camunda Receive Task

I have a question I have defined the following process in Camunda.
And deployed a servlet in the same application server that allows me to send signals to the deployed process easily.
However, everytime I call the signal method using the process instance id, I get an exception
03-Feb-2019 19:36:18.365 ERROR [o.c.b.engine.context] : ENGINE-16004 Exception while closing command context: cannot signal execution 9c3b0cbc-27da-11e9-ae94-0242ac140004: it has no current activity
org.camunda.bpm.engine.impl.pvm.PvmException: cannot signal execution 9c3b0cbc-27da-11e9-ae94-0242ac140004: it has no current activity
at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.signal(PvmExecutionImpl.java:716)
at org.camunda.bpm.engine.impl.cmd.SignalCmd.execute(SignalCmd.java:63)
at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:104)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.camunda.bpm.engine.impl.RuntimeServiceImpl.signal(RuntimeServiceImpl.java:423)
at systems.appollo.ams.camunda.signaler.SignalerServlet.doGet(SignalerServlet.java:26)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:94)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:502)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

Aborting a web service call from the server side

We have a web service where sometimes the clients are stuck in a loop and are harassing the server, using up threads and causing high CPU. We would like to force the client to hit a timeout, but not use a thread in our thread pool (by sleeping, for example). We're using Jetty to supply the web services:
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.sun.xml.ws.api.server.InstanceResolver$1.invoke(InstanceResolver.java:250)
at com.sun.xml.ws.server.InvokerTube$2.invoke(InvokerTube.java:149)
at com.sun.xml.ws.server.sei.SEIInvokerTube.processRequest(SEIInvokerTube.java:88)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:1063)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:979)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:950)
at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:825)
at com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:380)
at com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:651)
at com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:264)
at com.sun.xml.ws.transport.http.servlet.ServletAdapter.invokeAsync(ServletAdapter.java:218)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doGet(WSServletDelegate.java:159)
at com.sun.xml.ws.transport.http.servlet.WSServletDelegate.doPost(WSServletDelegate.java:194)
at com.sun.xml.ws.transport.http.servlet.WSServlet.doPost(WSServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:769)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at org.eclipse.jetty.server.handler.DebugHandler.handle(DebugHandler.java:81)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:485)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:290)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:606)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:535)
at java.lang.Thread.run(Thread.java:745)
Any way to have the web server abort the work but NOT return any response to the client?
UPDATE: For clarification - we control ONLY the server. We do not control the clients.
You may find this link informative!

Websockets on Atmosphere.js keep closing , possible bringing the server down

We have a GWT application deployed on AWS ELB.
We're using an Atmosphere.js and websocket set up to get number of live users every second to the UI.
I keep seeing this error on the UI:
Websocket closed, reason: Connection was closed abnormally (that is,
with no close frame being sent). - wasClean: false
and on the server side I see
02-Feb-2017 09:27:08.486 SEVERE [http-nio-8080-exec-5]
org.atmosphere.container.JSR356Endpoint.onError java.io.EOFException
at
org.apache.coyote.http11.upgrade.NioServletInputStream.doRead(NioServletInputStream.java:97) at
org.apache.coyote.http11.upgrade.AbstractServletInputStream.read(AbstractServletInputStream.java:124)
at
org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:60)
at
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsReadListener.onDataAvailable(WsHttpUpgradeHandler.java:185)
at
org.apache.coyote.http11.upgrade.AbstractServletInputStream.onDataAvailable(AbstractServletInputStream.java:198)
at
org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDispatch(AbstractProcessor.java:96) at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:647)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
EDIT :
Listener configuration:
At times when there is a lot of activity on this page, the server eventually crashes.
What could be the cause for the exception I see on the server side?