We have a GWT application deployed on AWS ELB.
We're using an Atmosphere.js and websocket set up to get number of live users every second to the UI.
I keep seeing this error on the UI:
Websocket closed, reason: Connection was closed abnormally (that is,
with no close frame being sent). - wasClean: false
and on the server side I see
02-Feb-2017 09:27:08.486 SEVERE [http-nio-8080-exec-5]
org.atmosphere.container.JSR356Endpoint.onError java.io.EOFException
at
org.apache.coyote.http11.upgrade.NioServletInputStream.doRead(NioServletInputStream.java:97) at
org.apache.coyote.http11.upgrade.AbstractServletInputStream.read(AbstractServletInputStream.java:124)
at
org.apache.tomcat.websocket.server.WsFrameServer.onDataAvailable(WsFrameServer.java:60)
at
org.apache.tomcat.websocket.server.WsHttpUpgradeHandler$WsReadListener.onDataAvailable(WsHttpUpgradeHandler.java:185)
at
org.apache.coyote.http11.upgrade.AbstractServletInputStream.onDataAvailable(AbstractServletInputStream.java:198)
at
org.apache.coyote.http11.upgrade.AbstractProcessor.upgradeDispatch(AbstractProcessor.java:96) at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:647)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
EDIT :
Listener configuration:
At times when there is a lot of activity on this page, the server eventually crashes.
What could be the cause for the exception I see on the server side?
Related
I am setting up WSO2 APIM with Analytics running on docker. I am getting the following error in the worker & no data is being published to the console. I'm using a MySQL database.
I am using the docker images # https://github.com/wso2/docker-apim/tree/v2.6.0.3/dockerfiles/centos
OpenJDK8U-jdk_x64_linux_hotspot_8u222b10
mysql-connector-java-5.1.47-bin.jar
[2019-10-05 04:58:27,208] ERROR {org.wso2.carbon.databridge.core.internal.queue.QueueWorker} - Dropping wrongly formatted event sent org.wso2.carbon.databridge.core.exception.EventConversionException: Error when converting org.wso2.metrics.stream.Gauge:1.0.0 of event bundle with events 4
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:188)
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.toEventList(ThriftEventConverter.java:90)
at org.wso2.carbon.databridge.core.internal.queue.QueueWorker.run(QueueWorker.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.wso2.carbon.databridge.core.exception.EventConversionException: No StreamDefinition for streamId org.wso2.metrics.stream.Gauge:1.0.0 present in cache
at org.wso2.carbon.databridge.receiver.thrift.converter.ThriftEventConverter.createEventList(ThriftEventConverter.java:171)
... 7 more
This issue occurs when the particular stream, "org.wso2.metrics.stream.Gauge:1.0.0", hasn't deployed properly from the capp. Below steps can be followed to resolve this.
Remove the /tmp directory which is available in the APIM_ANALYTICS_HOME directory.
Restart the server.
If there is any issue observed while deploying the capp, please backup the capp and remove it from the carbon console. (You can find it under the fault apps section. This is to remove any cache data related to the capp).
Then follow steps 1 and 2 again.
I am setting up a multi-region WSO2 APIM environment. The gateway component in the slave region is failing to connect with the API Analytics component in the Master region with the following error:
[2019-09-02 09:24:02,563] ERROR - DataEndpointConnectionWorker Error while trying to connect to the endpoint. Cannot borrow client for ssl://wso2apim-analytics-service.domain.com:7712.
org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: Cannot borrow client for ssl://wso2apim-analytics-service.domain.com:7712.
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:134)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: Error while trying to login to the data receiver.
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:54)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:128)
... 6 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:161)
at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:65)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.send_connect(ThriftSecureEventTransmissionService.java:104)
at org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:95)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:47)
... 7 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:750)
at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159)
Analytics is already enabled in the slave region. Also, we are able to telnet on port 7712 to the analytics service in the master region.
Why is the "java.net.SocketTimeoutException: Read timed out" occurring?
There are a lot of points that you should check.
Domain resolution on the slave region.
Check the connection from the slave to the master , firewall rules , etc.
Check the certificate on both sides and that it match with the domain name.
Use openssl s_client -connect wso2apim-analytics-service.domain.com:7712
Good luck.
Once the debug logs are enabled for below class, when graphs are selected for analytics, it will log the Siddhi queries it sends to the analytics server.
This would help us to capture exact queries publisher portal sends to the analytics server and try to identify any culprit points available.
Shutdown the server.
Add the following property to /repository/conf/log4j.properties
log4j.logger.org.wso2.carbon.apimgt.impl.utils.APIUtil=DEBUG
Restart the server
Please refer to [1] for more details on logging.
[1] https://docs.wso2.com/display/AM260/Setting+Up+Logging
Test cluster of two brokers, WKA membership scheme, PostgreSQL message store, working fine for a couple of days, then throwing following errors:
TID: [] [] [2016-07-19 12:09:24,738] ERROR {org.wso2.andes.server.protocol.MultiVersionProtocolEngine} - Error establishing session {org.wso2.andes.server.protocol.MultiVersionProtocolEngine}
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:218)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198)
at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45)
at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485)
at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51)
at java.lang.Thread.run(Thread.java:745)
Startup of Message Broker looks fine, no errors, JDBC connection to PostgreSQL DB is ok, Registry mount looks ok. Then after that error appears in wso2carbon.log several times/minute.
Anyone any ideas? As far as I know nothing's changed and I don't know what it's trying to connect to.
This usually happens when client's whom connected to MB tries to create connections per message. jms is heavy connection and not recommended to create connections per each message. Therefore, please go through client implementation and verify connections are not created per message.
If by any chance you are using wso2 esb to publish/subscribe queues/topics to mb there is a property "transport.jms.CacheLevel" connection caching in esb axis2.xml.Read the documentation and use appropriate caching level for your usecase.
There was bug in connection caching property to be ignored in esb 4.8.1 which is currently fixed in 4.9.0 as well.
These are the possible cases I can think of with the given information. If you need more info please provide a detailed usecase.
I created a web app which use mysql. I used spring for the persistence.
Every thing worked on my local tomcat server.
I upload it into cloudfoundary, I follow the instuction and I create a mysql service and I use it
as a service for my web-app.
I try to run the app and I got the following exceptions:
HTTP Status 500 -
--------------------------------------------------------------------------------
type Exception report
message
description The server encountered an internal error () that prevented it from fulfilling this request.
exception
org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:812)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:868)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:876)
com.tutorialspoint.StudentJDBCTemplate.create(StudentJDBCTemplate.java:19)
com.tutorialspoint.Hello.doGet(Hello.java:58)
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
root cause
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
java.sql.DriverManager.getConnection(DriverManager.java:582)
java.sql.DriverManager.getConnection(DriverManager.java:154)
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:812)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:868)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:876)
com.tutorialspoint.StudentJDBCTemplate.create(StudentJDBCTemplate.java:19)
com.tutorialspoint.Hello.doGet(Hello.java:58)
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
root cause
java.net.ConnectException: Connection refused
java.net.PlainSocketImpl.socketConnect(Native Method)
java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
java.net.Socket.connect(Socket.java:529)
java.net.Socket.connect(Socket.java:478)
java.net.Socket.<init>(Socket.java:375)
java.net.Socket.<init>(Socket.java:218)
com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
java.sql.DriverManager.getConnection(DriverManager.java:582)
java.sql.DriverManager.getConnection(DriverManager.java:154)
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriverManager(DriverManagerDataSource.java:173)
org.springframework.jdbc.datasource.DriverManagerDataSource.getConnectionFromDriver(DriverManagerDataSource.java:164)
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnectionFromDriver(AbstractDriverBasedDataSource.java:149)
org.springframework.jdbc.datasource.AbstractDriverBasedDataSource.getConnection(AbstractDriverBasedDataSource.java:119)
org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111)
org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77)
org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:812)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:868)
org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:876)
com.tutorialspoint.StudentJDBCTemplate.create(StudentJDBCTemplate.java:19)
com.tutorialspoint.Hello.doGet(Hello.java:58)
javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
note The full stack trace of the root cause is available in the Apache Tomcat/6.0.35 logs.
--------------------------------------------------------------------------------
Apache Tomcat/6.0.35
If the problem is still there: there are 2 ways to configure a connection to CF services.
1) Taking the advantage of "auto-reconfigure". For any Java app, if it is using Spring framework, the doc here specifically described the details: http://docs.cloudfoundry.com/frameworks/java/spring/spring.html
2) For Java apps, other than Spring apps, the details of connection to the CF provisioned services like hostname or password needs to be taken care explicitly. These can be retrieved by an environment variable named "VCAP_SERVICES". Simply this code snippet can achieve that:
System.getenv("VCAP_SERVICES")
After that set the properties to the connection configuration.
We are developing an application for an academic course of software architecture and we must use Java EE to implement that.
Essentially we have to implement a distributed Hearts game(you know the Windows game? I think so).
We want to use SOAP to make the client able to communicate with the server (glassfish in our case) but to avoid polling we need to implement the web service endpoint as a #WebServiceProvider.
Such kind of endopoit can implement the asyncProvider interface that allow us to decoplue soap request and soap response. But here we have 2 big problems:
The first one is that: glassfish (without any additional settings) can use a set of only 5 threads in the http-thread-pool. After the first five request we have the endopoint servlet "blocked". Is that the right mode to work of asyncProvider? After the invoke termination we were expecting that the servlet thread was released to process another one incoming request but, apparently, it does not work in this way. It's not a real "async" managment of the request. Are we wrong?
The second one: We are pretty sure that the problem we are introducing is strictly correlated with the first one. Making ten request with a thread pool size of 5 we have this scenario: The first five requests are well processed and, after a timeout of 20 seconds, they are correctly sent to the client. From the sixth to the last we have some mistake: in the server side everything is ok, but in the client-side and only sometimes we have an excpetion like this:
java.util.concurrent.ExecutionException: javax.xml.ws.WebServiceException: java.net.SocketException: Unexpected end of file from server
at com.sun.xml.ws.util.CompletedFuture.get(CompletedFuture.java:72)
at client.Client$1.handleResponse(Client.java:79)
at com.sun.xml.ws.client.AsyncResponseImpl.set(AsyncResponseImpl.java:125)
at com.sun.xml.ws.client.sei.AsyncMethodHandler$SEIAsyncInvoker$1.onCompletion(AsyncMethodHandler.java:202)
at com.sun.xml.ws.client.Stub$1.onCompletion(Stub.java:397)
at com.sun.xml.ws.api.pipe.Fiber.completionCheck(Fiber.java:503)
at com.sun.xml.ws.api.pipe.Fiber.run(Fiber.java:423)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.xml.ws.WebServiceException: java.net.SocketException: Unexpected end of file from server
at com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:214)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:162)
at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:93)
at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:629)
at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:588)
at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:573)
at com.sun.xml.ws.api.pipe.Fiber.run(Fiber.java:422)
... 3 more
Caused by: java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:769)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:766)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379)
at com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:210)
... 10 more
We want to highlight that the content of the soap response is validated and well-formed after checking it without any exception at the server side.
If we send a number of request lower than the http-thread-pool size we have no problem.
Any suggestion for us?