i am using Wso2ESb 4.7.0 latest version and Wso2DSS 3.0.0
its working fine some times clients sending parallel requests that time its giving some ERRORS and Warnings
like this
SourceHandler I/O error (Probably the connection was closed by the remote party):Broken pipe
WARN - SourceHandler Connection time out after request is read: http-incoming-2240
18,799] WARN - SourceHandler Connection time out while writing the response: http-incoming-2156
[2013-08-14 17:10:19,801] WARN - SourceHandler Connection time out while writing the response: http-incoming-2161
[2013-08-14 17:10:19,801] WARN - SourceHandler Connection time out while writing the response: http-incoming-2171
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2162
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2178
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2159
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2173
[2013-08-14 17:10:19,803] WARN - SourceHandler Connection time out while writing the response: http-incoming-2179
[2013-08-14 17:10:19,803] WARN - SourceHandler Connection time out while writing the response: http-incoming-2174
[2013-08-14 17:13:50,157] INFO - LogMediator To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:b522b7bb-c01c-4b6c-a214-3cb00cb6c663, Direction: response, clientid =
In this case my application not working how we can avoid this
for more scalability.Wso2esb not forwarding messages
Related
I have a Kafka cluster with:
2 brokers b-1 and b-2.
2 topics with both: PartitionCount:1 ReplicationFactor:2 min.insync.replicas=1
Here is what happened:
%6|1613807298.974|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/2: Disconnected (after 3829996ms in state UP)
%3|1613807299.011|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/2: Connect to ipv4#172.31.18.172:9096 failed: Connection refused (after 36ms in state CONNECT)
%3|1613807299.128|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/2: Connect to ipv4#172.31.18.172:9096 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%4|1613807907.225|REQTMOUT|rdkafka#producer-2| [thrd:sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/2: Timed out 0 in-flight, 0 retry-queued, 1 out-queue, 1 partially-sent requests
%3|1613807907.225|FAIL|rdkafka#producer-2| [thrd:sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-2.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/2: 1 request(s) timed out: disconnect (after 343439ms in state UP)
%5|1613807938.942|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/1: Timed out ProduceRequest in flight (after 60767ms, timeout #0)
%5|1613807938.943|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/1: Timed out ProduceRequest in flight (after 60459ms, timeout #1)
%5|1613807938.943|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/1: Timed out ProduceRequest in flight (after 60342ms, timeout #2)
%5|1613807938.943|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/1: Timed out ProduceRequest in flight (after 60305ms, timeout #3)
%5|1613807938.943|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/1: Timed out ProduceRequest in flight (after 60293ms, timeout #4)
%4|1613807938.943|REQTMOUT|rdkafka#producer-1| [thrd:sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/1: Timed out 6 in-flight, 0 retry-queued, 0 out-queue, 0 partially-sent requests
%3|1613807938.943|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.am]: sasl_ssl://b-1.xxxx.kafka.ap-northeast-1.amazonaws.com:9096/1: 6 request(s) timed out: disconnect (after 4468987ms in state UP)
Within the code, I got this error when my producer performed a poll around that time:
2021-02-20 07:59:08,174 - ERROR - Failed to deliver message due to error: KafkaError{code=REQUEST_TIMED_OUT,val=7,str="Broker: Request timed out"}
Broker b-2 logs have this:
[2021-02-20 07:57:24,781] WARN Client session timed out, have not heard from server in 15103ms for sessionid 0x2000190b5d40001 (org.apache.zookeeper.ClientCnxn)
[2021-02-20 07:57:24,782] WARN Client session timed out, have not heard from server in 12701ms for sessionid 0x2000190b5d40000 (org.apache.zookeeper.ClientCnxn)
[2021-02-20 07:57:24,931] INFO Client session timed out, have not heard from server in 12701ms for sessionid 0x2000190b5d40000, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2021-02-20 07:57:24,932] INFO Client session timed out, have not heard from server in 15103ms for sessionid 0x2000190b5d40001, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2021-02-20 07:57:32,884] INFO Opening socket connection to server INTERNAL_ZK_DNS/INTERNAL_IP. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-02-20 07:57:32,910] INFO Opening socket connection to server INTERNAL_ZK_DNS/INTERNAL_IP. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2021-02-20 07:57:33,032] INFO Socket connection established to INTERNAL_ZK_DNS/INTERNAL_IP, initiating session (org.apache.zookeeper.ClientCnxn)
[2021-02-20 07:57:33,032] INFO Socket connection established to INTERNAL_ZK_DNS/INTERNAL_IP, initiating session (org.apache.zookeeper.ClientCnxn
My understanding here is that (1) b-2 went down i.e. unable to connect to Zookeeper (2) Messages were produced to b-1 successfully during this time. (3) b-1 was also trying to forward messages to b-2during this downtime due to the replication factor set to 2 (4) All these forwarded messages (ProduceRequests) got timed-out after 600s
My question:
Is my understanding correct and how I can prevent this from happening again?
If I had 3 brokers here, would b-1 have tried to connect to b-3 right away rather than waiting for b-2? Is that a good workaround? (Assuming topic replication factor = 2 everywhere)
finished WSO2 installation and first time I start the service and try to log into the management console it's not accessible (hosted on a Amazon EC2 unix instance).
Based on the log file the server starts normally:
[2017-03-31 22:40:29,622] [EI-Core] INFO - CarbonUIServiceComponent Mgt Console URL : https://XXX.XXX.XXX.XXX:8243/carbon/
When I try logging into mgt console the browser is hang up and won't finish loading the page. On wso2carbon.log I get the log entries below:
[2017-03-31 22:42:51,456] [EI-Core] WARN - EndpointContext Endpoint : AnonymousEndpoint with address https://XXX.XXX.XXX.XXX:37045 will be marked SUSPENDED as it failed
[2017-03-31 22:42:51,457] [EI-Core] WARN - EndpointContext Suspending endpoint : AnonymousEndpoint with address https://XXX.XXX.XXX.XXX:37045 - current suspend duration is : 30000ms - Next retry after : Fri Mar 31 22:43:21 UTC 2017
[2017-03-31 22:43:44,246] [EI-Core] INFO - SourceHandler Writer null when calling informWriterError
[2017-03-31 22:43:44,247] [EI-Core] WARN - SourceHandler Connection time out after request is read: http-incoming-1 Socket Timeout : 180000 Remote Address : YYY.YYY.YYY.YYY/179.212.90.90:4529
[2017-03-31 22:45:51,671] [EI-Core] WARN - ConnectCallback Connection refused or failed for : /XXX.XXX.XXX.XXX:37045
[2017-03-31 22:45:51,672] [EI-Core] WARN - EndpointContext Endpoint : AnonymousEndpoint with address https://XXX.XXX.XXX.XXX:37045 will be marked SUSPENDED as it failed
[2017-03-31 22:45:51,672] [EI-Core] WARN - EndpointContext Suspending endpoint : AnonymousEndpoint with address https://XXX.XXX.XXX.XXX:37045 - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Fri Mar 31 22:46:21 UTC 2017
[2017-03-31 22:46:44,662] [EI-Core] INFO - SourceHandler Writer null when calling informWriterError
[2017-03-31 22:46:44,663] [EI-Core] WARN - SourceHandler Connection time out after request is read: http-incoming-2 Socket Timeout : 180000 Remote Address : 179.212.90.90/179.212.90.90:4592
[2017-03-31 22:48:52,151] [EI-Core] WARN - ConnectCallback Connection refused or failed for : /XXX.XXX.XXX.XXX:37045
[2017-03-31 22:48:52,151] [EI-Core] WARN - EndpointContext Endpoint : AnonymousEndpoint with address https://XXX.XXX.XXX.XXX:37045 will be marked SUSPENDED as it failed
[2017-03-31 22:48:52,151] [EI-Core] WARN - EndpointContext Suspending endpoint : AnonymousEndpoint with address https://XXX.XXX.XXX.XXX:37045 - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Fri Mar 31 22:49:22 UTC 2017
XXX.XXX.XXX.XXX is the local host IP. YYY.YYY.YYY.YYY is my local IP from where I am trying to connect to the Amazon instance.
I am just trying to open mgt console but it seems the ESB is trying to process my request as a regular endpoint request.
The most confusing piece is that I can open the Analytics dashboard (on 9444) and BPEL mgt console (on 9445) that ships inside EI 6.0.0. Only the Enterprise Integrator mgt console accessible under 8243 port is not reachable.
Thanks in advance.
During login into EI console (ESB+DSS+AS console on 8243 port) EI trying to connect some other services. In my cases I've got (in some installations) Name Notification error at this point.
In your case you received timeout. Probably these port (37045) is closed for IP XXX.XXX.XXX.XXX
Try to configure AWS Security Group to open this port
We already have Active MQ running on the server. As a result we are getting port conflict when running WSO2. Where can we change the port to avoid this conflict.
So far we have done the following:
1. Searched for references for port 5672 under /repository/conf and changed 5672 to 5673. Files changed included broker and jndi.
Looks like the conflict is resolved but some services are still trying to connect to port 5672. I have not been able to find where to change this.
[2017-02-03 08:37:36,342] INFO - QpidServiceComponent WSO2 Message Broker is started.
[2017-02-03 08:37:36,374] INFO - JMSConnectionFactory JMS ConnectionFactory : Siddhi-JMS-Consumer initialized
[2017-02-03 08:37:36,379] INFO - JMSTransportHandler Starting jms topic consumer thread...
[2017-02-03 08:37:36,800] INFO - PassThroughHttpListener Starting Pass-through HTTP Listener...
[2017-02-03 08:37:36,811] INFO - PassThroughListeningIOReactorManager Pass-through HTTP Listener started on 0:0:0:0:0:0:0:0:8280
[2017-02-03 08:37:36,815] INFO - PassThroughHttpSSLListener Starting Pass-through HTTPS Listener...
[2017-02-03 08:37:36,825] INFO - PassThroughListeningIOReactorManager Pass-through HTTPS Listener started on 0:0:0:0:0:0:0:0:8243
[2017-02-03 08:37:37,669] INFO - AMQConnection Unable to connect to broker at tcp://172.17.51.124:5672
org.wso2.andes.transport.TransportException: Could not open connection
at org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:216)
at org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:74)
at org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:130)
at org.wso2.andes.client.AMQConnection$2.run(AMQConnection.java:631)
at org.wso2.andes.client.AMQConnection$2.run(AMQConnection.java:628)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:628)
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:409)
at org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:351)
at org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:40)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSUtils.createConnection(JMSUtils.java:387)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSListener.checkJMSConnection(JMSListener.java:137)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSListener.start(JMSListener.java:79)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSListener.run(JMSListener.java:186)
at java.lang.Thread.run(Thread.java:745)
org.wso2.andes.AMQConnectionFailureException: Could not open connection
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:486)
at org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:351)
at org.wso2.andes.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:40)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSUtils.createConnection(JMSUtils.java:387)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSListener.checkJMSConnection(JMSListener.java:137)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSListener.start(JMSListener.java:79)
at org.wso2.carbon.apimgt.jms.listener.utils.JMSListener.run(JMSListener.java:186)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.andes.transport.TransportException: Could not open connection
at org.wso2.andes.transport.network.mina.MinaNetworkTransport$IoConnectorCreator.connect(MinaNetworkTransport.java:216)
at org.wso2.andes.transport.network.mina.MinaNetworkTransport.connect(MinaNetworkTransport.java:74)
at org.wso2.andes.client.AMQConnectionDelegate_8_0.makeBrokerConnection(AMQConnectionDelegate_8_0.java:130)
at org.wso2.andes.client.AMQConnection$2.run(AMQConnection.java:631)
at org.wso2.andes.client.AMQConnection$2.run(AMQConnection.java:628)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.andes.client.AMQConnection.makeBrokerConnection(AMQConnection.java:628)
at org.wso2.andes.client.AMQConnection.<init>(AMQConnection.java:409)
... 7 more
[2017-02-03 08:37:37,750] ERROR - JMSListener Unable to continue server startup as it seems the JMS Provider is not yet started. Please start the JMS provider now.
[2017-02-03 08:37:37,756] ERROR - JMSListener Connection attempt : 1 for JMS Provider failed. Next retry in 20 seconds
[2017-02-03 08:37:39,373] INFO - CarbonEventManagementService Starting polling event receivers
[2017-02-03 08:37:43,411] INFO - ThriftDataReceiver Thrift Server started at 0.0.0.0
[2017-02-03 08:37:43,451] INFO - ThriftDataReceiver Thrift SSL port : 7711
[2017-02-03 08:37:43,468] INFO - ThriftDataReceiver Thrift port : 7611
We resolved this by using port offsets. No need to manually change ports.
https://docs.wso2.com/display/AM200/Changing+the+Default+Ports+with+Offset
I used previous version of wso2 esb and now I've shifted all my
services to latest version of wso2 esb 4.7.0, but still I am getting same error.
If clients sending parallel requests that time its giving some ERRORS and
Warnings like this:
SourceHandler I/O error (Probably the connection was closed by the remote party):Broken pipe
WARN - SourceHandler Connection time out after request is read: http-incoming-2240
18,799] WARN - SourceHandler Connection time out while writing the response: http-incoming-2156
[2013-08-14 17:10:19,801] WARN - SourceHandler Connection time out while writing the response: http-incoming-2161
[2013-08-14 17:10:19,801] WARN - SourceHandler Connection time out while writing the response: http-incoming-2171
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2162
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2178
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2159
[2013-08-14 17:10:19,802] WARN - SourceHandler Connection time out while writing the response: http-incoming-2173
[2013-08-14 17:10:19,803] WARN - SourceHandler Connection time out while writing the response: http-incoming-2179
[2013-08-14 17:10:19,803] WARN - SourceHandler Connection time out while writing the response: http-incoming-2174
[2013-08-14 17:13:50,157] INFO - LogMediator To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:b522b7bb-c01c-4b6c-a214-3cb00cb6c663, Direction: response, clientid = ...
How can i solve this?
I am currently trying to do the example presented in this Link:http://wso2.org/library/articles/2013/03/configuring-wso2-esb-wso2-message-broker. I am trying to post message to topic of message broker but everytime i access my proxy server and try to send message from esb proxy, i get this error. I dont know what went wrong. Till morning this was working. My error is:
[2013-04-30 16:40:27,516] ERROR - ClientUtils The system cannot infer the transport information from the jms:/myTopic?&t
ransport.jms.DestinationType=topic URL.
[2013-04-30 16:40:27,526] ERROR - Axis2Sender Unexpected error during sending message out
org.apache.axis2.AxisFault: The system cannot infer the transport information from the jms:/myTopic?&transport.jms.Desti
nationType=topic URL.
at org.apache.axis2.description.ClientUtils.inferOutTransport(ClientUtils.java:81)
at org.apache.axis2.client.OperationClient.prepareMessageContext(OperationClient.java:288)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:249)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:456)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:57)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:316)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:329)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:59)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:175)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at org.apache.synapse.transport.nhttp.ServerWorker.processEntityEnclosingMethod(ServerWorker.java:476)
at org.apache.synapse.transport.nhttp.ServerWorker.run(ServerWorker.java:296)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
[2013-04-30 16:40:27,526] WARN - EndpointContext Endpoint : endpoint_1afcbfaeb203878a99e989ab5fa10f9e1385b22f66654582 w
ill be marked SUSPENDED as it failed
[2013-04-30 16:40:27,526] WARN - EndpointContext Suspending endpoint : endpoint_1afcbfaeb203878a99e989ab5fa10f9e1385b22
f66654582 - current suspend duration is : 30000ms - Next retry after : Tue Apr 30 16:40:57 IST 2013
Thanks in advance
Strange. You say it was working until today morning. But according to the exception, JMS transport sender is not enabled in your ESB's axis2.xml. Please check whether it is enabled.