My APIM version is 1.10.0 and DAS is 3.0.1.
At first ,I deploy no Clustering APIM + DAS with mysql. the stats shows good.
Then,I clustering APIM into publisher、store、keymanager and gateway.Configured APIM and DAS order this article,when I invoke an api , gateway node then show error
[2016-10-13 11:13:54,775] ERROR - APIMgtUsageHandler Cannot publish event. null
java.lang.NullPointerException
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher.publishEvent(APIMgtUsageDataBridgeDataPublisher.java:124)
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageHandler.handleRequest(APIMgtUsageHandler.java:169)
at org.apache.synapse.rest.API.process(API.java:322)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:65)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:317)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:149)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-10-13 11:13:54,807] ERROR - APIMgtResponseHandler Cannot publish response event. null
java.lang.NullPointerException
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher.publishEvent(APIMgtUsageDataBridgeDataPublisher.java:140)
at org.wso2.carbon.apimgt.usage.publisher.APIMgtResponseHandler.mediate(APIMgtResponseHandler.java:211)
at org.apache.synapse.mediators.ext.ClassMediator.mediate(ClassMediator.java:84)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
at org.apache.synapse.rest.Resource.process(Resource.java:297)
at org.apache.synapse.rest.API.process(API.java:335)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:52)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:529)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:172)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:251)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Then i change the way of DAS configuration. Following this blog ,i configured all the api-manager.xml,there is no error and no stats.And store node show info:
[2016-10-18 11:31:39,131] INFO - ReceiverGroup Resending the failed published data...
[2016-10-18 11:31:44,134] WARN - AccessConfiguration Error loading properties from file: access-log.properties
[2016-10-18 11:31:44,207] INFO - TimeoutHandler This engine will expire all callbacks after : 120 seconds, irrespective of the timeout
I google the reason ,in wso2 jira,they say this is not a bug.
But I cannot wait for my stats for hours.Did I configure something wrong?
Reguards.
Seems in store node, data are publishing are enabled and working. But gateway node data not publishing. Please try to verify the gateway node configuration. Also check gateway node can access the DAS node.
Related
I have a WSO2 API Manager 2.5.0 installation, deployed in two nodes as active-active. It also has WSO2 Analytics and WSO2 Identity Server as Key Manager.
When each node starts, it leaves three times the same error in logs:
2018-11-14 07:56:50,989 [-] [DisruptorInboundEventThread-8] ERROR AndesSubscriptionManager Could not add subscription: subscriptionId=3091b9ba-867b-4539-bf25-fee511d1813d,storageQueue=AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101,protocolType=AMQP,isActive=true,subscriberConnection=Y29ubmVjdGVkSVA9LzEwLjAuMC40OjMzMTgyLzEsY29ubmVjdGVkTm9kZT1OT0RFOk9wZW5EYXRhQXBpTTEvMTAuMC4wLjQscHJvdG9jb2xDaGFubmVsSUQ9Y2Y3NDI4MDgtZGY3MS00NzJhLWFiMTEtOTY1Nzc2ZTBkNTZl
org.wso2.andes.kernel.subscription.SubscriptionException: StorageQueue: AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101 is not registered while creating subscription id=3091b9ba-867b-4539-bf25-fee511d1813d
at org.wso2.andes.kernel.subscription.AndesSubscription.<init>(AndesSubscription.java:136)
at org.wso2.andes.kernel.subscription.AndesSubscriptionManager.reloadSubscriptionsFromStorage(AndesSubscriptionManager.java:921)
at org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent.reloadSubscriptions(InboundDBSyncRequestEvent.java:208)
at org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent.updateState(InboundDBSyncRequestEvent.java:76)
at org.wso2.andes.kernel.disruptor.inbound.InboundEventContainer.updateState(InboundEventContainer.java:268)
at org.wso2.andes.kernel.disruptor.inbound.StateEventHandler.onEvent(StateEventHandler.java:70)
at org.wso2.andes.kernel.disruptor.inbound.StateEventHandler.onEvent(StateEventHandler.java:40)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-14 07:56:50,991 [-] [DisruptorInboundEventThread-8] ERROR AndesSubscriptionManager Could not add subscription: subscriptionId=c9a279eb-6c1f-447c-84bc-c077d33a06e1,storageQueue=AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101,protocolType=AMQP,isActive=true,subscriberConnection=Y29ubmVjdGVkSVA9LzEwLjAuMC40OjM0NTA0LzEsY29ubmVjdGVkTm9kZT1OT0RFOk9wZW5EYXRhQXBpTTEvMTAuMC4wLjQscHJvdG9jb2xDaGFubmVsSUQ9NGI5YmVmZmUtNjk1OC00M2Q4LTg3NGYtMzA5YmE5M2IyNzMw
org.wso2.andes.kernel.subscription.SubscriptionException: StorageQueue: AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101 is not registered while creating subscription id=c9a279eb-6c1f-447c-84bc-c077d33a06e1
at org.wso2.andes.kernel.subscription.AndesSubscription.<init>(AndesSubscription.java:136)
at org.wso2.andes.kernel.subscription.AndesSubscriptionManager.reloadSubscriptionsFromStorage(AndesSubscriptionManager.java:921)
at org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent.reloadSubscriptions(InboundDBSyncRequestEvent.java:208)
at org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent.updateState(InboundDBSyncRequestEvent.java:76)
at org.wso2.andes.kernel.disruptor.inbound.InboundEventContainer.updateState(InboundEventContainer.java:268)
at org.wso2.andes.kernel.disruptor.inbound.StateEventHandler.onEvent(StateEventHandler.java:70)
at org.wso2.andes.kernel.disruptor.inbound.StateEventHandler.onEvent(StateEventHandler.java:40)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-14 07:56:50,992 [-] [DisruptorInboundEventThread-8] ERROR AndesSubscriptionManager Could not add subscription: subscriptionId=3091b9ba-867b-4539-bf25-fee511d1813d,storageQueue=AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101,protocolType=AMQP,isActive=true,subscriberConnection=Y29ubmVjdGVkSVA9LzEwLjAuMC40OjMzMTgyLzEsY29ubmVjdGVkTm9kZT1OT0RFOk9wZW5EYXRhQXBpTTEvMTAuMC4wLjQscHJvdG9jb2xDaGFubmVsSUQ9Y2Y3NDI4MDgtZGY3MS00NzJhLWFiMTEtOTY1Nzc2ZTBkNTZl
org.wso2.andes.kernel.subscription.SubscriptionException: StorageQueue: AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101 is not registered while creating subscription id=3091b9ba-867b-4539-bf25-fee511d1813d
at org.wso2.andes.kernel.subscription.AndesSubscription.<init>(AndesSubscription.java:136)
at org.wso2.andes.kernel.subscription.AndesSubscriptionManager.reloadSubscriptionsFromStorage(AndesSubscriptionManager.java:972)
at org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent.reloadSubscriptions(InboundDBSyncRequestEvent.java:208)
at org.wso2.andes.kernel.disruptor.inbound.InboundDBSyncRequestEvent.updateState(InboundDBSyncRequestEvent.java:76)
at org.wso2.andes.kernel.disruptor.inbound.InboundEventContainer.updateState(InboundEventContainer.java:268)
at org.wso2.andes.kernel.disruptor.inbound.StateEventHandler.onEvent(StateEventHandler.java:70)
at org.wso2.andes.kernel.disruptor.inbound.StateEventHandler.onEvent(StateEventHandler.java:40)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I have no idea about the problem. I have looked for the error on the internet, but there are no similar cases.
Does anyone have any ideas?
2018-11-14 07:56:50,989 [-] [DisruptorInboundEventThread-8] ERROR AndesSubscriptionManager Could not add subscription: subscriptionId=3091b9ba-867b-4539-bf25-fee511d1813d,storageQueue=AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101,protocolType=AMQP,isActive=true,subscriberConnection=Y29ubmVjdGVkSVA9LzEwLjAuMC40OjMzMTgyLzEsY29ubmVjdGVkTm9kZT1OT0RFOk9wZW5EYXRhQXBpTTEvMTAuMC4wLjQscHJvdG9jb2xDaGFubmVsSUQ9Y2Y3NDI4MDgtZGY3MS00NzJhLWFiMTEtOTY1Nzc2ZTBkNTZl
org.wso2.andes.kernel.subscription.SubscriptionException: StorageQueue: AMQP_Topic_throttledata_NODE:OpenDataApiM1/10.0.0.101 is not registered while creating subscription id=3091b9ba-867b-4539-bf25-fee511d1813d
You might have probably found out the answer for this error. But the possible cause for the issue is when the APIM is unable to identify the correct topic connection factory related to its own node due to the sharing of the 'WSO2_MB_STORE_DB' between the nodes.
If you refer to their Installing and configuring databases documentation, it has mentioned that each Traffic Manager node must have its own Message Broker database (WSO2_MB_STORE_DB).
Therefore, please check the database configurations under the <APIM_Home>/repository/conf/master-datasource.xml file in both nodes whether you have shared the 'WSO2_MB_STORE_DB' between the nodes. If that is the case, you can point them to two separate fresh databases to get rid of this error.
I have a fairly simple dataflow job that reads from BigQuery and writes to a Spanner instance. All the reading has completed but for some reason it is consistently failing with an EOFException when starting the write to Spanner and it is in the class MutationGroupEncoder. There are about 5.8M rows and 1.36GB of data that have been partitioned, but then after writing about 3500 it fails.
Exception trace below - none of our code is involved in the stack trace, it's all library code that is marshaling the output data around. I haven't been able to find anyone else reporting a similar bug. We are on version 2.5.0 of the Google Cloud Apache Beam SDK.
java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: java.io.EOFException
at com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:183)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
at com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:124)
at com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:53)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:115)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:113)
at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:391)
at com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:360)
at com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:288)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: java.io.EOFException
at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:146)
at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:181)
... 21 more
Caused by: java.lang.RuntimeException: java.io.EOFException
at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode(MutationGroupEncoder.java:271)
at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement(SpannerIO.java:1030)
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.readBytes(MutationGroupEncoder.java:475)
at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodePrimitive(MutationGroupEncoder.java:434)
at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeModification(MutationGroupEncoder.java:326)
at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeMutation(MutationGroupEncoder.java:280)
at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode(MutationGroupEncoder.java:264)
at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement(SpannerIO.java:1030)
at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement(Unknown Source)
at org.apache.beam.runners.core.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:185)
at org.apache.beam.runners.core.SimpleDoFnRunner.processElement(SimpleDoFnRunner.java:146)
at com.google.cloud.dataflow.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:323)
at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn$1.output(GroupAlsoByWindowsParDoFn.java:181)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner$1.outputWindowedValue(GroupAlsoByWindowFnRunner.java:102)
at com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:124)
at com.google.cloud.dataflow.worker.util.BatchGroupAlsoByWindowViaIteratorsFn.processElement(BatchGroupAlsoByWindowViaIteratorsFn.java:53)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:115)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
at com.google.cloud.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:113)
at com.google.cloud.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:43)
at com.google.cloud.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:48)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:200)
at com.google.cloud.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:158)
at com.google.cloud.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:75)
at com.google.cloud.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:391)
at com.google.cloud.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:360)
at com.google.cloud.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:288)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:134)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:114)
at com.google.cloud.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:101)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
had same issue and it's got resolved after changing to 2.6.0 , and was able to upload 100 GB (350 Million records). But i observed dataflow jobs are slow when compared with 2.4.0.
Using wso2am-2.2.0 with wum we are running gateway instances in a cluster as -Dprofile=gateway-manager
in the logs after the startup I see an exception
Caused by: java.lang.NoClassDefFoundError: org/wso2/carbon/apimgt/micro/gateway/usage/publisher/util/UsagePublisherException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.getDeclaredConstructor(Class.java:2178)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:78)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory
.java:1032)
... 135 more
Caused by: java.lang.ClassNotFoundException: org.wso2.carbon.apimgt.micro.gateway.usage.publisher.util.UsagePublisherException
at org.wso2.carbon.webapp.mgt.l
When the gateway is used (an API is invoked), I see following exception:
TID: [-1234] [] ERROR {org.wso2.carbon.apimgt.gateway.throttling.publisher.ThrottleDataPublisher} - Error while publishing throttling events to global policy server {org.wso2.carbon.apimgt.gateway.throttling.publisher.ThrottleDataPublisher}
java.lang.NullPointerException
at org.wso2.carbon.apimgt.gateway.throttling.publisher.ThrottleDataPublisher.publishNonThrottledEvent(ThrottleDataPublisher.java:121)
at org.wso2.carbon.apimgt.gateway.handlers.throttling.ThrottleHandler.doRoleBasedAccessThrottlingWithCEP(ThrottleHandler.java:349)
at org.wso2.carbon.apimgt.gateway.handlers.throttling.ThrottleHandler.doThrottle(ThrottleHandler.java:512)
at org.wso2.carbon.apimgt.gateway.handlers.throttling.ThrottleHandler.handleRequest(ThrottleHandler.java:460)
at org.apache.synapse.rest.API.process(API.java:325)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:90)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:69)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:303)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:92)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:337)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:158)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
and as well seems the throttling data are not propagated (I don't see them processed by the analytics)
I am running WSO2 IS 5.0.0. I have the SP for IS 5.0.0 applied along with all the other security patches issued for that version for Identity Server and Carbon 4.2.0. My environment consists of 4 machines that are creating a cluster (using the WKA membership scheme and Load Balancer(AWS ELB) with sticky session enabled). I am using MySQL(not the default H2 database). The machines on which the IS is deployed are Windows Server 2012 R2 (EC2 AWS machines).
I am constantly receiving in the console log files "Deployment synchronization commit for tenant -1234 failed". After I applied the changes that #ycr proposed in order to disable the Dep-Sync:
<DeploymentSynchronizer>
<Enabled>false</Enabled>
<AutoCommit>false</AutoCommit>
<AutoCheckout>false</AutoCheckout>
<RepositoryType>svn</RepositoryType>
<SvnUrl>http://svnrepo.example.com/repos/</SvnUrl>
<SvnUser>username</SvnUser>
<SvnPassword>password</SvnPassword>
<SvnUrlAppendTenantId>false</SvnUrlAppendTenantId>
and restarted all of my machines. I had no issues for about 2 weeks. Then suddenly I received the the same error but with additional stack trace on two of my machines today (29.09.2016):
[2016-09-29 04:42:24,000] ERROR {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} - Deployment synchronization commit for tenant -1234 failed
com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! InvocationFuture{invocation=BasicInvocation{ serviceName='hz:impl:mapService', op=GetOperation{}, partitionId=247, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[172.31.2.242]:4000}, done=false} No response has been send backups-expected: 0 backups-completed: 0 reinvocations: 0
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.newOperationTimeoutException(BasicInvocation.java:782)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.waitForResponse(BasicInvocation.java:760)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:697)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
at com.hazelcast.map.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:257)
at com.hazelcast.map.proxy.MapProxySupport.getInternal(MapProxySupport.java:161)
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53)
at org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:130)
at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:182)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache.getPathID(JDBCPathCache.java:299)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceID(JDBCResourceDAO.java:81)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:151)
at org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.resourceExists(EmbeddedRegistry.java:644)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:293)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777)
at org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108)
at org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
at ------ End remote and begin local stack-trace ------.(Unknown Source)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponse(BasicInvocation.java:862)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.resolveResponseOrThrowException(BasicInvocation.java:795)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:698)
at com.hazelcast.spi.impl.BasicInvocation$InvocationFuture.get(BasicInvocation.java:676)
at com.hazelcast.map.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:257)
at com.hazelcast.map.proxy.MapProxySupport.getInternal(MapProxySupport.java:161)
at com.hazelcast.map.proxy.MapProxyImpl.get(MapProxyImpl.java:53)
at org.wso2.carbon.core.clustering.hazelcast.HazelcastDistributedMapProvider$DistMap.get(HazelcastDistributedMapProvider.java:130)
at org.wso2.carbon.caching.impl.CacheImpl.get(CacheImpl.java:182)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache.getPathID(JDBCPathCache.java:299)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.getResourceID(JDBCResourceDAO.java:81)
at org.wso2.carbon.registry.core.jdbc.dao.JDBCResourceDAO.resourceExists(JDBCResourceDAO.java:151)
at org.wso2.carbon.registry.core.jdbc.Repository.resourceExists(Repository.java:134)
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.resourceExists(EmbeddedRegistry.java:644)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.resourceExists(CacheBackedRegistry.java:293)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExistsInternal(UserRegistry.java:777)
at org.wso2.carbon.registry.core.session.UserRegistry.access$800(UserRegistry.java:60)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:760)
at org.wso2.carbon.registry.core.session.UserRegistry$9.run(UserRegistry.java:757)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.registry.core.session.UserRegistry.resourceExists(UserRegistry.java:757)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getDeploymentSyncConfigurationFromRegistry(CarbonRepositoryUtils.java:262)
at org.wso2.carbon.deployment.synchronizer.internal.repository.CarbonRepositoryUtils.getActiveSynchronizerConfiguration(CarbonRepositoryUtils.java:108)
at org.wso2.carbon.deployment.synchronizer.internal.DeploymentSynchronizerServiceImpl.commit(DeploymentSynchronizerServiceImpl.java:96)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.deploymentSyncCommit(CarbonDeploymentSchedulerTask.java:207)
at org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask.run(CarbonDeploymentSchedulerTask.java:128)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Are there any other configuration files that I need to tweak?
Thanks in advance.
Are you using any secondary user stores? If not you do not need Dep-sync. You can simply disable it. If you are using Secondary user stores the best option is to avoid using Dep-sync and copy secondary user store related configurations manually.
I am currently trying to do the example presented in this Link:http://wso2.org/library/articles/2013/03/configuring-wso2-esb-wso2-message-broker. I am trying to post message to topic of message broker but everytime i access my proxy server and try to send message from esb proxy, i get this error. I dont know what went wrong. Till morning this was working. My error is:
[2013-04-30 16:40:27,516] ERROR - ClientUtils The system cannot infer the transport information from the jms:/myTopic?&t
ransport.jms.DestinationType=topic URL.
[2013-04-30 16:40:27,526] ERROR - Axis2Sender Unexpected error during sending message out
org.apache.axis2.AxisFault: The system cannot infer the transport information from the jms:/myTopic?&transport.jms.Desti
nationType=topic URL.
at org.apache.axis2.description.ClientUtils.inferOutTransport(ClientUtils.java:81)
at org.apache.axis2.client.OperationClient.prepareMessageContext(OperationClient.java:288)
at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:249)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.synapse.core.axis2.Axis2FlexibleMEPClient.send(Axis2FlexibleMEPClient.java:456)
at org.apache.synapse.core.axis2.Axis2Sender.sendOn(Axis2Sender.java:57)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:316)
at org.apache.synapse.endpoints.AbstractEndpoint.send(AbstractEndpoint.java:329)
at org.apache.synapse.endpoints.AddressEndpoint.send(AddressEndpoint.java:59)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:175)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at org.apache.synapse.transport.nhttp.ServerWorker.processEntityEnclosingMethod(ServerWorker.java:476)
at org.apache.synapse.transport.nhttp.ServerWorker.run(ServerWorker.java:296)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
[2013-04-30 16:40:27,526] WARN - EndpointContext Endpoint : endpoint_1afcbfaeb203878a99e989ab5fa10f9e1385b22f66654582 w
ill be marked SUSPENDED as it failed
[2013-04-30 16:40:27,526] WARN - EndpointContext Suspending endpoint : endpoint_1afcbfaeb203878a99e989ab5fa10f9e1385b22
f66654582 - current suspend duration is : 30000ms - Next retry after : Tue Apr 30 16:40:57 IST 2013
Thanks in advance
Strange. You say it was working until today morning. But according to the exception, JMS transport sender is not enabled in your ESB's axis2.xml. Please check whether it is enabled.