WSO2 APIM 2.0 Clustering manage/monitor in carbon UI - wso2

I deployed the cluster in one server, here is the node start log:
TID: [-1234] [] [2016-09-02 03:55:41,596] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Cluster domain: wso2.pubstore.domain {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [-1234] [] [2016-09-02 03:55:41,596] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Loading hazelcast configuration from axis2 clustering configuration {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [-1234] [] [2016-09-02 03:55:41,620] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Using wka based membership management scheme {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [-1234] [] [2016-09-02 03:55:41,628] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:10.222.47.201, Remote Host:null, Port: 4006, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils}
TID: [-1234] [] [2016-09-02 03:55:41,628] INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} - Added member: Host:10.222.47.201, Remote Host:null, Port: 4007, HTTP:-1, HTTPS:-1, Domain: null, Sub-domain:null, Active:true {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils}
TID: [-1] [] [2016-09-02 03:55:47,768] INFO {org.wso2.carbon.event.processor.manager.core.internal.CarbonEventManagementService} - Starting polling event receivers {org.wso2.carbon.event.processor.manager.core.internal.CarbonEventManagementService}
TID: [-1234] [] [2016-09-02 03:55:50,176] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Hazelcast initialized in 8544ms {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [-1234] [] [2016-09-02 03:55:50,274] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Local member: [faedd3de-2842-4447-a84e-0db43c86595d] - Host:10.222.47.201, Remote Host:null, Port: 4006, HTTP:8285, HTTPS:8248, Domain: wso2.pubstore.domain, Sub-domain:worker, Active:true {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
TID: [-1234] [] [2016-09-02 03:55:50,664] INFO {org.wso2.andes.server.cluster.coordination.hazelcast.HazelcastAgent} - Initializing Hazelcast Agent {org.wso2.andes.server.cluster.coordination.hazelcast.HazelcastAgent}
TID: [-1234] [] [2016-09-02 03:55:50,676] INFO {org.wso2.andes.server.cluster.coordination.hazelcast.HazelcastAgent} - Successfully initialized Hazelcast Agent {org.wso2.andes.server.cluster.coordination.hazelcast.HazelcastAgent}
TID: [-1234] [] [2016-09-02 03:55:50,686] WARN {org.wso2.carbon.event.processor.manager.core.internal.CarbonEventManagementService} - CEP started with clustering enabled, but SingleNode configuration given. {org.wso2.carbon.event.processor.manager.core.internal.CarbonEventManagementService}
TID: [-1234] [] [2016-09-02 03:55:50,697] INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent} - Cluster initialization completed {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent}
From carbon UI,
It seems carbon UI can't display cluster information.
So, my question is that can carbon UI display cluster information?

No, As far as I know you can't view cluster related information from UI. However, you can see whether the nodes are correctly joined or not by logs. Check for member joined log messages in all consoles.
Example:
INFO - WKABasedMembershipScheme Member joined [057c1105-97b2-4c99-8b1d-4164127b5684]: /192.168.1.100:4200
Note: To provide a complete answer kindly provide the post or article your following.
Hope this helps.

Related

WSO2 EI 6.2.0 CAR deployment broken - NoClassDefFoundError

we have a project, where we have been using WSO2 Enterprise Integrator for a year or so. The implementation became relatively complex over time and it involves combination of proxies, APIs, templates, custom mediators and message broker. This gets deployed via a single CAR deployment artifact. The common code has been shared as WSO2>Extensions>JavaLibraryProject (WSO2 DevStudio). This library gets archived into CAR as:
<artifact name="M2E-Commons" version="1.0.1" type="lib/library/bundle" serverRole="ApplicationServer">
<file>M2E-Commons-1.0.1.jar</file>
</artifact>
This had been working for us just fine for all of the past releases, but got broken with WSO2 EI 6.2.0
This is the beginning of the log while deploying on WSO2 EI 6.1.1:
TID: [-1234] [] [2018-04-09 14:33:52,337] INFO {org.wso2.carbon.application.deployer.internal.ApplicationManager} - Deploying Carbon Application : M2E-CAP_1.0.16.car... {org.wso2.carbon.application.deployer.internal.ApplicationManager}
TID: [-1234] [] [2018-04-09 14:33:52,350] INFO {org.wso2.carbon.application.deployer.handler.DefaultAppDeployer} - OSGi bundle in file:///opt/wso2ei/wso2/tmp/carbonapps/-1234/1523284432337M2E-CAP_1.0.16.car/M2E-Commons_1.0.1/M2E-Commons-1.0.1.jar location is about to be installed to Carbon Server. {org.wso2.carbon.application.deployer.handler.DefaultAppDeployer}
TID: [-1234] [] [2018-04-09 14:33:52,351] INFO {org.wso2.carbon.application.deployer.handler.DefaultAppDeployer} - OSGi bundle M2E-Commons installed to Carbon Server. {org.wso2.carbon.application.deployer.handler.DefaultAppDeployer}
TID: [-1234] [] [2018-04-09 14:33:52,369] INFO {org.wso2.carbon.application.deployer.handler.DefaultAppDeployer} - OSGi bundle M2E-Commons successfully started on Carbon Server. {org.wso2.carbon.application.deployer.handler.DefaultAppDeployer}
TID: [-1234] [] [2018-04-09 14:33:52,369] INFO {org.apache.synapse.deployers.ClassMediatorDeployer} - Deploying Class mediators from file : /opt/wso2ei/wso2/tmp/carbonapps/-1234/1523284432337M2E-CAP_1.0.16.car/M2E-Device_1.0.0/M2E-Device-1.0.0.jar {org.apache.synapse.deployers.ClassMediatorDeployer}
TID: [-1234] [] [2018-04-09 14:33:52,370] INFO {org.apache.synapse.deployers.ClassMediatorDeployer} - Deploying Class mediators from file : /opt/wso2ei/wso2/tmp/carbonapps/-1234/1523284432337M2E-CAP_1.0.16.car/M2E-Flic_1.0.0/M2E-Flic-1.0.0.jar {org.apache.synapse.deployers.ClassMediatorDeployer}
And this is the same CAR deployed on WSO2 EI 6.2.0:
TID: [-1234] [] [2018-04-06 22:29:46,703] INFO {org.wso2.carbon.application.deployer.internal.ApplicationManager} - Deploying Carbon Application : M2E-CAP_1.0.16.car... {org.wso2.carbon.application.deployer.internal.ApplicationManager}
TID: [-1234] [] [2018-04-06 22:29:46,810] INFO {org.apache.synapse.deployers.ClassMediatorDeployer} - Deploying Class mediators from file : /opt/wso2ei/wso2/tmp/carbonapps/-1234/1523053786703M2E-CAP_1.0.16.car/M2E-Device_1.0.0/M2E-Device-1.0.0.jar {org.apache.synapse.deployers.ClassMediatorDeployer}
TID: [-1234] [] [2018-04-06 22:29:46,810] INFO {org.apache.synapse.deployers.ClassMediatorDeployer} - Deploying Class mediators from file : /opt/wso2ei/wso2/tmp/carbonapps/-1234/1523053786703M2E-CAP_1.0.16.car/M2E-Flic_1.0.0/M2E-Flic-1.0.0.jar {org.apache.synapse.deployers.ClassMediatorDeployer}
TID: [-1234] [] [2018-04-06 22:29:46,810] INFO {org.apache.synapse.deployers.ClassMediatorDeployer} - Deploying Class mediators from file : /opt/wso2ei/wso2/tmp/carbonapps/-1234/1523053786703M2E-CAP_1.0.16.car/M2E-M2L_ISS_1.0.0/M2E-M2L_ISS-1.0.0.jar {org.apache.synapse.deployers.ClassMediatorDeployer}
TID: [-1234] [] [2018-04-06 22:29:46,810] INFO {org.apache.synapse.deployers.ClassMediatorDeployer} - Deploying Class mediators from file : /opt/wso2ei/wso2/tmp/carbonapps/-1234/1523053786703M2E-CAP_1.0.16.car/M2E-M2T_1.0.1/M2E-M2T-1.0.1.jar {org.apache.synapse.deployers.ClassMediatorDeployer}
TID: [-1234] [] [2018-04-06 22:29:46,881] ERROR {org.apache.synapse.deployers.AbstractSynapseArtifactDeployer} - Deployment of the Synapse Artifact from file : /opt/wso2ei-6.2.0/wso2/tmp/carbonapps/-1234/1523053786703M2E-CAP_1.0.16.car/M2E-ISS.LightOff_1.0.0/M2E-ISS.LightOff-1.0.0.xml : Failed! {org.apache.synapse.deployers.AbstractSynapseArtifactDeployer}
java.lang.NoClassDefFoundError: sk/m2ms/wso2/m2e/M2Mediator
Apparently the OSGi bundle M2E-Commons deployment was not prioritized properly to start before the rest of artifact deployments - resulting in NoClassDefFoundError.
Do you have any ideas?
Can this be solved via some OSGi compliant mechanism/concept?
Or this is a valid bug to be submitted against WSO2?
Thank you in advance,
Peter
An error like NoClassDefFoundError will never happen because of start ordering. It means that a bundle uses the package sk.m2ms.wso2.m2e but neither imports nor contains that package. That is an error in the construction of the bundle, so if this is not a bundle that you built yourself then I would suggest that this is indeed a bug that should be reported to the vendor.

API Manager Gateway 1.8 Incoming Connections Timing out

I am running API Manager Gateway version 1.8. Server is running CentOS 8 with Java(TM) SE Runtime Environment (build 1.7.0_67-b01). I am testing on a test and production server. To rule out differences between the servers the production server was cloned from the test server and updated to point to a separate DB, Key Manager & LDAP server. In addition it was synched with Gateway Management node.
I start up both of the Gateway servers. There are several dozen APIs deployed on each server. For testing I deployed an identical API on test and prod that point to the same backend service. The API does not require authentication so there is no token call. I execute a wget for the API directly to the Gateway worker in prod and test.
The call executes successfully in test returning a response in about 1 second.
However call to production hangs for a minute, then wget retries. Eventually after several retries the call succeeds.
I have made hundreds of calls to the service directly from command prompt on the production Gateway node and they are successful every time.
I am skipping the load balancer so all traffic in my test is via http to the Gateway server and to my backend service.
In production I see the following in the logs. The request for /csmjk followed by a 1 minute delay before the http-incoming-1 times out.
TID: [0] [AM] [2017-04-16 07:38:31,472] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-1: GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:38:31,479] DEBUG {org.apache.synapse.transport.nhttp.access} - - 10.40.1.161 - - [16/Apr/2017:07:38:31 -0500] "GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0" - - "-" "Wget/1.12 (linux-gnu)" {org.apache.synapse.transport.nhttp.access}
TID: [0] [AM] [2017-04-16 07:38:31,481] DEBUG {org.apache.synapse.transport.passthru.ServerWorker} - Starting a new Server Worker instance {org.apache.synapse.transport.passthru.ServerWorker}
TID: [0] [AM] [2017-04-16 07:39:31,547] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-1: Timeout {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:39:31,547] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-1 {org.apache.synapse.transport.passthru.SourceHandler}
In test I see the request followed immediately by a call to our backend service:
TID: [0] [AM] [2017-04-16 07:42:08,281] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-10: GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:42:08,286] DEBUG {org.apache.synapse.transport.nhttp.access} - - 10.40.1.161 - - [16/Apr/2017:07:42:08 -0500] "GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0" - - "-" "Wget/1.12 (linux-gnu)" {org.apache.synapse.transport.nhttp.access}
TID: [0] [AM] [2017-04-16 07:42:08,304] DEBUG {org.apache.synapse.transport.passthru.ServerWorker} - Starting a new Server Worker instance {org.apache.synapse.transport.passthru.ServerWorker}
TID: [0] [AM] [2017-04-16 07:42:08,394] INFO {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher} - Flushing the events from the queue 1 {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher}
TID: [0] [AM] [2017-04-16 07:42:08,426] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - Headers : {Accept=*/*, Connection=Keep-Alive, Host=TEST:8281, User-Agent=Wget/1.12 (linux-gnu), X-JWT-Assertion=null} {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,426] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - Message context:[MessageContext: logID=57478a056938f45377e3a24e79fae0781cbfcc13f4af60aa] {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,585] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - End user: null, API user: null {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,601] INFO {org.apache.synapse.core.axis2.TimeoutHandler} - This engine will expire all callbacks after : 120 seconds, irrespective of the timeout action, after the specified or optional timeout {org.apache.synapse.core.axis2.TimeoutHandler}
TID: [0] [AM] [2017-04-16 07:42:08,617] DEBUG {org.apache.synapse.transport.passthru.connections.TargetConnections} - Trying to get a connection {}->http://SERVICE:1111 {org.apache.synapse.transport.passthru.connections.TargetConnections}
This has been working fine in production for many months and suddenly stopped working reliably two days ago.
Any help would be greatly appreciated.
Thanks
I have been working on this for 3 days straight now. After enabling additional debugging I determined that the last thing done prior to the timeout was a call to Google Analytics. I disabled Google Analytics Tracking and everything is back to normal now. This is also configured in my test instance. More debugging to come but at least I am back to responding to requests.
Thanks

The service cannot be found for the endpoint reference while User Store Configuration Deployer initialization

from time to time our api throws 500 Http codes. I tried to track down the problem and found this kind of errors in the wso2carbon.log:
TID: [0] [AM] [2017-03-05 20:06:11,687] INFO {org.wso2.carbon.core.multitenancy.TenantAxisConfigurator} - Creating tenant AxisConfiguration for tenant: aTenant[2] {org.wso2.carbon.core.multitenancy.TenantAxisConfigurator}
TID: [0] [AM] [2017-03-05 20:06:11,726] INFO {org.wso2.carbon.identity.user.store.configuration.deployer.UserStoreConfigurationDeployer} - User Store Configuration Deployer initiated. {org.wso2.carbon.identity.user.store.configuration.deployer.UserStoreConfigurationDeployer}
TID: [0] [AM] [2017-03-05 20:06:11,730] ERROR {org.apache.axis2.engine.AxisEngine} - The service cannot be found for the endpoint reference (EPR) local://axis2services/some/api/path {org.apache.axis2.engine.AxisEngine}
org.apache.axis2.AxisFault: The service cannot be found for the endpoint reference (EPR) local://axis2services/some/api/path
at org.apache.axis2.engine.DispatchPhase.checkPostConditions(DispatchPhase.java:78)
at org.apache.axis2.engine.Phase.invoke(Phase.java:329)
at org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:261)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:167)
at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.processRESTRequest(MultitenantMessageReceiver.java:594)
at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.doNhttpREST(MultitenantMessageReceiver.java:534)
at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.doREST(MultitenantMessageReceiver.java:353)
at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.processRequest(MultitenantMessageReceiver.java:212)
at org.wso2.carbon.core.multitenancy.MultitenantMessageReceiver.receive(MultitenantMessageReceiver.java:79)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:344)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:168)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
TID: [0] [AM] [2017-03-05 20:06:11,803] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: wso2carbon-sts {aTenant[2]} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [AM] [2017-03-05 20:06:11,856] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: org.wso2.carbon.sts - {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [AM] [2017-03-05 20:06:11,860] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: wso2carbon-sts {aTenant[2]} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [AM] [2017-03-05 20:06:11,926] INFO {org.wso2.carbon.mediation.initializer.multitenancy.TenantServiceBusInitializer} - Intializing the ESB Configuration for the tenant domain : aTenant {org.wso2.carbon.mediation.initializer.multitenancy.TenantServiceBusInitializer}
After Deploying Axis2 service:... everything runs fine.
So it seems that api calls can not be processed while the UserStoreConfigurationDeployer is running. But this happens between 10 and 20 times a day, causing several errors.
The javadoc says:
/**
* This is to deploy a new User Store Management Configuration file dropped or created at repository/deployment/server/userstores
* or repository/tenant/<>tenantId</>/userstores. Whenever a new file with .xml extension is added/deleted or a modification is done to
* an existing file, deployer will automatically update the existing realm configuration org.wso2.carbon.identity.user.store.configuration
* according to the new file.
*/
But these files are never touched.
I'm not an expert for wso2, so can someone point me where to search? Can the UserStoreConfigurationDeployer be configured to run only at start or only once a day (at 2:00 am or something)?
We are currently using wso2am-1.8.0, the upgrade to 2.something is planned.

wso2 api manager wso2 BAM

Tables are created but the data is not populated . followed the below steps https://docs.wso2.com/display/AM170/Publishing+API+Runtime+Statistics this link is used to configure
below is the error i see in the wso2carbon.log on wso2 api manager
TID: [0] [AM] [2014-10-16 20:15:50,544] ERROR {JAGGERY.modules.statistics.usage:jag} - java.lang.NullPointerException: null {JAGGERY.modules.statistics.usage:jag}
TID: [0] [AM] [2014-10-16 20:15:59,210] ERROR {JAGGERY.modules.statistics.usage:jag} - java.lang.NullPointerException: null {JAGGERY.modules.statistics.usage:jag}
TID: [0] [AM] [2014-10-16 20:16:01,473] ERROR {JAGGERY.modules.statistics.usage:jag} - java.lang.NullPointerException: null {JAGGERY.modules.statistics.usage:jag}
No stats will be published if there is no activity on the API. Please:
1. create API activity
2. wait 10 minutes for the bam analysis scripts to run
3. try to access the stats through API Manager again

GREG 4.1.1 - customizing oracle

I have problem customizing $GREG_HOME/repository/conf/registry.xml file for oracle database 10, I am using ojdbc14_1.0.0.jar.
After recommendations at http://wso2.org/project/registry/4.1.1/docs/installation_guide/db_oracle.html
Edit $GREG_HOME/repository/conf/registry.xml and give the following:
oracle-db
jdbc:oracle:thin:#SERVER_NAME:PORT/DB_NAME
USER_NAME
PASSWORD
oracle.jdbc.driver.OracleDriver
80
60000
5
I have errors at the startup of carbon server ("function is not supported" in Russian):
" TID: [] [WSO2 Governance Registry] [2012-03-29 15:41:41,843]
INFO
{org.wso2.carbon.ldap.server.configuration.LDAPConfigurationBuilder} -
KDC server is disabled.
{org.wso2.carbon.ldap.server.configuration.LDAPConfigurationBuilder}
TID: [] [WSO2 Governance Registry] [2012-03-29 15:41:41,859] INFO
{org.wso2.carbon.ldap.server.DirectoryActivator} - Initializing
Directory Server with working directory
D:\Distr\WSO2greg\WSO2GR~1.1\WSO2GR~1.1\bin..\repository\data\org.wso2.carbon.directory
and port 10389 {org.wso2.carbon.ldap.server.DirectoryActivator} TID:
[] [WSO2 Governance Registry] [2012-03-29 15:41:50,265] ERROR
{org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache} - Failed to
insert resource to /. Не поддерживаемая функция
{org.wso2.carbon.registry.core.jdbc.dao.JDBCPathCache}
java.sql.SQLException: Не поддерживаемая функция at
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:125)
at
oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:162)
"
WSO2 Governance Registry(4.5.0 onward) supports for different encoding types
Eg:
If you want to use Governance Registry with Russian, you should start the server with proper encoding type.
Windows : wso2server.bat -Dcarbon.registry.character.encoding=windows-1251
Liux : sh wso2server.sh -Dcarbon.registry.character.encoding=windows-1251
More information : http://docs.wso2.org/wiki/display/Governance450/Supported+System+Properties