TID: [-1234] [] [2022-02-26 15:41:04,837] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Starting WSO2 Carbon...
TID: [-1] [] [2022-02-26 15:41:04,837] [] INFO {org.ops4j.pax.logging.spi.support.EventAdminConfigurationNotifier} - Sending Event Admin nofification (configuration successful) to org/ops4j/pax/logging/Configuration
TID: [-1234] [] [2022-02-26 15:41:04,843] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Operating System : Windows 10 10.0, amd64
TID: [-1234] [] [2022-02-26 15:41:04,843] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Home : C:\Program Files\Java\jdk1.8.0_321\jre
TID: [-1234] [] [2022-02-26 15:41:04,843] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Version : 1.8.0_321
TID: [-1234] [] [2022-02-26 15:41:04,844] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java VM : Java HotSpot(TM) 64-Bit Server VM 25.321-b07,Oracle Corporation
TID: [-1234] [] [2022-02-26 15:41:04,844] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Carbon Home : \PROGRA~1\WSO2\IDENTI~1\5.11.0
TID: [-1234] [] [2022-02-26 15:41:04,844] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Temp Dir : \PROGRA~1\WSO2\IDENTI~1\5.11.0\tmp
TID: [-1234] [] [2022-02-26 15:41:04,844] [] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - User : Sampath Kumar, en-IN, Asia/Calcutta
TID: [-1] [] [2022-02-26 15:41:05,070] [] INFO {org.wso2.carbon.event.output.adapter.kafka.internal.ds.KafkaEventAdapterServiceDS} - Successfully deployed the Kafka output event adaptor service
TID: [-1] [] [2022-02-26 15:41:05,451] [] INFO {org.wso2.carbon.identity.oauth.uma.grant.internal.UMA2GrantServiceComponent} - Policy evaluator registered successfully: DefaultPolicyEvaluator
TID: [-1] [] [2022-02-26 15:41:05,452] [] INFO {org.wso2.carbon.identity.oauth.uma.grant.internal.UMA2GrantServiceComponent} - UMA Grant component activated successfully.
TID: [-1234] [] [2022-02-26 15:41:05,801] [] INFO {org.wso2.carbon.ldap.server.configuration.LDAPConfigurationBuilder} - KDC server is disabled.
TID: [-1234] [] [2022-02-26 15:41:08,518] [] INFO {org.apache.directory.server.KERBEROS_LOG} - KeyDerivation Interceptor initialized
TID: [-1] [] [2022-02-26 15:41:09,571] [] INFO {org.wso2.carbon.mex.internal.Office365SupportMexComponent} - Office365Support MexServiceComponent bundle activated successfully..
TID: [-1] [] [2022-02-26 15:41:09,580] [] INFO {org.wso2.carbon.mex2.internal.DynamicCRMCustomMexComponent} - DynamicCRMSupport MexServiceComponent bundle activated successfully.
TID: [-1] [] [2022-02-26 15:41:10,624] [] ERROR {org.apache.catalina.core.ContainerBase} - A child container failed during start java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#9f03abb]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.wso2.carbon.tomcat.ext.service.ExtendedStandardService.startInternal(ExtendedStandardService.java:52)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.wso2.carbon.tomcat.internal.CarbonTomcat.start(CarbonTomcat.java:113)
at org.wso2.carbon.tomcat.internal.ServerManager$1.run(ServerManager.java:167)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#9f03abb]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4805)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4940)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
... 17 more
Caused by: java.lang.IllegalArgumentException: The main resource set specified [C:\Program Files\WSO2\Identity Server\5.11.0\lib\tomcat\PROGRA~1\WSO2\IDENTI~1\5.11.0\repository\deployment\server\webapps\PROGRA~1\WSO2\IDENTI~1\5.11.0\repository\conf\tomcat\carbon] is not valid
at org.apache.catalina.webresources.StandardRoot.createMainResourceSet(StandardRoot.java:752)
at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:709)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 26 more
TID: [-1] [] [2022-02-26 15:41:10,645] [] ERROR {org.apache.catalina.core.ContainerBase} - A child container failed during start java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: A child container failed during start
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.wso2.carbon.tomcat.ext.service.ExtendedStandardService.startInternal(ExtendedStandardService.java:52)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.wso2.carbon.tomcat.internal.CarbonTomcat.start(CarbonTomcat.java:113)
at org.wso2.carbon.tomcat.internal.ServerManager$1.run(ServerManager.java:167)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.catalina.LifecycleException: A child container failed during start
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:928)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
... 9 more
Caused by: java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#9f03abb]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
... 17 more
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#9f03abb]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4805)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4940)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
... 17 more
Caused by: java.lang.IllegalArgumentException: The main resource set specified [C:\Program Files\WSO2\Identity Server\5.11.0\lib\tomcat\PROGRA~1\WSO2\IDENTI~1\5.11.0\repository\deployment\server\webapps\PROGRA~1\WSO2\IDENTI~1\5.11.0\repository\conf\tomcat\carbon] is not valid
at org.apache.catalina.webresources.StandardRoot.createMainResourceSet(StandardRoot.java:752)
at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:709)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 26 more
TID: [-1] [] [2022-02-26 15:41:10,656] [] ERROR {org.wso2.carbon.tomcat.internal.ServerManager} - tomcat life-cycle exception org.apache.catalina.LifecycleException: A child container failed during start
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:928)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.wso2.carbon.tomcat.ext.service.ExtendedStandardService.startInternal(ExtendedStandardService.java:52)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.wso2.carbon.tomcat.internal.CarbonTomcat.start(CarbonTomcat.java:113)
at org.wso2.carbon.tomcat.internal.ServerManager$1.run(ServerManager.java:167)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: A child container failed during start
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
... 9 more
Caused by: org.apache.catalina.LifecycleException: A child container failed during start
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:928)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
... 9 more
Caused by: java.util.concurrent.ExecutionException: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#9f03abb]
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
... 17 more
Caused by: org.apache.catalina.LifecycleException: Failed to start component [org.apache.catalina.webresources.StandardRoot#9f03abb]
at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:440)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
at org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4805)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4940)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909)
... 17 more
Caused by: java.lang.IllegalArgumentException: The main resource set specified [C:\Program Files\WSO2\Identity Server\5.11.0\lib\tomcat\PROGRA~1\WSO2\IDENTI~1\5.11.0\repository\deployment\server\webapps\PROGRA~1\WSO2\IDENTI~1\5.11.0\repository\conf\tomcat\carbon] is not valid
at org.apache.catalina.webresources.StandardRoot.createMainResourceSet(StandardRoot.java:752)
at org.apache.catalina.webresources.StandardRoot.startInternal(StandardRoot.java:709)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
... 26 more
TID: [-1234] [] [2022-02-26 15:41:11,021] [] INFO {org.wso2.carbon.user.core.ldap.UniqueIDReadWriteLDAPUserStoreManager} - LDAP connection created successfully in read-write mode
TID: [-1234] [] [2022-02-26 15:41:11,090] [] INFO {org.wso2.carbon.consent.mgt.core.internal.ConsentManagerComponent} - ConsentManagerComponent is activated.
TID: [-1234] [] [2022-02-26 15:41:11,477] [] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Configured Registry in 47ms
TID: [-1234] [] [2022-02-26 15:41:11,500] [] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Connected to mount at configregistry in 4ms
TID: [-1234] [] [2022-02-26 15:41:11,501] [] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Connected to mount at govregistry in 5ms
TID: [-1234] [] [2022-02-26 15:41:11,546] [] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Connected to mount at configregistry in 0ms
TID: [-1234] [] [2022-02-26 15:41:11,552] [] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Connected to mount at govregistry in 1ms
TID: [-1234] [] [2022-02-26 15:41:11,560] [] INFO {org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent} - Registry Mode : READ-WRITE
TID: [-1234] [] [2022-02-26 15:41:11,616] [] INFO {org.wso2.carbon.metrics.impl.util.JmxReporterBuilder} - Creating JMX reporter for Metrics with domain 'org.wso2.carbon.metrics'
TID: [-1234] [] [2022-02-26 15:41:11,621] [] INFO {org.wso2.carbon.metrics.impl.reporter.AbstractReporter} - Started JMX reporter for Metrics
TID: [-1234] [] [2022-02-26 15:41:13,086] [] INFO {org.wso2.carbon.registry.indexing.solr.SolrClient} - Default Embedded Solr Server Initialized
TID: [-1234] [] [2022-02-26 15:41:13,214] [] INFO {org.wso2.carbon.user.core.internal.UserStoreMgtDSComponent} - Carbon UserStoreMgtDSComponent activated successfully.
As the error indicates, there is some problem with tomcat configuration.
Cross check if you have made any changes to your <IS_HOME>\repository\conf\deployment.toml wrt tomcat
try starting the wso2 IS after installing before making any change
you are getting this error, after any particular configuration? provide details to help.
we trying to add new user to AD as secondary store in WSO2 but we got this problem
we tried to get more details from log file but we found the following :
TID: [-1234] [] [2018-01-04 23:25:19,052] DEBUG {org.wso2.carbon.user.core.ldap.ActiveDirectoryUserStoreManager} - value after escaping special characters in user02 : user02
TID: [-1234] [] [2018-01-04 23:25:19,062] DEBUG {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager} - Searching for user with SearchFilter: (&(objectClass=user)(sAMAccountName=user02)) in SearchBase:
TID: [-1234] [] [2018-01-04 23:25:19,065] DEBUG {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager} - Name in space for user02 is CN=user02,CN=Users,DC=DEVDC,DC=sure,DC=com,DC=sa
TID: [-1234] [] [2018-01-04 23:25:19,072] DEBUG {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager} - Searching for role: rolhejazi
TID: [-1234] [] [2018-01-04 23:25:19,072] DEBUG {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager} - Using search filter: (&(objectcategory=group)(cn=rolhejazi))
TID: [-1234] [] [2018-01-04 23:25:19,077] DEBUG {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager} - Searching in CN=Users,DC=devdc,DC=sure,DC=com,DC=sa
TID: [-1234] [] [2018-01-04 23:25:19,080] DEBUG {org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager} - Is role: rolhejazi exist: true
TID: [-1234] [] [2018-01-04 23:25:19,082] DEBUG {org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager} - Modifying role: CN=rolhejazi with type: 1 user: CN=user02,CN=Users,DC=DEVDC,DC=sure,DC=com,DC=sa in search base: CN=Users,DC=devdc,DC=sure,DC=com,DC=sa
TID: [-1234] [] [2018-01-04 23:25:19,400] DEBUG {org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager} - User: CN=user02,CN=Users,DC=DEVDC,DC=sure,DC=com,DC=sa was successfully modified in LDAP group: CN=rolhejazi
TID: [-1234] [] [2018-01-04 23:25:19,402] WARN {org.wso2.carbon.user.core.ldap.ActiveDirectoryUserStoreManager} - Unsecured connection is being used. Enabling user account operation will fail
TID: [-1234] [] [2018-01-04 23:25:19,411] ERROR {org.wso2.carbon.user.mgt.UserRealmProxy} - Error while enabling the user account. Please check password policy at DC for user : user02
org.wso2.carbon.user.core.UserStoreException: Error while enabling the user account. Please check password policy at DC for user : user02
I see this intermittent exception in my log file on unix server. My wso2 api manager is a standalone installation of 1.10.0. please help me understand this exception.
Stack trace below:
TID: [-1234] [] [2017-11-16 08:19:40,350] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispat
ched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2017-11-16 08:21:29,962] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispat
ched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2017-11-16 08:33:48,425] WARN {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} -
Invalid session id for thrift authenticator. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2017-11-16 08:33:48,425] ERROR {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Error in invoking validate key via thrift.. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2017-11-16 08:33:48,426] WARN {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClientPool} - Login failed.. Authenticating again.. {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClientPool}
TID: [-1234] [] [2017-11-16 08:33:48,437] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2017-11-16 08:33:48,437-0500] from IP address {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1] [] [2017-11-16 08:33:48,476] INFO {org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher} - Session timed out for TCP,localhost:7614,TCP,localhost:7714,Thrift Session Expired Exception {org.wso2.carbon.databridge.agent.thrift.internal.publisher.client.EventPublisher}
From your above logs, I can see following error log related to thrift
TID: [-1234] [] [2017-11-16 08:33:48,425] ERROR {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Error in invoking validate key via thrift.. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
I don't really see a reason for above, but Can you please change to WSClient from ThriftClient. This configuration resides on api-manager.xml file.
There could be a connection issue related to thrift in your node.
Going thru the WSO2 API Manager logs, we noticed few occurrences where calls returned as HTTP 500 error and successful on subsequent retry by client. This does not happen all the time (at about 5% rate). And all these incidents are from the same source IP. Looking further, these calls that fails with HTTP 500, never get send to back-end endpoint for actual processing - Therefore, these calls are failed by WSO2 API Manager itself.
We have verified the CPU, Memory of WSO2 are at normal working level. Our WSO2 APIM environment is with no authorization token enabled (simple pass thru with throttle) - hence this can be counted out.
From log, the error 500 is returned almost immediate and retry typically happens within 200ms. And all these errors (about 49) happens within same hour.
We also not able to verify the exact error message as API calls is from one of our client.
Appreciate if you could list down all possibilities as why WSO2 may returned HTTP 500 in this case. Otherwise, how we able to investigate the scenario further. Thanks.
Here are relevant logs
WSO2 APIM AWS ELB access logs
$ grep 'fromIndex=394501&toIndex=395000' *.log
WSO2APIM_ELBACESS.log:2016-12-06T08:37:37.905822Z wso2-oss-server x.x.x.x:63598 x.x.x.98:8280 0.000042 0.001862 0.000023 500 500 0 293 "GET https://wso2apimanager:443/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "RestSharp/x.x.x.x" ECDHE-RSA-AES128-SHA TLSv1
WSO2APIM_ELBACESS.log:2016-12-06T08:37:38.102539Z wso2-oss-server x.x.x.x:63598 x.x.x.120:8280 0.000041 0.035397 0.000025 200 200 0 33 "GET https://wso2apimanager:443/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "RestSharp/x.x.x.x" ECDHE-RSA-AES128-SHA TLSv1
WSO2 APIM http access log
$ grep 'fromIndex=394501&toIndex=395000' *.log
http_access_2016-12-06.log_m1:x.x.x.x x.x.x.140 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "RestSharp/x.x.x.x"
http_access_2016-12-06.log_m1:x.x.x.x x.x.x.72 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "Synapse-PT-HttpComponents-NIO"
http_access_2016-12-06.log_m2:x.x.x.x x.x.x.140 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "RestSharp/x.x.x.x"
Back-end AWS ELB access log
$ grep 'fromIndex=394501&toIndex=395000' *.log
BACKEND_ELBACCESS.log:2016-12-06T08:37:38.104641Z vpc-connect-api x.x.x.120:46946 x.x.x.23:80 0.000045 0.032717 0.000022 200 200 0 33 "GET http://backendserver:80/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "Synapse-PT-HttpComponents-NIO" - -
WSO2 API Manager 1 (wso2carbon.log)
TID: [-1234] [] [2016-12-06 08:34:47,427] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected {org.wso2.carbon.databridge.core.DataBridge}
TID: [-1234] [] [2016-12-06 08:35:18,069] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:35:47,850] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1] [] [2016-12-06 08:37:13,485] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16702: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:13,486] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,486] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_28 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:43 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,486] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,487] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566702 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:37:13,961] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:18,342] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:18,007] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:40:17,933] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:41:17,725] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:43:17,811] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:45:17,892] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:47:18,363] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:50:47,850] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:51:17,864] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:53:17,753] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:55:18,025] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:57:18,333] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:59:17,949] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
WSO2 API Manager 2 (wso2carbon.log)
TID: [-1234] [] [2016-12-06 08:34:54,620] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected {org.wso2.carbon.databridge.core.DataBridge}
TID: [-1234] [] [2016-12-06 08:36:18,043] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,674] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16760: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:13,692] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,693] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_28 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:43 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,693] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,694] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566592 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:37:14,331] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1] [] [2016-12-06 08:37:14,620] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16759: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:14,621] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:14,621] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:44 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:14,621] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:14,818] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:15,255] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:29,543] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,137] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,568] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,990] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:36,409] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:37,899] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:38,327] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:41,129] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:41,545] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:42,969] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:38:09,156] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:38:17,817] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:39:16,163] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566604 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1] [] [2016-12-06 08:39:19,169] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566640 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:39:19,648] ERROR {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Failed to submit the response {org.apache.synapse.transport.passthru.PassThroughHttpSender}
java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:19,649] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/json,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Failed to submit the response
at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:613)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
Caused by: java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:19,650] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 0, ERROR_MESSAGE = Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/json,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:19,688] WARN {org.apache.synapse.core.axis2.Axis2Sender} - Trying to send a response to an already responded client request - Rest API Context : /rest {org.apache.synapse.core.axis2.Axis2Sender}
TID: [-1] [] [2016-12-06 08:39:20,249] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : internal-vpc-connect-api-1883227725.ap-southeast-1.elb.amazonaws.com/x.x.x.43:80 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2016-12-06 08:39:20,252] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:20,254] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:39:50 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:20,254] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:20,254] ERROR {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Failed to submit the response {org.apache.synapse.transport.passthru.PassThroughHttpSender}
java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:20,255] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Accept-Encoding:gzip, deflate,Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/xml,password:Hjfawx99##,username:lynn#heatherjames.com,X-Forwarded-For:76.79.103.18,X-Forwarded-Port:443,X-Forwarded-Proto:https,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><am:fault xmlns:am="http://wso2.org/apimanager"><am:code>101503</am:code><am:type>Status report</am:type><am:message>Runtime Error</am:message><am:description>Error connecting to the back end</am:description></am:fault></soapenv:Body></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Failed to submit the response
at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:613)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
Caused by: java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:20,255] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:20,256] WARN {org.apache.synapse.core.axis2.Axis2Sender} - Trying to send a response to an already responded client request - Rest API Context : /rest {org.apache.synapse.core.axis2.Axis2Sender}
TID: [-1] [] [2016-12-06 08:39:22,936] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : internal-vpc-connect-api-1883227725.ap-southeast-1.elb.amazonaws.com/x.x.x.43:80 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2016-12-06 08:39:22,942] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:22,945] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:39:52 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:22,945] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:23,363] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
x 34 of similiar logs
TID: [-1234] [] [2016-12-06 08:39:52,303] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:53,910] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:42:19,456] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:42:56,120] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:44:18,039] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:46:17,939] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:46:36,177] WARN {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Invalid session id for thrift authenticator. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2016-12-06 08:46:36,177] ERROR {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Error in invoking validate key via thrift.. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2016-12-06 08:46:36,177] WARN {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient} - Login failed.. Authenticating again.. {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient}
TID: [-1234] [] [2016-12-06 08:46:36,231] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2016-12-06 08:46:36,231+0000] from IP address {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1234] [] [2016-12-06 08:48:17,776] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:49:18,033] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:50:17,748] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:52:18,419] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:54:18,051] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:56:17,815] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:57:56,120] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:58:18,054] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
From your logs you can see that your backend was suspended and that is the cause of your 500 error. APIM will unsuspend the endpoint after a set time out period and allow you to try the endpoint again. It seems the backend was having some issues where it was most of the time working but sometimes unavailable causing the suspension of the endpoint.
Since this report is so old I'm using the most current documentation. You can see the same error code of 303001 on the Endpoint Suspension page.
I further created a quick demo of this situation using APIM 4.1.0 on my laptop and used an endpoint url of bogus.endpoint which of course will not resolve. I received the following message back using postman:
{
"code": "303001",
"type": "Status report",
"message": "Runtime Error",
"description": "Currently , Address endpoint : [ Name : DemoSuspention--vv1_APIproductionEndpoint ] [ State : SUSPENDED ]"
}
And it gave me a status result of: 500 Internal Server Error 40 ms 732 B
Normally when I run into this type of situation I reach out to the technical point of contact for the backend and report what I'm seeing. They normally can locate some kind of issue in their logs and then address it by either restarting a server or fixing a bug depending on what is found.
I don't know why my EndPoint was marked like SUSPENDED and my application does not receive return from Service.
This was shown on log:
TID: [0] [ESB] [2014-09-01 11:00:32,269] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscarCaminhoTask, SOAPAction: urn:BuscarCaminhoTask, MessageID: urn:uuid:78e22483-c74e-4673-8286-8a18294135b3, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:00:33,259] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-15546: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2014-09-01 11:00:33,260] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : MonitoramentoEndPoint will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:00:33,260] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : MonitoramentoEndPoint - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Mon Sep 01 11:01:03 BRT 2014 {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:01:01,196] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscaServicosMonitorar, SOAPAction: urn:BuscaServicosMonitorar, MessageID: urn:uuid:e5ce0a3c-e5d1-4364-a7aa-897363392c4d, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:01:01,203] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscaHardwaresMonitorar, SOAPAction: urn:BuscaHardwaresMonitorar, MessageID: urn:uuid:e74ab088-1eb0-47cc-8447-7b9e016edcec, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:01:01,208] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscarCaminhoTask, SOAPAction: urn:BuscarCaminhoTask, MessageID: urn:uuid:18dad64b-292e-4c79-908e-6d9b638e1d8a, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:01:01,291] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-15544: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2014-09-01 11:01:01,292] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : MonitoramentoEndPoint will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:01:01,292] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : MonitoramentoEndPoint - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Mon Sep 01 11:01:31 BRT 2014 {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:01:02,210] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-388 {org.apache.synapse.transport.passthru.SourceHandler}
The EndPoint come back automatically after some seconds, but how can I know whats wrong?
This is the error after receive my request:
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "POST /services/MonitoramentoDS HTTP/1.1[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Expect: 100-continue[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Content-Type: text/xml; charset=utf-8[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "SOAPAction: "urn:UpdateServidor"[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Transfer-Encoding: chunked[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Host: localhost:9763[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Connection: Keep-Alive[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "User-Agent: Synapse-PT-HttpComponents-NIO[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - >> "HTTP/1.1 100 Continue[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,747] DEBUG {org.apache.synapse.transport.http.wire} - >> "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "19a[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "<?xml version='1.0' encoding='utf-8'?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body><UpdateServidor xmlns="http://ws.wso2.org/dataservice"><situacao>0</situacao><id>2</id><sithard>1</sithard><sitserv>1</sitserv><sittask>0</sittask></UpdateServidor></soap:Body></soap:Envelope>[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "0[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "HTTP/1.1 202 Accepted[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Content-Type: text/xml;charset=utf-8[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Transfer-Encoding: chunked[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Date: Mon, 01 Sep 2014 18:27:42 GMT[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Server: WSO2 Carbon Server[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "0[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,753] ERROR {org.apache.axis2.transport.base.threads.NativeWorkerPool} - Uncaught exception {org.apache.axis2.transport.base.threads.NativeWorkerPool}
java.lang.UnsupportedOperationException: An access occurred that is not valid.
at org.apache.axis2.description.InOnlyAxisOperation.getMessage(InOnlyAxisOperation.java:117)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:283)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:222)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Endpoint is getting suspended, when ESB could not send out message through that particular endpoint. You can find endpoint suspension parameters in your endpoint configuration.
Here if you see http-outgoing-15544: Connection time out while in state: REQUEST_DONE occurs because, connection between ESB and the backend server times out. Default socket
timeout is 60 seconds. You can increase it to 120 seconds and check. To do that edit the
passthru-http.properties file as;
http.socket.timeout=120000
And enable ESB wirelogs so you can find more info.
# log4j properties file, add following line and restart the server.
log4j.logger.org.apache.synapse.transport.http.wire=DEBUG