I am running WSO2 IS 5.0.0. I have the SP for IS 5.0.0 applied along with all the other security patches issued for that version for Identity Server and Carbon 4.2.0. My environment consists of 4 machines that are creating a cluster (using the WKA membership scheme and Load Balancer(AWS ELB) with sticky session enabled). I am using MySQL(not the default H2 database). The machines on which the IS is deployed are Windows Server 2012 R2 (EC2 AWS machines).
Sporadically I receive the following error in the console log:
ERROR {org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultRequestCoordinator} - Exception in Authentication Framework
java.lang.NullPointerException
at org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultLogoutRequestHandler.sendResponse(DefaultLogoutRequestHandler.java:155)
at org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultLogoutRequestHandler.handle(DefaultLogoutRequestHandler.java:131)
at org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultRequestCoordinator.handle(DefaultRequestCoordinator.java:97)
at org.wso2.carbon.identity.application.authentication.framework.servlet.CommonAuthenticationServlet.doPost(CommonAuthenticationServlet.java:54)
at org.wso2.carbon.identity.application.authentication.framework.servlet.CommonAuthenticationServlet.doGet(CommonAuthenticationServlet.java:44)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:60)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I am not quite sure what might be causing this error to occur. It is not critical for my system because I have not noticed that it is causing some kind of troubles or downtime.
However I would like to know whether it is a bug or known issue? If so is there a way to be fixed (or applied a workaround) by updating some configuration files? May be there is some sort of patch that could fix it?
Thanks in advance.
I assume this happens when relyingParty is null. Try sending it as a query param of the logout request like this.
https://is_host.com/commonauth?commonAuthLogout=true&type=samlsso&sessionDataKey
=E294FEF4A64BF7E14940E2964F78E351&commonAuthCallerPath=https://is_home.com/
identity/main/logoutsso&relyingParty=SAMLSSOIdentity
Related
I have done mitigations steps by follows this WSO2 Documentation to avoid Cross Site Scripting (XSS) attacks in WSO2 EI 6.4.0.
In below mentioned files i have made required changes as mentioned here
edited <PRODUCT_HOME>/repository/conf/carbon.xml
Added the some configuration within the <Hosts> element of the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file.
Post restarting server, I have noticed couple of things as mentioned below
We're unable to modify artifacts like API, Sequence, task Scheduler via wso2 management console, But we can upload CARs, Connectors etc.
Whenever I try to edit files in management console, got below ERROR in logs
[2022-10-18 07:38:59,904] [-1234] [] [http-nio-9443-exec-34] ERROR {org.wso2.carbon.tomcat.ext.valves.CompositeValve
} - Could not handle request: /carbon/sequences/save_sequence.jsp
javax.servlet.ServletException: Possible XSS Attack. Suspicious code : eval($)
at org.wso2.carbon.ui.valve.XSSValve.validateParameters(XSSValve.java:110)
at org.wso2.carbon.ui.valve.XSSValve.invoke(XSSValve.java:86)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.
java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
In DSS, we could able to edit in management console(eg. adding new operation) , but newly added operation is not available when i do try it option
Concluding that whatever we want, we can do via CAR, this mitigation change is not allow us to edit config files via management console.
Is my understanding correct or Am I missing something to notice? I just need to know what was the product level impacts post this Mitigation.
Kindly clarify my doubt or share your thoughts on the same.
We need to skip the content validation of the artifact files. We can whitelist the resource paths used to modify artifacts such as API, Sequences etc. To do this you need to add the following patterns under the <Patterns> element of the <XSSPreventionConfig> element in carbon.xml config file.
<XSSPreventionConfig>
<Enabled>true</Enabled>
<Rule>allow</Rule>
<Patterns>
<Pattern>carbon/sequences</Pattern>
<Pattern>carbon/configadmin</Pattern>
<Pattern>carbon/localentries</Pattern>
<Pattern>carbon/api</Pattern>
<Pattern>carbon/proxyservices</Pattern>
<Pattern>carbon/resources</Pattern>
<Pattern>carbon/task</Pattern>
</Patterns>
</XSSPreventionConfig>
Reference: http://ravindraranwala.blogspot.com/2015/10/preventing-xss-and-csrf-vulnerabilities.html
I have setup an API Manager 1.10 cluster, with each component in a separate VM, fronted by an nginx proxy.
Whenever I try to generate a key I get the following error in store (logs from store):
ERROR - AMDefaultKeyManagerImpl Error while creating tokens - Connection timed out
java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
...
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
[2016-12-14 16:25:03,376] ERROR - APIConsumerImpl Error while re-generating AccessToken
org.wso2.carbon.apimgt.api.APIManagementException: Error while creating tokens - Connection timed out
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.handleException(AMDefaultKeyManagerImpl.java:592)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:369)
at org.wso2.carbon.apimgt.impl.APIConsumerImpl.renewAccessToken(APIConsumerImpl.java:659)
at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.renewAccessToken(UserAwareAPIConsumer.java:36)
at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_refreshToken(APIStoreHostObject.java:3800)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
.....
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:303)
... 70 more
[2016-12-14 16:25:03,377] ERROR - APIStoreHostObject Error while renewing Access Token for Consumer Key qZf8VvT4ts7a0TT6nE6ZHfY9t5sa and user admin
[2016-12-14 16:25:03,377] ERROR - subscription-add:jag org.wso2.carbon.apimgt.api.APIManagementException: Error while renewing Access Token for Consumer Key 88 and user admin
I have configured connections among the components by following this article The logs above are only showing up in the API store console.
I have also configured the Keymanager host and port in API Gateway in the following locations:
/repository/deployment/server/synapse-configs/default/api
AuthorizeAPI.xml
RevokeAPI.xml
TokenAPI.xml
What do I need to do to fix this?
In store's api-manager.xml, <RevokeAPIURL> should be pointing to gateway. make sure you have that configured properly.
I didn't have the port 8243 opened in my security group for the API gateway. So the problem had nothing to do with WSO2, it was my infrastructure.
My APIM version is 1.10.0 and DAS is 3.0.1.
At first ,I deploy no Clustering APIM + DAS with mysql. the stats shows good.
Then,I clustering APIM into publisher、store、keymanager and gateway.Configured APIM and DAS order this article,when I invoke an api , gateway node then show error
[2016-10-13 11:13:54,775] ERROR - APIMgtUsageHandler Cannot publish event. null
java.lang.NullPointerException
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher.publishEvent(APIMgtUsageDataBridgeDataPublisher.java:124)
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageHandler.handleRequest(APIMgtUsageHandler.java:169)
at org.apache.synapse.rest.API.process(API.java:322)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:65)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:317)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:149)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-10-13 11:13:54,807] ERROR - APIMgtResponseHandler Cannot publish response event. null
java.lang.NullPointerException
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher.publishEvent(APIMgtUsageDataBridgeDataPublisher.java:140)
at org.wso2.carbon.apimgt.usage.publisher.APIMgtResponseHandler.mediate(APIMgtResponseHandler.java:211)
at org.apache.synapse.mediators.ext.ClassMediator.mediate(ClassMediator.java:84)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
at org.apache.synapse.rest.Resource.process(Resource.java:297)
at org.apache.synapse.rest.API.process(API.java:335)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:52)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:529)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:172)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:251)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Then i change the way of DAS configuration. Following this blog ,i configured all the api-manager.xml,there is no error and no stats.And store node show info:
[2016-10-18 11:31:39,131] INFO - ReceiverGroup Resending the failed published data...
[2016-10-18 11:31:44,134] WARN - AccessConfiguration Error loading properties from file: access-log.properties
[2016-10-18 11:31:44,207] INFO - TimeoutHandler This engine will expire all callbacks after : 120 seconds, irrespective of the timeout
I google the reason ,in wso2 jira,they say this is not a bug.
But I cannot wait for my stats for hours.Did I configure something wrong?
Reguards.
Seems in store node, data are publishing are enabled and working. But gateway node data not publishing. Please try to verify the gateway node configuration. Also check gateway node can access the DAS node.
I try to connect WSO2 ML with WSO2 DAS using datasource cassandra. In ML_HOME/repository/conf/datasources/analytics-datasources.xml and DAS_HOME/repository/conf/datasources/analytics-datasources.xml I have the same configuration to use cassandra.
<datasource>
<name>WSO2_ANALYTICS_DS_CASSANDRA</name>
<description>The Cassandra datasource used for analytics</description>
<definition type="CASSANDRA">
<configuration>
<contactPoints>192.168.1.111</contactPoints>
<port>9042</port>
<!--<username>admin</username>
<password>admin</password>
<clusterName>cluster1</clusterName> -->
<compression>NONE</compression>
<poolingOptions>
<coreConnectionsPerHost hostDistance="LOCAL">8</coreConnectionsPerHost>
<maxSimultaneousRequestsPerHostThreshold hostDistance="LOCAL">1024</maxSimultaneousRequestsPerHostThreshold>
</poolingOptions>
<queryOptions>
<fetchSize>5000</fetchSize>
<consistencyLevel>ONE</consistencyLevel>
<serialConsistencyLevel>SERIAL</serialConsistencyLevel>
</queryOptions>
<socketOptions>
<keepAlive>false</keepAlive>
<sendBufferSize>150000</sendBufferSize>
<connectTimeoutMillis>12000</connectTimeoutMillis>
<readTimeoutMillis>12000</readTimeoutMillis>
</socketOptions>
</configuration>
</definition>
</datasource>
When I try to create a Dataset in ML using DAS I get error when try to list the tables of DAS, specifically in ConfigurationApiV11.java, method getDASTables when call analyticsDataApi.listTables()
This is my error in logs:
TID: [-1234] [] [2016-09-03 22:35:59,762] WARN {org.apache.cxf.phase.PhaseInterceptorChain} - Application {http://api.rest.ml.carbon.wso2.org/}Lo$
org.apache.cxf.interceptor.Fault: Error while connecting to the remote service. null
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:155)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:121)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:126)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:113)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
It seems that WSO2ML is trying to connect to the analytic service remotely. By default it perform automatic selection of the mode. If you are using a single ML node and /AnalyticsDataConfiguration/Mode is set to AUTO at repository/conf/analytics/analytics-data-config.xml. Then change it to LOCAL. If the ML server start before the Cassandra cluster becomes available it may cause the ML server to use the REMOTE mode.
I've got a website running on Amazon Web Services that is deployed using Elastic Beanstalk and runs on a minimum of 2 EC2 micro instances. An auto scaling policy is in place, so that it can scale up and scale down depending on the the traffic in the website. Because of this auto scaling policy, I wanted to avoid using sticky sessions and for that reason I am using memcached-session-manager. I am using Amazon ElastiCache (small instance) for the memcached server.
The configuration in the context.xml is as follows:
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="sessions.myinstancecode.0001.use1.cache.amazonaws.com:11211"
sticky="false"
sessionBackupAsync="false"
lockingMode="none"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory" />
This works fine when the traffic is low (i.e. less than 10 users online) but sometimes causes the EC2 instance to restart. You can imagine that if the website is currently running on two instances and they both decide to restart at the same time, the website becomes unreachable and it's a big problem. These are the last lines in the tail_catalina.log that is rotated on Amazon S3 before the EC2 instance decides to restart:
Jun 13, 2012 12:32:27 AM de.javakaffee.web.msm.BackupSessionTask handleException
WARNING: Could not store session 42F9761AC24F826E1FC3F2A834FBF442 in memcached.
Note that this session was relocated to this node because the original node was not available.
net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: sessions.myinstancecode.0001.use1.cache.amazonaws.com/10.194.23.99:11211
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:73)
at de.javakaffee.web.msm.BackupSessionTask.storeSessionInMemcached(BackupSessionTask.java:230)
at de.javakaffee.web.msm.BackupSessionTask.doBackupSession(BackupSessionTask.java:195)
at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:120)
at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:51)
at de.javakaffee.web.msm.BackupSessionService$SynchronousExecutorService.submit(BackupSessionService.java:339)
at de.javakaffee.web.msm.BackupSessionService.backupSession(BackupSessionService.java:198)
at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:967)
at de.javakaffee.web.msm.SessionTrackerValve.backupSession(SessionTrackerValve.java:226)
at de.javakaffee.web.msm.SessionTrackerValve.invoke(SessionTrackerValve.java:128)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:680)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Jun 13, 2012 12:32:28 AM de.javakaffee.web.msm.LockingStrategy onAfterBackupSession
WARNING: An error occurred during onAfterBackupSession.
net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: sessions.myinstancecode.0001.use1.cache.amazonaws.com/10.194.23.99:11211
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:73)
at de.javakaffee.web.msm.LockingStrategy.onAfterBackupSession(LockingStrategy.java:287)
at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:970)
at de.javakaffee.web.msm.SessionTrackerValve.backupSession(SessionTrackerValve.java:226)
at de.javakaffee.web.msm.SessionTrackerValve.invoke(SessionTrackerValve.java:128)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:680)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
It seems like the Amazon ElastiCache node is failing, but the thing is that, checking on Amazon CloudWatch, I can see that the CPU utilization never went over 8%. Is there any reason why the Amazon ElastiCache node fails, even though it's not being stressed that much? Also, why does Amazon decide to restart (or better: terminate and start a new instance) when Amazon ElastiChace node fails?
Any help greatly appreciated.
Thanks!
You should increase the sessionBackupTimeout of memcached-session-manager, from the documentation:
sessionBackupTimeout (optional, default 100)
The timeout in milliseconds after that a session backup is considered
as beeing failed. This property is only evaluated if sessions are
stored synchronously (set via sessionBackupAsync). The default value
is 100 milliseconds.