Fail to conecting ML and DAS of wso2 using cassandra - wso2

I try to connect WSO2 ML with WSO2 DAS using datasource cassandra. In ML_HOME/repository/conf/datasources/analytics-datasources.xml and DAS_HOME/repository/conf/datasources/analytics-datasources.xml I have the same configuration to use cassandra.
<datasource>
<name>WSO2_ANALYTICS_DS_CASSANDRA</name>
<description>The Cassandra datasource used for analytics</description>
<definition type="CASSANDRA">
<configuration>
<contactPoints>192.168.1.111</contactPoints>
<port>9042</port>
<!--<username>admin</username>
<password>admin</password>
<clusterName>cluster1</clusterName> -->
<compression>NONE</compression>
<poolingOptions>
<coreConnectionsPerHost hostDistance="LOCAL">8</coreConnectionsPerHost>
<maxSimultaneousRequestsPerHostThreshold hostDistance="LOCAL">1024</maxSimultaneousRequestsPerHostThreshold>
</poolingOptions>
<queryOptions>
<fetchSize>5000</fetchSize>
<consistencyLevel>ONE</consistencyLevel>
<serialConsistencyLevel>SERIAL</serialConsistencyLevel>
</queryOptions>
<socketOptions>
<keepAlive>false</keepAlive>
<sendBufferSize>150000</sendBufferSize>
<connectTimeoutMillis>12000</connectTimeoutMillis>
<readTimeoutMillis>12000</readTimeoutMillis>
</socketOptions>
</configuration>
</definition>
</datasource>
When I try to create a Dataset in ML using DAS I get error when try to list the tables of DAS, specifically in ConfigurationApiV11.java, method getDASTables when call analyticsDataApi.listTables()
This is my error in logs:
TID: [-1234] [] [2016-09-03 22:35:59,762] WARN {org.apache.cxf.phase.PhaseInterceptorChain} - Application {http://api.rest.ml.carbon.wso2.org/}Lo$
org.apache.cxf.interceptor.Fault: Error while connecting to the remote service. null
at org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:155)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:121)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:167)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:94)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:123)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:207)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:213)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:154)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:126)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:113)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

It seems that WSO2ML is trying to connect to the analytic service remotely. By default it perform automatic selection of the mode. If you are using a single ML node and /AnalyticsDataConfiguration/Mode is set to AUTO at repository/conf/analytics/analytics-data-config.xml. Then change it to LOCAL. If the ML server start before the Cassandra cluster becomes available it may cause the ML server to use the REMOTE mode.

Related

WSO2 EI 6.4.0 - Mitigating Cross Site Scripting Attacks

I have done mitigations steps by follows this WSO2 Documentation to avoid Cross Site Scripting (XSS) attacks in WSO2 EI 6.4.0.
In below mentioned files i have made required changes as mentioned here
edited <PRODUCT_HOME>/repository/conf/carbon.xml
Added the some configuration within the <Hosts> element of the <PRODUCT_HOME>/repository/conf/tomcat/catalina-server.xml file.
Post restarting server, I have noticed couple of things as mentioned below
We're unable to modify artifacts like API, Sequence, task Scheduler via wso2 management console, But we can upload CARs, Connectors etc.
Whenever I try to edit files in management console, got below ERROR in logs
[2022-10-18 07:38:59,904] [-1234] [] [http-nio-9443-exec-34] ERROR {org.wso2.carbon.tomcat.ext.valves.CompositeValve
} - Could not handle request: /carbon/sequences/save_sequence.jsp
javax.servlet.ServletException: Possible XSS Attack. Suspicious code : eval($)
at org.wso2.carbon.ui.valve.XSSValve.validateParameters(XSSValve.java:110)
at org.wso2.carbon.ui.valve.XSSValve.invoke(XSSValve.java:86)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.
java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
In DSS, we could able to edit in management console(eg. adding new operation) , but newly added operation is not available when i do try it option
Concluding that whatever we want, we can do via CAR, this mitigation change is not allow us to edit config files via management console.
Is my understanding correct or Am I missing something to notice? I just need to know what was the product level impacts post this Mitigation.
Kindly clarify my doubt or share your thoughts on the same.
We need to skip the content validation of the artifact files. We can whitelist the resource paths used to modify artifacts such as API, Sequences etc. To do this you need to add the following patterns under the <Patterns> element of the <XSSPreventionConfig> element in carbon.xml config file.
<XSSPreventionConfig>
<Enabled>true</Enabled>
<Rule>allow</Rule>
<Patterns>
<Pattern>carbon/sequences</Pattern>
<Pattern>carbon/configadmin</Pattern>
<Pattern>carbon/localentries</Pattern>
<Pattern>carbon/api</Pattern>
<Pattern>carbon/proxyservices</Pattern>
<Pattern>carbon/resources</Pattern>
<Pattern>carbon/task</Pattern>
</Patterns>
</XSSPreventionConfig>
Reference: http://ravindraranwala.blogspot.com/2015/10/preventing-xss-and-csrf-vulnerabilities.html

I am stuck with making a connection of AWS Elastic Beanstalk and RDS

I've been struggling here to connect my Tomcat AWS Elastic Beanstalk and RDS.
I've successfully launched a TOMCAT server and add RDS to it by following the instructions.
But I have no idea how to connect my tomcat server to added RDS server.
Connecting to a Database (Tomcat Platforms)
You will see Connecting to a Database (Tomcat Platforms). But I have no idea how and where to put the example source code.
So I just made a Test project with example JSP code at guide, built the project, and deployed to my Tomcat server.
But it doesn't work either showing the message:
HTTP Status 500 - An exception occurred processing JSP page /main.jsp at line 24
type Exception report
message An exception occurred processing JSP page /main.jsp at line 24
description The server encountered an internal error that prevented it from fulfilling this request.
exception
org.apache.jasper.JasperException: An exception occurred processing JSP page /main.jsp at line 24
21: Class.forName("oracle.jdbc.driver.OracleDriver");
22: System.out.println("Driver loaded!");
23: } catch (ClassNotFoundException e) {
24: throw new RuntimeException("Cannot find the driver in the classpath!", e);
25: }
26:
27: Connection conn = null;
Stacktrace:
org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:579)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:476)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause
java.lang.RuntimeException: Cannot find the driver in the classpath!
org.apache.jsp.main_jsp._jspService(main_jsp.java:134)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause
java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1335)
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1163)
org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:125)
org.apache.jasper.servlet.JasperLoader.loadClass(JasperLoader.java:62)
java.lang.Class.forName0(Native Method)
java.lang.Class.forName(Class.java:264)
org.apache.jsp.main_jsp._jspService(main_jsp.java:131)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
note The full stack trace of the root cause is available in the Apache Tomcat/8.0.50 logs.
Apache Tomcat/8.0.50
Could you guys just help me out? I've spent all my day but couldn't solve it yet.

WSO2 DAS + Clustering APIM with mysql

My APIM version is 1.10.0 and DAS is 3.0.1.
At first ,I deploy no Clustering APIM + DAS with mysql. the stats shows good.
Then,I clustering APIM into publisher、store、keymanager and gateway.Configured APIM and DAS order this article,when I invoke an api , gateway node then show error
[2016-10-13 11:13:54,775] ERROR - APIMgtUsageHandler Cannot publish event. null
java.lang.NullPointerException
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher.publishEvent(APIMgtUsageDataBridgeDataPublisher.java:124)
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageHandler.handleRequest(APIMgtUsageHandler.java:169)
at org.apache.synapse.rest.API.process(API.java:322)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:65)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:83)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:317)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:149)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-10-13 11:13:54,807] ERROR - APIMgtResponseHandler Cannot publish response event. null
java.lang.NullPointerException
at org.wso2.carbon.apimgt.usage.publisher.APIMgtUsageDataBridgeDataPublisher.publishEvent(APIMgtUsageDataBridgeDataPublisher.java:140)
at org.wso2.carbon.apimgt.usage.publisher.APIMgtResponseHandler.mediate(APIMgtResponseHandler.java:211)
at org.apache.synapse.mediators.ext.ClassMediator.mediate(ClassMediator.java:84)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:155)
at org.apache.synapse.rest.Resource.process(Resource.java:297)
at org.apache.synapse.rest.API.process(API.java:335)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:86)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:52)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:295)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:529)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:172)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:251)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Then i change the way of DAS configuration. Following this blog ,i configured all the api-manager.xml,there is no error and no stats.And store node show info:
[2016-10-18 11:31:39,131] INFO - ReceiverGroup Resending the failed published data...
[2016-10-18 11:31:44,134] WARN - AccessConfiguration Error loading properties from file: access-log.properties
[2016-10-18 11:31:44,207] INFO - TimeoutHandler This engine will expire all callbacks after : 120 seconds, irrespective of the timeout
I google the reason ,in wso2 jira,they say this is not a bug.
But I cannot wait for my stats for hours.Did I configure something wrong?
Reguards.
Seems in store node, data are publishing are enabled and working. But gateway node data not publishing. Please try to verify the gateway node configuration. Also check gateway node can access the DAS node.

Exception in Authentication Framework WSO2 IS

I am running WSO2 IS 5.0.0. I have the SP for IS 5.0.0 applied along with all the other security patches issued for that version for Identity Server and Carbon 4.2.0. My environment consists of 4 machines that are creating a cluster (using the WKA membership scheme and Load Balancer(AWS ELB) with sticky session enabled). I am using MySQL(not the default H2 database). The machines on which the IS is deployed are Windows Server 2012 R2 (EC2 AWS machines).
Sporadically I receive the following error in the console log:
ERROR {org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultRequestCoordinator} - Exception in Authentication Framework
java.lang.NullPointerException
at org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultLogoutRequestHandler.sendResponse(DefaultLogoutRequestHandler.java:155)
at org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultLogoutRequestHandler.handle(DefaultLogoutRequestHandler.java:131)
at org.wso2.carbon.identity.application.authentication.framework.handler.request.impl.DefaultRequestCoordinator.handle(DefaultRequestCoordinator.java:97)
at org.wso2.carbon.identity.application.authentication.framework.servlet.CommonAuthenticationServlet.doPost(CommonAuthenticationServlet.java:54)
at org.wso2.carbon.identity.application.authentication.framework.servlet.CommonAuthenticationServlet.doGet(CommonAuthenticationServlet.java:44)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:60)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I am not quite sure what might be causing this error to occur. It is not critical for my system because I have not noticed that it is causing some kind of troubles or downtime.
However I would like to know whether it is a bug or known issue? If so is there a way to be fixed (or applied a workaround) by updating some configuration files? May be there is some sort of patch that could fix it?
Thanks in advance.
I assume this happens when relyingParty is null. Try sending it as a query param of the logout request like this.
https://is_host.com/commonauth?commonAuthLogout=true&type=samlsso&sessionDataKey
=E294FEF4A64BF7E14940E2964F78E351&commonAuthCallerPath=https://is_home.com/
identity/main/logoutsso&relyingParty=SAMLSSOIdentity

memcached-session-manager on AWS

I've got a website running on Amazon Web Services that is deployed using Elastic Beanstalk and runs on a minimum of 2 EC2 micro instances. An auto scaling policy is in place, so that it can scale up and scale down depending on the the traffic in the website. Because of this auto scaling policy, I wanted to avoid using sticky sessions and for that reason I am using memcached-session-manager. I am using Amazon ElastiCache (small instance) for the memcached server.
The configuration in the context.xml is as follows:
<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="sessions.myinstancecode.0001.use1.cache.amazonaws.com:11211"
sticky="false"
sessionBackupAsync="false"
lockingMode="none"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory" />
This works fine when the traffic is low (i.e. less than 10 users online) but sometimes causes the EC2 instance to restart. You can imagine that if the website is currently running on two instances and they both decide to restart at the same time, the website becomes unreachable and it's a big problem. These are the last lines in the tail_catalina.log that is rotated on Amazon S3 before the EC2 instance decides to restart:
Jun 13, 2012 12:32:27 AM de.javakaffee.web.msm.BackupSessionTask handleException
WARNING: Could not store session 42F9761AC24F826E1FC3F2A834FBF442 in memcached.
Note that this session was relocated to this node because the original node was not available.
net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: sessions.myinstancecode.0001.use1.cache.amazonaws.com/10.194.23.99:11211
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:73)
at de.javakaffee.web.msm.BackupSessionTask.storeSessionInMemcached(BackupSessionTask.java:230)
at de.javakaffee.web.msm.BackupSessionTask.doBackupSession(BackupSessionTask.java:195)
at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:120)
at de.javakaffee.web.msm.BackupSessionTask.call(BackupSessionTask.java:51)
at de.javakaffee.web.msm.BackupSessionService$SynchronousExecutorService.submit(BackupSessionService.java:339)
at de.javakaffee.web.msm.BackupSessionService.backupSession(BackupSessionService.java:198)
at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:967)
at de.javakaffee.web.msm.SessionTrackerValve.backupSession(SessionTrackerValve.java:226)
at de.javakaffee.web.msm.SessionTrackerValve.invoke(SessionTrackerValve.java:128)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:680)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Jun 13, 2012 12:32:28 AM de.javakaffee.web.msm.LockingStrategy onAfterBackupSession
WARNING: An error occurred during onAfterBackupSession.
net.spy.memcached.internal.CheckedOperationTimeoutException: Timed out waiting for operation - failing node: sessions.myinstancecode.0001.use1.cache.amazonaws.com/10.194.23.99:11211
at net.spy.memcached.internal.OperationFuture.get(OperationFuture.java:73)
at de.javakaffee.web.msm.LockingStrategy.onAfterBackupSession(LockingStrategy.java:287)
at de.javakaffee.web.msm.MemcachedSessionService.backupSession(MemcachedSessionService.java:970)
at de.javakaffee.web.msm.SessionTrackerValve.backupSession(SessionTrackerValve.java:226)
at de.javakaffee.web.msm.SessionTrackerValve.invoke(SessionTrackerValve.java:128)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:680)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
It seems like the Amazon ElastiCache node is failing, but the thing is that, checking on Amazon CloudWatch, I can see that the CPU utilization never went over 8%. Is there any reason why the Amazon ElastiCache node fails, even though it's not being stressed that much? Also, why does Amazon decide to restart (or better: terminate and start a new instance) when Amazon ElastiChace node fails?
Any help greatly appreciated.
Thanks!
You should increase the sessionBackupTimeout of memcached-session-manager, from the documentation:
sessionBackupTimeout (optional, default 100)
The timeout in milliseconds after that a session backup is considered
as beeing failed. This property is only evaluated if sessions are
stored synchronously (set via sessionBackupAsync). The default value
is 100 milliseconds.