Unable to issue valid access token from WSO2 IS 5.0.0 - wso2

Question

I am running WSO2 IS 5.0.0. I have the SP for IS 5.0.0 applied along with all the other security patches issued for that version for Identity Server and Carbon 4.2.0. My environment consists of 4 machines that are creating a cluster (using the WKA membership scheme and Load Balancer with sticky session). I am using MySQL(not the default H2 database). The machines on which the IS is deployed are Windows Server 2012 R2 (EC2 AWS machines).
The PRODUCT_HOME/repository/conf/identity.xml has the following configuration:
<JDBCPersistenceManager>
<DataSource>
<Name>jdbc/WSO2CarbonDB</Name>
</DataSource>
<SessionDataPersist>
<Enable>true</Enable>
<RememberMePeriod>20060</RememberMePeriod>
<CleanUp>
<Enable>true</Enable>
<Period>1440</Period>
<TimeOut>20160</TimeOut>
</CleanUp>
<Temporary>false</Temporary>
</SessionDataPersist>
</JDBCPersistenceManager>
<SessionContextCache>
<Enable>true</Enable>
<Capacity>100000</Capacity>
</SessionContextCache>
<OAuth>
<AuthorizationCodeDefaultValidityPeriod>300</AuthorizationCodeDefaultValidityPeriod>
<AccessTokenDefaultValidityPeriod>1800</AccessTokenDefaultValidityPeriod>
<UserAccessTokenDefaultValidityPeriod>1800</UserAccessTokenDefaultValidityPeriod>
<RefreshTokenValidityPeriod>31540000</RefreshTokenValidityPeriod>
<TimestampSkew>10</TimestampSkew>
<EnableOAuthCache>false</EnableOAuthCache>
<RenewRefreshTokenForRefreshGrant>true</RenewRefreshTokenForRefreshGrant>
</OAuth>
Sporadically I receive the following error in the Identity Server console log(when issuing access token using the client_credentials grant type and openid scope):
ERROR {org.wso2.carbon.identity.openidconnect.DefaultIDTokenBuilder} - Error occurred while getting access token based information
ERROR {org.wso2.carbon.identity.oauth2.OAuth2Service} - Error when issuing the access token.
org.wso2.carbon.identity.oauth2.IdentityOAuth2Exception: Error occurred while getting access token based information
at org.wso2.carbon.identity.openidconnect.DefaultIDTokenBuilder.getAccessTokenIssuedTime(DefaultIDTokenBuilder.java:348)
at org.wso2.carbon.identity.openidconnect.DefaultIDTokenBuilder.buildIDToken(DefaultIDTokenBuilder.java:141)
at org.wso2.carbon.identity.oauth2.token.AccessTokenIssuer.issue(AccessTokenIssuer.java:212)
at org.wso2.carbon.identity.oauth2.OAuth2Service.issueAccessToken(OAuth2Service.java:177)
at org.wso2.carbon.identity.oauth.endpoint.token.OAuth2TokenEndpoint.getAccessToken(OAuth2TokenEndpoint.java:233)
at org.wso2.carbon.identity.oauth.endpoint.token.OAuth2TokenEndpoint.issueAccessToken(OAuth2TokenEndpoint.java:108)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:194)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:159)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I am still trying to isolate the exact scenario which is causing the system to behave this way.
When this particular error is logged in the Identity Server console log, I get the following in the http_access log (PRODUCT_HOME/repository/logs/http_access_DATE.log)
172.31.26.60 - - [09/Sep/2016:07:18:28 +0000] "POST /oauth2/token?grant_type=client_credentials&scope=openid HTTP/1.1" 400 82 "-" "-"
As end result I receive the following as response:
error: "server_error"
error_description: "Error when issuing the access token"
I tried updating the PRODUCT_HOME/repository/conf/identity.xml file by setting the EnableOAuthCache to TRUE:
<EnableOAuthCache>true</EnableOAuthCache>
Then I restarted all the Identity Server instances (as I mentioned I have 4 machines in WKA cluster). Everything was going well until one day we started receiving access token which is invalid.
After digging further into this particular issue, I was able to find out that for some reason the Identity Server was returning two access tokens. The one was valid and the other not. Still I was using the exact same POST request for retrieving access token with client_credentials grant type. Usually after issuing the access token I am executing get request to the userinfo endpoint and it turned out that only one of the access tokens is valid.
The mentioned above is also randomly happening. Usually after 10-15 mins, the invalid token is not returned anymore and the system is stabilized.
When the issue is experienced the following is logged in the Identity Server console log:
ERROR{org.wso2.carbon.identity.oauth2.dao.TokenPersistenceTask} - org.wso2.carbon.identity.oauth2.IdentityOAuth2Exception: Error when getting an Identity Persistence Store instance.
No additional information or stack trace.
Also the invalid access token which is returned does not exist in the database.
My system relies on these access tokens and when I am not able to retrieve these or I retrieve invalid token and not able to call the userinfo endpoint I am experiencing downtime. The last time it was about 15-20 mins.
So my questions are:
1. Has someone else observed such behavior or is it a known issue? If so is there a workaround for it or some sort of patch that could be applied?
2. If not do you think it might be some sort of configuration issue? If so which configuration files you think should be modified?
3. Do you think this behavior might be related to the WKA Cluster configuration?
4. Do you think this might be related to the usage of MySQL?
Thank you for your time and consideration.

Answer 1

It's better to use the latest release in the identity server. You can go to the WSO2 Identity Server page and download the latest version.