Why my WSO2 APIM APIKeyValidator responds with Erro? - wso2

Question

I am building an cluster with 2 WSO2 APIM All-in-One distributions. So at this point I have the APACHE as front end of my cluster and the others 2 APIM nodes shared the <APIM_HOME>/repository/deployment/server.
The problem now is the APIKeyValidator. After published and subscribed in test API, when I am try to consume an operation, I am receiving the message:
{
"fault": {
"code": 900900,
"message": "Unclassified Authentication Failure",
"description": "Resource forbidden"
}
}
My carbon log gave me the follow lines:
TID: [-1234] [] [2020-12-09 11:19:53,476] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator} - Received Token 44ce0123-2815-3a9c-8acf-9e12bc5ae2a6 {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator}
TID: [-1234] [] [2020-12-09 11:19:53,476] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator} - Default Version API invoked {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator}
TID: [-1234] [] [2020-12-09 11:19:53,476] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator} - Removing Authorization header from headers {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator}
TID: [-1234] [] [2020-12-09 11:19:53,477] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator} - Found resource in Cache for key: /apis/operadoras/3.0.0/3.0.0/operadoras:GET {org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator}
TID: [-1234] [] [2020-12-09 11:19:53,477] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator} - Matching resource is: /operadoras {org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator}
TID: [-1234] [] [2020-12-09 11:19:53,477] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient} - KeyValidation request from gateway to keymanager via web service call for:/apis/operadoras/3.0.0 with ID: urn:uuid:f345252f-24f5-48e6-a6e2-7d2154f6177d at [2020.12.09 11:19:53,477 BRT] {org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient}
TID: [-1234] [] [2020-12-09 11:19:53,489] DEBUG {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService} - KeyValidation request from gateway: requestTime= [2020.12.09 11:19:53,489 BRT] , for:/apis/operadoras/3.0.0 with accessToken=44ce0123-2815-3a9c-8acf-9e12bc5ae2a6 , transactionId= {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService}
TID: [-1234] [] [2020-12-09 11:19:53,489] DEBUG {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService} - Before calling Validate Token method... {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService}
TID: [-1234] [] [2020-12-09 11:19:53,489] DEBUG {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService} - **State after calling validateToken ... true** {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService}
TID: [-1234] [] [2020-12-09 11:19:53,491] DEBUG {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService} - **State after calling validateSubscription... true** {org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService}
TID: [-1234] [] [2020-12-09 11:19:53,492] **ERROR {org.apache.axis2.rpc.receivers.RPCMessageReceiver} - Invalid tenant domain null** {org.apache.axis2.rpc.receivers.RPCMessageReceiver}
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:173)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:147)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:232)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.owasp.csrfguard.CsrfGuardFilter.doFilter(CsrfGuardFilter.java:88)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:65)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter(HttpHeaderSecurityFilter.java:124)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.identity.context.rewrite.valve.TenantContextRewriteValve.invoke(TenantContextRewriteValve.java:80)
at org.wso2.carbon.identity.authz.valve.AuthorizationValve.invoke(AuthorizationValve.java:91)
at org.wso2.carbon.identity.auth.valve.AuthenticationValve.invoke(AuthenticationValve.java:65)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.event.receiver.core.internal.tenantmgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:48)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1115)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1775)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1734)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
**Caused by: org.wso2.carbon.identity.base.IdentityRuntimeException: Invalid tenant domain null**
at org.wso2.carbon.identity.base.IdentityRuntimeException.error(IdentityRuntimeException.java:63)
at org.wso2.carbon.identity.core.util.IdentityTenantUtil.getTenantId(IdentityTenantUtil.java:252)
at org.wso2.carbon.identity.oauth2.validators.JDBCScopeValidator.getTenantId(JDBCScopeValidator.java:294)
at org.wso2.carbon.identity.oauth2.validators.JDBCScopeValidator.validateScope(JDBCScopeValidator.java:150)
at org.wso2.carbon.apimgt.keymgt.handlers.DefaultKeyValidationHandler.validateScopes(DefaultKeyValidationHandler.java:180)
at org.wso2.carbon.apimgt.keymgt.service.APIKeyValidationService.validateKey(APIKeyValidationService.java:188)
... 59 more
TID: [-1234] [] [2020-12-09 11:19:53,496] DEBUG {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} - Call to Key Manager : API call failed reason=API_authentication_failure transactionId=urn:uuid:f345252f-24f5-48e6-a6e2-7d2154f6177d with userAgent=PostmanRuntime/7.26.8 for requestURI=/apis/operadoras/3.0.0/operadoras at time=Wed Dec 09 11:19:53 BRT 2020 from clientIP=10.19.52.80, elapsedTimeInMilliseconds=0 {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler}
TID: [-1234] [] [2020-12-09 11:19:53,496] ERROR {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler} - API authentication failure due to Unclassified Authentication Failure {org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler}
org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Resource forbidden
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.WSAPIKeyDataStore.getAPIKeyData(WSAPIKeyDataStore.java:51)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.doGetKeyValidationInfo(APIKeyValidator.java:323)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIKeyValidator.getKeyValidationInfo(APIKeyValidator.java:255)
at org.wso2.carbon.apimgt.gateway.handlers.security.oauth.OAuthAuthenticator.authenticate(OAuthAuthenticator.java:206)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.isAuthenticate(APIAuthenticationHandler.java:210)
at org.wso2.carbon.apimgt.gateway.handlers.security.APIAuthenticationHandler.handleRequest(APIAuthenticationHandler.java:158)
at org.apache.synapse.rest.API.process(API.java:325)
at org.apache.synapse.rest.RESTRequestHandler.apiProcessNonDefaultStrategy(RESTRequestHandler.java:149)
at org.apache.synapse.rest.RESTRequestHandler.dispatchToAPI(RESTRequestHandler.java:95)
at org.apache.synapse.rest.RESTRequestHandler.process(RESTRequestHandler.java:71)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:303)
at org.apache.synapse.core.axis2.SynapseMessageReceiver.receive(SynapseMessageReceiver.java:92)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ServerWorker.processNonEntityEnclosingRESTHandler(ServerWorker.java:337)
at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:158)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.wso2.carbon.apimgt.gateway.handlers.security.APISecurityException: Error while accessing backend services for API key validation
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient.getAPIKeyData(APIKeyValidatorClient.java:123)
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.WSAPIKeyDataStore.getAPIKeyData(WSAPIKeyDataStore.java:48)
... 18 more
Caused by: org.apache.axis2.AxisFault: Invalid tenant domain null
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:381)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:456)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:227)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.wso2.carbon.apimgt.keymgt.stub.validator.APIKeyValidationServiceStub.validateKey(APIKeyValidationServiceStub.java:531)
at org.wso2.carbon.apimgt.gateway.handlers.security.keys.APIKeyValidatorClient.getAPIKeyData(APIKeyValidatorClient.java:110)
... 19 more
I don´t find any information that could help me to deal with this issue...
Have any one an idea?
UPDATE: After a good investigation I have discovered that the problem happens only when we configure scopes. If we don´t use scopes The servers request have no problem.
UPDATE 2: After turn on the log4j.logger.org.wso2.carbon.identity.oauth2=DEBUG in the log4j.properties, I had discovery that, the scopes and tenants are created and available for resources but the resource has one more '/3.0.0' (version).
TID: [-1234] [] [2020-12-11 12:04:22,320] DEBUG {org.wso2.carbon.identity.oauth2.validators.DefaultOAuth2TokenValidator} - There is no scope validator registered for subscriber_DefaultApplication_PRODUCTION#carbon.super {org.wso2.carbon.identity.oauth2.validators.DefaultOAuth2TokenValidator}
TID: [-1234] [] [2020-12-11 12:04:22,322] DEBUG {org.wso2.carbon.identity.oauth2.dao.TokenManagementDAOImpl} - Retrieving tenant and scope for resource: /apis/myapi/3.0.0/3.0.0/info:GET {org.wso2.carbon.identity.oauth2.dao.TokenManagementDAOImpl}
TID: [-1234] [] [2020-12-11 12:04:22,324] DEBUG {org.wso2.carbon.identity.oauth2.dao.TokenManagementDAOImpl} - Found tenant id: -1234 and scope: SC_OPERADORAS_R for resource: /apis/myapi/3.0.0/3.0.0/info:GET {org.wso2.carbon.identity.oauth2.dao.TokenManagementDAOImpl}
TID: [-1234] [] [2020-12-11 12:04:22,325] ERROR {org.apache.axis2.rpc.receivers.RPCMessageReceiver} - Invalid tenant domain null {org.apache.axis2.rpc.receivers.RPCMessageReceiver}
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Na