WSO2 API Manager Store - Access token key generation failure - wso2

Question

I have setup an API Manager 1.10 cluster, with each component in a separate VM, fronted by an nginx proxy.
Whenever I try to generate a key I get the following error in store (logs from store):
ERROR - AMDefaultKeyManagerImpl Error while creating tokens - Connection timed out
java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
...
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
[2016-12-14 16:25:03,376] ERROR - APIConsumerImpl Error while re-generating AccessToken
org.wso2.carbon.apimgt.api.APIManagementException: Error while creating tokens - Connection timed out
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.handleException(AMDefaultKeyManagerImpl.java:592)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:369)
at org.wso2.carbon.apimgt.impl.APIConsumerImpl.renewAccessToken(APIConsumerImpl.java:659)
at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.renewAccessToken(UserAwareAPIConsumer.java:36)
at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_refreshToken(APIStoreHostObject.java:3800)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
.....
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:303)
... 70 more
[2016-12-14 16:25:03,377] ERROR - APIStoreHostObject Error while renewing Access Token for Consumer Key qZf8VvT4ts7a0TT6nE6ZHfY9t5sa and user admin
[2016-12-14 16:25:03,377] ERROR - subscription-add:jag org.wso2.carbon.apimgt.api.APIManagementException: Error while renewing Access Token for Consumer Key 88 and user admin
I have configured connections among the components by following this article The logs above are only showing up in the API store console.
I have also configured the Keymanager host and port in API Gateway in the following locations:
/repository/deployment/server/synapse-configs/default/api
AuthorizeAPI.xml
RevokeAPI.xml
TokenAPI.xml
What do I need to do to fix this?

Answer 1

In store's api-manager.xml, <RevokeAPIURL> should be pointing to gateway. make sure you have that configured properly.

Answer 2

I didn't have the port 8243 opened in my security group for the API gateway. So the problem had nothing to do with WSO2, it was my infrastructure.