API Manager Gateway 1.8 Incoming Connections Timing out - amazon-web-services

Question

I am running API Manager Gateway version 1.8. Server is running CentOS 8 with Java(TM) SE Runtime Environment (build 1.7.0_67-b01). I am testing on a test and production server. To rule out differences between the servers the production server was cloned from the test server and updated to point to a separate DB, Key Manager & LDAP server. In addition it was synched with Gateway Management node.
I start up both of the Gateway servers. There are several dozen APIs deployed on each server. For testing I deployed an identical API on test and prod that point to the same backend service. The API does not require authentication so there is no token call. I execute a wget for the API directly to the Gateway worker in prod and test.
The call executes successfully in test returning a response in about 1 second.
However call to production hangs for a minute, then wget retries. Eventually after several retries the call succeeds.
I have made hundreds of calls to the service directly from command prompt on the production Gateway node and they are successful every time.
I am skipping the load balancer so all traffic in my test is via http to the Gateway server and to my backend service.
In production I see the following in the logs. The request for /csmjk followed by a 1 minute delay before the http-incoming-1 times out.
TID: [0] [AM] [2017-04-16 07:38:31,472] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-1: GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:38:31,479] DEBUG {org.apache.synapse.transport.nhttp.access} - - 10.40.1.161 - - [16/Apr/2017:07:38:31 -0500] "GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0" - - "-" "Wget/1.12 (linux-gnu)" {org.apache.synapse.transport.nhttp.access}
TID: [0] [AM] [2017-04-16 07:38:31,481] DEBUG {org.apache.synapse.transport.passthru.ServerWorker} - Starting a new Server Worker instance {org.apache.synapse.transport.passthru.ServerWorker}
TID: [0] [AM] [2017-04-16 07:39:31,547] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-1: Timeout {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:39:31,547] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-1 {org.apache.synapse.transport.passthru.SourceHandler}
In test I see the request followed immediately by a call to our backend service:
TID: [0] [AM] [2017-04-16 07:42:08,281] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-10: GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:42:08,286] DEBUG {org.apache.synapse.transport.nhttp.access} - - 10.40.1.161 - - [16/Apr/2017:07:42:08 -0500] "GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0" - - "-" "Wget/1.12 (linux-gnu)" {org.apache.synapse.transport.nhttp.access}
TID: [0] [AM] [2017-04-16 07:42:08,304] DEBUG {org.apache.synapse.transport.passthru.ServerWorker} - Starting a new Server Worker instance {org.apache.synapse.transport.passthru.ServerWorker}
TID: [0] [AM] [2017-04-16 07:42:08,394] INFO {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher} - Flushing the events from the queue 1 {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher}
TID: [0] [AM] [2017-04-16 07:42:08,426] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - Headers : {Accept=*/*, Connection=Keep-Alive, Host=TEST:8281, User-Agent=Wget/1.12 (linux-gnu), X-JWT-Assertion=null} {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,426] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - Message context:[MessageContext: logID=57478a056938f45377e3a24e79fae0781cbfcc13f4af60aa] {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,585] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - End user: null, API user: null {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,601] INFO {org.apache.synapse.core.axis2.TimeoutHandler} - This engine will expire all callbacks after : 120 seconds, irrespective of the timeout action, after the specified or optional timeout {org.apache.synapse.core.axis2.TimeoutHandler}
TID: [0] [AM] [2017-04-16 07:42:08,617] DEBUG {org.apache.synapse.transport.passthru.connections.TargetConnections} - Trying to get a connection {}->http://SERVICE:1111 {org.apache.synapse.transport.passthru.connections.TargetConnections}
This has been working fine in production for many months and suddenly stopped working reliably two days ago.
Any help would be greatly appreciated.
Thanks

Answer 1

I have been working on this for 3 days straight now. After enabling additional debugging I determined that the last thing done prior to the timeout was a call to Google Analytics. I disabled Google Analytics Tracking and everything is back to normal now. This is also configured in my test instance. More debugging to come but at least I am back to responding to requests.
Thanks