Request still failing with WSO2 API Manager 3.2.0 high availability endpoints - wso2

Question

We have configured the high availability for API end points as mentioned here https://apim.docs.wso2.com/en/3.2.0/learn/design-api/endpoints/high-availability-for-endpoints/#configuring-load-balancing-endpoints
The configurations are as below
EndpointType: Load Balanced
Endpoint Suspension State:
ErrorCode = Connection Failed and Connection Closed
Initial Duration = 1800000
Max Duration = 1800000
Factor = 1
EndPoint Timeout State:
Retries Before Suspension = 2
Retry Delay = 15000
With above settings, some requests are still routed to the suspended endpoints and failing though the endpoint is already in suspended state.
Failure response
{
"fault": {
"code": 303000,
"type": "Status report",
"message": "Runtime Error",
"description": "Failover endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint - no ready child endpoints"
}
}
{
"fault": {
"code": 101503,
"type": "Status report",
"message": "Runtime Error",
"description": "Error connecting to the back end"
}
}
The error messages are as below
TID: [-1] [] [2023-01-09 12:25:48,776] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : /100.66.213.31:7010
TID: [-1234] [] [2023-01-09 12:25:48,777] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider is marked as TIMEOUT and will be retried : 1 more time/s after : Mon Jan 09 12:26:03 UTC 2023 until its marked SUSPENDED for failure
TID: [-1234] [] [2023-01-09 12:25:48,777] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint] Detect a Failure in a child endpoint : Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0]
TID: [-1234] [] [2023-01-09 12:25:48,778] INFO {org.apache.synapse.mediators.builtin.LogMediator} - {api:admin--NewMCMInboundChannel-RESTAPIService:vv2} STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end
TID: [-1] [] [2023-01-09 12:27:26,156] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : /100.66.213.31:7010
TID: [-1234] [] [2023-01-09 12:27:26,158] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider is marked as TIMEOUT and will be retried : 0 more time/s after : Mon Jan 09 12:27:41 UTC 2023 until its marked SUSPENDED for failure
TID: [-1234] [] [2023-01-09 12:27:26,158] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint] Detect a Failure in a child endpoint : Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0]
TID: [-1234] [] [2023-01-09 12:27:26,158] INFO {org.apache.synapse.mediators.builtin.LogMediator} - {api:admin--NewMCMInboundChannel-RESTAPIService:vv2} STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end
TID: [-1] [] [2023-01-09 12:31:36,455] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : /100.66.213.31:7010
TID: [-1234] [] [2023-01-09 12:31:36,459] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider has been marked for SUSPENSION, but no further retries remain. Thus it will be SUSPENDED.
TID: [-1234] [] [2023-01-09 12:31:36,459] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider - current suspend duration is : 1800000ms - Next retry after : Mon Jan 09 13:01:36 UTC 2023
TID: [-1234] [] [2023-01-09 12:31:36,460] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint] Detect a Failure in a child endpoint : Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0]
TID: [-1234] [] [2023-01-09 12:31:36,460] INFO {org.apache.synapse.mediators.builtin.LogMediator} - {api:admin--NewMCMInboundChannel-RESTAPIService:vv2} STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end

Answer 1

Let me describe the scenario using your error logs.
Let's say you have endpoint 0 and endpoint 1.
Endpoint 1 works as expected and the endpoint 0 is the erroneous endpoint. You have configured above timeout and suspension configurations for endpoint 2.
Endpoint_0 received the request, however it has timedout. Since you have configured 2 as the retry limit, it will try two more times and suspend the request until endpoint suspension duration. As it has been timedout, server will temporary suspend this endpoint for 15000ms(Retry Delay). Until then, endpoint_1 serves the traffic.
TID: [-1] [] [2023-01-09 12:25:48,776] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : /100.66.213.31:7010
TID: [-1234] [] [2023-01-09 12:25:48,777] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider is marked as TIMEOUT and will be retried : 1 more time/s after : Mon Jan 09 12:26:03 UTC 2023 until its marked SUSPENDED for failure
TID: [-1234] [] [2023-01-09 12:25:48,777] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint] Detect a Failure in a child endpoint : Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0]
TID: [-1234] [] [2023-01-09 12:25:48,778] INFO {org.apache.synapse.mediators.builtin.LogMediator} - {api:admin--NewMCMInboundChannel-RESTAPIService:vv2} STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end
After 15000ms, endpoint_0 will serve another request. This too timedout and will be marked suspended for another 15000ms.
TID: [-1] [] [2023-01-09 12:27:26,156] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : /100.66.213.31:7010
TID: [-1234] [] [2023-01-09 12:27:26,158] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider is marked as TIMEOUT and will be retried : 0 more time/s after : Mon Jan 09 12:27:41 UTC 2023 until its marked SUSPENDED for failure
TID: [-1234] [] [2023-01-09 12:27:26,158] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint] Detect a Failure in a child endpoint : Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0]
TID: [-1234] [] [2023-01-09 12:27:26,158] INFO {org.apache.synapse.mediators.builtin.LogMediator} - {api:admin--NewMCMInboundChannel-RESTAPIService:vv2} STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end
The third consecutive request to the endpoint_0 has been timedout and there aren't any retried left. Endpoint will be marked suspended until the suspension duration.
TID: [-1] [] [2023-01-09 12:31:36,455] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : /100.66.213.31:7010
TID: [-1234] [] [2023-01-09 12:31:36,459] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider has been marked for SUSPENSION, but no further retries remain. Thus it will be SUSPENDED.
TID: [-1234] [] [2023-01-09 12:31:36,459] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0 with address http://100.66.213.31:7010/mcm-provider - current suspend duration is : 1800000ms - Next retry after : Mon Jan 09 13:01:36 UTC 2023
TID: [-1234] [] [2023-01-09 12:31:36,460] WARN {org.apache.synapse.endpoints.LoadbalanceEndpoint} - Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint] Detect a Failure in a child endpoint : Endpoint [NewMCMInboundChannel-RESTAPIService--vv2_APIproductionEndpoint_0]
TID: [-1234] [] [2023-01-09 12:31:36,460] INFO {org.apache.synapse.mediators.builtin.LogMediator} - {api:admin--NewMCMInboundChannel-RESTAPIService:vv2} STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end