WSO2 API-Manager HTTP Requests terminate after 5mins with Error 101508 - wso2

Question

We deployed a long running HTTP service on WSO2 APIM 3.2.0 that needs approx. 6-15minutes to reply to a request. To support long running HTTP calls we configured the API endpoint timeouts to 15mins and also the APIM in the deployment.toml:
[synapse_properties]
'synapse.global_timeout_interval' = 900000
[passthru_http]
'http.socket.timeout' = 1000000
However, always after exactly 5 minutes we get the following error:
<am:fault xmlns:am=“http://wso2.org/apimanager”><am:code>101508</am:code><am:type>Status report</am:type><am:message>Runtime Error</am:message><am:description>Error in Sender</am:description></am:fault>
This indicates that our backend service is terminating the connection. However, if we call the service directly, i.e. without APIM Gateway it replies after 6-15min. Do we miss any timeout settings in APIM that it completes after 5mins ?
Thanks a lot for your help!
Cheers,
Sebastian

Answer 1

101508, this will happen because the backend endpoint is getting timed out. You could see the occurrences of the endpoint time out and in the same way, the entire HTTP socket also gets timeout in the wso2carbon.log.
Sample endpoint timeout log
TID: [-1234] [] [2022-02-22 21:30:43,187] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : <ENDPOINT> will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
Sample socket timeout
TID: [-1] [] [2022-02-22 21:30:43,186] WARN {org.apache.synapse.transport.passthru.TargetHandler} - Connection closed by target host before receiving the response {org.apache.synapse.transport.passthru.TargetHandler
Please check the endpoint and htp.socket.timeout values as mentioned in the WSO2 documentation[1].
You should increase the above values that should be more than 6-15 minutes. You can configure it in milliseconds.
[1] https://apim.docs.wso2.com/en/3.2.0/learn/design-api/endpoints/resiliency/endpoint-timeouts/
Note: The socket timeout value should be greater than both the Synapse global timeout and any endpoint timeouts given for your API.