WSO2 ESB: Address endpoint does not resume sending after backends recovery - wso2

Question

I am using WSO2 ESB with RabbitMQ, I have two proxy services:
AMQPProducerSample, it receives messages via HTTP transport and send it to RabbitMQ queue
AMQPProxy it works as consumer for RabbitMQ queue (via rabbitmq transport), consumed messages are send to HTTP endpoint SampleEndPoint
Everything works fine except one scenario:
My backend service set in SampleEndPoint goes down.
New messages arrives and are published via AMQPProducerSample, delivery fails (that is expected because my backend is down). In console I can see:
WARN - ConnectCallback Connection refused or failed for : mfb.localhost/127.0.0.1:80
WARN - FaultHandler ERROR_CODE : 101503
WARN - FaultHandler ERROR_MESSAGE : Error connecting to the back end
WARN - FaultHandler ERROR_DETAIL : Error connecting to the back end
WARN - FaultHandler ERROR_EXCEPTION : null
WARN - FaultHandler FaultHandler : Endpoint [SampleEndPoint]
My backend service recovers and is available
New messages arrives and are published via AMQPProducerSample but my endpoint does not receive any message any more. It works fine only if number of undelivered messages is below 5. In that case messages are successfully delivered to backend service. The delivery always stops working after 5 (five) undelivered messages.
How to solve this? Is there any option I need to set or change to make it work all the time?
I am using WSO2 ESB 4.8.1
Below is my full config:
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
<registry provider="org.wso2.carbon.mediation.registry.WSO2Registry">
<parameter name="cachableDuration">15000</parameter>
</registry>
<proxy name="AMQPProxy"
transports="rabbitmq"
startOnLoad="true"
trace="enable">
<description/>
<target>
<inSequence>
<log level="full"/>
<property name="OUT_ONLY" value="true"/>
<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
<send>
<endpoint key="SampleEndPoint"/>
</send>
</inSequence>
</target>
<parameter name="rabbitmq.queue.name">queue</parameter>
<parameter name="rabbitmq.connection.factory">AMQPConnectionFactory</parameter>
<parameter name="rabbitmq.exchange.name">exchange</parameter>
<parameter name="rabbitmq.queue.routing.key">route</parameter>
</proxy>
<proxy name="AMQPProducerSample"
transports="http"
startOnLoad="true"
trace="disable">
<description/>
<target>
<inSequence>
<property name="OUT_ONLY" value="true"/>
<property name="FORCE_SC_ACCEPTED" value="true" scope="axis2"/>
<property name="NO_KEEPALIVE" value="true" scope="axis2"/>
<send>
<endpoint>
<address uri="rabbitmq:/AMQPProxy?rabbitmq.server.host.name=localhost&rabbitmq.server.port=5672&rabbitmq.queue.name=queue&rabbitmq.queue.routing.key=route&rabbitmq.exchange.name=exchange"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<send/>
</outSequence>
</target>
</proxy>
<endpoint name="SampleEndPoint">
<address uri="http://mfb.localhost/">
<timeout>
<duration>1000</duration>
<responseAction>discard</responseAction>
</timeout>
<suspendOnFailure>
<errorCodes>-1</errorCodes>
<progressionFactor>1.0</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<errorCodes>-1</errorCodes>
</markForSuspension>
</address>
</endpoint>
<sequence name="fault">
<log level="full">
<property name="MESSAGE" value="Executing default 'fault' sequence"/>
<property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/>
<property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/>
</log>
<drop/>
</sequence>
<sequence name="main">
<in>
<log level="full"/>
<filter source="get-property('To')" regex="http://localhost:9000.*">
<send/>
</filter>
</in>
<out>
<send/>
</out>
<description>The main sequence for the message mediation</description>
</sequence>
</definitions>

Answer 1

This is a known bug which was there in ESB 4.8.1 fresh pack. Later on we have fixed it. So I recommend you to switch into ESB 4.9.0 release where this issue was fixed.
[1] https://docs.wso2.com/display/ESB490/Downloading+the+Product

Answer 2

You have to change the following parameter within transportReceiver for rabbitmq in axis2.xml
<parameter name="rabbitmq.connection.retry.count" locked="false">5</parameter>
"rabbitmq.connection.retry.count" is the number of times attempted to reconnect after each retry, I think you have set it to 5, that is why connection drop after five attempts. Increse the parameter value and try it.