Issue with Iterator Mediator in WSO2 ESB - wso2

I have an scenario like this:
A proxy service in ESB do a call to a service and receive a payload, between 2.3 and 4.9MB.
This payload have a list of Persons.
With this list I use the iterator mediator to iterate over it and inside each element I call a DBReport mediator to store the values inside the Person element.
When the payload have 2.3MB all work fine and the data is store in my DB in a few minutes, about 5000 elements.
But with the 4.9MB payload it´s really slowly as you can see rapidly the RAM take 2.7 of 3.0GB max. At that moment the JMX monitoring frozen and just 123 record of 10 000 elements in 3minutes was stored in DB
After a while the processing continue and stop for a while and you can see the delay between each DBReport call:
TID: [0] [ESB] [2013-06-12 16:00:00,480] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:6a11bc08-8e1f-4c37-a031-fe9c84e395da, Direction: response, TEST = 06469 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:00:09,591] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:439fa810-3333-4cbf-bb68-08eb843ca408, Direction: response, TEST = 06471 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:00:15,555] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:24b5dacd-53dd-4d78-8f57-24944cdf2a84, Direction: response, TEST = 06475 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:01:33,420] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:2a8c77a9-ae60-4f84-901d-3b4f76a6b480, Direction: response, TEST = 06477 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:01:41,489] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:1f0632c3-91ee-4fae-b47e-2c88b73aed69, Direction: response, TEST = 06478 {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:02:04,190] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:02b2b0ae-3d52-48c9-ad09-04e983501e3d, Direction: response, TEST = 06505 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:05,620] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:ae2b72d2-124d-4f34-8033-9036a2b0419a, Direction: response, TEST = 06507 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:07,411] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:ac951f44-c212-4a4d-93c2-9473677acf32, Direction: response, TEST = 06511 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:14,232] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:99e2804e-f70b-449f-9b5f-46f65d81638b, Direction: response, TEST = 06514 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:02:15,757] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:b0f1dfca-2eda-415c-9c11-3ea70168044e, Direction: response, TEST = 06516 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:11:57,247] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:9d92ecc9-3b70-440d-8195-c743da44ccd5, Direction: response, TEST = 06517 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:12:13,862] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:fac67fed-989c-44e2-91cb-c2b3b40460ee, Direction: response, TEST = 06522 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:12:28,531] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:635d74b9-23ae-435f-909e-c2c6ae171407, Direction: response, TEST = 06523 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:12:45,257] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an SOAPAction: , MessageID: urn:uuid:59a006e2-e2ef-4241-9439-10d9db5830c3, Direction: response, TEST = 06524 {org.apache.synapse.mediators.builtin.
TID: [0] [ESB] [2013-06-12 16:19:46,585] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/an onymous, WSAction: , SOAPAction: , MessageID: urn:uuid:5d867db0-3e88-4798-9360-0f9a49b9c5ae, Direction: response, TEST = 06525 {org.apache.synap se.mediators.builtin.LogMediator}
TID: [0] [ESB] [2013-06-12 16:20:06,866] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:ad48e642-eeb0-49ef-a25f-63f2f10534c0, Direction: response, TEST = 06530 {org.apache.synapse.mediators.builtin.LogMediator}
Any idea?


DBReport mediator is executed synchronously in WSO2 ESB which means the thread that execute this DBReport mediator is stucked until the Database operation is completed. That means the performance is lower than when the execution happens asynchronously.
Therefore to gain the maximum performance, please use WSO2 Data Services server and use DSS to execute the insert Database operation. That way you can gain the maximum performance.
It is better if you can add your configuration files. That will help to identify the root cause of this issue.

Related

WSO2 API Manager returns http 500 error code

Going thru the WSO2 API Manager logs, we noticed few occurrences where calls returned as HTTP 500 error and successful on subsequent retry by client. This does not happen all the time (at about 5% rate). And all these incidents are from the same source IP. Looking further, these calls that fails with HTTP 500, never get send to back-end endpoint for actual processing - Therefore, these calls are failed by WSO2 API Manager itself.
We have verified the CPU, Memory of WSO2 are at normal working level. Our WSO2 APIM environment is with no authorization token enabled (simple pass thru with throttle) - hence this can be counted out.
From log, the error 500 is returned almost immediate and retry typically happens within 200ms. And all these errors (about 49) happens within same hour.
We also not able to verify the exact error message as API calls is from one of our client.
Appreciate if you could list down all possibilities as why WSO2 may returned HTTP 500 in this case. Otherwise, how we able to investigate the scenario further. Thanks.
Here are relevant logs
WSO2 APIM AWS ELB access logs
$ grep 'fromIndex=394501&toIndex=395000' *.log
WSO2APIM_ELBACESS.log:2016-12-06T08:37:37.905822Z wso2-oss-server x.x.x.x:63598 x.x.x.98:8280 0.000042 0.001862 0.000023 500 500 0 293 "GET https://wso2apimanager:443/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "RestSharp/x.x.x.x" ECDHE-RSA-AES128-SHA TLSv1
WSO2APIM_ELBACESS.log:2016-12-06T08:37:38.102539Z wso2-oss-server x.x.x.x:63598 x.x.x.120:8280 0.000041 0.035397 0.000025 200 200 0 33 "GET https://wso2apimanager:443/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "RestSharp/x.x.x.x" ECDHE-RSA-AES128-SHA TLSv1
WSO2 APIM http access log
$ grep 'fromIndex=394501&toIndex=395000' *.log
http_access_2016-12-06.log_m1:x.x.x.x x.x.x.140 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "RestSharp/x.x.x.x"
http_access_2016-12-06.log_m1:x.x.x.x x.x.x.72 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "Synapse-PT-HttpComponents-NIO"
http_access_2016-12-06.log_m2:x.x.x.x x.x.x.140 - - [06/Dec/2016:08:37:37 +0000] "GET /rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" - - "-" "RestSharp/x.x.x.x"
Back-end AWS ELB access log
$ grep 'fromIndex=394501&toIndex=395000' *.log
BACKEND_ELBACCESS.log:2016-12-06T08:37:38.104641Z vpc-connect-api x.x.x.120:46946 x.x.x.23:80 0.000045 0.032717 0.000022 200 200 0 33 "GET http://backendserver:80/rest/request?view=full&fromIndex=394501&toIndex=395000&lastModifiedFrom=2016-12-05%2000%3A00%3A00 HTTP/1.1" "Synapse-PT-HttpComponents-NIO" - -
WSO2 API Manager 1 (wso2carbon.log)
TID: [-1234] [] [2016-12-06 08:34:47,427] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected {org.wso2.carbon.databridge.core.DataBridge}
TID: [-1234] [] [2016-12-06 08:35:18,069] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:35:47,850] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1] [] [2016-12-06 08:37:13,485] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16702: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:13,486] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,486] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_28 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:43 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,486] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,487] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566702 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:37:13,961] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:18,342] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:18,007] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:40:17,933] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:41:17,725] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:43:17,811] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:45:17,892] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:47:18,363] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:50:47,850] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:51:17,864] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:53:17,753] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:55:18,025] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:57:18,333] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:59:17,949] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
WSO2 API Manager 2 (wso2carbon.log)
TID: [-1234] [] [2016-12-06 08:34:54,620] INFO {org.wso2.carbon.databridge.core.DataBridge} - user admin connected {org.wso2.carbon.databridge.core.DataBridge}
TID: [-1234] [] [2016-12-06 08:36:18,043] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,674] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16760: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:13,692] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,693] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_28 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:43 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:13,693] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:37:13,694] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566592 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:37:14,331] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_28 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1] [] [2016-12-06 08:37:14,620] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-16759: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [-1234] [] [2016-12-06 08:37:14,621] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:14,621] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:37:44 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:37:14,621] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101507, ERROR_MESSAGE = Error in Sender {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:14,818] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:15,255] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:29,543] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,137] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,568] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:35,990] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:36,409] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:37,899] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:38,327] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:41,129] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:41,545] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:37:42,969] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:38:09,156] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:38:17,817] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:39:16,163] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566604 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1] [] [2016-12-06 08:39:19,169] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-566640 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1234] [] [2016-12-06 08:39:19,648] ERROR {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Failed to submit the response {org.apache.synapse.transport.passthru.PassThroughHttpSender}
java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:19,649] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/json,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Failed to submit the response
at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:613)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
Caused by: java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:19,650] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 0, ERROR_MESSAGE = Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/json,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:19,688] WARN {org.apache.synapse.core.axis2.Axis2Sender} - Trying to send a response to an already responded client request - Rest API Context : /rest {org.apache.synapse.core.axis2.Axis2Sender}
TID: [-1] [] [2016-12-06 08:39:20,249] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : internal-vpc-connect-api-1883227725.ap-southeast-1.elb.amazonaws.com/x.x.x.43:80 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2016-12-06 08:39:20,252] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:20,254] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:39:50 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:20,254] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:20,254] ERROR {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Failed to submit the response {org.apache.synapse.transport.passthru.PassThroughHttpSender}
java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:20,255] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Accept-Encoding:gzip, deflate,Access-Control-Allow-Headers:authorization,Access-Control-Allow-Origin,Content-Type,SOAPAction,Access-Control-Allow-Methods:GET,Access-Control-Allow-Origin:*,Content-Type:application/xml,password:Hjfawx99##,username:lynn#heatherjames.com,X-Forwarded-For:76.79.103.18,X-Forwarded-Port:443,X-Forwarded-Proto:https,<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><am:fault xmlns:am="http://wso2.org/apimanager"><am:code>101503</am:code><am:type>Status report</am:type><am:message>Runtime Error</am:message><am:description>Error connecting to the back end</am:description></am:fault></soapenv:Body></soapenv:Envelope> Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Failed to submit the response
at org.apache.synapse.transport.passthru.PassThroughHttpSender.handleException(PassThroughHttpSender.java:613)
at org.apache.synapse.transport.passthru.PassThroughHttpSender.invoke(PassThroughHttpSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
Caused by: java.lang.NullPointerException
TID: [-1234] [] [2016-12-06 08:39:20,255] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:20,256] WARN {org.apache.synapse.core.axis2.Axis2Sender} - Trying to send a response to an already responded client request - Rest API Context : /rest {org.apache.synapse.core.axis2.Axis2Sender}
TID: [-1] [] [2016-12-06 08:39:22,936] WARN {org.apache.synapse.transport.passthru.ConnectCallback} - Connection refused or failed for : internal-vpc-connect-api-1883227725.ap-southeast-1.elb.amazonaws.com/x.x.x.43:80 {org.apache.synapse.transport.passthru.ConnectCallback}
TID: [-1234] [] [2016-12-06 08:39:22,942] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:22,945] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : admin--XXX_APIproductionEndpoint_29 - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Tue Dec 06 08:39:52 UTC 2016 {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:39:22,945] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 101503, ERROR_MESSAGE = Error connecting to the back end {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:23,363] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
x 34 of similiar logs
TID: [-1234] [] [2016-12-06 08:39:52,303] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Executing default 'fault' sequence, ERROR_CODE = 303001, ERROR_MESSAGE = Currently , Address endpoint : [ Name : admin--XXX_APIproductionEndpoint_29 ] [ State : SUSPENDED ] {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:39:53,910] INFO {org.apache.synapse.endpoints.EndpointContext} - Endpoint : admin--XXX_APIproductionEndpoint_29 currently SUSPENDED will now be marked active since it processed its last message {org.apache.synapse.endpoints.EndpointContext}
TID: [-1234] [] [2016-12-06 08:42:19,456] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:42:56,120] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:44:18,039] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:46:17,939] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:46:36,177] WARN {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Invalid session id for thrift authenticator. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2016-12-06 08:46:36,177] ERROR {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl} - Error in invoking validate key via thrift.. {org.wso2.carbon.apimgt.keymgt.service.thrift.APIKeyValidationServiceImpl}
TID: [-1234] [] [2016-12-06 08:46:36,177] WARN {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient} - Login failed.. Authenticating again.. {org.wso2.carbon.apimgt.gateway.handlers.security.thrift.ThriftKeyValidatorClient}
TID: [-1234] [] [2016-12-06 08:46:36,231] INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} - 'admin#carbon.super [-1234]' logged in at [2016-12-06 08:46:36,231+0000] from IP address {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}
TID: [-1234] [] [2016-12-06 08:48:17,776] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:49:18,033] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:50:17,748] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:52:18,419] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:54:18,051] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-12-06 08:56:17,815] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1] [] [2016-12-06 08:57:56,120] INFO {org.wso2.andes.kernel.AndesRecoveryTask} - Running DB sync task. {org.wso2.andes.kernel.AndesRecoveryTask}
TID: [-1234] [] [2016-12-06 08:58:18,054] INFO {org.apache.synapse.mediators.builtin.LogMediator} - STATUS = Message dispatched to the main sequence. Invalid URL., RESOURCE = / {org.apache.synapse.mediators.builtin.LogMediator}

From your logs you can see that your backend was suspended and that is the cause of your 500 error. APIM will unsuspend the endpoint after a set time out period and allow you to try the endpoint again. It seems the backend was having some issues where it was most of the time working but sometimes unavailable causing the suspension of the endpoint.
Since this report is so old I'm using the most current documentation. You can see the same error code of 303001 on the Endpoint Suspension page.
I further created a quick demo of this situation using APIM 4.1.0 on my laptop and used an endpoint url of bogus.endpoint which of course will not resolve. I received the following message back using postman:
{
"code": "303001",
"type": "Status report",
"message": "Runtime Error",
"description": "Currently , Address endpoint : [ Name : DemoSuspention--vv1_APIproductionEndpoint ] [ State : SUSPENDED ]"
}
And it gave me a status result of: 500 Internal Server Error 40 ms 732 B
Normally when I run into this type of situation I reach out to the technical point of contact for the backend and report what I'm seeing. They normally can locate some kind of issue in their logs and then address it by either restarting a server or fixing a bug depending on what is found.

wso2 esb how to match response and request from log file

As I looked at the esb log and tried to find out which response corresponds to which request, I was unable to since esb uses event instead of thread. Any idea?
The ESB log same is below:
TID: [0] [ESB] [2015-05-31 00:03:23,147] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/EchoProx, MessageID: urn:uuid:4a300fc1-d5b1-4aeb-9068-164e69cdb5a0, Direction: request, Envelope: <?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body></soapenv:Body></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator} TID: [0] [ESB] [2015-05-31 00:03:27,123] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction: , MessageID: urn:uuid:ff8def98-3ce0-4cd4-a593-9c7b423ff88e, Direction: response, Envelope: <?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><jsonObject><result>true</result></jsonObject></soapenv:Body></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}

You can use the Corelation Identifier pattern for this [1].
[1] https://docs.wso2.com/display/IntegrationPatterns/Correlation+Identifier.
Refer to the same question which was answered before.
[2] WSO2 ESB Tracking Request-Response

WSO2 esb duplicate SOAP envelope tag

I created Proxy Service in WSO2 ESB, with xst transformation and when i to send request from esb i watch in logs next response from service:
"<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/[\r][\n]" {org.apache.synapse.transport.http.wire}
"[0x9]<soap:Header/>" {org.apache.synapse.transport.http.wire}
{org.apache.synapse.transport.http.wire} - >> "[0x9]<soap:Body> <m:putResponse xmlns:m="http://www.lmsoftmdm.ru">[\r][\n]" {org.apache.synapse.transport.http.wire}
{org.apache.synapse.transport.http.wire} - >> "[0x9]<m:return xmlns:xs="http://www.w3.org/2001/XMLSchema"[\r][\n]" {org.apache.synapse.transport.http.wire}
{org.apache.synapse.transport.http.wire} - >> "[0x9][0x9][0x9]xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">[\r][\n]" {org.apache.synapse.transport.http.wire}
{org.apache.synapse.transport.http.wire} - >> "[0x9][0x9]<m:isError>true</m:isError>[\r][\n]" {org.apache.synapse.transport.http.wire}
{org.apache.synapse.transport.http.wire} - >> "[0x9]</m:return>[\r][\n]" {org.apache.synapse.transport.http.wire}
{org.apache.synapse.transport.http.wire} - >> "</m:putResponse></soap:Body>[\r][\n]" {org.apache.synapse.transport.http.wire}
{org.apache.synapse.transport.http.wire} - >> "</soap:Envelope>'
But ESB change it before sending:
"<?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2015-03-05 18:45:24,018] DEBUG {org.apache.synapse.transport.http.wire} - << "[0x9]<soap:Header></soap:Header>[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2015-03-05 18:45:24,018] DEBUG {org.apache.synapse.transport.http.wire} - << "[0x9]<soap:Body> <m:putResponse xmlns:m="http://www.lmsoftmdm.ru">[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2015-03-05 18:45:24,018] DEBUG {org.apache.synapse.transport.http.wire} - << "[0x9]<m:return xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2015-03-05 18:45:24,018] DEBUG {org.apache.synapse.transport.http.wire} - << "[0x9][0x9]<m:isError>true</m:isError>[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2015-03-05 18:45:24,018] DEBUG {org.apache.synapse.transport.http.wire} - << "[0x9]</m:return>[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2015-03-05 18:45:24,018] DEBUG {org.apache.synapse.transport.http.wire} - << "</m:putResponse></soap:Body>[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2015-03-05 18:45:24,018] DEBUG {org.apache.synapse.transport.http.wire} - << "</soap:Envelope></soapenv:Body></soapenv:Envelope>
How can I fix this problem?
thanks in advance!

This because you have used soap envelope with in your xslt style sheet.
what ESB does is , It only pass the body elements to xslt mediator. so you can only manipulate the body elements using xslt mediator not the soap envelope or soap header. so after xslt transformation,transformed content will be added to body of the original soap. so if you have soap envelop in your transformation , that will be considered as body part. so use xslt mediator to only manipulate the body and use enrich mediator to manipulate the soap envelope

EndPoint will be marked SUSPENDED WSO2 ESB

I don't know why my EndPoint was marked like SUSPENDED and my application does not receive return from Service.
This was shown on log:
TID: [0] [ESB] [2014-09-01 11:00:32,269] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscarCaminhoTask, SOAPAction: urn:BuscarCaminhoTask, MessageID: urn:uuid:78e22483-c74e-4673-8286-8a18294135b3, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:00:33,259] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-15546: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2014-09-01 11:00:33,260] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : MonitoramentoEndPoint will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:00:33,260] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : MonitoramentoEndPoint - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Mon Sep 01 11:01:03 BRT 2014 {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:01:01,196] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscaServicosMonitorar, SOAPAction: urn:BuscaServicosMonitorar, MessageID: urn:uuid:e5ce0a3c-e5d1-4364-a7aa-897363392c4d, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:01:01,203] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscaHardwaresMonitorar, SOAPAction: urn:BuscaHardwaresMonitorar, MessageID: urn:uuid:e74ab088-1eb0-47cc-8447-7b9e016edcec, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:01:01,208] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/MonitoramentoServico.MonitoramentoServicoHttpSoap11Endpoint, WSAction: urn:BuscarCaminhoTask, SOAPAction: urn:BuscarCaminhoTask, MessageID: urn:uuid:18dad64b-292e-4c79-908e-6d9b638e1d8a, Direction: request {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2014-09-01 11:01:01,291] WARN {org.apache.synapse.transport.passthru.TargetHandler} - http-outgoing-15544: Connection time out while in state: REQUEST_DONE {org.apache.synapse.transport.passthru.TargetHandler}
TID: [0] [ESB] [2014-09-01 11:01:01,292] WARN {org.apache.synapse.endpoints.EndpointContext} - Endpoint : MonitoramentoEndPoint will be marked SUSPENDED as it failed {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:01:01,292] WARN {org.apache.synapse.endpoints.EndpointContext} - Suspending endpoint : MonitoramentoEndPoint - last suspend duration was : 30000ms and current suspend duration is : 30000ms - Next retry after : Mon Sep 01 11:01:31 BRT 2014 {org.apache.synapse.endpoints.EndpointContext}
TID: [0] [ESB] [2014-09-01 11:01:02,210] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-388 {org.apache.synapse.transport.passthru.SourceHandler}
The EndPoint come back automatically after some seconds, but how can I know whats wrong?
This is the error after receive my request:
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "POST /services/MonitoramentoDS HTTP/1.1[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Expect: 100-continue[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Content-Type: text/xml; charset=utf-8[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "SOAPAction: "urn:UpdateServidor"[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Transfer-Encoding: chunked[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Host: localhost:9763[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "Connection: Keep-Alive[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "User-Agent: Synapse-PT-HttpComponents-NIO[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - << "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,746] DEBUG {org.apache.synapse.transport.http.wire} - >> "HTTP/1.1 100 Continue[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,747] DEBUG {org.apache.synapse.transport.http.wire} - >> "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "19a[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "<?xml version='1.0' encoding='utf-8'?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap:Body><UpdateServidor xmlns="http://ws.wso2.org/dataservice"><situacao>0</situacao><id>2</id><sithard>1</sithard><sitserv>1</sitserv><sittask>0</sittask></UpdateServidor></soap:Body></soap:Envelope>[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "0[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - << "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "HTTP/1.1 202 Accepted[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Content-Type: text/xml;charset=utf-8[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Transfer-Encoding: chunked[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Date: Mon, 01 Sep 2014 18:27:42 GMT[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "Server: WSO2 Carbon Server[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "0[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,750] DEBUG {org.apache.synapse.transport.http.wire} - >> "[\r][\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2014-09-01 15:27:42,753] ERROR {org.apache.axis2.transport.base.threads.NativeWorkerPool} - Uncaught exception {org.apache.axis2.transport.base.threads.NativeWorkerPool}
java.lang.UnsupportedOperationException: An access occurred that is not valid.
at org.apache.axis2.description.InOnlyAxisOperation.getMessage(InOnlyAxisOperation.java:117)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:283)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:222)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)

Endpoint is getting suspended, when ESB could not send out message through that particular endpoint. You can find endpoint suspension parameters in your endpoint configuration.
Here if you see http-outgoing-15544: Connection time out while in state: REQUEST_DONE occurs because, connection between ESB and the backend server times out. Default socket
timeout is 60 seconds. You can increase it to 120 seconds and check. To do that edit the
passthru-http.properties file as;
http.socket.timeout=120000
And enable ESB wirelogs so you can find more info.
# log4j properties file, add following line and restart the server.
log4j.logger.org.apache.synapse.transport.http.wire=DEBUG

prefix cannot be "null" when creating a QName

I'm using the WSO2 ESB version 4.8.1. I tested sample no 658( Smooks Mediator transformation xml -> xml) I only changed in configuration paths to folders (I don't have folder '/home/lakmali/...':-)).This sample not work. Error from log:
java.lang.IllegalArgumentException: prefix cannot be "null" when creating a QName.
Full log with error:
TID: [0] [ESB] [2014-02-12 10:33:13,589] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Starting WSO2 Carbon... {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,605] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Operating System : Windows 7 6.1, amd64 {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,605] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Home : C:\Java\jdk1.7.0_40\jre {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,605] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Version : 1.7.0_40 {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,605] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java VM : Java HotSpot(TM) 64-Bit Server VM 24.0-b56,Oracle Corporation {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,605] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Carbon Home : C:\Java\WSO2ES~1.1\bin\.. {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,605] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - Java Temp Dir : C:\Java\WSO2ES~1.1\bin\..\tmp {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,605] INFO {org.wso2.carbon.core.internal.CarbonCoreActivator} - User : ******* {org.wso2.carbon.core.internal.CarbonCoreActivator}
TID: [0] [ESB] [2014-02-12 10:33:13,745] WARN {org.wso2.carbon.core.bootup.validator.util.ValidationResultPrinter} - The default keystore (wso2carbon.jks) is currently being used. To maximize security when deploying to a production environment, configure a new keystore with a unique password in the production server profile. {org.wso2.carbon.core.bootup.validator.util.ValidationResultPrinter}
TID: [0] [ESB] [2014-02-12 10:33:13,761] INFO {org.wso2.carbon.databridge.agent.thrift.AgentHolder} - Agent created ! {org.wso2.carbon.databridge.agent.thrift.AgentHolder}
TID: [0] [ESB] [2014-02-12 10:33:13,776] INFO {org.wso2.carbon.databridge.agent.thrift.internal.AgentDS} - Successfully deployed Agent Client {org.wso2.carbon.databridge.agent.thrift.internal.AgentDS}
TID: [0] [ESB] [2014-02-12 10:33:19,051] INFO {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService} - Configured Registry in 39ms {org.wso2.carbon.registry.core.jdbc.EmbeddedRegistryService}
TID: [0] [ESB] [2014-02-12 10:33:19,113] INFO {org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent} - Registry Mode : READ-WRITE {org.wso2.carbon.registry.core.internal.RegistryCoreServiceComponent}
TID: [0] [ESB] [2014-02-12 10:33:19,769] INFO {org.wso2.carbon.user.core.internal.UserStoreMgtDSComponent} - Carbon UserStoreMgtDSComponent activated successfully. {org.wso2.carbon.user.core.internal.UserStoreMgtDSComponent}
TID: [0] [ESB] [2014-02-12 10:33:21,563] INFO {org.apache.catalina.startup.TaglibUriRule} - TLD skipped. URI: http://tiles.apache.org/tags-tiles is already defined {org.apache.catalina.startup.TaglibUriRule}
TID: [0] [ESB] [2014-02-12 10:33:22,530] INFO {org.apache.axis2.deployment.ClusterBuilder} - Clustering has been disabled {org.apache.axis2.deployment.ClusterBuilder}
TID: [0] [ESB] [2014-02-12 10:33:22,826] INFO {org.wso2.carbon.stratos.landing.page.deployer.LandingPageWebappDeployer} - Deployed product landing page webapp: StandardEngine[Catalina].StandardHost[localhost].StandardContext[/home] {org.wso2.carbon.stratos.landing.page.deployer.LandingPageWebappDeployer}
TID: [0] [ESB] [2014-02-12 10:33:22,826] INFO {org.wso2.carbon.identity.user.store.configuration.deployer.UserStoreConfigurationDeployer} - User Store Configuration Deployer initiated. {org.wso2.carbon.identity.user.store.configuration.deployer.UserStoreConfigurationDeployer}
TID: [0] [ESB] [2014-02-12 10:33:22,873] INFO {org.apache.synapse.transport.passthru.PassThroughHttpSSLSender} - Initializing Pass-through HTTP/S Sender... {org.apache.synapse.transport.passthru.PassThroughHttpSSLSender}
TID: [0] [ESB] [2014-02-12 10:33:22,889] INFO {org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder} - HTTPS Loading Identity Keystore from : repository/resources/security/wso2carbon.jks {org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder}
TID: [0] [ESB] [2014-02-12 10:33:22,889] INFO {org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder} - HTTPS Loading Trust Keystore from : repository/resources/security/client-truststore.jks {org.apache.synapse.transport.nhttp.config.ClientConnFactoryBuilder}
TID: [0] [ESB] [2014-02-12 10:33:22,920] INFO {org.apache.synapse.transport.passthru.PassThroughHttpSSLSender} - Pass-through HTTPS Sender started... {org.apache.synapse.transport.passthru.PassThroughHttpSSLSender}
TID: [0] [ESB] [2014-02-12 10:33:22,920] INFO {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Initializing Pass-through HTTP/S Sender... {org.apache.synapse.transport.passthru.PassThroughHttpSender}
TID: [0] [ESB] [2014-02-12 10:33:22,920] INFO {org.apache.synapse.transport.passthru.PassThroughHttpSender} - Pass-through HTTP Sender started... {org.apache.synapse.transport.passthru.PassThroughHttpSender}
TID: [0] [ESB] [2014-02-12 10:33:22,935] INFO {org.apache.synapse.transport.vfs.VFSTransportSender} - VFS Sender started {org.apache.synapse.transport.vfs.VFSTransportSender}
TID: [0] [ESB] [2014-02-12 10:33:23,045] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: echo {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [ESB] [2014-02-12 10:33:23,310] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: Echo.aar - file:/C:/Java/WSO2ES~1.1/bin/../repository/deployment/server/axis2services/Echo.aar {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [ESB] [2014-02-12 10:33:23,575] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: echo {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [ESB] [2014-02-12 10:33:23,825] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: Version {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [ESB] [2014-02-12 10:33:23,934] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: Version.aar - file:/C:/Java/WSO2ES~1.1/bin/../repository/deployment/server/axis2services/Version.aar {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [ESB] [2014-02-12 10:33:24,043] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: Version {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [ESB] [2014-02-12 10:33:24,199] INFO {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener} - Initializing Pass-through HTTP/S Listener... {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener}
TID: [0] [ESB] [2014-02-12 10:33:24,667] INFO {org.apache.synapse.transport.passthru.PassThroughHttpListener} - Initializing Pass-through HTTP/S Listener... {org.apache.synapse.transport.passthru.PassThroughHttpListener}
TID: [0] [ESB] [2014-02-12 10:33:24,683] WARN {org.apache.synapse.transport.vfs.PollTableEntry} - transport.vfs.FileURI parameter is missing in the proxy service configuration {org.apache.synapse.transport.vfs.PollTableEntry}
TID: [0] [ESB] [2014-02-12 10:33:24,870] INFO {org.apache.axis2.deployment.ModuleDeployer} - Deploying module: addressing-1.6.1-wso2v10 - file:/C:/Java/WSO2ES~1.1/bin/../repository/deployment/client/modules/addressing-1.6.1-wso2v10.mar {org.apache.axis2.deployment.ModuleDeployer}
TID: [0] [ESB] [2014-02-12 10:33:24,885] INFO {org.apache.axis2.deployment.ModuleDeployer} - Deploying module: rampart-1.6.1-wso2v8 - file:/C:/Java/WSO2ES~1.1/bin/../repository/deployment/client/modules/rampart-1.6.1-wso2v8.mar {org.apache.axis2.deployment.ModuleDeployer}
TID: [0] [ESB] [2014-02-12 10:33:24,901] INFO {org.apache.axis2.transport.tcp.TCPTransportSender} - TCP Sender started {org.apache.axis2.transport.tcp.TCPTransportSender}
TID: [0] [ESB] [2014-02-12 10:33:25,712] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: org.wso2.carbon.message.processor - {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [ESB] [2014-02-12 10:33:25,712] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: org.wso2.carbon.message.store - {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [ESB] [2014-02-12 10:33:26,321] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: wso2carbon-sts {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [ESB] [2014-02-12 10:33:26,430] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: org.wso2.carbon.sts - {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [ESB] [2014-02-12 10:33:26,617] INFO {org.apache.axis2.deployment.DeploymentEngine} - Deploying Web service: org.wso2.carbon.tryit - {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [ESB] [2014-02-12 10:33:26,820] INFO {org.wso2.carbon.core.init.CarbonServerManager} - Repository : C:\Java\WSO2ES~1.1\bin\../repository/deployment/server/ {org.wso2.carbon.core.init.CarbonServerManager}
TID: [0] [ESB] [2014-02-12 10:33:26,960] INFO {org.wso2.carbon.core.internal.permission.update.PermissionUpdater} - Permission cache updated for tenant -1234 {org.wso2.carbon.core.internal.permission.update.PermissionUpdater}
TID: [0] [ESB] [2014-02-12 10:33:26,991] INFO {org.wso2.carbon.mediation.initializer.ServiceBusInitializer} - Starting ESB... {org.wso2.carbon.mediation.initializer.ServiceBusInitializer}
TID: [0] [ESB] [2014-02-12 10:33:27,007] INFO {org.wso2.carbon.mediation.initializer.ServiceBusInitializer} - Initializing Apache Synapse... {org.wso2.carbon.mediation.initializer.ServiceBusInitializer}
TID: [0] [ESB] [2014-02-12 10:33:27,007] INFO {org.apache.synapse.SynapseControllerFactory} - Using Synapse home : C:\Java\WSO2ES~1.1\. {org.apache.synapse.SynapseControllerFactory}
TID: [0] [ESB] [2014-02-12 10:33:27,007] INFO {org.apache.synapse.SynapseControllerFactory} - Using synapse.xml location : C:\Java\WSO2ES~1.1\.\.\repository\deployment\server\synapse-configs\default {org.apache.synapse.SynapseControllerFactory}
TID: [0] [ESB] [2014-02-12 10:33:27,007] INFO {org.apache.synapse.SynapseControllerFactory} - Using server name : localhost {org.apache.synapse.SynapseControllerFactory}
TID: [0] [ESB] [2014-02-12 10:33:27,023] INFO {org.apache.synapse.SynapseControllerFactory} - The timeout handler will run every : 15s {org.apache.synapse.SynapseControllerFactory}
TID: [0] [ESB] [2014-02-12 10:33:27,023] INFO {org.apache.synapse.Axis2SynapseController} - Initializing Synapse at : Wed Feb 12 10:33:27 CET 2014 {org.apache.synapse.Axis2SynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,023] INFO {org.wso2.carbon.mediation.initializer.CarbonSynapseController} - Loading the mediation configuration from the file system {org.wso2.carbon.mediation.initializer.CarbonSynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,023] INFO {org.apache.synapse.config.xml.MultiXMLConfigurationBuilder} - Building synapse configuration from the synapse artifact repository at : .\.\repository/deployment/server/synapse-configs\default {org.apache.synapse.config.xml.MultiXMLConfigurationBuilder}
TID: [0] [ESB] [2014-02-12 10:33:27,023] INFO {org.apache.synapse.config.xml.XMLConfigurationBuilder} - Generating the Synapse configuration model by parsing the XML configuration {org.apache.synapse.config.xml.XMLConfigurationBuilder}
TID: [0] [ESB] [2014-02-12 10:33:27,101] INFO {org.apache.synapse.config.SynapseConfigurationBuilder} - Loaded Synapse configuration from the artifact repository at : .\.\repository/deployment/server/synapse-configs\default {org.apache.synapse.config.SynapseConfigurationBuilder}
TID: [0] [ESB] [2014-02-12 10:33:27,116] INFO {org.apache.synapse.Axis2SynapseController} - Loading mediator extensions... {org.apache.synapse.Axis2SynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,116] INFO {org.apache.synapse.Axis2SynapseController} - Deploying the Synapse service... {org.apache.synapse.Axis2SynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,116] INFO {org.apache.synapse.Axis2SynapseController} - Deploying Proxy services... {org.apache.synapse.Axis2SynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,116] INFO {org.apache.synapse.core.axis2.ProxyService} - Building Axis service for Proxy service : ssXMLProxy {org.apache.synapse.core.axis2.ProxyService}
TID: [0] [ESB] [2014-02-12 10:33:27,116] INFO {org.apache.synapse.core.axis2.ProxyService} - Adding service ssXMLProxy to the Axis2 configuration {org.apache.synapse.core.axis2.ProxyService}
TID: [0] [ESB] [2014-02-12 10:33:27,132] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: ssXMLProxy {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [ESB] [2014-02-12 10:33:27,225] INFO {org.apache.synapse.core.axis2.ProxyService} - Successfully created the Axis2 service for Proxy service : ssXMLProxy {org.apache.synapse.core.axis2.ProxyService}
TID: [0] [ESB] [2014-02-12 10:33:27,225] INFO {org.apache.synapse.Axis2SynapseController} - Deployed Proxy service : ssXMLProxy {org.apache.synapse.Axis2SynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,225] INFO {org.apache.synapse.core.axis2.ProxyService} - Building Axis service for Proxy service : FirmaXMLProxy {org.apache.synapse.core.axis2.ProxyService}
TID: [0] [ESB] [2014-02-12 10:33:27,241] INFO {org.apache.synapse.core.axis2.ProxyService} - Adding service FirmaXMLProxy to the Axis2 configuration {org.apache.synapse.core.axis2.ProxyService}
TID: [0] [ESB] [2014-02-12 10:33:27,241] INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} - Deploying Axis2 service: FirmaXMLProxy {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [ESB] [2014-02-12 10:33:27,303] INFO {org.apache.synapse.core.axis2.ProxyService} - Successfully created the Axis2 service for Proxy service : FirmaXMLProxy {org.apache.synapse.core.axis2.ProxyService}
TID: [0] [ESB] [2014-02-12 10:33:27,303] INFO {org.apache.synapse.Axis2SynapseController} - Deployed Proxy service : FirmaXMLProxy {org.apache.synapse.Axis2SynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,303] INFO {org.apache.synapse.Axis2SynapseController} - Deploying EventSources... {org.apache.synapse.Axis2SynapseController}
TID: [0] [ESB] [2014-02-12 10:33:27,319] INFO {org.apache.synapse.ServerManager} - Server ready for processing... {org.apache.synapse.ServerManager}
TID: [0] [ESB] [2014-02-12 10:33:27,350] INFO {org.wso2.carbon.bam.mediationstats.data.publisher.internal.MediationStatisticsComponent} - Statistic Reporter is Disabled {org.wso2.carbon.bam.mediationstats.data.publisher.internal.MediationStatisticsComponent}
TID: [0] [ESB] [2014-02-12 10:33:27,350] INFO {org.wso2.carbon.bam.mediationstats.data.publisher.internal.MediationStatisticsComponent} - Can't register an observer for mediationStatisticsStore. If you have disabled StatisticsReporter, please enable it in the Carbon.xml {org.wso2.carbon.bam.mediationstats.data.publisher.internal.MediationStatisticsComponent}
TID: [0] [ESB] [2014-02-12 10:33:27,397] INFO {org.wso2.carbon.rule.kernel.internal.ds.RuleEngineConfigDS} - Successfully registered the Rule Config service {org.wso2.carbon.rule.kernel.internal.ds.RuleEngineConfigDS}
TID: [0] [ESB] [2014-02-12 10:33:27,865] INFO {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener} - Starting Pass-through HTTPS Listener... {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener}
TID: [0] [ESB] [2014-02-12 10:33:27,865] INFO {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener} - Pass-through HTTPS Listener started on 0:0:0:0:0:0:0:0:8243 {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener}
TID: [0] [ESB] [2014-02-12 10:33:27,865] INFO {org.apache.synapse.transport.passthru.PassThroughHttpListener} - Starting Pass-through HTTP Listener... {org.apache.synapse.transport.passthru.PassThroughHttpListener}
TID: [0] [ESB] [2014-02-12 10:33:27,881] INFO {org.apache.synapse.transport.passthru.PassThroughHttpListener} - Pass-through HTTP Listener started on 0:0:0:0:0:0:0:0:8280 {org.apache.synapse.transport.passthru.PassThroughHttpListener}
TID: [0] [ESB] [2014-02-12 10:33:27,881] INFO {org.apache.synapse.transport.vfs.VFSTransportListener} - VFS listener started {org.apache.synapse.transport.vfs.VFSTransportListener}
TID: [0] [ESB] [2014-02-12 10:33:27,881] INFO {org.apache.tomcat.util.net.NioSelectorPool} - Using a shared selector for servlet write/read {org.apache.tomcat.util.net.NioSelectorPool}
TID: [0] [ESB] [2014-02-12 10:33:28,130] INFO {org.apache.tomcat.util.net.NioSelectorPool} - Using a shared selector for servlet write/read {org.apache.tomcat.util.net.NioSelectorPool}
TID: [0] [ESB] [2014-02-12 10:33:28,146] INFO {org.wso2.carbon.registry.eventing.internal.RegistryEventingServiceComponent} - Successfully Initialized Eventing on Registry {org.wso2.carbon.registry.eventing.internal.RegistryEventingServiceComponent}
TID: [0] [ESB] [2014-02-12 10:33:28,520] INFO {org.wso2.carbon.core.init.JMXServerManager} - JMX Service URL : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi {org.wso2.carbon.core.init.JMXServerManager}
TID: [0] [ESB] [2014-02-12 10:33:28,520] INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} - Server : WSO2 Enterprise Service Bus-4.8.1 {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent}
TID: [0] [ESB] [2014-02-12 10:33:28,520] INFO {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent} - WSO2 Carbon started in 22 sec {org.wso2.carbon.core.internal.StartupFinalizerServiceComponent}
TID: [0] [ESB] [2014-02-12 10:33:28,910] INFO {org.wso2.carbon.ui.internal.CarbonUIServiceComponent} - Mgt Console URL : https://10.104.0.44:9443/carbon/ {org.wso2.carbon.ui.internal.CarbonUIServiceComponent}
TID: [0] [ESB] [2014-02-12 10:35:38,588] ERROR {org.wso2.carbon.mediator.transform.SmooksMediator} - Failed to filter source. {org.wso2.carbon.mediator.transform.SmooksMediator}
org.milyn.SmooksException: Failed to filter source.
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:86)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:61)
at org.milyn.Smooks._filter(Smooks.java:516)
at org.milyn.Smooks.filterSource(Smooks.java:475)
at org.wso2.carbon.mediator.transform.SmooksMediator.mediate(SmooksMediator.java:123)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.base.AbstractTransportListener.handleIncomingMessage(AbstractTransportListener.java:328)
at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:597)
at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:328)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:158)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:107)
at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.lang.IllegalArgumentException: prefix cannot be "null" when creating a QName
at javax.xml.namespace.QName.<init>(QName.java:251)
at com.sun.xml.internal.stream.events.XMLEventAllocatorImpl.getQName(XMLEventAllocatorImpl.java:254)
at com.sun.xml.internal.stream.events.XMLEventAllocatorImpl.getXMLEvent(XMLEventAllocatorImpl.java:76)
at com.sun.xml.internal.stream.events.XMLEventAllocatorImpl.allocate(XMLEventAllocatorImpl.java:53)
at com.sun.xml.internal.stream.XMLEventReaderImpl.nextEvent(XMLEventReaderImpl.java:84)
at com.sun.xml.internal.stream.XMLEventReaderImpl.next(XMLEventReaderImpl.java:248)
at org.wso2.carbon.mediator.transform.stream.IOElementPipe.populateEvents(IOElementPipe.java:90)
at org.wso2.carbon.mediator.transform.stream.IOElementPipe.getData(IOElementPipe.java:68)
at org.wso2.carbon.mediator.transform.stream.ElementInputStream.read(ElementInputStream.java:61)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.skipString(Unknown Source)
at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.milyn.delivery.sax.SAXParser.parse(SAXParser.java:70)
at org.milyn.delivery.sax.SmooksSAXFilter.doFilter(SmooksSAXFilter.java:75)
... 19 more
TID: [0] [ESB] [2014-02-12 10:35:38,604] ERROR {org.apache.synapse.transport.vfs.VFSTransportListener} - Error processing File URI : file:///c:/java/test/toconvert/s481/input-message-658.xml {org.apache.synapse.transport.vfs.VFSTransportListener}
org.wso2.carbon.mediator.service.MediatorException: Failed to filter source. Caused by Failed to filter source.
at org.wso2.carbon.mediator.transform.SmooksMediator.handleException(SmooksMediator.java:242)
at org.wso2.carbon.mediator.transform.SmooksMediator.mediate(SmooksMediator.java:137)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:77)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:47)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:131)
at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:166)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.base.AbstractTransportListener.handleIncomingMessage(AbstractTransportListener.java:328)
at org.apache.synapse.transport.vfs.VFSTransportListener.processFile(VFSTransportListener.java:597)
at org.apache.synapse.transport.vfs.VFSTransportListener.scanFileOrDirectory(VFSTransportListener.java:328)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:158)
at org.apache.synapse.transport.vfs.VFSTransportListener.poll(VFSTransportListener.java:107)
at org.apache.axis2.transport.base.AbstractPollingTransportListener$1$1.run(AbstractPollingTransportListener.java:67)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Thanks in advance for response, Grzegorz

the issue has been fixed with the wso2 enterprise service bus 4.9.0 for sample 658. Once the server stats in sample mode kindly update the folder paths from /home/lakmali in[1] to a folder location relevant for you and in the proxy service named synapse_sample_658.xml[2] available in path /wso2esb-4.9.0/repository/samples
[1]repository/samples/resources/smooks/smooks-config-658.xml within the location <file:destinationDirectoryPattern>
[2]
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
<proxy name="SmooksSample" transports="vfs" startOnLoad="true" trace="disable">
<description />
<target>
<inSequence>
<smooks config-key="smooks-key">
<input type="xml" />
<output type="xml" />
</smooks>
</inSequence>
</target>
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.PollInterval">5</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.FileURI">file:///home/shavantha/dev/test/smooks/in</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///home/shavantha/dev/test/smooks/original</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///home/shavantha/dev/test/smooks/original</parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
<parameter name="transport.vfs.ContentType">application/xml</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
</proxy>
<localEntry key="smooks-key" src="file:repository/samples/resources/smooks/smooks-config-658.xml" />
<sequence name="fault">
<log level="full">
<property name="MESSAGE" value="Executing default "fault" sequence" />
<property name="ERROR_CODE" expression="get-property('ERROR_CODE')" />
<property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')" />
</log>
<drop />
</sequence>
<sequence name="main">
<log />
<drop />
</sequence>
</definitions>
The public jira for the issue you have raised is available on https://wso2.org/jira/browse/ESBJAVA-3608
regards,shavantha