WSO2 ESB FileConnector MergeFiles does not produce expected results - wso2

I am working with the WSO2 ESB 6.5.0 and am using the fileconnector-2.0.21 version for file operations. The proxy service that I'm building is intended to read the files from a certain source path and merge the contents into a file at the destination path.
The content of the proxy service artifact is below
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="FileMergeProxy" startOnLoad="true" transports="http https" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log description="Combine files" level="custom" separator="| ">
<property name="statusMessage" value="Processing complete"/>
<property name="fileCreation" value="Initiating file merge"/>
<property expression="$ctx:mergeDirPath" name="sourceDir"/>
<property expression="$ctx:finalDirPath" name="destDir"/>
<property expression="$ctx:outputFilePattern" name="pattern"/>
<property expression="fn:concat('file:///', $ctx:mergeDirPath)" name="sourceDir1" scope="default" type="STRING"/>
property expression="fn:concat('file:///', $ctx:finalDirPath)" name="destDir1" scope="default" type="STRING"/>
</log>
<fileconnector.mergeFiles>
<source>{$ctx:mergeDirPath}</source>
<destination>{$ctx:finalDirPath}</destination>
<filePattern>{$ctx:outputFilePattern}</filePattern>
</fileconnector.mergeFiles>
<log description="Processing complete" level="custom" separator="| ">
<property name="doneMessage" value="File merge processing complete"/>
</log>
</inSequence>
<outSequence/>
<faultSequence>
<log level="custom">
<property name="text" value="An unexpected error occured"/>
<property expression="get-property('ERROR_MESSAGE')" name="message"/>
<property expression="get-property('ERROR_DETAIL')" name="errordetail"/>
</log>
<send description="Send Error Information"/>
</faultSequence>
</target>
</proxy>
I'm invoking the service with the following parameters as a JSON file
{
"mergeDirPath": "C://temp//merge//",
"finalDirPath": "C://temp//final//finalcontent.txt",
"outputFilePattern": "\\*txt"
}
When I call the service after deploying the artifacts to the ESB engine, it creates an empty file in the destination path without merging the contents of the files in the source directory. This is the extract from the WSO2 logs.
[2020-03-23 12:48:36,683] [] INFO - LogMediator To: /services/FileMergeProxy,MessageID: urn:uuid:b569ec6a-d4fe-4763-a0ca-fb2eb868a31e correlation_id : a1c542bb-7fac-4e0b-9582-8cb1a605f618,Direction: request,Payload: { "mergeDirPath": "C://temp//merge//", "finalDirPath": "C://temp//final//finalcontent.txt", "outputFilePattern": "\\*txt" }
[2020-03-23 12:49:00,422] [] INFO - LogMediator statusMessage = Combine files| fileCreation = Initiating file merge| sourceDir = C://temp//merge//| destDir = C://temp//final//finalcontent.txt| pattern = \*txt| sourceDir1 = file:///C://temp//merge//| destDir1 = file:///C://temp//final//finalcontent.txt
[2020-03-23 12:49:02,943] [] INFO - LogMediator doneMessage = File merge processing complete
Unless I'm missing something here, shouldn't the mergeFiles be doing exactly that - merge the contents of files in a specified directory ? Any helpful suggestions or pointers are welcome. Thanks in advance.

According to the proxy you have shared, you have defined properties inside the log mediator. Define the properties outside the log mediator.

Related

Paralel processing in WSO2 EI 6.2

I'm developing a healthcheck API that needs to call several endpoints. The idea is that, if any of those EPs fail, I have to capture the fault and incorporate that into the response. I thought about creating a main sequence that calls other leaf sequences, and each leaf sequence has it own fault sequence.
This diagram should make things clearer:
seqA (faultSsqA)
/
mainSeq - seqB (faultSeqB)
\ seqC (faultSeqC)
If all the sequences process successfully, everything goes well and I'm able to process the response, however, if there's a fault within one of the leaf sequences, the corresponding faultsequence is executed, but I don't know how to return the flow of execution back to the main sequence. What I want is that, if a exception occurs in sequence B, fault sequence B is executed and then the execution flow goes back to the main sequence and then to sequence C.
Even better it would be if I could execute all the leaf sequences in pararel, and just gather all the results in the end. I know that iterate mediator sort of do that, but from what I understand the semantics are not the same of what I want.
Anyone has thoughts on this?
EDIT: so it seems the clone mediator is what I'm looking for, however I'm still not able to get it working the way I need to. Since I'll be connecting to different platforms, the response formats are distinct. Also, I need to treat situations such as timeouts, 500 response codes, etc. Therefore, I can't simply call an endpoint from each target - for each backend platform I've created a sequence that does all this logic, and populate some properties so that I can populate the final response ("platform A = up", etc).
My expectation would be that all the target sequences would be processed in paralel
This is my main sequence. I use a clone mediator and target it to 2 different sequences, which do the actual invocation of the EP and treat the response (I'm using only 2 for now, but in the future will be many more). The sequences also update some variables to reflect the status of the backend platforms ("platform A = up", etc).
<sequence name="inSequence_healthCheck" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<clone id="healthCHeck">
<target>
<sequence>
<sequence key="healthCheck_wacs"/>
</sequence>
</target>
<target>
<sequence>
<sequence key="healthCheck_thesys"/>
</sequence>
</target>
</clone>
<loopback/>
</sequence>
This is one of the target sequences (it' very simple for now, but there should be more logic in it). Notice that I populate the property wacsStatus depending on the response.
<sequence name="healthCheck_wacs" onError="faultSequence_healthCheck_wacs" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<call>
<endpoint key="gov:ClientEquipments/endpoints/WACS/wacs_healthCheck.endpoint.xml"/>
</call>
<switch source="$axis2:HTTP_SC">
<case regex="2\d\d|4\d\d">
<property name="wacsStatus" scope="default" type="BOOLEAN" value="true"/>
</case>
<default>
<property name="wacsStatus" scope="default" type="BOOLEAN" value="false"/>
</default>
</switch>
</sequence>
And this is my outSequence, that does the aggregate:
<sequence name="outSequence_healthCheck" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<log>
<property name="step" value="START outSequence_healthCheck"/>
<property expression="$ctx:wacsStatus" name="wacsStatus"/>
<property expression="$ctx:thesysStatus" name="thesysStatus"/>
<property expression="$ctx:phStatus" name="phStatus"/>
<property expression="$ctx:naStatus" name="naStatus"/>
</log>
<property name="info" scope="default">
<ns:Information xmlns:ns="http://wso2.com"/>
</property>
<aggregate id="healthCHeck">
<completeCondition>
<messageCount max="-1" min="2"/>
</completeCondition>
<onComplete enclosingElementProperty="info" expression="s11:Body/child::* | s12:Body/child::*"
xmlns:m0="http://services.samples" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:s12="http://www.w3.org/2003/05/soap-envelope">
<log level="full"/>
</onComplete>
</aggregate>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<payloadFactory media-type="json">
<format>
{
"WACS" : "$1",
"thesys" : "$2"
}
</format>
<args>
<arg evaluator="xml" expression="$ctx:wacsStatus"/>
<arg evaluator="xml" expression="$ctx:thesysStatus"/>
</args>
</payloadFactory>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<property name="HTTP_SC" scope="axis2" type="STRING" value="200"/>
<respond/>
</sequence>
My expectation would be that the aggregate mediator would wait for all the target sequences to execute, by then all the properties would be properly populated and I could use the payloadFactory to properly format the response. However, from the logs I can see that, even though the aggregate does aggregate all the responses from the target sequences, the properties are not properly populated and therefore the response is not correct.
Thanks.
Pedro
You should try an iterate mediator/clone & aggregate mediator (scatter-gather pattern) construction. The iterate allows you to execute multiple sequences in paralel while the aggregate will collect the responses. You can have the separate sequences deal with fault handling and return either the expected response or some fault. You can then check the aggregated result for any faults.
Here are a few samples:
Yenlo Blog
WSO2 Architecture Team

WSO2ESB concat input file name with SYSTEM_DATE

I am going to concat input file name with SYSTEM_DATE below is the code
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="importclassroomcourse"
transports="https,http,local"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<log level="custom">
<property name="transport.vfs.ReplyFileName"
expression="fn:concat(get-property('SYSTEM_DATE', 'yyMMddHHmmss'), '-', get-property('filename'), '.xml')"/>
</log>
</inSequence>
<outSequence>
<send/>
</outSequence>
<endpoint>
<address uri="https://qa-abc.com/api/classroom/xyz"/>
</endpoint>
</target>
<description/>
</proxy>
But it won't concat the file name below is the log
LogMediator transport.vfs.ReplyFileName = 160316122250-.xml
Kindly guide me how can i get the file name because file name is missing.I have sent the file test.xlsx
I can't find any initialisation of property 'filename' in your mediation : this property does not exist and therefore, has no value when used in 'concat'.
What do you mean by "I have sent the file text.xlsx" ? Your proxy use http and local transport, it's not a VFS proxy scanning a directory to load files ?
You can define this property :
<property name="transport.vfs.ReplyFileName" expression="fn:concat(get-property('SYSTEM_DATE', 'yyMMddHHmmss'), '-test.xml')"/>
but after that, you must use send with a vfs endpoint inside the inSequence to write the file (in your case, there is a https endpoint)
If your proxy were a VFS proxy, you could use FILE_NAME on scope transport to get incoming filename.

Log into proxy log file each proxy duration

We need to log into the logs file of WSO2ESB , the duration of each call of proxy instances.
For example :
....
10:10:14,736 [MessageID : xxxxxxxxx] [duration : 259 ms]
....
It means that the proxy call associated to MessageID xxxxxxxx had a duration of 259 ms.
I don't want to active statistics mediator or BAM mediator.
Any idea to dot it easily ?
Tks
Nicolas
You can use SYSTEM_TIME property which is a Synapse Message Context Property. Using this you can get the current time in milliseconds in the desired places of the message flow and set the values as a properties. Then you can use the Script Mediator to write a small script (Javascript or Ruby) which reads the time values (stored in propeties) and do the neccessary calculations. You can display the result either in the script mediator or you can set it to a new propety so that it can be loged inside the proxy.
Synapse Message Context Propeties
http://docs.wso2.org/display/ESB470/Synapse+Message+Context+Properties
Using Script Mediator
http://docs.wso2.org/pages/viewpage.action?pageId=26838871
Log Mediator
http://docs.wso2.org/display/ESB470/Log+Mediator
Following sample might help you.
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="SampleTimeProxy"
transports="https,http"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<property name="TIME_1"
expression="get-property('SYSTEM_TIME')"
scope="default"
type="LONG"/>
</inSequence>
<outSequence>
<send/>
<property name="TIME_2"
expression="get-property('SYSTEM_TIME')"
scope="default"
type="LONG"/>
<script language="js">var time1 = mc.getProperty("TIME_1");
var time2 = mc.getProperty("TIME_2");
var timeTaken = time2 - time1;
print("-------------- " + timeTaken + " ms -----------------");
mc.setProperty("RESPONSE_TIME", timeTaken);
</script>
<log>
<property name="time" expression="get-property('RESPONSE_TIME')"/>
</log>
</outSequence>
<endpoint>
<address uri="http://localhost:8080/axis2/services/SimpleStockQuoteService"/>
</endpoint>
</target>
<publishWSDL uri="http://localhost:8080/axis2/services/SimpleStockQuoteService?wsdl"/>
<description/>
</proxy>

Can we store a string in a text file in wso2esb?

I have a text file on my local system I wish to append the data in particular file as synchronously.
I have tried many ways, but it's not working.
ESB has this future in Oracle SOA. We can add in FILE ADAPTER. In ESB it's neither giving errors nor expected result.
My configuration is like this:
<proxy xmlns="http://ws.apache.org/ns/synapse" name="FileWrite" transports="http,vfs" statistics="disable" trace="disable" startOnLoad="true">
<target>
<inSequence>
<log>
<property name="OUT_ONLY" value="true"/>
</log>
</inSequence>
<outSequence>
<log>
<property name="OUT_ONLY" value="true"/>
</log>
<payloadFactory>
<format>
<error>error404</error>
</format>
</payloadFactory>
<send>
<endpoint>
<address uri="vfs:file:///home/youtility2/Desktop/Errorlog"/>
</endpoint>
</send>
</outSequence>
</target>
<parameter name="transport.vfs.ReplyFileURI">file:///home/user/test/out? transport.vfs.Append=true</parameter>
<parameter name="transport.PollInterval">10</parameter>
<parameter name="transport.vfs.FileNamePattern">Errorlog.text</parameter>
<parameter name="transport.vfs.ContentType">text/xml</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.ReplyFileName">Errorlog.xml</parameter>
<description></description>
</proxy>
Actually I kept a log mediator in outSequence. The inSequence mediator is not sending the data in to the outSequence process not forwarding into outSequence. That's why I think that the above configuration is not working.
Any reference for this?
I tried above configuration in inSequence also. It's giving errors like this:
ERROR - Axis2Sender Unexpected error during sending message out
org.apache.axis2.AxisFault:
The VFS transport doesn't support synchronous responses. Please use
the appropriate (out only) message exchange pattern
Please refer to this link.
The issue is, you are setting a property inside a log mediator which is a predefined property, (ie:OUT_ONLY) which is used to indicate the request is only the "out-only" request. So, system, wont expect the response back. That is why, you are not getting anything in your outsequence.
DO NOT USE the predefined properties in the Log mediator,which will cause issues.
Keep some text in the log meditaor to indictae the flow of the message.
eg:
<inSequence>
<log>
<property name="INSEQUENCEEEEEEE" value="********"/>
</log>
</inSequence>
Like wise keep different descriptive log in the outsequence and see whether you are getting the message there, without any issue.

WSO2 ESB 4.5.0 vfs TransportSender as Endpoint Errors with 'Access is denied'

I am trying to put together a proof of concept for the use of wso2 esb. The proof of concept will rely on the ESB picking up a csv file dropped into a folder, converting the details to xml, posting them to a 3rd party web service, then converting the response, which should contain the binary for a pdf, into a pdf and dropping it into a folder.
The current problem with this is that when I configure a folder as an endpoint in wso2 esb 4.5.0, any file that I send to that end-point errors. A stripped down version of my configuration is as defined below: -
<proxy name="PDFPoller"
transports="vfs"
startOnLoad="true"
trace="enable"
statistics="enable">
<description/>
<target>
<inSequence>
<log level="custom">
<property name="status" value="PDF Receieved"/>
</log>
<log level="full"/>
<property name="transport.vfs.ReplyFileName"
expression="test1.pdf"
scope="transport"/>
<property name="OUT_ONLY" value="true"/>
<send>
<endpoint name="FileEpr">
<address uri="vfs:file:///c:/wso2/processed"/>
</endpoint>
</send>
</inSequence>
</target>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///C:/wso2/output</parameter>
<parameter name="transport.vfs.FileURI">file:///C:/wso2/PollFolder</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///C:/wso2/Failed</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.pdf</parameter>
<parameter name="transport.vfs.ContentType">application/pdf</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
The error that I get from the ESB is as follows: -
2012-10-29 09:46:00,642 [-] [Axis2 Task] ERROR VFSTransportSender IO Error while
org.apache.commons.vfs2.FileSystemException: Could not write to "file:///c:/wso2/processed".
at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1440)
at org.apache.commons.vfs2.provider.DefaultFileContent.getOutputStream(DefaultFileContent.java:462)
at org.apache.synapse.transport.vfs.VFSTransportSender.populateResponseFile(VFSTransportSender.java:232)
at org.apache.synapse.transport.vfs.VFSTransportSender.sendMessage(VFSTransportSender.java:173)
at org.apache.axis2.transport.base.AbstractTransportSender.invoke(AbstractTransportSender.java:112)
at org.apache.axis2.engine.AxisEngine$TransportNonBlockingInvocationWorker.run(AxisEngine.java:627)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.FileNotFoundException: c:\wso2\processed (Access is denied)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
at java.io.FileOutputStream.<init>(FileOutputStream.java:116)
at org.apache.commons.vfs2.provider.local.LocalFile.doGetOutputStream(LocalFile.java:220)
at org.apache.commons.vfs2.provider.AbstractFileObject.getOutputStream(AbstractFileObject.java:1432)
... 8 more
I don't believe that the problem is a local folder permissions problem, because I have a) checked the permissions on the folder manually and b) The ESB will move the file that I place in the 'pollfolder' to either 'Output' or 'Processed' if I set the transport.vfs.MoveAfterProcess property to that value, so it is possible to write to both these folders.
Any help would be appreciated.
You configuration seems fine. But I suspect the following section.
<property name="transport.vfs.ReplyFileName"
expression="test1.pdf"
scope="transport"/>
You can try without that property. Then it will create a file as same as the input file name inside the "processed" folder. If it works then try the above property as follows.
<property name="transport.vfs.ReplyFileName"
value="test1.pdf"
scope="transport"/>
As you can see, I have replaced "expression" attribute with "value". Expression should be used when you want to execute something, for example concatenating two strings to build the response file name etc.