File Processing :Have issues while File proceesing in wso2 - wso2

Respected Sir,
I have to process only the files which are coming to the folder newly ( there could be files which are already existing in the folder which should not be picked for processing )
The client does not want to move the files to another folder after processing. So if the file remains in the same folder after processing. They should not be re-processed and should remain in the same folder with same extension
Below is the code I am trying. But I am not able to achieve above .Please guide.
<?xml version="1.0" encoding="UTF-8"?>
<proxy name`enter code here`="FlatFileProxy" startOnLoad="true" transports="vfs" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<property name="FORCE_SC_ACCEPTED" scope="axis2" type="STRING" value="true"/>
<property name="REST_URL_POSTFIX" scope="axis2" type="STRING" value=""/>
<property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
<datamapper config="gov:datamapper/xmlToCsv1.dmc" inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json" inputType="XML" outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json" outputType="CSV"/>
<property expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.csv')" name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
<property name="OUT_ONLY" scope="default" type="STRING" value="true"/>
<property name="messageType" scope="axis2" type="STRING" value="application/csv"/>
<property action="remove" name="LAST_MODIFIED" scope="transport"/>
<send>
<endpoint>
<address uri="vfs:file:///C:/Flatfile/output"/>
</endpoint>
</send>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.PollInterval">50ms</parameter>
<parameter name="transport.vfs.ContentType">application/xml</parameter>
<parameter name="transport.vfs.FileURI">file:///C:/Flatfile/input1</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///C:/Flatfile/failure</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
<parameter name="transport.vfs.Locking">disable</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///C:/Flatfile/orgFilesProcessedSuccessfully</parameter>
</proxy>
With Regards,
Aditya

With vfs Transport you have the option to use transport.vfs.ActionAfterProcess NONE. But then you'll have to find a way in your inSequence to find a way to not process file that was already processed (maybe based on file date with get-property('transport', 'LAST_MODIFIED')).
Take care on the sorting in this case to process the last file first (transport.vfs.FileSortAttribute = Lastmodifiedtimestamp and transport.vfs.FileSortAsscending = false)

Related

vfs transport transport is not stable in WSO2

I using EI version 6.6.0 and OS - CentOS Linux 7 (Core).
I have a proxy service and one part of its task is to copy a file with content to a backup. To do this I use the transport.vfs.replyfilename property with the OUT_ONLY property and the endpoint with the url where I want to save the backup file.
This is example my proxy :
<target>
<inSequence>
<log level="custom" separator="
">
<property name="STEP" value="Start operation"/>
<property expression="$trp:FILE_LENGTH" name="LENGTH"/>
<property expression="$trp:LAST_MODIFIED" name="MODIFIED"/>
<property expression="$trp:FILE_URI" name="URI"/>
<property expression="$trp:FILE_PATH" name="PATH"/>
<property expression="$trp:FILE_NAME" name="NAME"/>
</log>
<property expression="//*[local-name()='text']" name="INPUT_FILE" scope="default" type="STRING"/>
<property expression="$trp:FILE_NAME" name="uri.var.load_file" scope="default" type="STRING"/>
<log level="custom">
<property name="STEP" value="Copy input file to another location"/>
<property name="LOAD_FILE" expression="get-property('uri.var.load_file')"/>
</log>
<property expression="get-property('uri.var.load_file')"
name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
<property name="OUT_ONLY" scope="default" type="STRING" value="true"/>
<send>
<endpoint name="fileEndpoint">
<address trace="disable" uri="vfs:file:///relaitive_path_of_backup_folder"/>
</endpoint>
</send>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.PollInterval">60</parameter>
<parameter name="transport.vfs.FailedRecordsFileDestination">file:///opt/wso2ei-6.6.0/repository/conf</parameter>
<parameter name="transport.vfs.MaxRetryCount">5</parameter>
<parameter name="transport.vfs.ContentType">text/plain;charset=UTF-8</parameter>
<parameter name="transport.vfs.FileURI">vfs:ftp://login:password#host/outgoing_test?transport.vfs.ReconnectTimeout=10&transport.vfs.MaxRetryCount=5&transport.vfs.ConnectTimeout=5000</parameter>
<parameter name="transport.vfs.ActionAfterProcess">DELETE</parameter>
<parameter name="transport.vfs.FailedRecordsFileName">transferFails.log</parameter>
<parameter name="transport.vfs.MoveAfterFailure">vfs:ftp://login:password#host/outgoing_test/error?transport.vfs.ReconnectTimeout=10&transport.vfs.MaxRetryCount=5&transport.vfs.ConnectTimeout=5000</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.csv</parameter>
<parameter name="transport.vfs.Locking">disable</parameter>
<parameter name="transport.vfs.MoveFailedRecordTimestampFormat">yyyy-MM-dd'T'HH:mm:ss.SSSZ_</parameter>
But I have encountered that this method is not stable and does not always copy the file. There is nothing in the logs about this, no errors. Just sometimes the file is not copied to the specified path.
How to make this process guaranteed ?
What I do wrong ?

How do I transfer files from server to server using wso2 vfs?

I have created a wso2 file transfer service which transfers files from one folder to another on my local machine, this is done using a proxy and a sequence on integration studio, I then export a carbon application and deploy it on enterprise integrator. However, the moment I try to transfer files from server to server or from folder to folder within the same server the files do not transfer even though there are no errors in the error log. I have created my own FTP server using iis. I have also encoded the FTP file path. Please see my proxy and sequence below, any help will be greatly appreciated.
Proxy service,
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="YenloProxy" startOnLoad="true" transports="http https" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log level="custom">
<property name="sequence" value="Proxy"/>
</log>
<clone>
<target sequence="YenloSequence"/>
</clone>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.FileURI">vfs:ftp://ftp-user%3AP%40ssw0rd01%40192.168.1.36%3A21%2FIncoming%2F</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterFailure">vfs:ftp://ftp-user%3AP%40ssw0rd01%40192.168.1.36%3A21%2FFailure%2F</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.dat</parameter>
<parameter name="transport.vfs.MoveAfterProcess">vfs:ftp://ftp-user%3AP%40ssw0rd01%40192.168.1.36%3A21%2FOutgoing%2F</parameter>
</proxy>
Sequence,
<?xml version="1.0" encoding="UTF-8"?>
<sequence name="YenloSequence" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<log level="custom">
<property name="sequence" value="YenloSequence"/>
</log>
<property expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.txt')" name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
<property name="OUT_ONLY" scope="default" type="STRING" value="true"/>
<property name="ClientApiNonBlocking" scope="axis2" type="STRING" value="true"/>
<send>
<endpoint name="FileEpr">
<address uri="vfs:ftp://ftp-user%3AP%40ssw0rd01%40192.168.1.36%3A21%2FOutput%2F">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>1</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</address>
</endpoint>
</send>
</sequence>
In the proxy service, the transports attribute is used to define the transport protocols that are used to receive messages. Since we are using VFS Transport to receive the file, you need to set the value of the transport as follows, transports="vfs". You may refer to this example for more information.

Guaranteed Delivery of Messages with Active MQ and WSO2 EI

Issue: i have a proxy service which consumes only XML message, but sometimes if we are getting XML message with not proper format,just syntax error,our proxy should send that message to a different queue,the message should not loss.
i have followed the below link but unable to get the expected output
https://www.yenlo.com/blog/guaranteed-message-deliveries-part-3-monitoring-the-redelivery-policy
Below is my proxy:
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="RollbackProxy" startOnLoad="true" transports="jms" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log description="" level="full">
<property name="****************" value="******************************"/>
</log>
</inSequence>
<outSequence/>
<faultSequence>
<property name="SET_ROLLBACK_ONLY" scope="axis2" type="STRING" value="true"/>
<log category="ERROR" level="full">
<property expression="$ctx:ERROR_CODE" name="error_code"/>
<property expression="$ctx:ERROR_MESSAGE" name="error_message"/>
<property expression="$ctx:ERROR_DETAIL" name="error_detail"/>
</log>
</faultSequence>
</target>
<parameter name="transport.jms.Destination">QueueName</parameter>
<parameter name="transport.jms.ContentType">
<rules xmlns="">
<jmsProperty>contentType</jmsProperty>
<default>application/xml</default>
</rules>
</parameter>
<parameter name="transport.jms.ConnectionFactory">myQueueConnectionFactory</parameter>
</proxy>
axis2.xml and activemq.xml i have properly updated.
we are using active MQ.
Can somebody guide me on this?
Thanks in Advance

How to send an email on my personal email id fro WSO2 ESB

I am trying to send an email from the WSO2 ESB and configured as per one wos2 blog. I am getting error " MailTransportListener Checking mail for account : synapse.demo.1#gmail.com". I a not sure what is wrong in code and can I also send am email on my personal email id so that I can come to know that everything is working fine.
Synapse Code:
<proxy name="StockQuoteProxy" transports="mailto">
<parameter name="transport.mail.Address">synapse.demo.1#gmail.com</parameter>
<parameter name="transport.mail.Protocol">pop3</parameter>
<parameter name="transport.PollInterval">5</parameter>
<parameter name="mail.pop3.host">pop.gmail.com</parameter>
<parameter name="mail.pop3.port">995</parameter>
<parameter name="mail.pop3.user">synapse.demo.1</parameter>
<parameter name="mail.pop3.password">mailpassword</parameter>
<parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
<parameter name="mail.pop3.socketFactory.fallback">false</parameter>
<parameter name="mail.pop3.socketFactory.port">995</parameter>
<parameter name="transport.mail.ContentType">application/xml</parameter>
<target>
<inSequence>
<property name="senderAddress" expression="get-property('transport', 'From')"/>
<log level="full">
<property name="Sender Address" expression="get-property('senderAddress')"/>
</log>
<send>
<endpoint>
<address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<property name="Subject" value="Custom Subject for Response" scope="transport"/>
<property name="messageType" value="text/plain" scope="axis2-client"/>
<script language="js"><![CDATA[
mc.setPayloadXML(<ns:text xmlns:ns="http://ws.apache.org/commons/ns/payload">Plain text received!</ns:text>);
]]></script>
<header name="To" expression="fn:concat('mailto:', get-property('senderAddress'))"/>
<log level="full">
<property name="message" value="Response message"/>
<property name="Sender Address" expression="get-property('senderAddress')"/>
</log>
<send/>
</outSequence>
</target>
<publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
Thanks for your response. All I want is to send an email once I hit this service. my new code is :
<proxy name="SendEmailService" transports="http" startOnLoad="true" statistics="enable" trace="disable">
<target inSequence="SendEmailService_IN" />
<publishWSDL key="SendEmailService_wsdl"/>
</proxy>
<localEntry key="SendEmailService_wsdl" src="file:repository/conf/employee/OverseaseEmployee.wsdl"/>
<sequence name="SendEmailService_IN">
<property name="Subject" value="Custom Subject for Response" scope="transport"/>
<property name="ContentType" value="text/plain" scope="axis2"/>
<property name="OUT_ONLY" value="true" scope="default" type="STRING"/>
<send>
<endpoint>
<address uri="06.shrikant#gmail.com"/>
</endpoint>
</send>
<parameter name="mail.smtp.user">shrikant</parameter>
<parameter name="transport.mail.Protocol">smtp</parameter>
<parameter name="transport.mail.Address">06.shrikant#gmail.com</parameter>
<parameter name="mail.smtp.port">587</parameter>
<parameter name="mail.smtp.password">pass</parameter>
<parameter name="transport.mail.ContentType">text/plain</parameter>
<parameter name="mail.smtp.starttls.enable">true</parameter>
<parameter name="mail.smtp.auth">true</parameter>
<parameter name="mail.smtp.from">06.shrikant#gmail.com</parameter>
<parameter name="mail.smtp.host">smtp.gmail.com</parameter>
<send/>
</sequence>
Here is a working proxy to call a service and send the response to an email address. Here, transports="http,https" should be changed depending on how you want to call the proxy service.
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="MyPxy"
transports="http,https"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<send>
<endpoint>
<address uri="http://www.mocky.io/v2/57cd173e1200008620bb7826"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<property name="Subject"
value="Custom Subject for Response"
scope="transport"/>
<property name="ContentType" value="text/plain" scope="axis2"/>
<property name="OUT_ONLY" value="true" scope="default" type="STRING"/>
<send>
<endpoint>
<address uri="mailto:toemailuser#gmail.com"/>
</endpoint>
</send>
<respond/>
</outSequence>
</target>
<parameter name="mail.smtp.user">fromemailuser</parameter>
<parameter name="transport.mail.Protocol">smtp</parameter>
<parameter name="transport.mail.Address">fromemailuser#gmail.com</parameter>
<parameter name="mail.smtp.port">587</parameter>
<parameter name="mail.smtp.password">pass</parameter>
<parameter name="transport.mail.ContentType">text/plain</parameter>
<parameter name="mail.smtp.starttls.enable">true</parameter>
<parameter name="mail.smtp.auth">true</parameter>
<parameter name="mail.smtp.from">fromemailuser#gmail.com</parameter>
<parameter name="mail.smtp.host">smtp.gmail.com</parameter>
<description/>
</proxy>
You can refer this blog for sending email with wso2 esb http://elilsivanesan.blogspot.com/2016/06/mail-transport-with-wso2-esb.html

Send Mediator with VFS error WSO2 ESB

I create a proxy and sequence to read file from source folder then write it to target folder with send mediator with VFS. My Proxy look like this :
<proxy name="XXX" transports="vfs" startOnLoad="true" trace="disable">
<description/>
<target>
<inSequence>
<property name="filename"
expression="fn:concat(get-property('transport', 'FILE_NAME'), '')"/>
<sequence key="write"/>
<log level="custom">
<property name="finish" value="finish"/>
</log>
</inSequence>
</target>
<parameter name="transport.PollInterval">1</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///after</parameter>
<parameter name="transport.vfs.FileURI">file:///process</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
and my "write" sequence look like this :
<sequence name="write">
<property name="transport.vfs.ReplyFileName"
expression="get-property('filename')"
scope="transport"/>
<property name="OUT_ONLY" value="true"/>
<send>
<endpoint name="FileEpr">
<address uri="vfs:file:///Target"/>
</endpoint>
</send>
When I tried this proxy and sequence, the send mediator success to create a file in the "target" folder, but the problem is the content of the file is not written. So the size is 0 byte. My original file is 1000bytes. If I add this config in the sequence
<log level="full"/>
it works perfectly fine. My question is, do I need to use the log full config? but when I look at the File Processing sample in WSO2 web it doesn't use any log full config. So how can I use send mediator to write file without using any "log" property?
Thanks,