vfs transport transport is not stable in WSO2 - wso2

I using EI version 6.6.0 and OS - CentOS Linux 7 (Core).
I have a proxy service and one part of its task is to copy a file with content to a backup. To do this I use the transport.vfs.replyfilename property with the OUT_ONLY property and the endpoint with the url where I want to save the backup file.
This is example my proxy :
<target>
<inSequence>
<log level="custom" separator="
">
<property name="STEP" value="Start operation"/>
<property expression="$trp:FILE_LENGTH" name="LENGTH"/>
<property expression="$trp:LAST_MODIFIED" name="MODIFIED"/>
<property expression="$trp:FILE_URI" name="URI"/>
<property expression="$trp:FILE_PATH" name="PATH"/>
<property expression="$trp:FILE_NAME" name="NAME"/>
</log>
<property expression="//*[local-name()='text']" name="INPUT_FILE" scope="default" type="STRING"/>
<property expression="$trp:FILE_NAME" name="uri.var.load_file" scope="default" type="STRING"/>
<log level="custom">
<property name="STEP" value="Copy input file to another location"/>
<property name="LOAD_FILE" expression="get-property('uri.var.load_file')"/>
</log>
<property expression="get-property('uri.var.load_file')"
name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
<property name="OUT_ONLY" scope="default" type="STRING" value="true"/>
<send>
<endpoint name="fileEndpoint">
<address trace="disable" uri="vfs:file:///relaitive_path_of_backup_folder"/>
</endpoint>
</send>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.PollInterval">60</parameter>
<parameter name="transport.vfs.FailedRecordsFileDestination">file:///opt/wso2ei-6.6.0/repository/conf</parameter>
<parameter name="transport.vfs.MaxRetryCount">5</parameter>
<parameter name="transport.vfs.ContentType">text/plain;charset=UTF-8</parameter>
<parameter name="transport.vfs.FileURI">vfs:ftp://login:password#host/outgoing_test?transport.vfs.ReconnectTimeout=10&transport.vfs.MaxRetryCount=5&transport.vfs.ConnectTimeout=5000</parameter>
<parameter name="transport.vfs.ActionAfterProcess">DELETE</parameter>
<parameter name="transport.vfs.FailedRecordsFileName">transferFails.log</parameter>
<parameter name="transport.vfs.MoveAfterFailure">vfs:ftp://login:password#host/outgoing_test/error?transport.vfs.ReconnectTimeout=10&transport.vfs.MaxRetryCount=5&transport.vfs.ConnectTimeout=5000</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.csv</parameter>
<parameter name="transport.vfs.Locking">disable</parameter>
<parameter name="transport.vfs.MoveFailedRecordTimestampFormat">yyyy-MM-dd'T'HH:mm:ss.SSSZ_</parameter>
But I have encountered that this method is not stable and does not always copy the file. There is nothing in the logs about this, no errors. Just sometimes the file is not copied to the specified path.
How to make this process guaranteed ?
What I do wrong ?

Related

File Processing :Have issues while File proceesing in wso2

Respected Sir,
I have to process only the files which are coming to the folder newly ( there could be files which are already existing in the folder which should not be picked for processing )
The client does not want to move the files to another folder after processing. So if the file remains in the same folder after processing. They should not be re-processed and should remain in the same folder with same extension
Below is the code I am trying. But I am not able to achieve above .Please guide.
<?xml version="1.0" encoding="UTF-8"?>
<proxy name`enter code here`="FlatFileProxy" startOnLoad="true" transports="vfs" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<property name="FORCE_SC_ACCEPTED" scope="axis2" type="STRING" value="true"/>
<property name="REST_URL_POSTFIX" scope="axis2" type="STRING" value=""/>
<property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
<datamapper config="gov:datamapper/xmlToCsv1.dmc" inputSchema="gov:datamapper/xmlToCsv1_inputSchema.json" inputType="XML" outputSchema="gov:datamapper/xmlToCsv1_outputSchema.json" outputType="CSV"/>
<property expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.csv')" name="transport.vfs.ReplyFileName" scope="transport" type="STRING"/>
<property name="OUT_ONLY" scope="default" type="STRING" value="true"/>
<property name="messageType" scope="axis2" type="STRING" value="application/csv"/>
<property action="remove" name="LAST_MODIFIED" scope="transport"/>
<send>
<endpoint>
<address uri="vfs:file:///C:/Flatfile/output"/>
</endpoint>
</send>
</inSequence>
<outSequence/>
<faultSequence/>
</target>
<parameter name="transport.vfs.Streaming">true</parameter>
<parameter name="transport.PollInterval">50ms</parameter>
<parameter name="transport.vfs.ContentType">application/xml</parameter>
<parameter name="transport.vfs.FileURI">file:///C:/Flatfile/input1</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///C:/Flatfile/failure</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
<parameter name="transport.vfs.Locking">disable</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///C:/Flatfile/orgFilesProcessedSuccessfully</parameter>
</proxy>
With Regards,
Aditya
With vfs Transport you have the option to use transport.vfs.ActionAfterProcess NONE. But then you'll have to find a way in your inSequence to find a way to not process file that was already processed (maybe based on file date with get-property('transport', 'LAST_MODIFIED')).
Take care on the sorting in this case to process the last file first (transport.vfs.FileSortAttribute = Lastmodifiedtimestamp and transport.vfs.FileSortAsscending = false)

Guaranteed Delivery of Messages with Active MQ and WSO2 EI

Issue: i have a proxy service which consumes only XML message, but sometimes if we are getting XML message with not proper format,just syntax error,our proxy should send that message to a different queue,the message should not loss.
i have followed the below link but unable to get the expected output
https://www.yenlo.com/blog/guaranteed-message-deliveries-part-3-monitoring-the-redelivery-policy
Below is my proxy:
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="RollbackProxy" startOnLoad="true" transports="jms" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log description="" level="full">
<property name="****************" value="******************************"/>
</log>
</inSequence>
<outSequence/>
<faultSequence>
<property name="SET_ROLLBACK_ONLY" scope="axis2" type="STRING" value="true"/>
<log category="ERROR" level="full">
<property expression="$ctx:ERROR_CODE" name="error_code"/>
<property expression="$ctx:ERROR_MESSAGE" name="error_message"/>
<property expression="$ctx:ERROR_DETAIL" name="error_detail"/>
</log>
</faultSequence>
</target>
<parameter name="transport.jms.Destination">QueueName</parameter>
<parameter name="transport.jms.ContentType">
<rules xmlns="">
<jmsProperty>contentType</jmsProperty>
<default>application/xml</default>
</rules>
</parameter>
<parameter name="transport.jms.ConnectionFactory">myQueueConnectionFactory</parameter>
</proxy>
axis2.xml and activemq.xml i have properly updated.
we are using active MQ.
Can somebody guide me on this?
Thanks in Advance

How to send an email on my personal email id fro WSO2 ESB

I am trying to send an email from the WSO2 ESB and configured as per one wos2 blog. I am getting error " MailTransportListener Checking mail for account : synapse.demo.1#gmail.com". I a not sure what is wrong in code and can I also send am email on my personal email id so that I can come to know that everything is working fine.
Synapse Code:
<proxy name="StockQuoteProxy" transports="mailto">
<parameter name="transport.mail.Address">synapse.demo.1#gmail.com</parameter>
<parameter name="transport.mail.Protocol">pop3</parameter>
<parameter name="transport.PollInterval">5</parameter>
<parameter name="mail.pop3.host">pop.gmail.com</parameter>
<parameter name="mail.pop3.port">995</parameter>
<parameter name="mail.pop3.user">synapse.demo.1</parameter>
<parameter name="mail.pop3.password">mailpassword</parameter>
<parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
<parameter name="mail.pop3.socketFactory.fallback">false</parameter>
<parameter name="mail.pop3.socketFactory.port">995</parameter>
<parameter name="transport.mail.ContentType">application/xml</parameter>
<target>
<inSequence>
<property name="senderAddress" expression="get-property('transport', 'From')"/>
<log level="full">
<property name="Sender Address" expression="get-property('senderAddress')"/>
</log>
<send>
<endpoint>
<address uri="http://localhost:9000/services/SimpleStockQuoteService"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<property name="Subject" value="Custom Subject for Response" scope="transport"/>
<property name="messageType" value="text/plain" scope="axis2-client"/>
<script language="js"><![CDATA[
mc.setPayloadXML(<ns:text xmlns:ns="http://ws.apache.org/commons/ns/payload">Plain text received!</ns:text>);
]]></script>
<header name="To" expression="fn:concat('mailto:', get-property('senderAddress'))"/>
<log level="full">
<property name="message" value="Response message"/>
<property name="Sender Address" expression="get-property('senderAddress')"/>
</log>
<send/>
</outSequence>
</target>
<publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
Thanks for your response. All I want is to send an email once I hit this service. my new code is :
<proxy name="SendEmailService" transports="http" startOnLoad="true" statistics="enable" trace="disable">
<target inSequence="SendEmailService_IN" />
<publishWSDL key="SendEmailService_wsdl"/>
</proxy>
<localEntry key="SendEmailService_wsdl" src="file:repository/conf/employee/OverseaseEmployee.wsdl"/>
<sequence name="SendEmailService_IN">
<property name="Subject" value="Custom Subject for Response" scope="transport"/>
<property name="ContentType" value="text/plain" scope="axis2"/>
<property name="OUT_ONLY" value="true" scope="default" type="STRING"/>
<send>
<endpoint>
<address uri="06.shrikant#gmail.com"/>
</endpoint>
</send>
<parameter name="mail.smtp.user">shrikant</parameter>
<parameter name="transport.mail.Protocol">smtp</parameter>
<parameter name="transport.mail.Address">06.shrikant#gmail.com</parameter>
<parameter name="mail.smtp.port">587</parameter>
<parameter name="mail.smtp.password">pass</parameter>
<parameter name="transport.mail.ContentType">text/plain</parameter>
<parameter name="mail.smtp.starttls.enable">true</parameter>
<parameter name="mail.smtp.auth">true</parameter>
<parameter name="mail.smtp.from">06.shrikant#gmail.com</parameter>
<parameter name="mail.smtp.host">smtp.gmail.com</parameter>
<send/>
</sequence>
Here is a working proxy to call a service and send the response to an email address. Here, transports="http,https" should be changed depending on how you want to call the proxy service.
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="MyPxy"
transports="http,https"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<send>
<endpoint>
<address uri="http://www.mocky.io/v2/57cd173e1200008620bb7826"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<property name="Subject"
value="Custom Subject for Response"
scope="transport"/>
<property name="ContentType" value="text/plain" scope="axis2"/>
<property name="OUT_ONLY" value="true" scope="default" type="STRING"/>
<send>
<endpoint>
<address uri="mailto:toemailuser#gmail.com"/>
</endpoint>
</send>
<respond/>
</outSequence>
</target>
<parameter name="mail.smtp.user">fromemailuser</parameter>
<parameter name="transport.mail.Protocol">smtp</parameter>
<parameter name="transport.mail.Address">fromemailuser#gmail.com</parameter>
<parameter name="mail.smtp.port">587</parameter>
<parameter name="mail.smtp.password">pass</parameter>
<parameter name="transport.mail.ContentType">text/plain</parameter>
<parameter name="mail.smtp.starttls.enable">true</parameter>
<parameter name="mail.smtp.auth">true</parameter>
<parameter name="mail.smtp.from">fromemailuser#gmail.com</parameter>
<parameter name="mail.smtp.host">smtp.gmail.com</parameter>
<description/>
</proxy>
You can refer this blog for sending email with wso2 esb http://elilsivanesan.blogspot.com/2016/06/mail-transport-with-wso2-esb.html

How to invoke DSS service from ESB - I feel that I'm close, I just need a nudge

I think i'm getting close with this. I'm trying to invoke an insert call on a DSS service from and ESB in WSO2.
I have the DSS service setup and I am able to insert data into the table from the 'try it' link. I copied the WSDL to the ESB and referenced the endpoint. I can see the insert operation from the ESB try it service. I put in my data and click send. I see a 'success' response come back but nothing is being added to the table.
Is anyone willing to nudge me in the right direction with this?
Thank you!
Response from try it service
<success details="in-only operation"/>
proxy
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="MyProxy"
transports="https,http,local,vfs"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<property name="targetfilename" value="TITLES"/>
<log level="full"/>
<clone/>
</inSequence>
</target>
<publishWSDL key="InsertServiceWSDL"/>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.FileURI">file:///var/process
/rrin</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///var/process
/rroriginal</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///var/process
/rrfault</parameter>
<parameter name="transport.vfs.FileNamePattern">TITLES.xml</parameter>
<parameter name="transport.vfs.ContentType">application/xml</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<description/>
</proxy>
You'll need to add an address endpoint pointing to the DSS service. Refer to this sample which is on how to define a proxy service for an axis2 web service. Your scenario is very similar to this.
Here is how I did it. The call will now hit the DSS and insert the data into the table. I am seeing some errors in the log for each XML row it processes and send to the DSS. I'm not sure why yet. I'm still researching that.
Edit: I changed the call mediator to a send mediator and that fixed this issue. I am no longer getting this error message.
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[1,1]
Message: Premature end of file.
Here is my sequence.
<sequence xmlns="http://ws.apache.org/ns/synapse" name="MySequence">
<log level="custom">
<property name="sequence" value="MySequence"></property>
</log>
<property xmlns:ns="http://org.apache.synapse/xsd" name="filename" expression="get-property('transport', 'FILE_NAME')"></property>
<log level="custom">
<property xmlns:ns="http://org.apache.synapse/xsd" name="show-name" expression="get-property('filename')"></property>
<property xmlns:ns="http://org.apache.synapse/xsd" name="file-name" expression="get-property('targetfilename')"></property>
</log>
<iterate xmlns:ns2="http://org.apache.synapse/xsd" xmlns:ns="http://org.apache.synapse/xsd" xmlns:z="RowsetSchema" expression="//z:row" id="It1">
<target>
<sequence>
<property name="Id" expression="//z:row/#ID"></property>
<property name="vch" expression="//z:row/#vch"></property>
<log level="custom">
<property name="showids" expression="get-property('Id')"></property>
<property name="showvch" expression="get-property('vch')"></property>
</log>
<filter xpath="//z:row[starts-with(#vch, '978')]">
<then>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:p="http://ws.wso2.org/dataservice">
<soapenv:Body>
<p:insert_AR_operation>
<p:ID xmlns:xs="http://ws.wso2.org/dataservice">$1</p:ID>
<p:vch xmlns:xs="http://ws.wso2.org/dataservice">$2</p:vch>
</p:insert_AR_operation>
</soapenv:Body>
</soapenv:Envelope>
</format>
<args>
<arg expression="get-property('Id')" evaluator="xml"></arg>
<arg expression="get-property('vch')" evaluator="xml"></arg>
</args>
</payloadFactory>
<log level="custom">
<property name="sequence" value="Calling LevelsAR_ISBNService"></property>
</log>
<property name="HTTP_METHOD" value="POST" scope="axis2"></property>
<property name="SOAPAction" value="insert_AR_operation" scope="transport"></property>
<send>
<endpoint>
<address uri="http://*.*.*.*:****/services/AR_Service.HTTPEndpoint/"></address>
</endpoint>
</send>
</then>
<else>
<log level="custom">
<property name="sequence" value="Condition Drop"></property>
</log>
<drop></drop>
</else>
</filter>
</sequence>
</target>
</iterate>
<property xmlns:ns2="http://org.apache.synapse/xsd" xmlns:ns="http://org.apache.synapse/xsd" name="transport.vfs.ReplyFileName" expression="fn:concat(get-property('SYSTEM_DATE', 'yyMMddHHmmss'), '-', get-property('filename'))" scope="transport"></property>
<property name="OUT_ONLY" value="true"></property>
</sequence>

Pass property from inSequence to outSequence

I'm sending a message with a Proxy to a HL7 TCP/IP port and get the response in the outSequence. But my problem is that all properties set in the inSequence are not anymore available. All of them are null. I tested with all the different scopes (transport, axis2, axis2-client), but none of them worked.
I saw in this post that it should be possible. Is the HL7 sender destroying the properties?
How can use my properties from the inSequence in the outSequence?
Example of my Proxy (get message from ActiveMQ JMS and sends to HL7 port 4000):
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" xmlns:hl7="http://wso2.org/hl7" xmlns:urn="urn:hl7-org:v2xml" name="demo_toHL7" transports="jms" startOnLoad="true" trace="disable">
<parameter name="transport.jms.Destination">demo_qFilter</parameter>
<parameter name="transport.jms.ConnectionFactory">queueBlocking</parameter>
<parameter name="transport.jms.DestinationType">queue</parameter>
<parameter name="transport.jms.ContentType">
<rules>
<jmsProperty>contentType</jmsProperty>
<default>application/edi-hl7</default>
</rules>
</parameter>
<target faultSequence="rollbackSequence">
<inSequence>
<log level="full"/>
<property name="ClientApiNonBlocking" scope="axis2" action="remove"/>
<property name="testProperty" value="blabla" scope="transport"/>
<property name="messageType" value="application/edi-hl7" scope="axis2"/>
<property name="ContentType" value="application/edi-hl7" scope="axis2"/>
<send>
<endpoint>
<address uri="hl7://localhost:4000"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<log level="custom">
<property name="PROPERTY" expression="get-property('transport','testProperty')"/>
</log>
</outSequence>
</target>
</proxy>
I'm using WSO2 ESB 4.0.3 and installed the HL7 Feature. As receiver I use the 7edit application.
Try with property scope as "default/synapse"
FiveO edit comment:
Try with property scope as "default":
Sending a transport property from the inSequence to the outSequence (on behalf of the default scope):
<inSequence>
...
<property name="myPropertyInTransport" value="myValue" scope="transport"/>
<property name="myPropertyInDefault" expression="get-property('transport','myPropertyInTransport')" scope="default"/>
...
</inSequence>
<outSequence>
...
<property name="myPropertyInTransport" expression="get-property('default', 'myPropertyInDefault')" scope="transport"/>
<!-- Now myProperty is also available in the outSequence -->
...
</outSequence>