Currently working for a company on wso2 integrator, I need to retrieve files periodically created into a directory so that they can be processed, modified and sent to a remote API.
The problem is that we need to use wso2 exclusively and absolutely can't code side programs to adapt to the situation. We can use script though only if they are embedded within wso2.
Does someone have a clue?
This is a simple synapse configuration that:
Wait for incoming CSV files on a given directory
Convert its contents to XML
Do some transformation on its elements
Sends the result to a another endpoint (an output directory in this case)
I hope it helps to get you going (comments inline):
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
<!-- ======================================== -->
<!-- Smooks configuration to parse sample CSV -->
<!-- ======================================== -->
<localEntry key="smooks-orders-csv">
<smooks-resource-list
xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd">
<resource-config selector="org.xml.sax.driver">
<resource>org.milyn.csv.CSVReader</resource>
<param name="fields">OrderId,SKU,Quantity,UnitPrice,TotalPrice</param>
<param name="skip-line-count">1</param> <!-- Skip Header line -->
<param name="rootElementName">orders</param>
<param name="recordElementName">order</param>
</resource-config>
</smooks-resource-list>
</localEntry>
<!-- ====================================================== -->
<!-- XML processing logic: Replaces SKU number 0002 to 0003 -->
<!-- ====================================================== -->
<localEntry key="change-sku">
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="#*|node()">
<xsl:copy>
<xsl:apply-templates select="#*|node()"/>
</xsl:copy>
</xsl:template>
<xsl:template match="SKU/text()[.='0002']">0003</xsl:template>
</xsl:stylesheet>
<description/>
</localEntry>
<!-- *********************** -->
<!-- Process Orders sequence -->
<!-- *********************** -->
<sequence name="process_orders" onError="fault" statistics="enable" trace="enable">
<!-- Convert incoming file into a stream of orders -->
<smooks config-key="smooks-orders-csv">
<input type="text"/>
<output type="xml"/>
</smooks>
<!-- General Synapse properties that must be set for a One-Way exchange pattern -->
<property name="FORCE_SC_ACCEPTED" scope="axis2" value="true"/>
<property name="OUT_ONLY" value="true"/>
<!-- VFS send will fail unless you remove this property -->
<property action="remove" name="ClientApiNonBlocking" scope="axis2"/>
<!-- The "magic" property 'transport.vfs.ReplyFileName' is used by the VFS transport to define output file name
when the specified URI is a directory
-->
<property
expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')"
name="transport.vfs.ReplyFileName" scope="transport" xmlns:ns="http://org.apache.synapse/xsd"/>
<!-- Create output file -->
<send>
<endpoint>
<address format="pox" uri="vfs:file:///mnt/c/transfer/out"/>
</endpoint>
</send>
</sequence>
<!-- Inbound polling endpoint -->
<inboundEndpoint name="received_orders" onError="fault"
protocol="file" sequence="process_orders" suspend="false">
<parameters>
<parameter name="interval">1000</parameter>
<parameter name="sequential">true</parameter>
<parameter name="coordination">true</parameter>
<!-- Input directory -->
<parameter name="transport.vfs.FileURI">file:///mnt/c/transfer/in</parameter>
<!-- CSV files are plain text messages -->
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<!-- RegExp used to filter files in input directory -->
<parameter name="transport.vfs.FileNamePattern">.*\.txt</parameter>
<parameter name="transport.vfs.Locking">disable</parameter>
<!-- Move processed files to the 'processed' subdirectory -->
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///mnt/c/transfer/in/processed</parameter>
<!-- Move failed files to the 'rejected subdirectory -->
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///mnt/c/transfer/in/rejected</parameter>
<parameter name="transport.vfs.AutoLockRelease">true</parameter>
<parameter name="transport.vfs.LockReleaseSameNode">false</parameter>
<parameter name="transport.vfs.DistributedLock">false</parameter>
<parameter name="transport.vfs.FileSortAttribute">NONE</parameter>
<parameter name="transport.vfs.FileSortAscending">true</parameter>
<parameter name="transport.vfs.CreateFolder">true</parameter>
<parameter name="transport.vfs.Streaming">false</parameter>
<parameter name="transport.vfs.Build">false</parameter>
</parameters>
</inboundEndpoint>
</definitions>
Related
Please someone could help. I have a set of data as csv(from excel multiple sheets i have formed csv)just i want to insert those data into multiple table depends the sheets of an excel.
please help me to sort out my issues.
I expected those csv in each iteration to form xml.
First, we can build the XML from CSV data and iterate through the XML. Please find a sample config below. In this sample, it will pick the CSV files from the source directory and process.
Smooks Config:
<?xml version="1.0"?>
<smooks-resource-list
xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd"
xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd">
<resource-config selector="org.xml.sax.driver">
<resource>org.milyn.csv.CSVReader</resource>
<param name="fields">firstname,lastname,gender,age,country</param>
<param name="rootElementName">people</param>
<param name="recordElementName">person</param>
</resource-config>
</smooks-resource-list>
Proxy:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="CSVToXML"
transports="https,http,vfs"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<smooks config-key="conf:/repository/smooks-config.xml">
<input type="text"/>
<output type="xml"/>
</smooks>
<!-- Here iterate through the XML -->
</inSequence>
<outSequence/>
</target>
<parameter name="transport.PollInterval">1000</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="Operation">urn:placeOrder</parameter>
<parameter name="transport.vfs.FileURI">file://<SOURCE_DIR></parameter>
<parameter name="transport.vfs.MoveAfterProcess">file://<PROCESSED_FILES_DIR></parameter>
<parameter name="transport.vfs.MoveAfterFailure">file://<FAILED_FILES_DIR></parameter>
<parameter name="transport.vfs.FileNamePattern">.*\.csv</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<description/>
</proxy>
References:
1. https://docs.wso2.com/display/EI640/VFS+Transport
2. https://docs.wso2.com/display/EI640/Smooks+Mediator
3. https://docs.wso2.com/display/EI640/Iterate+Mediator
I was working with the wso2 esb mail transport. I configured my mail id and all the details in the axis2.xml. When I restarted the Wso2 esb, I found that my Inbox is emptied.
Here is my axix2.xml:
<!-- ================================================= -->
<!-- Transport Ins (Listeners) -->
<!-- ================================================= -->
<transportReceiver name="mailto" class="org.apache.axis2.transport.mail.MailTransportListener">
<!-- configure any optional POP3/IMAP properties
check com.sun.mail.pop3 and com.sun.mail.imap package documentation for more details-->
</transportReceiver>
<!-- ================================================= -->
<!-- Transport Outs (Senders) -->
<!-- ================================================= -->
<!-- Uncomment and configure the SMTP server information
check com.sun.mail.smtp package documentation for descriptions of properties-->
<transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">
<parameter name="mail.smtp.host">mum.*******.com</parameter>
<parameter name="mail.smtp.port">587</parameter>
<parameter name="mail.smtp.starttls.enable">true</parameter>
<parameter name="mail.smtp.auth">true</parameter>
<parameter name="mail.smtp.user">*******</parameter>
<parameter name="mail.smtp.password">*********</parameter>
<parameter name="mail.smtp.from">*******#*****.com</parameter>
</transportSender>
Here is my proxy:
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="MailSample"
startOnLoad="true"
trace="disable">
<description/>
<target>
<inSequence>
<log>
<property name="hii" value="hi************************"/>
</log>
<property name="senderAddress" expression="get-property('transport', 'From')"/>
<log level="full">
<property name="Sender Address" expression="get-property('senderAddress')"/>
</log>
<send>
<endpoint>
<address uri="mailto:janiyani.richa#gmail.com"/>
</endpoint>
</send>
</inSequence>
</target>
<parameter name="transport.jms.ContentType">
<rules>
<jmsProperty>contentType</jmsProperty>
<default>application/xml</default>
</rules>
</parameter>
<parameter name="transport.jms.Destination">mailSender</parameter>
</proxy>
Do you know where did my mails go?
Is it possible to retrieve the mails?
Thanks in Advance.
By default, most POP3 mail servers keep messages only until they are downloaded by the mail client and then those are deleted. So the above behaviour can be due to that reason and you can prevent this from happening by doing the necessary configurations to leave the messages on server without deleting them even after they are read.
For setting this property at the client (axis2) side, see the mail.pop3.rsetbeforequit property in the com.sun.mail.pop3 documentation available here. (Also depending on the server, there can be server side configurations instead that will make sure that mails will always be kept at the server after being read.)
I have created a simple vfs proxy which consumes a file and places it on the JMS queue. I want to be able to pick up one file at a time. I have my pollinterval set for every 10 minutes.
Are there any properties or settings I am missing to control this functionality?
Sample Proxy Below
<proxy xmlns="http://ws.apache.org/ns/synapse" name="VFSImportProxy" transports="https,http,vfs" statistics="disable" trace="disable" startOnLoad="true">
<target>
<inSequence>
<log separator="," />
<sequence key="smooks_sequence" />
<property name="OUT_ONLY" value="true" scope="default" type="STRING" />
<send>
<endpoint key="MessageStore_EP" />
</send>
</inSequence>
<outSequence />
<faultSequence />
</target>
<parameter name="transport.PollInterval">600000</parameter>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.vfs.FileURI">vfs:sftp://locationpickup/local2</parameter>
<parameter name="transport.vfs.MoveAfterProcess">smb://location/archive</parameter>
<parameter name="transport.vfs.MoveAfterFailure">smb://location/failed</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<enableSec />
</proxy>
Yes, the parameter name is transport.vfs.FileProcessCount and it's available in version 4.8.0 and up.
https://docs.wso2.com/display/ESB480/VFS+Transport
I am using wso2esb4.8.0 .I wish insert the data into tables i am getting the data from client in .CSV format i have changed the VFS transport details in AXIS2 file and my configuration working fine for single CSV file
But my scenario is i have multiple tables like client,company,branch,divsion,party
these are dependent each other it means if i want insert into company i need clientid same as all tables
client give me sample data my clientid is auto genrated function in database it means remaining fields i need to insert and i will get clientid for that i used this query in dblookup mediator
insert into client(clientcode,clientname,clientshortname,createdbyid,modifiedbyid) values(?,?,?,?,?) RETURNING clientid;
and i need to insert company using this id how would i do this i tried for client insertion and unable get any idea for next insertion my configuration is
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="Smook"
transports="https,http,vfs"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<smooks config-key="Smook">
<input type="text"/>
<output type="xml"/>
</smooks>
<dblookup>
<connection>
<pool>
<password>Youtility11</password>
<user>youtilitydba</user>
<url>jdbc:postgresql://localhost:5432/USCProduction</url>
<driver>org.postgresql.Driver</driver>
</pool>
</connection>
<statement>
<sql>insert into mclient(clientcode,clientname,clientshortname,createdbyid,modifiedbyid) values(?,?,?,?,?) RETURNING clientid;</sql>
<parameter expression="//clientcode/text()" type="CHAR"/>
<parameter expression="//clientname/text()" type="CHAR"/>
<parameter expression="//clientshortname/text()" type="CHAR"/>
<parameter expression="//createdbyid/text()" type="BIGINT"/>
<parameter expression="//modifiedbyid/text()" type="BIGINT"/>
<result name="clientid" column="clientid"/>
</statement>
</dblookup>
<log level="full">
<property name="message" expression="get-property('clientid')"/>
</log>
</inSequence>
<outSequence/>
</target>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.PollInterval">5</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///home/youtility2/Desktop/smooks/out</parameter>
<parameter name="transport.vfs.FileURI">file:///home/youtility2/Desktop/smooks/in</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///home/youtility2/Desktop/smooks/failure</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.csv</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
<description/>
</proxy>
my smooke mediator is like this
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:csv="http://www.milyn.org/xsd/smooks/csv-1.2.xsd">
<resource-config selector="org.xml.sax.driver">
<resource>org.milyn.csv.CSVReader</resource>
<param name="fields">clientcode,clientname,clientshortname,createdbyid,modifiedbyid</param>
<param name="rootElementName">people</param>
<param name="recordElementName">person</param>
</resource-config>
</smooks-resource-list>
i have clientid but my company csv is same as like above if i insert into company i will get companyid how would i do this any have any idea.
I want to read some values from a file that being processed by the VFS transport.
My file is :
<sales>
<header>
<source>STORE1</source>
<target>HEADOFFICE</target>
</header>
So I use a property mediator to get the value I want using XPath. But when I print that property in Log Mediator it prints nothing. My proxy look like this :
<proxy name="Try1"
transports="vfs"
startOnLoad="true"
trace="disable">
<target>
<inSequence>
<property xmlns:ns="http://org.apache.synapse/xsd"
name="source"
expression="//header/source/text()"
scope="default"
type="STRING"/>
<log level="full">
<property xmlns:ns="http://org.apache.synapse/xsd"
name="Value_source"
expression="get-property('source')"/>
</log>
</inSequence>
</target>
<parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
<parameter name="transport.PollInterval">15</parameter>
<parameter name="transport.vfs.MoveAfterProcess">file:///Users/Target</parameter>
<parameter name="transport.vfs.FileURI">file:///Users/Source</parameter>
<parameter name="transport.vfs.MoveAfterFailure">file:///Users/Failure</parameter>
<parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
<parameter name="transport.vfs.ContentType">text/plain</parameter>
<parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
The log mediator result is
[2013-06-28 16:59:49,939] INFO - LogMediator Value_source=
So how can I get the values in file using XPath?
The issue is due to the ContentType set for the transport parameter:
Switch
<parameter name="transport.vfs.ContentType">text/plain</parameter>
with
<parameter name="transport.vfs.ContentType">application/xml</parameter>
since you're doing XPath operations on the message.
The problem is with the content of your file. Since it is not well-formed, the xpath expression fails. Isn't it possible for you to have the well-formed xml in the input content?
<sales>
<header>
<source>STORE1</source>
<target>HEADOFFICE</target>
</header>
</sales>