How to use WSO2 ESB to mediate fixed length text data - wso2

I have the needs to use WSO2 ESB to mediate the incoming fixed length text data as received via MQ, one line of text per message, into XML format and then send the transformed data onto an SOAP endpoint via HTTP.
I understand that I could use WSO2 ESB admin console to configure an InSequence to do the data parsing and mapping with substring function and then set up a proxy service to include this configured sequenece.
I would need helps about how to do all these in details in steps. Greatly appreciate if someone could provide some examples or links to some webpages about how-to.
Thanks!

You should have a look to smooks, I think this is the best solution :
CSV : http://wso2.com/library/blog-post/2013/09/csv-to-xml-transformation-with-wso2-esb-smooks-mediator/
Fixed Lengh text : http://vvratha.blogspot.fr/2014/05/processing-large-text-file-using-smooks.html
An other solution would be to write your own messageBuilder, search "org.apache.axis2.format.PlainTextBuilder" to find the source code...

I agree with Jean-Michel, that smooks would be a good solution. But, this is also possible to do within a single simple proxy service. Set up a simple pass-through proxy to your endpoint. Then, open it up in source view (or the wizard) and configure the insequence to add a PayloadMediator.
Here is an example of how to use Payload Mediator [1]
Here is an excerpted example of what that would look like with a few xpath expressions to extract fixed-length fields from your input:
<payloadFactory media-type="xml">
<format>
<m:body xmlns:m="http://services.samples">
<m:field1>$1</m:field1>
<m:field2>$2</m:field2>
</m:body>
</format>
<args>
<arg expression="substring(//*,0,10)"/>
<arg expression="substring(//*,10,10)"/>
</args>
</payloadFactory>
You may also need to use the content type property in your sequence, because you're changing the content type to xml:
<property name="ContentType" value="text/xml" scope="axis2"/>
Best of luck!
[1] https://docs.wso2.com/pages/viewpage.action?pageId=33136018

For those who are interested in a working solution, here is my smooks configuration:
<?xml version="1.0"?>
<smooks-resource-list xmlns="http://www.milyn.org/xsd/smooks-1.1.xsd" xmlns:fl="http://www.milyn.org/xsd/smooks/fixed-length-1.3.xsd">
<fl:reader fields="price[5]?trim,quantity[5]?trim,symbol[5]?trim.upper_case,comment[10]?trim" recordElementName="order">
<fl:listBinding beanId="order" class="test.Order" />
</fl:reader>
</smooks-resource-list>
Also, need to add the jar file of test.Order to the classpath of WSO2 ESB.

Related

Configurable Values In WSO2 ESB similar to .properties file

I would like to know if there is a way to maintain certain custom configuration values in a .properties file (in Java) and load the properties at ESB startup and use the custom property values within a mediation flow? any ideas on this would be really helpful.
This might be what you are looking for:
https://dzone.com/articles/retrieve-values-xml-config (page dated ~2013)
It shows how you can have a file in the WSO2 registry and read it in your proxy.
(the file stored in the registry can contain your properties)
I would agree with the suggestion to save the properties in an XML file which you then upload to your registry, independent of the code.
That way, the properties can be updated or deleted without having to touch the code.
If you are constrained to use Java .properties files, then it might be advisable to create a JAR that is deployed as a library, which you then call from your ESB sequence.
For a similar use case, we had a custom configuration values / parameters or constants into a global_parameter.xml file and have it managed in a governance registry (decide based on your stack).
For example, the global_parameter.xml can be like the following.
<custom>
<Version>2.3</Version>
<Type>FOR</Type>
</custom>
We can load the parameters through a sequence and then use the parameters by reusing the sequence file.
<sequence xmlns="http://ws.apache.org/ns/synapse" name="loadProperties_seq">
<property xmlns:ns="http://org.apache.synapse/xsd" name="localProperties" expression="get-property('registry', 'gov:/common/utils/properties/global_parameter.xml')" scope="default" type="OM"/>
</sequence>
Refer the properties..
<property name="url_reg" expression="//custom/Version"/>
<property name="user_reg" expression="//constants/Type"/>

WSO2 : how to preserve Content-Type in http header with mediation

I'm using wso2am with 1.10.0 version.
I have a API (Content-Type: multipart/form-data)
but when request invoked, endpoint(real api server) can't receive Content-Type header.
I found a solution.
http://prabu-lk.blogspot.kr/2015/09/how-to-preserving-http-headers-in-wso2.html
but this solution is a global setting.
The reason I can not use this solution is because of the other api.
This api changes the content-type in customhandler.
So i want use mediation to be applied in units of api.
If you know how to fix it, please help me.
Have you tried to add a custom sequence to API and set the header from there?
Refer document here for more info. This should add as an in-sequence.
Your sequence should look like this.
<?xml version="1.0" encoding="UTF-8"?>
<sequence xmlns="http://ws.apache.org/ns/synapse" name="header-sequence">
<header name="Content-Type" value="multipart/form-data"/>
</sequence>

Why I am incurring in these problems performing a lot of HTTP POST request (toward a web service) in a WSO2 ESB application?

I am very new in WSO2 and I have the following problem.
I am developing an ESB application that achieve the following tasks:
Perform some DSS queries on a local database.
Use the retrieved data to build some XML documents.
Send all these documents to a remote web service (implemented in PHP).
Obtain the web service responses (a specific response for all request that contains one of these generated XML document) and use its content to store a value on the previous local databse.
The web service receiving the previous request containin an XML document stores the document content on another database.
I have implemented this ESB application, and it seems to works (but I am finding some problem that I will explain later) using the following logic:
I have created an API containing an in sequence that perform the queries, build the XML documents, send these documents to the web service, something like this (I can't post the entire code because the XML creation logic is pretty big):
<?xml version="1.0" encoding="UTF-8"?>
<api context="/glisTest2" name="glisTest2" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="GET" outSequence="glisOutSequence">
<inSequence>
<payloadFactory media-type="xml">
<format>
<body/>
</format>
<args/>
</payloadFactory>
<header name="Action" scope="default" value="urn:FindNotProcessed"/>
<log level="full"/>
<callout endpointKey="prgfasEndpoint">
<source xmlns:ns="http://org.apache.synapse/xsd" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
<target xmlns:ns="http://org.apache.synapse/xsd" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xmlns:s12="http://www.w3.org/2003/05/soap-envelope" xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
</callout>
<log level="full"/>
<property expression="count(//ds:Sample)" name="total_samples" scope="default" type="STRING" xmlns:ds="http://ws.wso2.org/dataservice"/>
<log level="custom">
<property expression="$ctx:total_samples" name="total samples: "/>
</log>
<!-- Iterate throug samples -->
<iterate expression="$body//ds:Sample" id="ITR_AGG" sequential="true" xmlns:ds="http://ws.wso2.org/dataservice">
<target sequence="sampleDataSequence"/>
<!-- This is my main sequence. It will call some other sequece in cascade: -->
</iterate>
</inSequence>
<faultSequence/>
</resource>
</api>
The callout mediator is used to obtain to call the DSS service that performs query to obtain data.
Now in this section of code:
<!-- Iterate throug samples -->
<iterate expression="$body//ds:Sample" id="ITR_AGG" xmlns:ds="http://ws.wso2.org/dataservice">
<target sequence="sampleDataSequence"/>
</iterate>
I am iterating on each retrieved element and for each element I will basically build an XML that will be send to the remote web service. All this logic (pretty big because the XML documents contains many fields) are contained into the sampleDataSequence sequence (this sequence will be performed at each iteration to build and send an XML document).
I am not attaching the entire code, at the end of the sampleDataSequence sequence I perform the web service call (a POST request) passing to it the current XML document (generated in the current iteration), in this way (previously I put the current XML document in the body request):
<property name="messageType" scope="axis2" type="STRING" value="application/xml"/>
<property name="HTTP_METHOD" scope="axis2" type="STRING" value="post"/>
<send>
<endpoint key="glisEndpoint"/>
</send>
So, for each iteration, I build a brand new XML document and I send to it to my web service (this works, the web service receive it).
So the in sequence end, then there is the out sequence that that receives and collects all the web service responses so then it can parse these response one by one and write a record into a result table on my local database.
I have done it in this way:
<?xml version="1.0" encoding="UTF-8"?>
<sequence name="glisOutSequence" trace="disable" xmlns="http://ws.apache.org/ns/synapse">
<aggregate id="ITR_AGG">
<completeCondition>
<messageCount max="-1" min="-1"/>
</completeCondition>
<onComplete expression="s11:Body/child::*[position()=1] | s12:Body/child::*[position()=1]" xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" xmlns:s12="http://www.w3.org/2003/05/soap-envelope">
<property name="it_count" scope="operation" type="STRING" value="0"/>
<!-- Iterate over responses. For correct count calculations they should
be sequential -->
<iterate expression="$body//response" id="ITR_RES" sequential="true">
</aggregate>
<log level="custom">
<property expression="$body" name="AT THE END"/>
</log>
<send/>
</sequence>
So basically I am using the 2 Enterprise Integration Pattern provided by WSO2 ESB: Splitter and Aggregator to achieve this task. It seems to me that this should be the standard solution for this type of task (when you have to send n message to an external web service and then collect and operate on the n response from this web service). Is it my architecture the standard one for this type of task?
I am asking it because now I am finding the following problem:
I have deployed this application on a test environment and we are obtaining the following problem.
The test environment have both this WSO2 ESB application and the contacted web service installed on the same laptop computer (but in the production environment these system will be on different computer because there will be many WSO2 ESB application and a single central web service that will receive message from these ESB applications).
The problem is that the WSO2 application generate a lot of XML documents (something like 1000). So it will be perform 1000 consecutive POST requests to the web service.
The web service receives all these requests (containing the 1000 XML documents that have to be inserted into another database) but after a certain number of DB inserton, the DB (Postgree) give a too many clients exception.
So this exception could be related to: connection not closes, result set not closed or something like this.
My personal opinion (it could be wrong) is that it should be a problem related to the back end of the web service and not to my WSO2 ESB application. But the person that are testing it says to me that maybe the WSO2 application doesn't close the connections.
It seems strange to me baceusa it is an error on the DB and not on the web service but I really have no idea.
Another thing is that I fix this problem putting a delay of some milliseconds before send the POST request to the web service, in this way:
<!-- Inserted a short delay to prevent flooding the GLIS server -->
<script language="js">java.lang.Thread.sleep(200);</script>
<property name="messageType" scope="axis2" type="STRING" value="application/xml"/>
<property name="HTTP_METHOD" scope="axis2" type="STRING" value="post"/>
<property name="ClientApiNonBlocking" value="true" scope="axis2" action="remove"/>
<send>
<endpoint key="glisEndpoint"/>
</send
Putting a short delay before send the request it seems to works fine. My idea is that maybe the problem is that in this test environment both the ESB application and the web service application are installed on the same laptop so the comuncation is very very quick (there is not a net in the middle) so the ESB application shot 1000 request very quick, the web service can receive it but it can't write on the database.
Could be the problem?
Another doubt is: who are testing the application says to me that maybe I am using the wrong enterprise integration pattern to achieve this taks and that I should perform the web service call and elaborate the response one by one (for each XML document). In my personal opinion this should be impossible using ESB logic because if I send a single XML to the web service, when it give me a response ESB enter in the out sequence and so the ESB application will end after processing a single document (because I think that I can't come back to the in sequence after that the out sequence is completed). Is it my reasoning correct?
So, at the end: could be a problem of the test environment (both application on the same PC), or a problem of the PHP web service back end application (result set not closed or something like this, it seems strange because using the delay it works) or could be something related to my ESB application?
My personal opinion - I do not see how "not closing connections" from WSO2 side (acts as a client for PHP-service)
may product problems between PHP and database.
I do not see any problems with integration patterns, you used in your solution - everything looks ok.
So, I think problem in PHP web-service or it's specific configuration on local test laptop.
For example java-applications are always used connection-pool to get DB connections -
and settings of this connection-pool can control number of connections to DB.
Not sure - but probably PHP also has some kind of connection-pools.
Also, database usually has many options about possible number of connections at same time.
So - it may be everything is configured by properly in production system, but configured "by default" in test laptop,
so - as result we have this kind of problem.
Anyway - we have situation then called service has some specific SLA - for example number of connections it may accept at same time.
Also, it looks like, it may happens this service may be unavailable or return error (for example because of database connection problem).
your JS delay is really dirty solution - WSO2 ESB (as any other ESB) has much better solution -
called "Guaranteed Delivery".
In WSO2 it may be implemented by followed way: instead of sending message to glisEndpoint, you should put it into
Message Store.
Then you need configure
Message Processor :
it will get messages from message store and send to endpoint.
If calling endpoint will be failed (by some reason) - it will put message back to Message Store.
There are many configuration options in Message Processor - so, you can configure for example to send only 1 message per second, or any other pattern.
Please - keep in mind - there are many types of message stores: starting from simplest In Memory Message store and finishing using ActiveMQ or RabbitMQ.
You should choose proper message store type depending from your requirements.

Customize WSO2 gateway error messages

The WSO2 Gateway returns this format of error message:
{"fault":{"code":404,"type":"Status report","message":"Not Found",
"description":"The requested resource (/account-info/1.0/) is not available."}}
But we have a standard format for all error messages that our microservices (which implement the APIs served by WSO2) return. We return a simple JSON object with two fields, error & error_description. So that we can present a consistent interface to our users, in this particular case we would like to see the WSO2 Gateway return exactly this:
{"error":"SERVICE_UNAVAILABLE","error_description":
"The requested resource (/account-info/1.0/) is not available."}
and return the HTTP status code only in the header.
Can you please tell me what I need to do to achieve this. Thank you.
These error messages are generated from default sequences shipped with WSO2 API Manager.
If the requested resource is not found in API Manager, it will execute the main sequence ($AM_HOME/repository/deployment/server/synapse-configs/default/sequences/main.xml). If you open this file, you will see that the above default error message is generated using a payload factory mediator as shown below.
<payloadFactory>
<format>
<am:fault xmlns:am="http://wso2.org/apimanager">
<am:code>404</am:code>
<am:type>Status report</am:type>
<am:message>Not Found</am:message>
<am:description>The requested resource (/$1) is not available.</am:description>
</am:fault>
</format>
<args>
<arg expression="$axis2:REST_URL_POSTFIX" />
</args>
</payloadFactory>
So if you want to customize the error message, then you need to modify the above payload factory mediator according to your need.
Likewise, if there is any faults occur, the fault sequence ($AM_HOME/repository/deployment/server/synapse-configs/default/sequences/fault.xml) will be executed. You might want to change the message format there as well.

WSO2 ESB send to multiple endpoints

Can a message be sent to multiple endpoints from within the send mediator in a proxy service?
This link from the WSO2 ESB Send Mediator documentation says under the Syntax chapter that If the message is to be sent to one or more endpoints, then the following is used:
<send>
(endpointref | endpoint)+
</send>
where the endpointref token refers to the following:
<endpoint key="name"/>
I've tried to include two endpoints under send, but the second one gets removed automatically when saving the proxy service (inside the Developer Studio or straight in the ESB Stratos interface). I did go to the Synapse page for the Send Mediator to see if they say anything special and their format says:
(endpointref | endpoint)?
Now assuming these characters represent regular expression, ? stands for 0 or 1 times, + is 1 or more times. Did WSO2 implement this extra "one or more endpoints" feature on top of Synapse Send Mediator or is it just a mistake on the documentation pages. If they did, what's the exact syntax to make it work?
Thank you!
Actually you can use Recipienlist endpoint to send a single message to multiple endpoints.
After defining recipient list store taht as localentry and provide that as endpoint key.
You can do something like this:
<send>
<endpoint key="jmsMBendpoint1"/>
</send>
<send>
<endpoint key="jmsMBendpoint2"/>
</send>
.I have used this approach and is working for me.
You can use the clone mediator to send to multiple endpoints with specifying respective endpoints as in the below configuration.
<sequence xmlns="http://ws.apache.org/ns/synapse" name="send_to_all">
<clone sequential="false">
<target endpoint="endpoint1"/>
<target endpoint="endpoint2"/>
<target endpoint="endpoint3"/>
</clone>
<drop/>
</sequence>