Combine json responses from iterate mediator in WSO2 - wso2

I have the following json which I iterate in order to get each token and id:{
"Response": {
"Data": [{
"Token": "12345",
"Code": "10148117"
}, {
"Token": "123465",
"Code": "10148118"
}]
}
}
<iterate expression="//Data" id="it1">
<target>
<sequence>
<property expression="//Data/Token" name="SaveFireToken"
scope="default" type="STRING" />
<property expression="//Data/Code" name="SaveCustCode"
scope="default" type="STRING" />
<filter regex="(.|\s)*\S(.|\s)*" source="get-property('SaveFireToken')">
<then />
<else>
<payloadFactory media-type="json">
<format>{
"OutputTimeStamp": "$1",
"OutputRequestID": "$2",
"StatusCode" :"3",
"StatusMessage" : "No token found"
}
</format>
<args>...
</args>
</payloadFactory>
<respond />
</else></filter>
<script language="js">...</script>
<payloadFactory description="Request"
requst to backend
</payloadFactory>
<call>
<endpoint>
<address uri="https://..."/>
</endpoint>
</call>
</sequence>
</target>
</iterate>
Let's say I've main 2 calls to the backend, then I get 2 responses that look like:
{"id":0000,"success":1,"failure":0,"can_ids":0,"results":[{"message_id":"00000"}]}
I've tried to aggregate them but with no success:
<aggregate id="it1">
<completeCondition>
<messageCount min="-1" max="-1" />
</completeCondition>
<onComplete expression="$body/*[1]">
<log>...<log>
</onComplete>
</aggregate>
Can someone show me where I do something wrong? I suspect that I use it wrong- xpression="$body/*[1]", or if there's way to combine the result property in the json from the responses. Also the aggregate should be inside or outside of the iterate mediator, I've seen it both ways?
EDIT:
After a while I ended up using this:
<aggregate id="it1">
<completeCondition>
<messageCount max="-1" min="-1"/>
</completeCondition>
<onComplete expression="//jsonObject/success" xmlns:ns="http://org.apache.synapse/xsd">
<log level="full">...</log>
</onComplete>
</aggregate>
I don't know if it's the best solution but it worked for me and helped me extract the values that I needed from the response and aggregate them.

My first thought when I see this question is whether the expression you have given for the iterate mediator is correct. Assuming you have included the exact json payload you are iterating, the expression should be as follow :
<iterate expression="//Response/data" id="it1">
.....
</iterate>
For your second question, your expression for aggregate mediator is correct. And you have to have the aggregate mediator outside the iterate mediator. The only issue I see is that you don't have an enclosing element to merge the individual responses. In other words, we need to have an outer array. To achieve please do something like below. (Add an enclosingElementProperty attribute to the onComplete tag)
<property name="Aggregated_Responses" scope="default">
<jsonObject />
</property>
<aggregate id="it1">
<completeCondition>
<messageCount min="-1" max="-1" />
</completeCondition>
<onComplete expression="$body/*[1]" enclosingElementProperty="Aggregated_Responses">
<log>...<log>
</onComplete>
</aggregate>
Iterate mediator processes the json payload as xml in the implementation level. Hence we need to have such workarounds to achieve this functionality. For more references please refer to this blog. https://medium.com/#nirothipanram/wso2-esb-iterate-and-aggregate-json-requests-ab5fc3617f

Related

What is the best way to iterate inside WSO2 EI?

After reading of WSO2 EI References, I still confuse about how use iterators inside a EI sequence.
In my case I have a payload like this....
{
...
"array": [
{"cpf": "12345678911"},
{"cnpj":"12345678912346"}
]
}
So I have to iterate to check if those guys exist using another web services. in order to achieve that flow, I am using the iterate mediator to split the message and then building the logic to make those validations as follows..
The code that implements this image is following:
<iterate description="" expression="//jsonObject/array" id="myid">
<target>
<sequence>
<property expression="json-eval($.array.cpf)" name="tipoCPF" scope="default" type="STRING"/>
<filter description="" xpath="boolean(get-property('tipoCPF'))">
<then>
<property expression="json-eval($.array.cpf)" name="uri.var.cpf" scope="default" type="STRING"/>
<call>
<endpoint>
<http method="get" uri-template="http://endpoint/service/{uri.var.cpf}"/>
</endpoint>
</call>
<filter regex="200" source="get-property('axis2','HTTP_SC')">
<then/>
<else>
<payloadFactory description="" media-type="json">
<format>{
"code":"400",
"error":"CPF inexistente"
}</format>
<args/>
</payloadFactory>
<property name="HTTP_SC" scope="axis2" type="STRING" value="400"/>
<respond/>
</else>
</filter>
</then>
<else>
<property expression="json-eval($.array.cnpj)" name="tipoCNPJ" scope="default" type="STRING"/>
<filter xpath="boolean(get-property('tipoCNPJ'))">
<then>
<property expression="json-eval($.array.cnpj)" name="uri.var.cnpj" scope="default" type="STRING"/>
<header name="Authorization" scope="transport" value="Basic Y29yZS5jb25zdWx0aW5nOm8xNXRyZWI="/>
<call>
<endpoint>
<http method="get" uri-template="http://endpoint/service/cnpj/{uri.var.cnpj}"/>
</endpoint>
</call>
<filter regex="200" source="get-property('axis2','HTTP_SC')">
<then/>
<else>
<payloadFactory media-type="json">
<format>{
"code":"400",
"error":"CNPJ inexistente"
}</format>
<args/>
</payloadFactory>
<property name="HTTP_SC" scope="axis2" type="STRING" value="400"/>
<respond/>
</else>
</filter>
</then>
<else>
<call>
<endpoint>
<http method="get" uri-template="http://endpoint/service/info"/>
</endpoint>
</call>
</else>
</filter>
</else>
</filter>
</sequence>
</target>
</iterate>
This iterator work as part inside of the an 'insequence'. The 'Insequence' is deigned to allow to insert new contracts information inside the Database.
Problem: after add this iterator, the service starts to make duplicated insertions inside Database. It´s looks like the iteration don´t finish in the edge of tags 'iterator'. It´s like the iteration continues to the rest of insequence.
Try: In order to solve this problem i try to add an aggregator mediator after the iterator. But or doesn't have any effect end the duplicated insert persist, or I receive an error message.
So What is the correct whey to make this iterations inside WSO2 EI?
As you have mentioned, Iteration will occur even outside the iterate tag, until the Aggregate mediator is used. To resolve this issue, you need to add the aggregate mediator inside the iterate mediator. This will stop the iteration within the iterator tag itself.
For your use case, you may need to set continueParent="true" in the IterateMediator, so that the mediation will continue after the iterate mediator for the insertion Operation in database.
Thanks for the helping Arunan!
After your answer I try to add the Aggregator as follows
The config of aggregator is following:
...
<aggregate id="NIRO">
<completeCondition>
<messageCount max="-1" min="-1"/>
</completeCondition>
<onComplete expression="//jsonObject">
<log description="" level="full" separator=";">
<property expression="json-eval($.)" name="jsonObject"/>
</log>
</onComplete>
</aggregate>
</sequence>
</target>
</iterate>
As you sad I change the iterator property 'Continue Parent' to 'true'. But the problem persists....
As proposed in the other answer, you need to
use an Aggregate mediator to close the iteration
If and only if the Iterator is set to continueParent="true"
However, I am not sure that putting it inside the <iterate> works. Here is a working solution using an Aggregate mediator right after your Iterator.
<sequence>
<iterate continueParent="true" description="" expression="//jsonObject/array" id="myid">
<target>
<your_sequence />
</target>
</iterate>
<aggregate>
<completeCondition>
<messageCount max="-1" min="-1" />
</completeCondition>
<onComplete enclosingElementProperty="//jsonObject/array" expression="/whatever/you/want"/>
</aggregate>
</sequence>
Notice expression="//jsonObject/array" you used in your Iteration, you'll need to use it in the Aggregator's enclosingElementProperty. This is how your EI will know from which iterator it should aggregate from (not 100% sure about this point, more of an empirical consideration).

Call multiple sequences in paralell and merge in payloadfactory

I have an endpoint in wso2 integrator, wich receive some parameters, call some rest apis, and produce a new response.
I created a sequence for each rest api call, wich will get the properties and make the specific call. Then with a script mediator, i'm creating a new payload with the response, and putting it in a property. Example: myResponseA, myResponseB, myResponseC.
My main endpoint have an IN sequence with only a clone mediator and a loopback tag. The clone mediator have a target for each sequence described above, like:
<clone continueParent="false" sequential="true">
<target sequence="mySequenceA">
</target>
<target sequence="mySequenceB">
</target>
</clone>
<loopback />
My main endpoint have an OUT sequence with a payloadFactory and a send tag, like:
<payloadFactory media-type="json">
<format>
<![CDATA[
{
"myResponseA": $1,
"myResponseB": $2
}
]]>
</format>
<args>
<arg expression="get-property('myResponseA')"/>
<arg expression="get-property('myResponseB')"/>
</args>
</payloadFactory>
<property name="HTTP_SC" scope="axis2" type="STRING" value="200"/>
<send/>
The problem is, the OUT sequence is called multiple times, one for each sequence in Clone mediator.
I tried to use the Aggregate mediator with no luck, 'coz my apis is restful and i don't know how to use the aggregate expression. I dont even need it, because i put the responses in different properties and read it in my payloadFactory.
How to execute my OUT sequence only one single time when all my sequences return in clone mediator? or there is another mediator i should use?
Obs.: I have to call those apis in parallel, because each one will take some time, and i will call 5~10 each time.
Can you try this as a startup :-D
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="PoCCloneAggregate"
transports="http"
startOnLoad="true"
statistics="enable"
trace="enable">
<description/>
<target>
<inSequence>
<property name="enclosing_element" scope="default">
<result xmlns=""/>
</property>
<clone continueParent="false" sequential="true">
<target>
<sequence>
<log level="custom">
<property name="CLON" value="clon 1"/>
</log>
<payloadFactory media-type="json">
<format>
{"data":
{"temperatura":"10",
"id":"1"}}
</format>
<args>
</args>
</payloadFactory>
<loopback/>
</sequence>
</target>
<target>
<sequence>
<log level="custom">
<property name="CLON" value="clon 2"/>
</log>
<payloadFactory media-type="json">
<format>
{"data":
{"temperatura":"20",
"id":"2"}}
</format>
<args>
</args>
</payloadFactory>
<loopback/>
</sequence>
</target>
<target>
<sequence>
<log level="custom">
<property name="CLON" value="clon 3"/>
</log>
<payloadFactory media-type="json">
<format>
{"data":
{"temperatura":"30",
"id":"3"}}
</format>
<args>
</args>
</payloadFactory>
<loopback/>
</sequence>
</target>
</clone>
</inSequence>
<outSequence>
<log level="full"/>
<aggregate>
<completeCondition>
<messageCount min="-1" max="-1"/>
</completeCondition>
<onComplete expression="$body/jsonObject" xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/" enclosingElementProperty="enclosing_element">
<log level="custom" separator=",">
<property name="MessageFlow" value="======================= Respuestas Agregadas. ==============="/>
</log>
<log level="full" separator=","/>
<send/>
</onComplete>
</aggregate>
</outSequence>
<faultSequence/>
</target>
</proxy>
You have to use aggragate mediator, the goal is to route each response to a single sequence containing this aggregate.
For example, if you use send mediator inside mySequenceA and mySequenceB, define the "receive" attribute to route the response in a dedicated sequence with <send receive="myResponseSequence"> (the same "myResponseSequence" must be used in mySequenceA and B).
If you use call mediator, then call a dedicated sequence with <sequence key="myResponseSequence"> (the same "myResponseSequence" must be used in mySequenceA and B)
inside myResponseSequence, use aggregate mediator :
the completeCondition can be let to default, the ESB will wait to receive the same number of response than the number of targets in your clone mediator. It gives you someting like :
<completeCondition>
<messageCount min="-1" max="-1"/>
</completeCondition>
the onComplete contains the mediation sequence that will be executed as soon as all the responses will be there : this sequence must contain the send mediator that will send a single response to the caller. The "expression" attribute on this "onComplete" node will contain an xpath that tell which node from the response must arrive inside the onComplete sequence.
inside the onComplete sequence, you can use payloadMediator to compose your single response
<onComplete xmlns:ns="http://org.apache.synapse/xsd" expression="//values">
<payloadFactory media-type="xml">
<format>
<myCustomResponse>
<result>$1</result>
</myCustomResponse>
</format>
<args>
<arg evaluator="xml" expression="//values"/>
</args>
</payloadFactory>
<send/>
</onComplete>
The fact that your api is restful should not be a problem with aggregate mediator. Perhaps the format is json and not xml : use json-eval if you want for the xpath or media-type=json" inside payloadFactory

WSO2 ESB: How to Aggregate Payloads from Nested Iterators

I'm attempting to produce a stream of comments from a Facebook page. Ultimately I'd like a response from WSO2 like this:
<comments>
<comment>
<post_id>123</post_id>
<comment_id>456</comment_id>
<from_name>Bob Brown</from_name>
<from_id>789</from_id>
<message>This is a comment on a post.</message>
<created_time>2015-12-17T15:25:30+0000</created_time>
</comment>
</comments>
I'm using the API module for WSO2 ESB to provide an abstraction layer over a Facebook page to get a simple stream of all the comments on a page after a given timestamp.
The logic I'm working on right now is taking all the posts on a given Facebook page (using the WSO2 Facebook Connector), iterating over all the posts (using an iterate mediator), checking if the post has comments (using the filter mediator), if there are comments I'm then iterating over the comments and restructuring them into a simple XML element (using the PayloadFactory mediator). This is where I'm getting stuck.
I've figured out that within an iterate mediator I can't update properties external to the iterator. My initial instinct was to enrich an external property with the comment payload generated in the second iterator as a child element, but no dice.
I'm now attempting to aggregate the outputs of the second iterator as shown below but I'm not able to aggregate the payloads:
<?xml version="1.0" encoding="UTF-8"?>
<inSequence xmlns="http://ws.apache.org/ns/synapse">
<property expression="$ctx:query.param.limit" name="limit"
scope="default" type="STRING"/>
<property expression="$ctx:query.param.since" name="since"
scope="default" type="STRING"/>
<property name="fields" scope="default" type="STRING" value="id,comments{id,message,from,created_time}"/>
<call-template target="facebook_getFeed">
<with-param name="pageId" value="{get-property('uri.var.page')}"/>
<with-param name="maxPosts" value="{get-property('limit')}"/>
<with-param name="since" value="{get-property('since')}"/>
<with-param name="fields" value="{get-property('fields')}"/>
</call-template>
<log level="custom">
<property name="S3_MESSAGE" value="Iterating through posts"/>
</log>
<property name="comment_stream" scope="default">
<comments xmlns=""/>
</property>
<iterate continueParent="true" expression="//jsonObject/data"
id="fb_api_comments_post_iterate" sequential="true" xmlns:ns1="http://cache.services">
<target>
<sequence>
<filter xmlns:ns="http://org.apache.synapse/xsd" xpath="count(//data/comments)>0">
<then>
<property expression="//data/id"
name="fb_post_id" scope="operation" type="STRING"/>
<iterate continueParent="true"
expression="//data/comments/data"
id="fb_api_comments_comment_iterate" sequential="true">
<target>
<sequence>
<log level="custom">
<property name="S3_MESSAGE" value="Transforming comment"/>
</log>
<payloadFactory media-type="xml">
<format>
<comments xmlns="">
<post_id>$1</post_id>
<comment_id>$2</comment_id>
<from_name>$3</from_name>
<from_id>$4</from_id>
<message>$5</message>
<created_time>$6</created_time>
</comments>
</format>
<args>
<arg evaluator="xml" expression="get-property('fb_post_id')"/>
<arg evaluator="xml" expression="//data/id"/>
<arg evaluator="xml" expression="//data/from/name"/>
<arg evaluator="xml" expression="//data/from/id"/>
<arg evaluator="xml" expression="//data/message"/>
<arg evaluator="xml" expression="//data/created_time"/>
</args>
</payloadFactory>
</sequence>
</target>
</iterate>
</then>
</filter>
</sequence>
</target>
</iterate>
<aggregate>
<correlateOn expression="fb_api_comments_comment_iterate"/>
<completeCondition>
<messageCount max="-1" min="-1"/>
</completeCondition>
<onComplete enclosingElementProperty="comment_stream" expression="//comments">
<loopback/>
</onComplete>
</aggregate>
</inSequence>
Any assistance would be greatly appreciated.
In your question, you mentioned property update was falling outside the iterate mediator. According to my knowledge this happens due to this reason.
Before calling each iteration cycle message context will be cloned and it will only be available in the context inside iterator target. Hence, we are unable to get the messages/properties updated inside the iterator which are in the default/synapse scope. If you want to store something global, use property with operation scope.
How solved this problem was,
Before iterator - assign the property to operation scope
Inside iterator - do modifications
Outside the iterator - assign the property to default scope (if you
want to take it back to default scope)
Use below structure as an example for the proxy service:
<!--assign the property to operation scope-->
<property name="ITERATOR_DATA_PAYLOAD"
expression="get-property('DATA_PAYLOAD')"
scope="operation"
type="OM"/>
<iterate xmlns:ns="http://org.apache.synapse/xsd"
continueParent="true"
expression="//bookstore/book"
sequential="true">
<target>
<sequence>
<!--if want to assign the property-->
<property name="DATA_PAYLOAD"
expression="get-property('operation','ITERATOR_DATA_PAYLOAD')"
type="OM"/>
</sequence>
</target>
</iterate>
<!--Outside the iterator-->
<property xmlns:ns="http://org.apache.synapse/xsd"
name="NEW_DATA_PAYLOAD"
expression="get-property('operation','ITERATOR_DATA_PAYLOAD')"
type="OM"/>
Whilst I'm happy for someone to post technically more correct answer, I bit the bullet because of time constraints and begrudgingly wrote some arbitrary JavaScript in a script mediator:
var payload = mc.getPayloadXML();
print("Starting FB Post Loop...");
var comments = new XML("<comments></comments>");
comments.comment = new XMLList();
for (var i = 0; i < payload[0].data.length(); i++){
print("Checking if post has comments...");
if(payload[0].data[i].comments.data.length() > 0){
print("Comments exist on this post...");
print("Post ID: " + payload[0].data[i].id);
for (var ii = 0; ii < payload[0].data[i].comments.data.length(); ii++){
print("Building comment entry...");
var comment = <comment/>;
comment.post_id = payload[0].data[i].id.toString();
comment.comment_id = payload[0].data[i].comments.data[ii].id.toString();
comment.from_id = payload[0].data[i].comments.data[ii].from.id.toString();
comment.from_name = payload[0].data[i].comments.data[ii].from.name.toString();
comment.message = payload[0].data[i].comments.data[ii].message.toString();
comment.created_time = payload[0].data[i].comments.data[ii].created_time.toString();
print(comment);
comments.comment += comment;
}
}
}
mc.setPayloadXML(comments);
IMHO having to fall back to custom code is fine, but often means a world of hurt from a maintainability perspective. The other side of the coin is that I'd genuinely like to leverage WSO2/Synapse mediators where possible, since that's what using the WSO2 ESB is all about!
Looking forward to reading/discussing other solutions.

WSO2ESB Parallel SOAP calls

I have the following problem:
Develop a WSDL Based Proxy that calls several different SOAP Web services in parallel and returns the responses:
<inSequence>
<log level="full"/>
<iterate xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:agg="http://www.test.ro/aggregate"
preservePayload="true"
attachPath="//soapenv:Body"
expression="//agg:AggregateRequest/agg:messageRequest">
<target>
<sequence>
<property name="messageId"
expression="//soapenv:Body/agg:messageRequest/agg:messageId[node()]"/>
<property name="endpoint"
expression="//soapenv:Body/agg:messageRequest/agg:endpoint[node()]"/>
<xslt key="CleanPayload" source="/"/>
<send>
<endpoint key="MirrorEndpoint"/>
</send>
</sequence>
</target>
</iterate>
</inSequence>
<outSequence>
<property name="resp" scope="default">
<agg:AggregateResponse xmlns:agg="http://www.test.ro/aggregate"/>
</property>
<aggregate>
<completeCondition timeout="10">
<messageCount min="-1" max="-1"/>
</completeCondition>
<onComplete xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
expression="$body/*"
enclosingElementProperty="resp">
<log level="full"/>
<send/>
</onComplete>
</aggregate>
</outSequence>
My challenges are:
how do I pass the messageId from inSequence to outSequence so I can match the response to the request (similar to local environment from Message Broker)
how do I set the endpoint key with the text from agg:endpoint
I know this may be a newbie question but I'm having a hard time finding good tutorials on this topic.
Thank you
When you set a property (scope default) inside the In Sequence, you can find back it's value inside the Out Sequence.
Set this property inside your In Sequence :
<property name="IN_MESSAGE_ID" expression="get-property('MessageID')"/>
you can use get-property('IN_MESSAGE_ID') in your Out Sequence
If you want to send a message to a dynamic address, you can set "To" header and use send mediator :
<header name="To" expression="get-property('MY_DESTINATION')"/>
<send/>
Try setting the messageId as a HTTP header property,
<header name="MessageId" value="0001" scope="transport"/>
And retrieve this header in the outSequence
<property name="MessageId" expression="get-property('MessageId')"/>
I havnt tested this out, so give a feedback on whether it works

Getting error in Aggregate Mediator?

I am using iterate mediater and aggregate mediator.
My request is:
<p:GetPersonDataOperation xmlns:p="http://tempuri.org">
<!--1 or more occurrences-->
<xs:ID xmlns:xs="http://tempuri.org">1</xs:ID>
</p:GetPersonDataOperation>
and response is :
<GetPersonDataCollection xmlns="http://tempuri.org">
<GetPersonData>
<AppInstanceID>1</AppInstanceID>
<RecordID>349</RecordID>
<ID>1</ID>
<Name>name</Name>
<LastName>lastname</LastName>
<Descr>description</Descr>
<Address>Park Street</Address>
</GetPersonData>
</GetPersonDataCollection>
If i don't use Aggregate mediator then i get the above response, But if i use Aggregate mediator i get request timeOut Exception
My in Sequence is :
<sequence xmlns="http://ws.apache.org/ns/synapse" name="GetPersonDataOperationSeq">
<iterate xmlns:xs="http://tempuri.org" xmlns:ns="http://org.apache.synapse/xsd" xmlns:p="http://tempuri.org" preservePayload="true" attachPath="//p:GetPersonData" expression="//p:GetPersonData/xs:ID" id="Iterator1">
<target>
<sequence>
<property name="ID" expression="//xs:ID" scope="default" type="STRING"/>
<payloadFactory>
<format>
<p:GetPersonData>
<xs:ID>$1</xs:ID>
</p:GetPersonData>
</format>
<args>
<arg expression="get-property('ID')"/>
</args>
</payloadFactory>
<send receive="AggregatorSeq">
<endpoint key="GetPersonDataEP"/>
</send>
</sequence>
</target>
</iterate>
</sequence>
And From The above in sequence i am redirecting to another Sequence called AggregatorSeq and my AggregatorSeq is:
<sequence xmlns="http://ws.apache.org/ns/synapse" name="AggregatorSeq">
<log level="custom">
<property name="CamHereProp" value="*******************Yes??????????????**********************************************"/>
</log>
<aggregate>
<completeCondition>
<messageCount min="-1" max="-1"/>
</completeCondition>
<onComplete xmlns:ns="http://org.apache.synapse/xsd" xmlns:p="http://tempuri.org" expression="//p:GetPersonDataCollection">
<send/>
</onComplete>
</aggregate>
</sequence>
What am i doing wrong.Looking forward to your answers.Thanks in advance
Try adding the ID of the iterator to your aggregator. In your case it should be like,
<aggregate id="Iterator1">
Also, if each of your response body starts with <GetPersonData> then you need to add that to the expression in onComplete.
<onComplete xmlns:ns="http://org.apache.synapse/xsd" xmlns:p="http://tempuri.org" expression="//p:GetPersonData">
Can you specify the issue with the code you have given. You can use <log level="full"/> to debug till what level is your configuration gets executed.