WSO2 ESB: How to Aggregate Payloads from Nested Iterators - wso2

Question

I'm attempting to produce a stream of comments from a Facebook page. Ultimately I'd like a response from WSO2 like this:
<comments>
<comment>
<post_id>123</post_id>
<comment_id>456</comment_id>
<from_name>Bob Brown</from_name>
<from_id>789</from_id>
<message>This is a comment on a post.</message>
<created_time>2015-12-17T15:25:30+0000</created_time>
</comment>
</comments>
I'm using the API module for WSO2 ESB to provide an abstraction layer over a Facebook page to get a simple stream of all the comments on a page after a given timestamp.
The logic I'm working on right now is taking all the posts on a given Facebook page (using the WSO2 Facebook Connector), iterating over all the posts (using an iterate mediator), checking if the post has comments (using the filter mediator), if there are comments I'm then iterating over the comments and restructuring them into a simple XML element (using the PayloadFactory mediator). This is where I'm getting stuck.
I've figured out that within an iterate mediator I can't update properties external to the iterator. My initial instinct was to enrich an external property with the comment payload generated in the second iterator as a child element, but no dice.
I'm now attempting to aggregate the outputs of the second iterator as shown below but I'm not able to aggregate the payloads:
<?xml version="1.0" encoding="UTF-8"?>
<inSequence xmlns="http://ws.apache.org/ns/synapse">
<property expression="$ctx:query.param.limit" name="limit"
scope="default" type="STRING"/>
<property expression="$ctx:query.param.since" name="since"
scope="default" type="STRING"/>
<property name="fields" scope="default" type="STRING" value="id,comments{id,message,from,created_time}"/>
<call-template target="facebook_getFeed">
<with-param name="pageId" value="{get-property('uri.var.page')}"/>
<with-param name="maxPosts" value="{get-property('limit')}"/>
<with-param name="since" value="{get-property('since')}"/>
<with-param name="fields" value="{get-property('fields')}"/>
</call-template>
<log level="custom">
<property name="S3_MESSAGE" value="Iterating through posts"/>
</log>
<property name="comment_stream" scope="default">
<comments xmlns=""/>
</property>
<iterate continueParent="true" expression="//jsonObject/data"
id="fb_api_comments_post_iterate" sequential="true" xmlns:ns1="http://cache.services">
<target>
<sequence>
<filter xmlns:ns="http://org.apache.synapse/xsd" xpath="count(//data/comments)>0">
<then>
<property expression="//data/id"
name="fb_post_id" scope="operation" type="STRING"/>
<iterate continueParent="true"
expression="//data/comments/data"
id="fb_api_comments_comment_iterate" sequential="true">
<target>
<sequence>
<log level="custom">
<property name="S3_MESSAGE" value="Transforming comment"/>
</log>
<payloadFactory media-type="xml">
<format>
<comments xmlns="">
<post_id>$1</post_id>
<comment_id>$2</comment_id>
<from_name>$3</from_name>
<from_id>$4</from_id>
<message>$5</message>
<created_time>$6</created_time>
</comments>
</format>
<args>
<arg evaluator="xml" expression="get-property('fb_post_id')"/>
<arg evaluator="xml" expression="//data/id"/>
<arg evaluator="xml" expression="//data/from/name"/>
<arg evaluator="xml" expression="//data/from/id"/>
<arg evaluator="xml" expression="//data/message"/>
<arg evaluator="xml" expression="//data/created_time"/>
</args>
</payloadFactory>
</sequence>
</target>
</iterate>
</then>
</filter>
</sequence>
</target>
</iterate>
<aggregate>
<correlateOn expression="fb_api_comments_comment_iterate"/>
<completeCondition>
<messageCount max="-1" min="-1"/>
</completeCondition>
<onComplete enclosingElementProperty="comment_stream" expression="//comments">
<loopback/>
</onComplete>
</aggregate>
</inSequence>
Any assistance would be greatly appreciated.

Answer 1

In your question, you mentioned property update was falling outside the iterate mediator. According to my knowledge this happens due to this reason.
Before calling each iteration cycle message context will be cloned and it will only be available in the context inside iterator target. Hence, we are unable to get the messages/properties updated inside the iterator which are in the default/synapse scope. If you want to store something global, use property with operation scope.
How solved this problem was,
Before iterator - assign the property to operation scope
Inside iterator - do modifications
Outside the iterator - assign the property to default scope (if you
want to take it back to default scope)
Use below structure as an example for the proxy service:
<!--assign the property to operation scope-->
<property name="ITERATOR_DATA_PAYLOAD"
expression="get-property('DATA_PAYLOAD')"
scope="operation"
type="OM"/>
<iterate xmlns:ns="http://org.apache.synapse/xsd"
continueParent="true"
expression="//bookstore/book"
sequential="true">
<target>
<sequence>
<!--if want to assign the property-->
<property name="DATA_PAYLOAD"
expression="get-property('operation','ITERATOR_DATA_PAYLOAD')"
type="OM"/>
</sequence>
</target>
</iterate>
<!--Outside the iterator-->
<property xmlns:ns="http://org.apache.synapse/xsd"
name="NEW_DATA_PAYLOAD"
expression="get-property('operation','ITERATOR_DATA_PAYLOAD')"
type="OM"/>

Answer 2

Whilst I'm happy for someone to post technically more correct answer, I bit the bullet because of time constraints and begrudgingly wrote some arbitrary JavaScript in a script mediator:
var payload = mc.getPayloadXML();
print("Starting FB Post Loop...");
var comments = new XML("<comments></comments>");
comments.comment = new XMLList();
for (var i = 0; i < payload[0].data.length(); i++){
print("Checking if post has comments...");
if(payload[0].data[i].comments.data.length() > 0){
print("Comments exist on this post...");
print("Post ID: " + payload[0].data[i].id);
for (var ii = 0; ii < payload[0].data[i].comments.data.length(); ii++){
print("Building comment entry...");
var comment = <comment/>;
comment.post_id = payload[0].data[i].id.toString();
comment.comment_id = payload[0].data[i].comments.data[ii].id.toString();
comment.from_id = payload[0].data[i].comments.data[ii].from.id.toString();
comment.from_name = payload[0].data[i].comments.data[ii].from.name.toString();
comment.message = payload[0].data[i].comments.data[ii].message.toString();
comment.created_time = payload[0].data[i].comments.data[ii].created_time.toString();
print(comment);
comments.comment += comment;
}
}
}
mc.setPayloadXML(comments);
IMHO having to fall back to custom code is fine, but often means a world of hurt from a maintainability perspective. The other side of the coin is that I'd genuinely like to leverage WSO2/Synapse mediators where possible, since that's what using the WSO2 ESB is all about!
Looking forward to reading/discussing other solutions.