WSO2 Data Service Not Binding Inputs - wso2

I've trying to create a simple data service that inserts an entry in our database with the following definition:
<data name="RESTLoggerDataService" serviceNamespace="" transports="http https">
<config id="default">
<property name="driverClassName">com.microsoft.sqlserver.jdbc.SQLServerDriver</property>
<property name="url">jdbc:sqlserver://SERVER:1433;databaseName=DB</property>
<property name="username">user</property>
<property name="password">pass</property>
</config>
<query id="CreateLog" useConfig="default">
<sql>INSERT INTO Log (Date, Username, ServiceID) VALUES (?,?,?)</sql>
<param name="Date" paramType="SCALAR" sqlType="STRING" type="IN" ordinal="1"/>
<param name="Username" paramType="SCALAR" sqlType="STRING" type="IN" ordinal="2"/>
<param name="ServiceID" paramType="SCALAR" sqlType="STRING" type="IN" ordinal="3"/>
</query>
<resource method="POST" path="Log">
<call-query href="CreateLog">
<with-param name="Date" query-param="Date"/>
<with-param name="Username" query-param="Username"/>
<with-param name="ServiceID" query-param="ServiceID"/>
</call-query>
</resource>
</data>
It seems this is not quite valid as none of the parameters I'm passing in via a CURL call are binding. I've tried:
curl -X POST -H 'Accept: application/xml' -H 'Content-Type: application/xml' --data "#test.xml" http://localhost:90/services/RESTLoggerDataService/Log
with the following XML:
<_postlog>
<Date>2020-09-15 09:06:00.000</Date>
<Username>KIRBJJ</Username>
<ServiceID>WA233</ServiceID>
</_postlog>
but get the following back:
curl: (52) Empty reply from server
curl: (52) Empty reply from server
<axis2ns19:DataServiceFault xmlns:axis2ns19="http://ws.wso2.org/dataservice"><axis2ns19:current_params>{Username=null, Date=null, ServiceID=null}</axis2ns19:current_params><axis2ns19:source_data_service><axis2ns19:data_service_name>RESTLoggerDataService</axis2ns19:data_service_name><axis2ns19:description>Exposing the data service as a REST service.</axis2ns19:description><axis2ns19:location>C:\PROGRA~1\WSO2\ENTERP~1\7154FB~1.0\MICRO-~1\bin\..\tmp\carbonapps\-1234\1600259919991RestLoggerCompositeExporter_1.0.0.car\RESTDataService_1.0.0\RESTDataService-1.0.0.dbs</axis2ns19:location><axis2ns19:default_namespace>http://ws.wso2.org/dataservice</axis2ns19:default_namespace></axis2ns19:source_data_service><axis2ns19:ds_code>DATABASE_ERROR</axis2ns19:ds_code><axis2ns19:nested_exception>com.microsoft.sqlserver.jdbc.SQLServerException: Cannot insert the value NULL into column 'Date', table 'LOGS.dbo.LogEntry'; column does not allow nulls. INSERT fails.</axis2ns19:nested_exception><axis2ns19:current_request_name>_postlog</axis2ns19:current_request_name></axis2ns19:DataServiceFault>
<axis2ns8:DataServiceFault xmlns:axis2ns8="http://ws.wso2.org/dataservice">
<axis2ns8:current_params>{}</axis2ns8:current_params>
<axis2ns8:source_data_service>
<axis2ns8:data_service_name>RESTLoggerDataService</axis2ns8:data_service_name>
<axis2ns8:description>Exposing the data service as a REST service.</axis2ns8:description>
<axis2ns8:location>C:\PROGRA~1\WSO2\ENTERP~1\7154FB~1.0\MICRO-~1\bin\..\tmp\carbonapps\-1234\1600263784524RestLoggerCompositeExporter_1.0.0.car\RESTDataService_1.0.0\RESTDataService-1.0.0.dbs</axis2ns8:location>
<axis2ns8:default_namespace>http://ws.wso2.org/dataservice/LoggingAPI</axis2ns8:default_namespace>
</axis2ns8:source_data_service>
<axis2ns8:ds_code>INCOMPATIBLE_PARAMETERS_ERROR</axis2ns8:ds_code>
<axis2ns8:current_request_name>_postlog</axis2ns8:current_request_name>
</axis2ns8:DataServiceFault>
It appears the input parameters are not binding correctly. Can anybody see what I'm doing wrong?

I have created a similar data service with MySQL and did not face any issues. Could you please try the following and see if it makes any changes. Change the payload by adding namespaces as follows and try the same mediation.
<p:_postlog xmlns:p="http://ws.wso2.org/dataservice">
<xs:Date xmlns:xs="http://ws.wso2.org/dataservice">2020-09-15 09:06:00.000</xs:Date>
<xs:Username xmlns:xs="http://ws.wso2.org/dataservice">KIRBJJ</xs:Username>
<xs:ServiceID xmlns:xs="http://ws.wso2.org/dataservice">WA233</xs:ServiceID>
</p:_postlog>
If this does not work please enable wire logs and confirm that the correct payload is passed to the data service.

Related

WSO2 MI How to create an Insert or Update PUT resource?

I want to create a PUT resource in WSO2 Micro Integrator, so I can either insert or update event records, but it's not even accepting my http request. Could you help me configure this correctly?
What I was expecting was that when I'd call something like this, it would insert in batch into the database.
curl --location --request PUT 'http://wso2-mi.mkwtf.com/services/APIDataService/eventRecord?calendar_date=2022-12-01' \
--header 'Content-Type: application/xml' \
--data-raw '
<event_record_list>
<event_record>
<event_type>1000124</event_type>
<event_count>217140</event_count>
</event_record>
<event_record>
<event_type>1000127</event_type>
<event_count>1567</event_count>
<event_record>
</event_record>
<event_type>1000129</event_type>
<event_count>31</event_count>
</event_record>
</event_record_list>'
But in reality it responds with this:
<axis2ns52:DataServiceFault xmlns:axis2ns52="http://ws.wso2.org/dataservice">
<axis2ns52:current_params>{event_type=1000124, event_count=217140}</axis2ns52:current_params>
<axis2ns52:source_data_service>
<axis2ns52:data_service_name>APIDataService</axis2ns52:data_service_name>
<axis2ns52:description></axis2ns52:description>
<axis2ns52:location>/home/wso2/qa/wso2mi/tmp/carbonapps/-1234/1671624403635APICompositeExporter_1.0.0.car/APIDataService_1.0.0/APIDataService-1.0.0.dbs</axis2ns52:location>
<axis2ns52:default_namespace>http://ws.wso2.org/dataservice</axis2ns52:default_namespace>
</axis2ns52:source_data_service>
<axis2ns52:ds_code>INCOMPATIBLE_PARAMETERS_ERROR</axis2ns52:ds_code>
<axis2ns52:current_request_name>_puteventrecord</axis2ns52:current_request_name>
</axis2ns52:DataServiceFault>
However, when I make this request, I'm able to make the request, and it returns me with 202, but I'm only able to insert or update one row at a time.
curl --location --request PUT 'http://wso2-mi.mkwtf.com/services/APIDataService/eventRecord?calendar_date=2022-12-01&event_type=1000124&event_count=217140' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'calendar_date=2022-12-01' \
--data-urlencode 'event_type=1000124' \
--data-urlencode 'event_count=217140'
Here is the Data Service definition:
<data name="APIDataService" serviceNamespace="" serviceGroup="" transports="http https local">
<description />
<config id="postgresDataService">
<property name="carbon_datasource_name">APIPostgres</property>
</config>
<query id="eventRecord" useConfig="postgresDataService">
<sql>
INSERT INTO event_record(calendar_date, event_type, event_count)
VALUES (:calendar_date, :event_type, :event_count)
ON CONFLICT (calendar_date, event_type)
DO UPDATE SET event_count = EXCLUDED.event_count
</sql>
<param name="calendar_date" sqlType="date" />
<param name="event_type" sqlType="integer" />
<param name="event_count" sqlType="integer" />
<properties>
<property name="forceJDBCBatchRequests">true</property>
</properties>
</query>
<resource method="PUT" path="eventRecord">
<call-query href="eventRecord">
<with-param name="calendar_date" query-param="calendar_date" />
<with-param name="event_type" query-param="event_type" />
<with-param name="event_count" query-param="event_count" />
</call-query>
</resource>
</data>
Due to a limitation of MI[1], you cannot use both the request body and URL params to call a dataservice. When the content type is set in the request it tries to dispatch to the data service operation based on the request body and ignores the URL params. As a result, you will observe the INCOMPATIBLE_PARAMETERS_ERROR error. Therefore you will need to pass all the required parameters of the eventRecord operation in your request body. Can you try to send a request as follows,
curl --location --request PUT 'http://wso2-mi.mkwtf.com/services/APIDataService/eventRecord' \
--header 'Content-Type: application/xml' \
--data-raw '
<event_record_list>
<event_record>
<calendar_date>2022-12-01</calendar_date>
<event_type>1000124</event_type>
<event_count>217140</event_count>
</event_record>
<event_record>
<calendar_date>2022-12-01</calendar_date>
<event_type>1000127</event_type>
<event_count>1567</event_count>
<event_record>
</event_record>
<calendar_date>2022-12-01</calendar_date>
<event_type>1000129</event_type>
<event_count>31</event_count>
</event_record>
</event_record_list>'
[1] - https://github.com/wso2/product-ei/issues/2811

Error while building Passthrough stream First Element must contain the local name, Envelope , but found alerts

Scenario
I've created a proxy service which calls an endpoint. The endpoints gives XML response. I'm using Foreach Mediator to iterate through the XML response using the defined XPath expression. Finally, each record will be inserted to the database using the DB Report Mediator.
Proxy Service
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="alertsIoscoProxy" startOnLoad="true" statistics="enable" trace="enable" transports="http https" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<send>
<endpoint>
<address uri="https://www.XXXXXX/investor_protection/investor_alerts/xml-feed">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>1</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</address>
</endpoint>
</send>
</inSequence>
<outSequence>
<foreach expression="//alerts/alert" id="foreach_alert">
<sequence>
<dbreport>
<connection>
<pool>
<driver>org.postgresql.Driver</driver>
<url>jdbc:postgresql://localhost:5432/postgres</url>
<user>postgres</user>
<password>admin</password>
</pool>
</connection>
<statement>
<sql><![CDATA[insert into IOSCO_RESPONSE (ID, DATE_POSTED, COMPANY,REGULATOR, JURISDICTION, LINK, SUBJECT) values (?,?,?,?,?,?,?)]]></sql>
<parameter expression="//id" type="VARCHAR" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"/>
<parameter expression="//datePosted" type="VARCHAR" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"/>
<parameter expression="//company" type="VARCHAR" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"/>
<parameter expression="//regulator" type="VARCHAR" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"/>
<parameter expression="//jurisdiction" type="VARCHAR" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"/>
<parameter expression="//link" type="VARCHAR" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"/>
<parameter expression="//subject" type="VARCHAR" xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"/>
</statement>
</dbreport>
</sequence>
</foreach>
<log level="custom">
<property name="add" value="Records added!!!!"/>
</log>
<sequence key="Lookup"/>
<respond/>
</outSequence>
<faultSequence/>
</target>
</proxy>
Sample XML
<alerts>
<message>Results are capped at 500 records.</message>
<alert>
<id>23091</id>
<datePosted>20221209</datePosted>
<company>Trado Banco</company>
<regulator>Financial Conduct Authority</regulator>
<jurisdiction>United Kingdom</jurisdiction>
<link>https://www.fca.org.uk/news/warnings/trado-banco</link>
<subject>Regarding fraudulent or manipulative practices (insider dealing, market manipulation, misrepresentation of material information, etc.)<br /><br />Regarding registration of issuance, offer or sale of securities/derivatives, and reporting requirements<br /><br />Regarding market intermediaries (investment and trading advisers, collective investment schemes, brokers, dealers, and transfer agents)<br /><br />Regarding markets, exchanges, and clearing and settlement entities<br /><br />Miscellaneous</subject>
<comments/>
<attachments/>
</alert>
.
.
.
</alerts>
Error
[2022-12-13 23:42:55,323] INFO {TRACE_LOGGER} - {proxy:alertsIoscoProxy} Proxy Service alertsIoscoProxy received a new message...
[2022-12-13 23:42:55,323] INFO {TRACE_LOGGER} - {proxy:alertsIoscoProxy} Message To: null
[2022-12-13 23:42:55,323] INFO {TRACE_LOGGER} - {proxy:alertsIoscoProxy} SOAPAction: null
[2022-12-13 23:42:55,323] INFO {TRACE_LOGGER} - {proxy:alertsIoscoProxy} WSA-Action: null
[2022-12-13 23:42:55,324] INFO {TRACE_LOGGER} - Setting specified anonymous fault-sequence for proxy
[2022-12-13 23:42:55,324] INFO {TRACE_LOGGER} - {proxy:alertsIoscoProxy} Using the anonymous in-sequence of the proxy service for mediation
[2022-12-13 23:42:55,878] INFO {TRACE_LOGGER} - Setting specified anonymous fault-sequence for proxy
[2022-12-13 23:42:57,175] ERROR {RelayUtils} - Error while building Passthrough stream org.apache.axiom.soap.SOAPProcessingException: First Element must contain the local name, Envelope , but found alerts
at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.constructNode(StAXSOAPModelBuilder.java:305)
at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.createOMElement(StAXSOAPModelBuilder.java:252)
at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.createNextOMElement(StAXSOAPModelBuilder.java:234)
at org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java:249)
at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java:204)
at org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.<init>(StAXSOAPModelBuilder.java:154)
at org.apache.axiom.om.impl.AbstractOMMetaFactory.createStAXSOAPModelBuilder(AbstractOMMetaFactory.java:73)
at org.apache.axiom.om.impl.AbstractOMMetaFactory.createSOAPModelBuilder(AbstractOMMetaFactory.java:79)
at org.apache.axiom.om.OMXMLBuilderFactory.createSOAPModelBuilder(OMXMLBuilderFactory.java:196)
at org.apache.axis2.builder.SOAPBuilder.processDocument(SOAPBuilder.java:65)
at org.apache.synapse.transport.passthru.util.DeferredMessageBuilder.getDocument(DeferredMessageBuilder.java:153)
at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:169)
at org.apache.synapse.transport.passthru.util.RelayUtils.buildMessage(RelayUtils.java:122)
at org.apache.synapse.transport.util.PassThroughMessageHandler.buildMessage(PassThroughMessageHandler.java:103)
at org.apache.synapse.mediators.AbstractListMediator.buildMessage(AbstractListMediator.java:155)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:96)
at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:72)
at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:158)
at org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:377)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:627)
at org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:208)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.synapse.transport.passthru.ClientWorker.run(ClientWorker.java:298)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Question
Why proxy Service is considering the response as Soap while its HTTP response with xml based formatting? How could I solve this issue?
One reason for this issue is, your response coming from https://www.XXXXXX/investor_protection/investor_alerts/xml-feed maybe having the Content-Type set to text/xml which is by default considered as a SOAP message and built with the SOAPMessageBuilder. If you can change the backend to send application/xml as the content type this would resolve the issue. Else you will have to change the Message Builder configurations to build text/xml messages with ApplicationXMLBuilder. For that you need to add the following to the deployment.toml.
[[custom_message_builders]]
content_type = "text/xml"
class="org.apache.axis2.builder.ApplicationXMLBuilder"
Note that this will affect all the SOAP 1.1 calls as well, which use text/xml content type. To avoid this you can create a custom message builder as well. More details are on the aforementioned doc.

Rule mediator integration with wso2 ei 6.2.0

I want to create a rest api where i send the json request payload and it should change the json key value based on rule defined using rule mediator
i have already tried the Sample 600 and modified with below code to achieve my requirement.
<rule xmlns:brs="http://wso2.org/carbon/rules">
<source xpath="//val"/>
<target action="replace" resultXpath="//val" xpath="//val"/>
<ruleSet>
<properties/>
<rule resourceType="regular" sourceType="inline"><![CDATA[
rule Invokeval
when
val: String()
eval(val.equals("someval"))
then update(drools.getWorkingMemory().getFactHandle(val),"WORKINGDROOL");
end
]]></rule>
</ruleSet>
<input namespace="http://services.samples" wrapperElementName="jsonObject">
<fact elementName="val" namespace="http://services.samples" type="java.lang.String" xpath="//val"/>
</input>
<output namespace="http://services.samples" wrapperElementName="jsonObject">
<fact elementName="val" namespace="http://services.samples" type="java.lang.String"/>
</output>
</rule>
getting RuleMediator invalde source value Error

save a dynamic value in wso2 esb for next use

i need to save result of web service login method as a property in wso2 esb. and use it whenever i want. token will be expired in 20 min so i want to save token in wso2 esb and when it expire regenerate it. how can i save a property dynamic in wso2 esb and retrieve it. my sequence to generate tokekn is :
<sequence name="x" xmlns="http://ws.apache.org/ns/synapse">
<payloadFactory media-type="xml">
<format>
<p:login xmlns:p="http://corebankingservice.endpoint.webservicegateway.core.channelmanager.caspian.com/">
<chUserInfoRequestBean xmlns="">
<password>****</password>
<username>user</username>
</chUserInfoRequestBean>
<channelServiceType xmlns="">***</channelServiceType>
</p:login>
</format>
</payloadFactory>
<send>
<endpoint>
<address uri="http://x.x.x.x:8280/services/..."/>
</endpoint>
</send>
</property name="SessionId" value = ...>
</sequence>
result of this sequence is SessionId so i need to save SessionId in memory or db or .... and use it for another method until it will be expired. how can i save such a this property in registery or cashe or database and use it later.
The only "official" way I know (which I used), is to use the dblookup mediator (retrieve the value later) and dbreport meditaor (to store the value) in a database.
Write value to a db:
<dbreport>
<connection>
<pool>
<dsName>datasource_name</dsName>
</pool>
</connection>
<statement>
<sql>
insert into table (attributName1, attributName2) values ('value1','value2')</sql>
</statement>
</dbreport>
The values can be passed from a property like in this example.
<sql>
insert into dbo.sais_config_db (serviceName, configKey, configValue) values ('staticVal1','staticVal2',?)</sql>
<parameter expression="get-property('myPropertyName')" type="CHAR"/>
Read value:
<dblookup>
<connection>
<pool>
<dsName>sais_config_db</dsName>
</pool>
</connection>
<statement>
<sql>
SELECT value1 FROM tableName where ....</sql>
<result name="resultFromDb" column="value1"/>
</statement>
</dblookup>
Hope that helps. There is another option, but I don't know if that works (haven't tried it).
Link
With WSO2 Enterprise Integrator 6.x.x and up, it's possible to set a property with scope "registry" that will persist the value between calls and being globally visible.
For example to persist the access token received in the JSON response:
<property description="propAccessToken" expression="json-eval($.access_token)" name="access_token" scope="registry" type="STRING"/>
Then to get the property use the following expression:
get-property('registry', 'access_token')

wso2 esb sequence DB update

I have a problem with the DBreport mediator which output a syntax error, but I can't see any error in the SQL statement
<sequence xmlns="http://ws.apache.org/ns/synapse" name="_dbcount">
<dblookup>
<connection>
<pool>
<password>1234</password>
<user>root</user>
<url>jdbc:mysql://localhost:3306/new_db</url>
<driver>com.mysql.jdbc.Driver</driver>
</pool>
</connection>
<statement>
<sql>
<![CDATA[ select * from consume where username= ? and id_api+ ?]]></sql>
<parameter value="riccardo" type="VARCHAR" />
<parameter value="1" type="INTEGER" />
<result name="result_use" column="use" />
<result name="result_user" column="username" />
</statement>
</dblookup>
<dbreport>
<connection>
<pool>
<password>1234</password>
<user>root</user>
<url>jdbc:mysql://localhost:3306/new_db</url>
<driver>com.mysql.jdbc.Driver</driver>
<property name="autocommit" value="false" />
</pool>
</connection>
<statement>
<sql>
<![CDATA[UPDATE consume SET use=21 WHERE username='riccardo' AND id_api='1']]></sql>
</statement>
</dbreport>
<log level="custom">
<property xmlns:ns="http://org.apache.synapse/xsd" xmlns:ns3="http://org.apache.synapse/xsd" name="result for client" expression="get-property('result_user')" />
<property xmlns:ns="http://org.apache.synapse/xsd" xmlns:ns3="http://org.apache.synapse/xsd" name="result for use" expression="get-property('result_use')" />
</log>
</sequence>
The first DB lookup works fine but the second SQL statement in the DBreport cannot update the value:
ERROR - DBReportMediator Error execuring insert statem
ent : UPDATE consume SET use='21' WHERE username='riccardo' AND id_api='1' again
st DataSource : jdbc:mysql://localhost:3306/new_db
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in
your SQL syntax; check the manual that corresponds to your MySQL server version
for the right syntax to use near 'use='21' WHERE username='riccardo' AND id_api=
'1'' at line 1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)...
My tables are as follows:
consume (int id, varchar username, int id_api, int use)
users (int id, varchar username, varchar password)
many thanks
EDIT I think there are major issues with DBREPORT AND DBLOOKUP mediators when writing on a Mysql DB, reading is fine, but Update and Insert just don't work...
tell me if I am wrong
Nuvio,
I think this issue arises not because of something wrong in the DBReport mediator. While going through the reported stack trace and the sequence configuration I can see you have defined "use" as a column name. I'm not quite sure how you created a database column with the term "use" because it is a MySQL keyword and MySQL query compiler should throw a SQLException whenever you have a keyword as a column or any other user defined attribute in your SQL query. Having a non-keyword name for in place of the column "use" would fix your problem.
Cheers,
Prabath