WSO2 MI, integration timeouts while trying to reach backend - wso2

i'm trying to build my integration with Integration Studio, it has to pass through multiple services and send the response to a backend endpoint. This works very well until it tries to call backend, then it goes on timeout state.
Here the XML
<?xml version="1.0" encoding="UTF-8"?>
<api context="/salesforce" name="SFApi" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="POST" uri-template="/anagrafiche">
<inSequence>
<property expression="$body//username" name="uri.var.username" scope="default" type="STRING"/>
<property expression="$body//password" name="uri.var.password" scope="default" type="STRING"/>
<property expression="$body//client_id" name="uri.var.client_id" scope="default" type="STRING"/>
<property expression="$body//client_secret" name="uri.var.client_secret" scope="default" type="STRING"/>
<property expression="$body//grant_type" name="uri.var.grant_type" scope="default" type="STRING"/>
<call>
<endpoint key="SFAuth"/>
<source type="body"/>
</call>
<property expression="json-eval($.access_token)" name="uri.var.accessToken" scope="default" type="STRING"/>
<header expression="fn:concat('Bearer ', $ctx:uri.var.accessToken)" name="Authorization" scope="transport"/>
<call>
<endpoint key="SFAnagrafiche"/>
</call>
<property expression="json-eval($.records)" name="uri.var.records" scope="default" type="STRING"/>
<call>
<endpoint key="BEAnagrafiche"/>
<source type="body"/>
<target type="body"/>
</call>
<respond/>
</inSequence>
<outSequence/>
<faultSequence/>
</resource>
</api>
The schema goes on until it calls BEAnagrafiche's endpoint, which has the following http REST url to a laravel project in POST method:
http://localhost/webidoo-project/public/api/anagrafiche
Here BEAnagrafiche's XML:
<?xml version="1.0" encoding="UTF-8"?>
<endpoint name="BEAnagrafiche" xmlns="http://ws.apache.org/ns/synapse">
<http method="post" uri-template="http://localhost/webidoo-project/public/api/anagrafiche">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>1.0</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</http>
<property name="data" value="$ctx:uri.var.records"/>
</endpoint>
And the backend service:
public function saveAnagrafiche(Request $request) {
return 'hello';
try {
DB::beginTransaction();
$data = $request->json();
foreach($data as $d) {
if(isset($d['attributes'])) unset($d['attributes']);
$user = new User($d);
$user->save();
}
DB::commit();
return response()->json([
'status' => true,
'message' => 'Anagrafiche salvate correttamente'
], 200);
} catch(\Exception $e) {
DB::rollback();
return response()->json([
'status' => false,
'message' => 'Errore durante il salvataggio delle anagrafiche',
'error' => $e->getMessage()
], 500);
}
}
Here below also last messages from mi's console:
passthru-http.properties:
#
# Copyright (c) 2019, WSO2 Inc. (http://www.wso2.org) All Rights Reserved.
#
# WSO2 Inc. licenses this file to you under the Apache License,
# Version 2.0 (the "License"); you may not use this file except
# in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
#
##############################################
#
# WARNING: Don't edit the file manually unless you are not using the deployment.toml file.
#
##############################################
## This file contains the configuration parameters used by the Pass-through HTTP transport
rest.dispatcher.service=__MultitenantDispatcherService
rest_uri_api_regex=\\w+://.+:\\d+/t/.*|\\w+://.+\\w+/t/.*|^(/t/).*
rest_uri_proxy_regex=\\w+://.+:\\d+/services/t/.*|\\w+://.+\\w+/services/t/.*|^(/services/t/).*
http.socket.reuseaddr=true
http.user.agent.preserve = false
worker_pool_size_max = 400
valid.max.message.size.in.bytes = 81920
worker_pool_size_core = 400
http.max.connection.per.host.port = 32767
io_buffer_size = 16384
http.server.preserve = true
http.connection.disable.keepalive = false
http.socket.timeout=600000
force.json.message.reverseProxyMode = false
worker_pool_queue_length = -1
http.headers.preserve=Content-Type
synapse.properties:
#
# Copyright (c) 2019, WSO2 Inc. (http://www.wso2.org) All Rights Reserved.
#
# WSO2 Inc. licenses this file to you under the Apache License,
# Version 2.0 (the "License"); you may not use this file except
# in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
##############################################
#
# WARNING: Don't edit the file manually unless you are not using the deployment.toml file.
#
##############################################
opentracing.enable = false
inbound.threads.core = 20
synapse.debugger.port.command = 9005
synapse.debugger.port.event = 9006
internal.https.api.port = 9154
synapse.xpath.func.extensions = org.wso2.micro.integrator.mediation.security.vault.xpath.SecureVaultLookupXPathFunctionProvider,org.wso2.micro.integrator.mediation.security.vault.external.hashicorp.HashiCorpVaultLookupXPathFunctionProvider
jaeger.sender.agent.host = localhost
synapse.commons.json.preserve.namespace = false
jaeger.reporter.log.spans = false
internal.http.api.enabled = true
inbound.port.offset.enable = false
synapse.temp_data.chunk.size = 3072
mediation.flow.statistics.enable = false
mediation.flow.statistics.collect.all = false
internal.http.api.port = 9191
synapse.threads.core = 20
opentracing.zipkin.backend.url = http://localhost:9411/api/v2/spans
jaeger.reporter.flush.interval = 10000
inbound.threads.max = 100
synapse.global_timeout_interval = 120000
jaeger.sampler.manager.host = localhost
opentracing.zipkin.enable = false
synapse.script.mediator.pool.size = 15
jaeger.reporter.max.queue.size = 100
mediation.flow.statistics.tracer.collect.properties = false
synapse.threads.max = 100
jaeger.sender.agent.port = 6831
statistics.clean.enable = true
mediation.flow.statistics.tracer.collect.payloads = false
statistics.clean.interval = 1000
jaeger.sampler.manager.port = 5778
synapse.sal.endpoints.sesssion.timeout.default=600000
synapse.carbon.ext.tenant.info=org.wso2.micro.integrator.initializer.handler.MITenantInfoConfigurator
mediation.flow.statistics.event.consume.interval=1000
mediation.flow.statistics.event.clean.interval=15000

According to the shared screenshot of the error message, it seems a socket timeout has occurred between the client and the server. The timeout has occurred only after the server has accepted request headers and body.
Have you tried invoking the backend service directly?
If it is working properly, how much time it takes to respond?
Have you observed any latency at your client-side?
Enable the wire logs as in this doc and share it.

Related

Remove activityid Parameter from response Header in WSO2- API Manager

As you know, by default, WSO2 add a "activityid" Parameter to each response header which would be passed from API-Manager.
Reference:
https://docs.wso2.com/display/EMM210/Getting+Operation+Details+by+the+Activity+ID
For some reason, I need to remove this parameter from the response for client. As I searched, I find out the following link
https://apim.docs.wso2.com/en/4.0.0/deploy-and-publish/deploy-on-gateway/api-gateway/message-mediation/removing-specific-request-headers-from-response/
So I added following code into Main.XML and Fault.XML, However it doesn't work
<property name="activityid" scope="transport" action="remove"/>
I also add the following property to the data flow of a specific API through Integrator Studio. However it doesnt work
<resource methods="POST" uri-template="/attachment">
<inSequence>
<call>
<endpoint key="attachment"/>
</call>
<send/>
</inSequence>
<outSequence>
<property action="remove" name="TRANSPORT_HEADERS" scope="default"/>
<property action="remove" name="activityid" scope="default"/>
</outSequence>
<faultSequence/>
</resource>
Anybody can Give me some advice about this matter??
You can use the message mediation for API Manager. Add the above property mediator to the response path 1 of the API.
Under runtime configurations of the API, there is a message mediation section for both request and response paths. Here you can upload a mediation sequence which will be executed in the request flow as well as response flow. You can save the mediation you need to an xml file and upload the file for the response flow.
1 - https://apim.docs.wso2.com/en/4.0.0/deploy-and-publish/deploy-on-gateway/api-gateway/message-mediation/changing-the-default-mediation-flow-of-api-requests/

How to read a property injected from file.properties in WSO2 - micro integrator?

Is there a way to read the properties from file in wso2 using property mediator?
I'm injecting an address uri from the file.properties, and passing the file path as -Dproperties.file.path argument to the startup script, it is able to resolve the varibale only within the address endpoint's uri attribute using the syntax $FILE:variableName as below:
<call>
<endpoint>
<address uri="$FILE:uploadPath">
<suspendOnFailure>
<initialDuration>-1</initialDuration>
<progressionFactor>1</progressionFactor>
</suspendOnFailure>
<markForSuspension>
<retriesBeforeSuspension>0</retriesBeforeSuspension>
</markForSuspension>
</address>
</endpoint>
</call>
But I also have the need to use the same property, do a minor transformation and pass it to a class mediator,
I've tried the below approaches and none of them are working.
<log level="custom">
<property expression="$FILE:uploadPath" name="file-path-1"/>
<property expression="$ctx:uploadPath" name="file-path-2"/>
<property expression="$axis2:uploadPath" name="file-path-3"/>
<property expression="$trp:uploadPath" name="file-path-4"/>
<property expression="$axis2:POST_TO_URI" name="POST_TO_URI"/>
</log>
Below are the versions of OS and tools that I'm using.
OS: Mac os catalina,
WSO2: micro-integrator 1.2.0, Integration sutdio - 7.1.0
Some time ago, i was struggling with similar issue, to try using $FILE: in dockered microintegrator . After checking sourcecode i realized that there is no implemented reading in property mediator, and that works only in endpoints... I belive that is because of security reason?
But, there is some workaround, which I'm not a big fan. You could use Property and ScriptMediator:
<property xmlns:ns="http://org.apache.synapse/xsd" name="propPath" expression="get-property('system','properties.file.path')" scope="default" type="STRING"/>
<script language="js"><![CDATA[
var path = java.nio.file.Paths.get(mc.getProperty('propPath'));
var fromFile = java.nio.file.Files.readAllLines(path, java.nio.charset.StandardCharsets.UTF_8);
mc.setProperty('line1',fromFile.get(0).toString());
mc.setProperty('line2',fromFile.get(1).toString());
]]></script>
Of course you got full line, but if you tweak this ScriptMediator, you can have what you want in separate properties.

WSO2 ESB FileConnector MergeFiles does not produce expected results

I am working with the WSO2 ESB 6.5.0 and am using the fileconnector-2.0.21 version for file operations. The proxy service that I'm building is intended to read the files from a certain source path and merge the contents into a file at the destination path.
The content of the proxy service artifact is below
<?xml version="1.0" encoding="UTF-8"?>
<proxy name="FileMergeProxy" startOnLoad="true" transports="http https" xmlns="http://ws.apache.org/ns/synapse">
<target>
<inSequence>
<log description="Combine files" level="custom" separator="| ">
<property name="statusMessage" value="Processing complete"/>
<property name="fileCreation" value="Initiating file merge"/>
<property expression="$ctx:mergeDirPath" name="sourceDir"/>
<property expression="$ctx:finalDirPath" name="destDir"/>
<property expression="$ctx:outputFilePattern" name="pattern"/>
<property expression="fn:concat('file:///', $ctx:mergeDirPath)" name="sourceDir1" scope="default" type="STRING"/>
property expression="fn:concat('file:///', $ctx:finalDirPath)" name="destDir1" scope="default" type="STRING"/>
</log>
<fileconnector.mergeFiles>
<source>{$ctx:mergeDirPath}</source>
<destination>{$ctx:finalDirPath}</destination>
<filePattern>{$ctx:outputFilePattern}</filePattern>
</fileconnector.mergeFiles>
<log description="Processing complete" level="custom" separator="| ">
<property name="doneMessage" value="File merge processing complete"/>
</log>
</inSequence>
<outSequence/>
<faultSequence>
<log level="custom">
<property name="text" value="An unexpected error occured"/>
<property expression="get-property('ERROR_MESSAGE')" name="message"/>
<property expression="get-property('ERROR_DETAIL')" name="errordetail"/>
</log>
<send description="Send Error Information"/>
</faultSequence>
</target>
</proxy>
I'm invoking the service with the following parameters as a JSON file
{
"mergeDirPath": "C://temp//merge//",
"finalDirPath": "C://temp//final//finalcontent.txt",
"outputFilePattern": "\\*txt"
}
When I call the service after deploying the artifacts to the ESB engine, it creates an empty file in the destination path without merging the contents of the files in the source directory. This is the extract from the WSO2 logs.
[2020-03-23 12:48:36,683] [] INFO - LogMediator To: /services/FileMergeProxy,MessageID: urn:uuid:b569ec6a-d4fe-4763-a0ca-fb2eb868a31e correlation_id : a1c542bb-7fac-4e0b-9582-8cb1a605f618,Direction: request,Payload: { "mergeDirPath": "C://temp//merge//", "finalDirPath": "C://temp//final//finalcontent.txt", "outputFilePattern": "\\*txt" }
[2020-03-23 12:49:00,422] [] INFO - LogMediator statusMessage = Combine files| fileCreation = Initiating file merge| sourceDir = C://temp//merge//| destDir = C://temp//final//finalcontent.txt| pattern = \*txt| sourceDir1 = file:///C://temp//merge//| destDir1 = file:///C://temp//final//finalcontent.txt
[2020-03-23 12:49:02,943] [] INFO - LogMediator doneMessage = File merge processing complete
Unless I'm missing something here, shouldn't the mergeFiles be doing exactly that - merge the contents of files in a specified directory ? Any helpful suggestions or pointers are welcome. Thanks in advance.
According to the proxy you have shared, you have defined properties inside the log mediator. Define the properties outside the log mediator.

How to call python script file from wso2 proxy services

How to call python script file from wso2 proxy service.
We tried with send mediator to call the python script file which is located in my local machine.
<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
name="FilepythonTest"
transports="http https"
startOnLoad="true">
<description/>
<target >
<inSequence>
<send>
<endpoint>
<address uri="local:///Users/vikashsaharan/Desktop/python/testpy.py"/>
</endpoint>
</send>
</inSequence>
<outSequence>
<log level="full"/>
</outSequence>
</target>
</proxy>
We are unable to call with this call. Please guide me how can i call python script from wso2
WSO2 EI has the inbuilt capability to execute a python script using the Script mediator. Following is a sample configuration.
**sample api configuration**
<api xmlns="http://ws.apache.org/ns/synapse" name="api" context="/api-context">
<resource methods="POST GET">
<inSequence>
<log level="full">
<property name="Message" value="Before transformation"/>
</log>
<script language="py" key="conf:/repository/script/stockquoteTransformResponse.py" function="transformRequest"/>
<log level="full">
<property name="Message" value="After transformation"/>
</log>
<respond/>
</inSequence>
</resource>
</api>
**stockquoteTransformResponse.py file saved in carbon registry.**
from org.apache.synapse.util.xpath import SynapseXPath
def transformRequest(mc):
symbolXPath = SynapseXPath("//*[local-name()='Code']/text()")
symbol = symbolXPath.stringValueOf(mc)
mc.setPayloadXML('''
<m:getQuote xmlns:m="http://services.samples">
<m:request>
<m:symbol>''' + symbol + '''</m:symbol>
</m:request>
</m:getQuote>''')
We need to add the jython jar to WSO2EI_HOME/lib directory. This was tested with jython-2.2.1.jar from http://central.maven.org/maven2/org/python/jython/2.2.1/jython-2.2.1.jar
Following output can be seen once we invoke the above api.
You can use a class mediator and execute a python script from there. Following is a sample class mediator that would do this.
public boolean mediate(MessageContext context) {
String command = "python /path/to/script.py";
try {
Process p = Runtime.getRuntime().exec(command);
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String ret = in.readLine();
System.out.println("value is : "+ret);
} catch (IOException e) {
// handle exception
}
return true;
}
You can refer Running a .py file from Java

Dynamically replace the PATH value of an Endpoint (WSO2 - API Manager)

I have created an API in the WSO2 API Manager (1.10.0) with the next info:
POST:
/regularPath/*
API URL:
http://<ip-address-1>/t/tenant.com/api/1.0.0/
HTTP-Endpoint:
http://<ip-address-2>:8181/{uri.var.newRestVar}
The issue is that the HTTP Endpoint has more than one path, e.g.:
http://<ip-address-2>:8181/mainService/cityInfo
http://<ip-address-2>:8181/country/cityInfo
http://<ip-address-2>:8181/country/dataKey/amountOfUsers
http://<ip-address-2>:8181/main/city/data/users
And I want that the resulting API URLs look like this:
http://<ip-address-1>/t/tenant.com/api/1.0.0/regularPath/mainService/cityInfo
http://<ip-address-1>/t/tenant.com/api/1.0.0/regularPath/country/cityInfo
http://<ip-address-1>/t/tenant.com/api/1.0.0/regularPath/country/dataKey/amountOfUsers
http://<ip-address-1>/t/tenant.com/api/1.0.0/regularPath/main/city/data/users
My initial approach was to use the REST_URL_PREFIX variable in order to capture the path after the 1.0.0 part of the API URL (e.g.: regularPath/country/dataKey/amountOfUsers) and then, assign that value to the uri.var.newRestVar variable.
Next, is the modified Service Bus Configuration of the API Manager (Carbon) I've created in the API Manager:
...
<inSequence>
<filter regex="PRODUCTION" source="$ctx:AM_KEY_TYPE">
<then>
<property expression="get-property('SYSTEM_TIME')" name="api.ut.backendRequestTime"/>
<log>
<property expression="$trp:To" name="ToURL"/>
</log>
<property expression="$axis2:REST_URL_POSTFIX" name="restURL" scope="default" type="STRING"/>
<log>
<property
expression="get-property('restURL')" name="logRestURL"/>
</log>
<script description="JS" language="js">
<![CDATA[
var unmodifiedRestPostfix = new String(mc.getProperty("restURL"));
print("value = " + unmodifiedRestPostfix);
unmodifiedRestPostfix = unmodifiedRestPostfix.replace("/regularPath/", "");
mc.setProperty("uri.var.newRestVar", unmodifiedRestPostfix);
]]>
</script>
<log>
<property expression="get-property('uri.var.newRestVar')" name="URI_VAR_NEWRESTVAR"/>
</log>
<send>
<endpoint name="admin-AT-tenant.com--API-Main_APIproductionEndpoint_0">
<http uri-template="http://<ip-address-2>:8181/{uri.var.newRestVar}"/>
</endpoint>
</send>
...
But it does not work. I checked the logs but everything seems to be OK:
TID: [35] [] [2016-04-11 16:47:01,955] #bank.com [35] [AM] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: local://axis2services/api/1.0.0/regularPath/mainService/cityInfo, MessageID: urn:uuid:091613ce-9fd3-4094-8638-5b112
a4214ad, Direction: request, logRestURL = /regularPath/mainService/cityInfo {org.apache.synapse.mediators.builtin.LogMediator}
a4214ad, Direction: request, URI_VAR_NEWRESTVAR = /regularPath/mainService/cityInfo {org.apache.synapse.mediators.builtin.LogMediator}
What configuration should I change in order to successfully access the HTTP-Endpoint using the paths of the API I've mentioned?
You can do the same thing without modifying the synapse configuration. Following are the steps. Hope it would be a solution for you
Create the api as following
API context : api/{version}/regularPath
Note: you can define the version in the context as a template in AM 1.10. once you define the version, above context will get the version
resources:
POST mainService/cityInfo
POST country/cityInfo
POST country/dataKey/amountOfUsers
POST main/city/data/users
HTTP Endpoint
http://<ip-address-2>:8181
After that you would be able to call a backend . No need to add custom modifications to synapse config
http://<ip-address-2>:8181/country/dataKey/amountOfUsers
with
http://<ip-address-1>/t/tenant.com/api/1.0.0/regularPath/country/dataKey/amountOfUsers
Additional information
If something went wrong, you can enable the wire logs and check what kind of request coming and going out in the gateway. see http://mytecheye.blogspot.com/2013/09/wso2-esb-all-about-wire-logs.html on how to enable wirelogs and debug using it.