How to identify wso2 request to backend - wso2

I tried to find a relevant source on the internet but i'm unable to find some info on this.
I want to understand how i can identify wso2 calls to a backend server.
I can set any custom http headers, but anyone can override that in cURL requests. So that can't be a solution. Any other ideas?
EDIT
If its not clear enough - When i will receive a call in my backend server (the API endpoint) then how would i identify if its coming from wso2 or from somewhere else. May be you will say 'HTTP_USER_AGENT' but that can be overridden automatically.

You can create a custom mediation sequence (per API) or a custom handler (global) to inject an http header automatically.

You have to enable wire logs for WSO2 API manager. To do this go to
/repository/conf/log4j.properties file and uncomment
#log4j.logger.org.apache.synapse.transport.http.wire=DEBUG
And restart the API Manager.
Outgoing messages from API Manager will be logged with <<symbol.
example:
[2017-08-17 10:10:49,007] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "GET /am/sample/pizzashack/v1/api/menu HTTP/1.1[\r][\n]"
[2017-08-17 10:10:49,007] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "Origin: https://10.100.5.154:9443[\r][\n]"
[2017-08-17 10:10:49,007] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "Accept: application/json[\r][\n]"
[2017-08-17 10:10:49,008] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "Referer: https://10.100.5.154:9443/store/apis/info?name=PizzaShackAPI&version=1.0.0&provider=admin[\r][\n]"
[2017-08-17 10:10:49,008] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "Accept-Encoding: gzip, deflate, br[\r][\n]"
[2017-08-17 10:10:49,008] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "Accept-Language: en-US,en;q=0.8[\r][\n]"
[2017-08-17 10:10:49,008] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "Host: localhost:9443[\r][\n]"
[2017-08-17 10:10:49,008] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "Connection: Keep-Alive[\r][\n]"
[2017-08-17 10:10:49,008] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "User-Agent: Synapse-PT-HttpComponents-NIO[\r][\n]"
[2017-08-17 10:10:49,008] DEBUG - wire HTTPS-Sender I/O dispatcher-1 << "[\r][\n]"

When a request served through WSO2 API Manager or ESB or EI, there are ways that you can log the request and response.
Please enable the wire logs in conf/log4j.properties file as follows.
#log4j.logger.org.apache.synapse.transport.http.wire=DEBUG
There are four logs types that will be printed when you enable that and will be marked by relevant mark.
Request comes to server ">>"
Request goes out to backend "<<"
Response comes to server ">>"
Response goes to client "<<"

Related

C++ library to populate SSL stream with valid HTTP requests

I am using Boost.Asio ssl streams, and got a working encrypted socket from which I can send and receive bytes.
I successfully did a GET request with the following code :
// Construct HTTP request (using vanilla std::ostream)
std::ostream request_stream(&request);
request_stream << "GET / HTTP/1.0\r\n";
request_stream << "Host: " << argv[1] << "\r\n";
...
// Send request
ssl::stream<tcp::socket> socket
boost::asio::write(socket, request);
And I would now love to find a small C++ library that would provide an easy way to get the ostream loaded with a valid HTTP request !
Since you're already using Boost.Asio, consider using Boost.Beast which is a low-level HTTP library. Example of sending a GET request using Boost.Beast:
using namespace boost::beast;
// Set up an HTTP GET request message
http::request<http::empty_body> req{http::verb::get, "/", 11};
req.set(http::field::host, "www.example.com");
req.set(http::field::user_agent, "Beast/1.0);
// Send the HTTP request to the remote host
http::write(socket, req);
The full example is here:
https://github.com/boostorg/beast/blob/master/example/http/client/sync/http_client_sync.cpp
Beast is available in Boost versions 1.66 and later. Here is the documentation page, which includes many examples:
http://www.boost.org/doc/libs/1_66_0/libs/beast/doc/html/index.html
If you really want to write the HTTP request to a std::ostream, Beast supports operator<< for HTTP messages (it is mainly there for debugging), but I think you are better off just writing it directly to the ip::tcp::socket or ssl::stream using Beast.

Callback requests at random ports in WSO2 EI 6.0

I have a proxy to an external web services being called by a schedule task in WSO2 EI 6.0. I can see the request being sent out of the ESB as an instance gets logged in ESB Analytics but it's not clear to me what is happening with the response payload.
These log messages show up in wso2carbon.log file:
TID: [-1] [] [2017-05-08 18:22:27,875] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-85 Socket Timeout : 180000 Remote Address : /XX.XX.XX.XX:35380 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1] [] [2017-05-08 18:17:27,896] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-84 Socket Timeout : 180000 Remote Address : /XX.XX.XX.XX:57507 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [-1] [] [2017-05-08 18:12:28,364] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-83 Socket Timeout : 180000 Remote Address : /XX.XX.XX.XX:50387 {org.apache.synapse.transport.passthru.SourceHandler}
XX.XX.XX.XX is the remote address where the webserices are hosted. I can understand from this the the response is getting lost because these ports are not open in the firewall hosting this WSO2 instance, but for security reasons I cannot open all ports so these random ports would always work.
I need to understand why WSO2 is expecting the response at these ports and not on the same port where the request connection was made (443 in my case), and even why a separate connection need to be established since this is a synchronous web service.
Please let me know any resource available containing more details on how the transport pass trough is affecting this.
If this is some functionality from WSO2 or Synapse is there a way to disable it?
If I end up having to open all ports in the firewall to support this, is there a way to limit the random port numbers to an specific range?
Thanks in advance.

API Manager Gateway 1.8 Incoming Connections Timing out

I am running API Manager Gateway version 1.8. Server is running CentOS 8 with Java(TM) SE Runtime Environment (build 1.7.0_67-b01). I am testing on a test and production server. To rule out differences between the servers the production server was cloned from the test server and updated to point to a separate DB, Key Manager & LDAP server. In addition it was synched with Gateway Management node.
I start up both of the Gateway servers. There are several dozen APIs deployed on each server. For testing I deployed an identical API on test and prod that point to the same backend service. The API does not require authentication so there is no token call. I execute a wget for the API directly to the Gateway worker in prod and test.
The call executes successfully in test returning a response in about 1 second.
However call to production hangs for a minute, then wget retries. Eventually after several retries the call succeeds.
I have made hundreds of calls to the service directly from command prompt on the production Gateway node and they are successful every time.
I am skipping the load balancer so all traffic in my test is via http to the Gateway server and to my backend service.
In production I see the following in the logs. The request for /csmjk followed by a 1 minute delay before the http-incoming-1 times out.
TID: [0] [AM] [2017-04-16 07:38:31,472] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-1: GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:38:31,479] DEBUG {org.apache.synapse.transport.nhttp.access} - - 10.40.1.161 - - [16/Apr/2017:07:38:31 -0500] "GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0" - - "-" "Wget/1.12 (linux-gnu)" {org.apache.synapse.transport.nhttp.access}
TID: [0] [AM] [2017-04-16 07:38:31,481] DEBUG {org.apache.synapse.transport.passthru.ServerWorker} - Starting a new Server Worker instance {org.apache.synapse.transport.passthru.ServerWorker}
TID: [0] [AM] [2017-04-16 07:39:31,547] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-1: Timeout {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:39:31,547] WARN {org.apache.synapse.transport.passthru.SourceHandler} - Connection time out after request is read: http-incoming-1 {org.apache.synapse.transport.passthru.SourceHandler}
In test I see the request followed immediately by a call to our backend service:
TID: [0] [AM] [2017-04-16 07:42:08,281] DEBUG {org.apache.synapse.transport.passthru.SourceHandler} - http-incoming-10: GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0 {org.apache.synapse.transport.passthru.SourceHandler}
TID: [0] [AM] [2017-04-16 07:42:08,286] DEBUG {org.apache.synapse.transport.nhttp.access} - - 10.40.1.161 - - [16/Apr/2017:07:42:08 -0500] "GET /csmjk/1.0/iscontentavailable/publisher/ISE.json?publisherdocumentid=10.1504/WRSTSD.2013.050791 HTTP/1.0" - - "-" "Wget/1.12 (linux-gnu)" {org.apache.synapse.transport.nhttp.access}
TID: [0] [AM] [2017-04-16 07:42:08,304] DEBUG {org.apache.synapse.transport.passthru.ServerWorker} - Starting a new Server Worker instance {org.apache.synapse.transport.passthru.ServerWorker}
TID: [0] [AM] [2017-04-16 07:42:08,394] INFO {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher} - Flushing the events from the queue 1 {org.wso2.carbon.databridge.agent.thrift.AsyncDataPublisher}
TID: [0] [AM] [2017-04-16 07:42:08,426] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - Headers : {Accept=*/*, Connection=Keep-Alive, Host=TEST:8281, User-Agent=Wget/1.12 (linux-gnu), X-JWT-Assertion=null} {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,426] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - Message context:[MessageContext: logID=57478a056938f45377e3a24e79fae0781cbfcc13f4af60aa] {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,585] INFO {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler} - End user: null, API user: null {org.wso2.carbon.apimgt.gateway.handlers.security.CCCAPIAuthenticationHandler}
TID: [0] [AM] [2017-04-16 07:42:08,601] INFO {org.apache.synapse.core.axis2.TimeoutHandler} - This engine will expire all callbacks after : 120 seconds, irrespective of the timeout action, after the specified or optional timeout {org.apache.synapse.core.axis2.TimeoutHandler}
TID: [0] [AM] [2017-04-16 07:42:08,617] DEBUG {org.apache.synapse.transport.passthru.connections.TargetConnections} - Trying to get a connection {}->http://SERVICE:1111 {org.apache.synapse.transport.passthru.connections.TargetConnections}
This has been working fine in production for many months and suddenly stopped working reliably two days ago.
Any help would be greatly appreciated.
Thanks
I have been working on this for 3 days straight now. After enabling additional debugging I determined that the last thing done prior to the timeout was a call to Google Analytics. I disabled Google Analytics Tracking and everything is back to normal now. This is also configured in my test instance. More debugging to come but at least I am back to responding to requests.
Thanks

Wso2 ESB Integration with SFDC

I am integrating WSO2 ESB 4.8.1 with SFDC.
Using SFDC connector 1.0
In WSO2 i have written the code <salesforce.logout/>, according to WSO2 Documentation they say that it closes the current connection.
<salesforce.logout/> produces below soap message which i identified in WSO2 ESB log
TID: [0] [ESB] [2016-08-30 07:55:39,442] DEBUG {org.apache.synapse.transport.http.wire} -  << "<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:urn="urn:partner.soap.sforce.com"><soapenv:Header><urn:SessionHeader><urn:sessionId>00D17000000BPGr!AQcAQDIggW.ikXtsb0Ckm8c8pKKDlF_8QN42jL31WUa6hDLOdEeNIjrYsevKW0FeZLDzlrjcDLwMni_7gYaZgNfdN4zv9Cgj</urn:sessionId></urn:SessionHeader></soapenv:Header><soapenv:Body><urn:logout></urn:logout></soapenv:Body></soapenv:Envelope>[\r][\n]" {org.apache.synapse.transport.http.wire}
But few times i am getting below error (INVALID_SESSION_ID: Invalid Session ID found in SessionHeader: Illegal Session. Session not found, missing session hash:) when <salesforce.logout/> is executed
TID: [0] [ESB] [2016-08-30 07:55:39,529] DEBUG {org.apache.synapse.transport.http.wire} -  >> "<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:sf="urn:fault.partner.soap.sforce.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><soapenv:Fault><faultcode>sf:INVALID_SESSION_ID</faultcode><faultstring>INVALID_SESSION_ID: Invalid Session ID found in SessionHeader: Illegal Session. Session not found, missing session hash: je59etMAEPM+m9VdYJb0AW==[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2016-08-30 07:55:39,529] DEBUG {org.apache.synapse.transport.http.wire} -  >> "This is expected, it can happen if the session has expired and swept away, or if the user logs out, or if its just someone trying to hack in. </faultstring><detail><sf:UnexpectedErrorFault xsi:type="sf:UnexpectedErrorFault"><sf:exceptionCode>INVALID_SESSION_ID</sf:exceptionCode><sf:exceptionMessage>Invalid Session ID found in SessionHeader: Illegal Session. Session not found, missing session hash: je59etMAEPM+m9VdYJb0AW==[\n]" {org.apache.synapse.transport.http.wire}
TID: [0] [ESB] [2016-08-30 07:55:39,529] DEBUG {org.apache.synapse.transport.http.wire} -  >> "This is expected, it can happen if the session has expired and swept away, or if the user logs out, or if its just someone trying to hack in. </sf:exceptionMessage></sf:UnexpectedErrorFault></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>[\r][\n]" {org.apache.synapse.transport.http.wire}
Is it SFDC issue/WSO2 SFDC connector issue/WSO2 ESB configuration issue.?
For upsert operation we are using configkey attribute in entire project, below is the code
<salesforce.upsert configKey="sfdc_connection_dtls">
          <allOrNone>0</allOrNone>
          <allowFieldTruncate>0</allowFieldTruncate>
          <sobjects xmlns:sfdc="sfdc">{//sfdc:sObjects}</sobjects>
        </salesforce.upsert>
So when i use <salesforce.logout/> in the respective sequence, does it closes only the current connection which is available in sequence.? or it closes all connection which are existing.?
Where ever i am using salesforce.upsert(below is the skeleton code) can i use <salesforce.logout/> after salesforce.upsert call.?
<salesforce.upsert configKey="sfdc_connection_dtls">
<!-- sobject goes here -->
</salesforce.upsert>
Looking forward for your help
Thanks,
Tejas
In WSO2 SFDC connector, there's a single SF connection created per flow per init configuration. Therefore, if you issue a logout, your subsequent requests will fail. It is not necessary to issue logout since the connection is anyway terminated at the end of the flow.

WSO2 ESB Proxy service call Backend service endpoint(WCF)(Big IP) have error 403/404

I create WSO2 ESB Proxy service for back end service. I can success call my DEV WCF service endpoint which hosted in DEV server and the endpoint is the DEV server name.
But i use same way to call the test environment endpoint, test endpoint is a Big Ip endpoints. It's seem WSO2 can't access the Big Ip endpoints.
From the debug info.(My big ip endpoint is https://hostname/201507/servicename.svc)
[2015-07-08 11:48:21,786] DEBUG - headers http-outgoing-3 >> POST /201507/servicename.svc HTTP/1.1
[2015-07-08 11:48:21,786] DEBUG - headers http-outgoing-3 >> Content-Type: application/soap+xml
[2015-07-08 11:48:21,787] DEBUG - headers http-outgoing-3 >> Transfer-Encoding: chunked
[2015-07-08 11:48:21,787] DEBUG - headers http-outgoing-3 >> Host: hostname:80
[2015-07-08 11:48:21,787] DEBUG - headers http-outgoing-3 >> Connection: Keep-Alive
[2015-07-08 11:48:21,787] DEBUG - headers http-outgoing-3 >> User-Agent: Synapse-PT-HttpComponents-NIO
It's seems WSO2 can't success post SOAP to right endpoint, have wrong port(80).
I also try to hard code the port like this https://hostname:443/201507/servicename.svc, but no luck same error.
Does anyone know the reason about this?
EDIT:
This may cause by default WSO2 using the chunked-encoding to call the back end service.
Issue: Port 80 for https. This is a bug in wso2 ESB4.8.1 (https://wso2.org/jira/browse/ESBJAVA-3336)
My issue may cause by wso2 default using HTTP 1.1 with Transfer-Encoding: chunked to call back end endpoint.
But my test environment backend servcie can't support chunking Encoding. This cause the error.
Add
<property name="DISABLE_CHUNKING" value="true" scope="axis2"/>
For Pass Through Proxy:
Content length is not right when disable chunking in ESB.
see bug in https://wso2.org/jira/browse/ESBJAVA-3297
For other:
It's working! but if your client is SOAP11 will send ContentType as "text/xml" and you back end accept "application/soap+xml". you need to add
<property name="messageType" value="application/soap+xml" scope="axis2" type="STRING"/>
<property name="ContentType" value="application/soap+xml" scope="axis2" type="STRING"/>
(default not needed property ContentType)