Why am I obtaining this strange behavior when I call an API implemented in WSO2 ESB by WSO2 API Manager? ("Accept: application/json" header issue) - wso2

Question

I am very new in WSO2 API Manager and I have the following problem registering some API (implemented using WSO2 ESB component) on an old WSO2 API Manager version 1.9.0).
The problem is the following one:
I register an API on the API Manager and in the try it tool it generate and perform a request like this:
curl -X GET --header "Accept: application/json" --header "Authorization: Bearer c86b19dac9dfd7406ebe9013373c9de9" "https://api.MY-COMPANY.org/api/dsa2/v1.0.0/market/12?lang=1"
This request is sent to the endpoint representing the ESB API implementation (I see it in the log). This ESB implementation contains a script mediator that do some works on a JSON object that the API will return to the user.
I am attaching the code fo my API for clarity:
<?xml version="1.0" encoding="UTF-8"?>
<api context="/market" name="market_details" xmlns="http://ws.apache.org/ns/synapse">
<resource methods="GET" protocol="http" uri-template="/{marketId}?lang={lang_id}">
<inSequence>
<log level="full"/>
<property expression="get-property('uri.var.marketId')" name="marketId" scope="default" type="STRING"/>
<property expression="get-property('uri.var.lang_id')" name="lang_id" scope="default" type="STRING"/>
<log level="custom">
<property expression="$ctx:marketId" name="Market ID"/>
<property expression="$ctx:lang_id" name="Lang ID"/>
</log>
<property name="messageType" scope="axis2" type="STRING" value="application/xml"/>
<payloadFactory media-type="xml">
<format>
<ds:GetMarketDetails xmlns:ds="http://ws.wso2.org/dataservice">
<ds:market_id>$1</ds:market_id>
<ds:language_id>$2</ds:language_id>
</ds:GetMarketDetails>
</format>
<args>
<arg evaluator="xml" expression="$ctx:marketId"/>
<arg evaluator="xml" expression="$ctx:lang_id"/>
</args>
</payloadFactory>
<header name="Action" scope="default" value="urn:GetMarketDetails"/>
<call>
<endpoint key="agrimarketprice_Endpoint"/>
</call>
<property name="messageType" scope="axis2" type="STRING" value="application/json"/>
<property expression="json-eval($.)" name="JSONPayload" scope="default" type="STRING"/>
<script language="js"><![CDATA[var log = mc.getServiceLog();
// Problem is: null values returned from DSS as #nil=true and converted as Object into JSON.
// See https://wso2.org/jira/browse/ESBJAVA-4467 as example of bug, reported into JIRA
// So, we need convert values, what may be nulls with using this function
function checkForNull(value) {
if (value instanceof Object && "#nil" in value) {
return null;
}
return value;
}
// stange workaround for getting JSON Payload. getPayloadJSON returned null.
var pl_string = mc.getProperty("JSONPayload");
log.info("PAYLOAD STRING: " + pl_string);
var payload = JSON.parse(pl_string);
// create new response
var response = payload.Markets.Market;
//log.info("RESPONSE: " + JSON.stringify(response));
//response.id = mc.getProperty("MarketId");
response.id = mc.getProperty("marketId");
// convert null values
response.id = checkForNull(response.id);
response.regione = checkForNull(response.regione);
response.province = checkForNull(response.province);
response.city = checkForNull(response.city);
response.district = checkForNull(response.district);
response.town = checkForNull(response.town);
response.village = checkForNull(response.village);
if(response.commodities && response.commodities.commoditiesList) {
// convert array of commodities into required HATEOS format
var commodity = new Array();
for (i = 0; i < response.commodities.commoditiesList.length; ++i) {
var el = response.commodities.commoditiesList[i];
var newEl = new Object();
newEl.commodity_details_id = checkForNull(el.commodity_details_id);
newEl.commodity_name_en = checkForNull(el.commodity_name_en);
newEl.commodity_name = checkForNull(el.commodity_name);
newEl.description = checkForNull(el.description);
newEl.image_link = checkForNull(el.image_link);
newEl.market_commodity_details_id = checkForNull(el.market_commodity_details_id);
newEl.price_series_id = checkForNull(el.price_series_id);
newEl.last_price_date = checkForNull(el.last_price_date);
newEl.last_avg_price = checkForNull(el.last_avg_price);
newEl.currency = checkForNull(el.currency);
newEl.measure_unit = checkForNull(el.measure_unit);
commodityDetailsLinks = [];
commodityDetailsRefObj = {};
commodityDetailsRefObj.rel = "commodity_details";
commodityDetailsRefObj.href = "http://5.249.148.180:8280/commodity_details/" + checkForNull(el.commodity_details_id);
commodityDetailsRefObj.type = "GET";
commodityDetailsLinks.push(commodityDetailsRefObj);
newEl.links = commodityDetailsLinks;
commodity.push(newEl);
}
response.commodities.commoditiesList = commodity;
}
log.info("PROCESSED RESPONSE:");
//var jsonStr = JSON.stringify(response, null, 2); // spacing level = 2
//log.info(jsonStr);
selfLinks = [];
selfsRefObj = {};
selfsRefObj.rel = "self";
selfsRefObj.href = "http://5.249.148.180:8280/market/" + checkForNull(response.id);
selfsRefObj.type = "GET";
selfLinks.push(selfsRefObj);
response.links = selfLinks;
delete response.id;
var jsonStr = JSON.stringify(response, null, 2); // spacing level = 2
log.info("RESPONSE PROCESSED:");
log.info(jsonStr);
// put payload back
mc.setPayloadJSON(response);
log.info("This is the end");]]></script>
<property name="RESPONSE" scope="default" type="STRING" value="true"/>
<header action="remove" name="To" scope="default"/>
<send/>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence/>
</resource>
</api>
In the JavaScript mediator happens something very strange.
Calling my API as done by the API Manager try it tool (IMPORTANT: the problem seems to be the --header "Accept: application/json" header):
curl -X GET --header "Accept: application/json" --header "Authorization: Bearer c86b19dac9dfd7406ebe9013373c9de9" "https://api.MY-COMPANY.org/api/dsa2/v1.0.0/market/12?lang=1"
I obtain this wrong JSON as value of the payload value (the value of my pl_string):
{
"Markets": {
"Market": [{
"market_id": 12,
"market_name": "Kouthiaba",
"market_description": "Kouthiaba Market",
"localization_id": "2",
"long": null,
"lat": null,
"country": "Senegal",
"regione": null,
"province": null,
"city": null,
"district": null,
"town": null,
"village": null,
"commodities": {
"commoditiesList": [{
"commodity_details_id": 43,
"commodity_name_en": "Millet",
"commodity_name": "Millet",
"description": "Millet",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fmillet.png?alt=media&token=ff9c67ab-e07a-4097-95ae-826fe1aa49ed",
"market_commodity_details_id": 178,
"price_series_id": 2494,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "150.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 13,
"commodity_name_en": "Sorghum",
"commodity_name": "Sorghum",
"description": "Sorghum",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fsorghum.png?alt=media&token=ae33f8e8-50c4-4b8e-868f-1997d50d7ad4",
"market_commodity_details_id": 179,
"price_series_id": 2495,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "160.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 11,
"commodity_name_en": "Maize",
"commodity_name": "Maize",
"description": "Maize is a staple food in many parts of the world, with a total production of 1040M tonnes. However, not all of the maize produced is for human consumption but it is also utilised in bio fuel production. Some of the maize produced is used for corn ethanol, animal feed and other maize products such as corn-starch and corn syrup. Maize for human consumption is used in five different forms 1) Popcorn 2) Flint corn 3) Dent corn 4) Floury corn and 5) Sweet corn.",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fmaize.png?alt=media&token=34d5a149-1721-47e9-863b-c939c7fd7419",
"market_commodity_details_id": 180,
"price_series_id": 2496,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "125.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 38,
"commodity_name_en": "Rice Ordinary",
"commodity_name": "Rice Ordinary",
"description": "Rice Ordinary",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Friz.png?alt=media&token=c35e7648-1793-423b-acd2-52d8a1e58c53",
"market_commodity_details_id": 181,
"price_series_id": 2497,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "285.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 39,
"commodity_name_en": "Rice Perfumed",
"commodity_name": "Rice Perfumed",
"description": "Rice Perfumed",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Friz.png?alt=media&token=c35e7648-1793-423b-acd2-52d8a1e58c53",
"market_commodity_details_id": 182,
"price_series_id": 2498,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "450.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 40,
"commodity_name_en": "Black Eyed Pea",
"commodity_name": "Black Eyed Pea",
"description": "Black Eyed Pea",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fblack_eyed_pea.png?alt=media&token=ab397785-68da-413a-978b-e0ebab8407b4",
"market_commodity_details_id": 183,
"price_series_id": 2499,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "325.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 42,
"commodity_name_en": "Peanut with Shell",
"commodity_name": "Peanut with Shell",
"description": "Peanut with Shell",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fpeanut.png?alt=media&token=6d5ded68-3126-44df-b429-89fe32483d2d",
"market_commodity_details_id": 184,
"price_series_id": 2500,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "175.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 41,
"commodity_name_en": "Peanut without Shell",
"commodity_name": "Peanut without Shell",
"description": "Peanut without Shell",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fpeanut_open.png?alt=media&token=d5af0a5c-6327-418e-9fac-19d34fcedaf5",
"market_commodity_details_id": 185,
"price_series_id": 2501,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": "325.0000",
"currency": "XOF",
"measure_unit": "kilogram"
}
]
}
}
]
}
}
This JSON broke my script because I have:
{
"Markets": {
"Market": [
..................
]
}
}
instead (as expected):
{
"Markets": {
"Market": {
................
}
}
}
The strange thing is that if I perform the same call to my API manager (using cURL in my shell) without set the --header "Accept: application/json" header, in this way:
curl -X GET --header "Authorization: Bearer c86b19dac9dfd7406ebe9013373c9de9" "https://api.MY-COMPANY.org/api/dsa/v1.0.0/market/12?lang=1"
I am obtaining the correct JSON and my script is not broken, infact my JSON payload (the value of my pl_string) now is:
{
"Markets": {
"Market": {
"market_id": 12,
"market_name": "Kouthiaba",
"market_description": "Kouthiaba Market",
"localization_id": 2,
"long": {
"#nil": "true"
},
"lat": {
"#nil": "true"
},
"country": "Senegal",
"regione": {
"#nil": "true"
},
"province": {
"#nil": "true"
},
"city": {
"#nil": "true"
},
"district": {
"#nil": "true"
},
"town": {
"#nil": "true"
},
"village": {
"#nil": "true"
},
"commodities": {
"commoditiesList": [{
"commodity_details_id": 43,
"commodity_name_en": "Millet",
"commodity_name": "Millet",
"description": "Millet",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fmillet.png?alt=media&token=ff9c67ab-e07a-4097-95ae-826fe1aa49ed",
"market_commodity_details_id": 178,
"price_series_id": 2494,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 150.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 13,
"commodity_name_en": "Sorghum",
"commodity_name": "Sorghum",
"description": "Sorghum",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fsorghum.png?alt=media&token=ae33f8e8-50c4-4b8e-868f-1997d50d7ad4",
"market_commodity_details_id": 179,
"price_series_id": 2495,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 160.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 11,
"commodity_name_en": "Maize",
"commodity_name": "Maize",
"description": "Maize is a staple food in many parts of the world, with a total production of 1040M tonnes. However, not all of the maize produced is for human consumption but it is also utilised in bio fuel production. Some of the maize produced is used for corn ethanol, animal feed and other maize products such as corn-starch and corn syrup. Maize for human consumption is used in five different forms 1) Popcorn 2) Flint corn 3) Dent corn 4) Floury corn and 5) Sweet corn.",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fmaize.png?alt=media&token=34d5a149-1721-47e9-863b-c939c7fd7419",
"market_commodity_details_id": 180,
"price_series_id": 2496,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 125.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 38,
"commodity_name_en": "Rice Ordinary",
"commodity_name": "Rice Ordinary",
"description": "Rice Ordinary",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Friz.png?alt=media&token=c35e7648-1793-423b-acd2-52d8a1e58c53",
"market_commodity_details_id": 181,
"price_series_id": 2497,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 285.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 39,
"commodity_name_en": "Rice Perfumed",
"commodity_name": "Rice Perfumed",
"description": "Rice Perfumed",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Friz.png?alt=media&token=c35e7648-1793-423b-acd2-52d8a1e58c53",
"market_commodity_details_id": 182,
"price_series_id": 2498,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 450.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 40,
"commodity_name_en": "Black Eyed Pea",
"commodity_name": "Black Eyed Pea",
"description": "Black Eyed Pea",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fblack_eyed_pea.png?alt=media&token=ab397785-68da-413a-978b-e0ebab8407b4",
"market_commodity_details_id": 183,
"price_series_id": 2499,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 325.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 42,
"commodity_name_en": "Peanut with Shell",
"commodity_name": "Peanut with Shell",
"description": "Peanut with Shell",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fpeanut.png?alt=media&token=6d5ded68-3126-44df-b429-89fe32483d2d",
"market_commodity_details_id": 184,
"price_series_id": 2500,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 175.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}, {
"commodity_details_id": 41,
"commodity_name_en": "Peanut without Shell",
"commodity_name": "Peanut without Shell",
"description": "Peanut without Shell",
"image_link": "https://firebasestorage.googleapis.com/v0/b/my-company-digital-services-portfolio.appspot.com/o/img%2Ficons%2Fagrimarket%2Fcommodity%2Fpeanut_open.png?alt=media&token=d5af0a5c-6327-418e-9fac-19d34fcedaf5",
"market_commodity_details_id": 185,
"price_series_id": 2501,
"last_price_date": "2018-03-18+01:00",
"last_avg_price": 325.0000,
"currency": "XOF",
"measure_unit": "kilogram"
}
]
}
}
}
}
and this not broke my script.
So, my questions are:
1) Why setting this header to my call my JSON (that is obtained by the result of a DSS query call converted in JSON format) is different?
2) Can I say tO WSO2 API MANAGER to avoid so set this ****--header "Accept: application/json"** header in the call?
Practically the best thing to do is that my API manager perform a call like this:
curl -X GET --header "Authorization: Bearer c86b19dac9dfd7406ebe9013373c9de9" "https://api.MY-COMPANY.org/api/dsa/v1.0.0/market/12?lang=1"
3) At the moment my API manager is performing this call:
curl -X GET --header "Accept: application/json" --header "Authorization: Bearer c86b19dac9dfd7406ebe9013373c9de9" "https://api.MY-COMPANY.org/api/dsa/v1.0.0/market/12?lang=1"
I think that this --header "Accept: application/json" header is propagated to the call to the ESB implementation of my API (correct me if I am doing wrong assertion).
In case can I remove this header into my API definition at the beginning of my API flow? Could be an idea?

Answer 1

the problem seems to be the --header "Accept: application/json" header
I assume you are trying the APIs out from the API Store UI, where the json response type is default. You can change it default value (or set list of supported values) in the the API Publusher's definition filling the response types for each resource
1) Why setting this header to my call my JSON (that is obtained by the result of a DSS query call converted in JSON format) is different?
I believe DSS uses different library (or library version) to convert XML to JSON. According to my experience try to avoid this conversion if possible (as XML doesn't explicitly contain value type information)
you can even disable this automatic conversion somewhere in axis2. xml (for API, DSS and ESB)
2) Can I say tO WSO2 API MANAGER to avoid so set this ****--header "Accept: application/json"** header in the call? Practically the best thing to do is that my API manager
as already mentioned you can set the default or supported values in the Publisher. The set response types are used only by the API Store, the clients may send anything..
3) At the moment my API manager is performing this call:
... --header "Accept: application/json" header is propagated to the call to the ESB implementation of my API (correct me if I am doing wrong assertion).
all headers (except Authorization) are propagated