Can ZOHO deluge script getUrl() function read HTTP response headers? - amazon-web-services

Question

When trying to use getUrl() to grab a CSV file from a URL with basic .htaccess authorization, I am redirected to an Amazon S3 location. The getURL() function passes the original HTTP headers (for the auth) to Amazon S3 which Amazon thinks is an Amazon token; this causes the following error in the response:
Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified
I can't see these issues talked about anywhere other than an advisory from Thompson Reuters: https://community.developers.thomsonreuters.com/questions/29247/aws-download-x-direct-download-returns-invalid-arg.html
The fix is to receive the redirect back from the remote server, look at the response and pull out the new (redirected) URL and grab the CSV file from there without the auth details in the header.
Is there a way in deluge script ZOHO to do this? The getUrl() function seems really basic and the documentation is very thin.
The other way to do this is a 'middleware' application that can use CURL, save the CSV's on a remote server then use ZOHO getUrl() to pull these CSV files. This is not an optimal solution but unless ZOHO gives access to some HTTP client functions then I don't see another way.

Answer 1

To get the detail of the response headers include detailed:true in the invokeurl request.
Example:
// parameters is a Map
// header is a Map
response = invokeurl
[
url :url
type :POST
parameters:parameters
headers:header
detailed:true
];
// To see all headers and content
info response;
// To see the http response code
info response.get('responseCode');
// With detailed:true any html or json returned will be put in responseText
// info response.get('responseText');
// To see the all http response headers
info response.get('responseHeader');
// To see a specific http response header
// Note: case matters in the response headers name
// "Content-Type" won't find "content-type"
info response.get('responseHeader').get('content-type');
// was the url redirected to another url?
info response.get('responseHeader').get('location');
// get the redirect url
redirect_url = response.get('responseHeader').get('location')
from there you can process the redirect url and pass it to the next http request.
Recommendation:
After working for months both including detailed:true and not including it, I now lean toward always including it. detailed:true includes more useful information and has a helpful regular structure: {responseCode: <code>, responseHeaders: <headers>, responseText: <returned-data>}.

Answer 2

This is possible in Deluge using the invoke URL task - https://www.zoho.com/deluge/help/web-data/invokeurl-task.html#response.
invokeURL can hand over the response headers to you from which you can get the redirect URL and then proceed with the authentication.