In a web service (web api) setting, what is the way to prove that a digitally signed response is related to the particular request. For example, I send some parameters and the response returns "true" or "false" but I want to prove that the response is linked to the parameters that I sent. The only way I can think of is that the response should include these parameters. Is there another option?
You have to link the request to the response including the request itself (or the main parts) or simply a hash of the URL and parameters. For example SHA256 (path +param1 + params2 + param3) depending on the data you want to preserve. The digital signature protects the integrity of the response in both cases.
If you include a hash in response you need also to store the request, or at least the parameters of the request, to be able to verify that the hash corresponds to the parameters sent. Include also a identifier to link the request and response in an easy way
Related
In my postman mock server I would like to use data from the request. Is this possible? I can't seem to find any reference to this scenario.
For example, my request includes a documentId value. I would like to capture that value and use it in the response.
Thanks.
Postman supports capture of URL path parameters for use in the response body, so e.g. if your example has https://my.example.com/v1/users/{{user_id}} in the URL, then you can use {{user_id}} in the response.
That's about as far as it goes though. You can't at present use data from query parameters, headers or the request body in your responses.
If you need to use other types of request data in your mock responses, you might want to check out MockLab. I've written up a detailed comparison of Postman mock servers and MockLab including a specific on dynamic responses and request data.
Update: Collected my thoughts better
I'm generating a unique identifier (UUID) for each user in the Viewer Request Lambda, and then selecting a cached page to return based upon that UUID. This works.
Ideally, this user would always have the same UUID.
I must generate that UUID in the Viewer Request if it is not present in a cookie on that Viewer Request. I also need that UUID to be set as a cookie, which of course happens in the response not the request.
Without caching, my server simply handles taking a custom header and creating a Set-Cookie in the response header.
I am not finding a way to handle this if I want to cache the page. I can ignore the request header for caching and serve the correct cached page, but then the user does not persist that UUID as no cookie is set to be utilized in their next request.
Has anyone accomplished something like this?
Things I'm trying
There are a few angles I'm working on with this, but haven't been able to get to work yet:
Some sort of setting in Cloudfront I'm unaware of that handles the header or other data pass-through from Viewer Request to Viewer Response, which could be used in a second lambda in Cloudfront.
Modify the response object headers preemptively in the Viewer Request. I don't think this is possible, as they return headers are not yet created, unless there's some built-in Cloudfront methodology I'm missing.
An existing pass-through header of some sort, I don't know if that's even a thing since I'm not intimately familiar with this aspect of request-response handling, but worth a shot.
Possibly (haven't tried yet though) I could create the entire response object in the Client Request lambda and somehow serve the cached page from there, modifying the response headers then passing it into the callback method.
Tobin's answer actually works, but is not a solid solution. If the user is not storing or serving their cookies it becomes an infinite loop, plus I'd rather not throw a redirect up in front of all of my pages if I can avoid it
Somewhat-working concept
Viewer Request Lambda, when UUID not present in cookies, generates UUID
Viewer Request Lambda sets UUID in cookies on header in request object. Callback with updated request object passed in
Presence of UUID cookie busts Cloudfront cache
Origin Request Lambda is triggered with UUID present
Origin Request Lambda calls original request URL again via http.get with UUID cookie set (40KB limit makes doing this in the Viewer Request Lambda impractical)
Second scenario for Viewer Request Lambda, seeing UUID now present, strips the UUID cookie then continues the request normally
Second Origin Request if not yet cached - Cached response if cached, as cache-busting UUID is not present - returns actual page HTML to First Origin Request
First Origin Request receives response from http.get containing HTML
First Origin Request creates custom response object containing response body from http.get and Set-Cookie header set with our original UUID
Subsequent calls, having the UUID already set, will strip the UUID from the cookie (to prevent cache busting) and skip directly to the second-scenario in the Viewer Request Lambda which will directly load the cached version of the page.
I say "somewhat" because when I try to hit my endpoint, I get a binary file downloaded.
EDIT
This is because I was not setting the content-type header. I now have only a 302 redirect problem... if I overcome this I'll post a full answer.
Original question
I have a function on the Viewer Request that picks an option and sets some things in the request before it's retrieved from the cache or server.
That works, but I want it to remember that choice for future users. The thought is to simply set a cookie I can read the next time that user comes through. As this is on the Viewer Request and not the Viewer Response I haven't figured out how to make that happen, or if it even is possible via the Lambda itself.
Viewer Request ->
Lambda picks options (needs to set cookie) ->
gets corresponding content ->
returns to Viewer with set-cookie header intact
I have seen the examples and been able to set cookies successfully in the Viewer Response via a Lambda. That doesn't help me much as the decision needs to be made on the request. Quite unsurprisingly adding this code into the Viewer Request shows nothing in the response.
I would argue that the really correct way to set a nonexistent cookie would be to return a 302 redirect to the same URI with Set-Cookie, and let the browser redo the request. This probably would not have much of an impact since the browser can reuse the same connection to "follow" the redirect.
But if you insist on not doing it that way, then you can inject the cookie into the request with your Viewer Request trigger and then emit a Set-Cookie with the same value in your Viewer Response trigger.
The request object, in a viewer response event, can be found at the same place where it's found in the original request event, event.Records[0].cf.request.
In a viewer-response trigger, this part of the structure contains the "request that CloudFront received from the viewer and that might have been modified by the Lambda function that was triggered by a viewer request event."
Use caution to ensure that you handle the cookie header correctly. The Cookie request header requires careful and accurate manipulation because the browser can use multiple formats when multiple cookies exist.
Once upon a time, cookies were required to be sent as a single request header.
Cookie: foo=bar; buzz=fizz
Parse these by splitting the values on ; followed by <space>.
But the browser may also split them with multiple headers, like this:
Cookie: foo=bar
Cookie: buzz=fizz
In the latter case, the array event.Records[0].cf.request.headers.cookie will contain multiple members. You need to examine the value attribute of each object in that array, check for multiple values within each, as well as accommodating the fact that the array will be completely undefined (not empty) if no cookies exist.
Bonus: Here's a function I wrote, that I believe correctly handles all cases including the case where there are no cookies. It will extract the cookie with the name you are looking for. Cookie names are case-sensitive.
// extract a cookie value from request headers, by cookie name
// const my_cookie_value = extract_cookie(event.Records[0].cf.request.headers,'MYCOOKIENAME');
// returns null if the cookie can't be found
// https://stackoverflow.com/a/55436033/1695906
function extract_cookie(headers, cname) {
const cookies = headers['cookie'];
if(!cookies)
{
console.log("extract_cookie(): no 'Cookie:' headers in request");
return null;
}
// iterate through each Cookie header in the request, last to first
for (var n = cookies.length; n--;)
{
// examine all values within each header value, last to first
const cval = cookies[n].value.split(/;\ /);
const vlen = cval.length;
for (var m = vlen; m--;)
{
const cookie_kv = cval[m].split('=');
if(cookie_kv[0] === cname)
{
return cookie_kv[1];
}
} // for m (each value)
} // for n (each header)
// we have no match if we reach this point
console.log('extract_cookie(): cookies were found, but the specified cookie is absent');
return null;
}
Are you able to add another directory: with the first cookie setter request, return (from the lambda) a redirect which includes the cookie-set header, that redirects to your actual content?
OK, long way round but:
Take cookie instruction from the incoming request
Set this somewhere (cache, etc)
Let the request get your object
on the Response, also call a function that reads the (cache) and sets the set-cookie header on the response if needed?
It's been more than one year since the question was published. I hope you found a solution and you can share it with us!
I am facing the same problem and I've thinking also about the infinite loop... What about this?
The viewer request event sends back a 302 response with the cookie set, e.g. uuid=whatever and a GET parameter added to the URL in the Location header, e.g. _uuid_set_=1.
In the next viewer request where the GET parameter _uuid_set_ is set (and equals 1, but this is not needed), there will be two options:
Either the cookie uuid is not set, in which case you can send back a response 500 to break the loop, or whatever fits your needs,
or the cookie is set, in which case you send another 302 back with the parameter _uuid_set_ removed, so that it is never seen by the end user and cannot be copy-pasted and shared and we all can sleep at night.
When trying to use getUrl() to grab a CSV file from a URL with basic .htaccess authorization, I am redirected to an Amazon S3 location. The getURL() function passes the original HTTP headers (for the auth) to Amazon S3 which Amazon thinks is an Amazon token; this causes the following error in the response:
Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified
I can't see these issues talked about anywhere other than an advisory from Thompson Reuters: https://community.developers.thomsonreuters.com/questions/29247/aws-download-x-direct-download-returns-invalid-arg.html
The fix is to receive the redirect back from the remote server, look at the response and pull out the new (redirected) URL and grab the CSV file from there without the auth details in the header.
Is there a way in deluge script ZOHO to do this? The getUrl() function seems really basic and the documentation is very thin.
The other way to do this is a 'middleware' application that can use CURL, save the CSV's on a remote server then use ZOHO getUrl() to pull these CSV files. This is not an optimal solution but unless ZOHO gives access to some HTTP client functions then I don't see another way.
To get the detail of the response headers include detailed:true in the invokeurl request.
Example:
// parameters is a Map
// header is a Map
response = invokeurl
[
url :url
type :POST
parameters:parameters
headers:header
detailed:true
];
// To see all headers and content
info response;
// To see the http response code
info response.get('responseCode');
// With detailed:true any html or json returned will be put in responseText
// info response.get('responseText');
// To see the all http response headers
info response.get('responseHeader');
// To see a specific http response header
// Note: case matters in the response headers name
// "Content-Type" won't find "content-type"
info response.get('responseHeader').get('content-type');
// was the url redirected to another url?
info response.get('responseHeader').get('location');
// get the redirect url
redirect_url = response.get('responseHeader').get('location')
from there you can process the redirect url and pass it to the next http request.
Recommendation:
After working for months both including detailed:true and not including it, I now lean toward always including it. detailed:true includes more useful information and has a helpful regular structure: {responseCode: <code>, responseHeaders: <headers>, responseText: <returned-data>}.
This is possible in Deluge using the invoke URL task - https://www.zoho.com/deluge/help/web-data/invokeurl-task.html#response.
invokeURL can hand over the response headers to you from which you can get the redirect URL and then proceed with the authentication.
We are using an onPrem S3 compatible storage server in an intranet network and we want to expose this intranet url to internet so we used a ReverseProxy with a mapping to the intranet url. When we test the intranet url it works perfectly but when we test the internet url we get the 403 error:
The request signature we calculated does not match the signature you provided. Check your Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 0a440c7f:15cc604b1e2:12d3af:24d; S3 Extended Request ID: null), S3 Extended Request ID: null
After debugging, we found that the proxy modifies the host header used to calculate the signature in order to redirect the request to the intranet url...
So my question is how to supress some headers from the V4 signature calculation using AWS SDK or Boto3 client. or is there a better architecture to expose an onPrem S3 service.
Thanks in advance.
Amir.
There are essentially two solutions to this.
The first one is easier: sign the request for the internal URL, then just use simple string prefix replacement to rewrite the host part of the signed URL to point it to the hostname of the external proxy. When the proxy rewrites the Host header, it will end up rewriting it back to exactly what you signed.
It is, I assume, common knowledge that signed URLs are immune to tampering, for all practical purposes: you can't change anything about a signed URL without invalidating it... but that's not what this is. The change is temporary, and the proxy's net effect is to undo the change.
The alternate solution requires the proxy or another service in the chain (before the storage service) to know the signing keys and secrets, so that it can first validate the incoming request, and if valid, modify the request and then generate a new signature that the service will accept. I once wrote a service to do this so that when a request was for HEAD, the proxy would use the same key and secret (which it knew) to generate a signature for the same request, but with GET. If it matched the signature in the incoming request, the proxy would replace the existing signature with a signature for a HEAD request -- thus allowing the client to use a URL originally signed for a GET request, to make either a GET or a HEAD request -- something S3 does not natively support, since a GET and a HEAD for the same object require two different signed URLs. The concept is the same, though -- generate a signature in the proxy for what the client is requesting, to validate the incoming signature, and then re-sign the request as needed. The solution I built used HAProxy's Lua integration to examine and modify the request in flight.
Can the API Gateway evaluate requests and route or return errors (not 200 statuses) on specific parameters?
What I want to achieve here is to NOT have millions of requests hitting our backend API since we already know (by evaluating the parameters) that we are not interested in returning responses on all requests but only a few percent.
I gave set up an API Proxy in the API Gateway with a complete set of requests, responses and backend.
The proxy is fully operational and up and running with throttling etc. What I would like is for the API Gateway to evaluate the requests querystring and depending on the values of certain parameters take different actions?
Let's say the complete URL and querystring is:
https://abc123.execute-api.eu-central-1.amazonaws.com/prod?param1=a¶m2=b¶m3=c
Now depending on the values of param1, param2 and param3 I might want to:
Forward the request to my actual API and return a response back to the client.
Drop the request OR return an empty (or templated) response with a specific HTTP-status (404, 503 etc - exact value not that important).
Is this achievable with the API Gateway or do I need to actually set up a host with a reverse proxy and let that handle this logic?
Request parameter and model validation has been a longstanding feature request and we are actively working on it. We'll update this post with more details when the feature has launched.
Update: Request parameter and body validation is now available as of early April 2017. see more details on this blog post.