AWS Cloudfront remove particular cookie before sending to origin - amazon-web-services

I want to remove particular cookie in aws cloudfront before sending to the origin server.
I have to send all cookies to origin except cookie named "_x_ad_zone".
I could not find any option to remove particular cookie in cloud front configuration. I believe that we have to achieve with lambda, but I have no clue how to do it.
Please let me know how can I achieve the same.
[EDIT]
Based on the answer, I wrote the following lambda#edge to solve my issue.
exports.handler = (event, context, callback) => {
const request = event.Records[0].cf.request;
const headers = request.headers;
const cookieName = '__a_x_id';
/*
* Lambda at the Edge headers are array objects.
*
* Client may send multiple Cookie headers, i.e.:
* > GET /viewerRes/test HTTP/1.1
* > User-Agent: curl/7.18.1 (x86_64-unknown-linux-gnu) libcurl/7.18.1 OpenSSL/1.0.1u zlib/1.2.3
* > Cookie: First=1; Second=2
* > Cookie: ClientCode=abc
* > Host: example.com
*
* You can access the first Cookie header at headers["cookie"][0].value
* and the second at headers["cookie"][1].value.
*
* Header values are not parsed. In the example above,
* headers["cookie"][0].value is equal to "First=1; Second=2"
*/
console.log(headers.cookie);
// Delete the cookie if found
if (headers.cookie) {
for (let i = 0; i < headers.cookie.length; i++) {
console.log(headers.cookie[i].value);
if (headers.cookie[i].value.indexOf(cookieName) >= 0) {
console.log('Adblocker cookie found and delete: '+headers.cookie[i].value);
headers.cookie[i].value = "0";
break;
}
}
request.headers = headers;
}
callback(null, request);
};

Quick disclaimer: When cookies are forwarded to the origin server, CloudFront caches the response against not only the URI and the headers (and the query string, if configured to do so), but also against the unique combination of cookie values (or missing cookies) presented by the browser -- so a response can only be served from cache when they contain (or are missing) exactly the same combination of cookies and values. This is not good for your cache hit rate, but of course, it also is completely correct design on the part of CloudFront -- if a different cookie is presented, CloudFront has no option but to assume that the cookie will potentially modify the response returned from the origin, so cookies must become components of the cache key.
It you must forward cookies, it's best to forward specific cookies.
However, CloudFront has a number of applications that are unrelated to caching, so there likely are valid use cases for a solution like this.
Your solution will only pass simplistic and optimistic tests. There are a number of edge cases it does not handle correctly. The example script for cookie manipulation is only a simple illustration, and includes something of a disclaimer to that effect:
* Header values are not parsed.
The first problem is that browsers are free to combine multiple cookies in a single Cookie: header, and your test of headers.cookie[i].value.indexOf(cookieName) will not only match a header with the cookie you want, it will match headers with that cookie plus others... and remove all the cookies in that particular header entry.
If used in a Viewer Request trigger, there's a significant risk of removing too many cookies with this solution. In an Origin Request trigger, it's even higher, since the cookies have already been stripped and re-canonicalized by the cookie forwarding configuration of the matched Cache Behavior, and CloudFront does combine multiple cookies on a single header row, at least under some conditions.
The second problem is related to the first: the simple string matching of indexOf() will match cookie values, as well as cookie names, so it's possible to get a false match on a cookie value -- which you don't want to be examining.
The third problem is that you're not really generating a valid replacement value. CloudFront seems to accept that, for now, but since it's technically invalid, there's a possibility that this could be "fixed" in the future.
I've written a Lambda#Edge script that I believe fully handles cookie semantics and will remove only and exactly the cookies you want to remove, and leave the data structures clean. Because I find this to be an interesting use case, I wrote it so that it will match as many cookies as you like -- not just one cookie -- with an exact-string, case-sensitive match, on the cookie name only.
The cookies are configured in an array near the top.
In your case, with a cookie named __a_x_id would look like this:
const discard = [ '__a_x_id' ];
Adding multiple cookie names to the array will block all of them.
This uses Node.js 6.10 and works with either a Viewer Request trigger or an Origin Request trigger. If you are doing any caching at all, you'll probably want to use it as an Origin Request trigger, since that means it fires less often.
I'm also pleased to report that in spite of looking a little complicated and doing a fair amount of string splitting and having multiple nested loops, this code has a Lambda execution time in a warm container of consistently less than 1 millisecond, whether or not any cookies are matched and removed.
'use strict';
// source: https://stackoverflow.com/a/45970883/1695906
// iterate through all Cookie: headers in a request trigger,
// removing any cookies on the "discard" list, while preserving
// the integrity of any other cookies, including those appearing on the same
// header line, and confirm the resulting "cookie" array to CloudFront
// requirements by removing any now-empty elements, or the entire array
// if no cookies remain
// configure with one or more cookies to be removed from all requests;
// cookie names are case-sensitive
const discard = [ 'grover', 'big_bird' ]; // friends of cookie monster
exports.handler = (event, context, callback) => {
const request = event.Records[0].cf.request;
const headers = request.headers;
// does the request have any cookies? skip to the end, if not
if(headers.cookie)
{
const cookies = headers.cookie;
// iterate each Cookie: header, from last to first;
// last-to-first makes it simple to splice-out an array element
// because we then need not keep track of the reduced array length
for (var n = cookies.length; n--;)
{
// there may be multiple cookies per header line; examine them all
const cval = cookies[n].value.split(/;\ /);
const vlen = cval.length; // how many we started with
// check individual cookies on this line, backwards
for (var m = vlen; m--;)
{
// cookie name is to the left of "="
const cookie_kv = cval[m].split('=')[0];
// run though each member of "discard" array,
// removing the cookie if it's a match,
// again last to first but for no particular reason, here
for(var di = discard.length; di--;)
{
if(cookie_kv == discard[di])
{
cval.splice(m,1); // cookie removed!
break; // no need to check any other matches, already gone
}
}
} // for m
// if our array of cookies for this header line has now changed in size,
// we must have deleted some or all of it, so we need to reassemble
// what remains, or eliminate the entire line
if(cval.length != vlen)
{
if(cval.length === 0) // did we remove everything?
{
// yes? we can eliminate this entire line
cookies.splice(n,1);
}
else
{
// no? reassemble the remaining cookies
headers.cookie[n].value = cval.join('; ');
}
}
} // for n
// if the only cookies present in the request were cookies we removed,
// we now have a completely empty array in headers.cookie, which
// CloudFront should consider invalid; clean it up
if(cookies.length === 0)
{
delete headers.cookie;
}
}
// return control to CloudFront, possibly with our modified request
return callback(null, request);
};

For this you need to write an Lambda#Edge function to conditionally filter the cookie in CloudFront.
Check this example to get an insight on required operations. Also note that you need to alter the request header in origin request event, using Lambda#Edge.

Related

"Secure" setting is preventing cookie from being set

I am fairly new to cookies and had been tasked with figuring out why a cookie warning kept popping up even after the user clicked accept. It turned out that the cookie wasn't being set in some browsers (mostly Webkit, but also one time in Firefox).
After some exploring I discovered that by removing the setting "Secure" in the cookie string I could make it work as intended, but I don't have enough understanding of cookies to determine why that would prevent the cookie from being set in the first place. Can anyone explain?
This is my cookie for reference:
acceptCookies() {
let expiry = new Date();
let months = 1;
let d = expiry.getDate();
expiry.setMonth( expiry.getMonth() + months );
if ( expiry.getDate() !== d ) {
expiry.setDate( 0 );
}
document.cookie = 'acceptsCookies=1 expires=' + expiry.toUTCString() + '; SameSite=lax; Secure';
}
Answering my own question: It turned out that the cookie was being set on https, but not on http, which makes sense since the "Secure" setting only allows it to be sent over HTTPS.
More info here:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies

400 Bad Request Request Header Or Cookie Too Large using Sustainsys.Saml2

I'm getting a browser error when using SustainSys.Saml2 library with my app:
400 Bad Request
Request Header Or Cookie Too Large
nginx/1.14.0
I think that reducing my cookie size might help and I only really need the email from the claim data, so I thought that if I could just save the email claim and remove the other claims, that it might reduce my cookie size and fix this error.
I read the response to a similar question (SustainSys.Saml2 Request length header too long) and looked for some information on how to implement AcsCommandResultCreated to remove unused claims (and hopefully reduce cookie size). I didn't find a lot of documentation, but did piece together some ideas and code to try and take a stab at it.
I've tried this code in my global.asax as well as in a controller action (that I made the "returnUrl" after Saml2/Acs). It doesn't look like my FedAuth cookie (set by Saml2/Acs) is any smaller. Any comments or suggestions? Thank you.
// Check if email claim exists
var principal = ClaimsPrincipal.Current;
var userEmail = principal.Claims.FirstOrDefault(claim => claim.Type == ClaimTypes.Email)?.Value;
// Create new command result that only contains the email claim
if (userEmail != null)
{
var emailClaim = principal.Claims.FirstOrDefault(claim => claim.Type == ClaimTypes.Email);
Sustainsys.Saml2.Configuration.Options.FromConfiguration.Notifications.AcsCommandResultCreated =
(commandResult, response) =>
{
var newCommandResult = new Sustainsys.Saml2.WebSso.CommandResult();
newCommandResult.Principal.Claims.Append(emailClaim);
commandResult = newCommandResult;
};
}
UPDATE:
It turned out that the test environment that I was using (which used nginx) needed to increase the request header buffer size. Adding these cookies increased the size to around 9500 bytes and nginx by default has a request header buffer size that is lower than that (I think 8000). Contacting the code owners of the test server running nginx, and increasing this solved my problem, without me having to reduce my cookie size.
Do you have a lot of failed authentication attempts? That can leave a lot of Saml2.XYZ correlation cookies around on the domain. Try checking the browser dev tools and clean those up.
The "headers too large" is usually something that happens when a user has tried signing in several times with a failure and those cookies get stuck. The real issue is usually something else - causing the authentication to fail and those correlation cookies to be accumulating.

How to get CookieManager from JMeter SampleResult

I have a backend listener that posts JMeter results to an external server. It works for most parts. However, I'd like to get cookie info for failed requests. Backend listener only gets SampleResult and I don't see an API to access cookie info from SampleResult. Does anyone have an idea?
View Result Tree listener is able to print out request cookies, so there must be a way. However, it's not obvious what's the class name for that particular listener. If anyone can point me to that source code, it'll be a starting point.
With current JMeter implementation it is not possible unless your create your own AbstractBackendListenerClient implementation which will be cookies-aware.
Also depending on how do you parse result you can substitute a "not interesting" SampleResult field (for example Response Message) with cookie values for failed requests. It can be done using i.e. JSR223 PostProcessor and the following Groovy code
if (!prev.isSuccessful()) {
def manager = sampler.getCookieManager()
def cookies = new StringBuilder()
for (int i = 0; i < manager.getCookieCount(); i++) {
cookies.append(manager.get(i).getName()).append('=').append(manager.get(i).getValue())
}
prev.setResponseMessage(cookies.toString())
}
Hard to help without seeing some code, but one general direction could be this: you can check if your SampleResult is of type HTTPSampleResult, and if yes, get its cookies, using getCookies() method. Whole process could be wrapped as the following function:
public String getCookiesForHttpResult(SampleResult sampleResult) {
if (sampleResult instanceof HTTPSampleResult) {
HTTPSampleResult httpSampleResult = (HTTPSampleResult) sampleResult;
return httpSampleResult.getCookies();
}
// not HTTP result
return null;
}
Something like that is done in JMeter's RequestViewHTTP.java
Getting CookieManager in backend listener can be problematic, since it's mutable: by the time backend listener reads from CookieManager, list of cookies may be different from the time of request execution.

Postman - how to loop request until I get a specific response?

I'm testing API with Postman and I have a problem:
My request goes to sort of middleware, so either I receive a full 1000+ line JSON, or I receive PENDING status and empty array of results:
{
"meta": {
"status": "PENDING",
"missing_connectors_count": 0,
"xxx_type": "INTERNATIONAL"
},
"results": []
}
The question is, how to loop this request in Postman until I will get status SUCCESS and results array > 0?
When I'm sending those requests manually one-by-one it's ok, but when I'm running them through Collection Runner, "PENDING" messes up everything.
I found an awesome post about retrying a failed request by Christian Baumann which allowed me to find a suitable approach to the exact same problem of first polling the status of some operation and only when it's complete run the actual tests.
The code I'd end up if I were you is:
const maxNumberOfTries = 3; // your max number of tries
const sleepBetweenTries = 5000; // your interval between attempts
if (!pm.environment.get("tries")) {
pm.environment.set("tries", 1);
}
const jsonData = pm.response.json();
if ((jsonData.meta.status !== "SUCCESS" && jsonData.results.length === 0) && (pm.environment.get("tries") < maxNumberOfTries)) {
const tries = parseInt(pm.environment.get("tries"), 10);
pm.environment.set("tries", tries + 1);
setTimeout(function() {}, sleepBetweenTries);
postman.setNextRequest(request.name);
} else {
pm.environment.unset("tries");
// your actual tests go here...
}
What I liked about this approach is that the call postman.setNextRequest(request.name) doesn't have any hardcoded request names. The downside I see with this approach is that if you run such request as a part of the collection, it will be repeated a number of times, which might bloat your logs with unnecessary noise.
The alternative I was considering is writhing a Pre-request Script which will do polling (by sending a request) and spinning until the status is some kind of completion. The downside of this approach is the need for much more code for the same logic.
When waiting for services to be ready, or when polling for long-running job results, I see 4 basic options:
Use Postman collection runner or newman and set a per-step delay. This delay is inserted between every step in the collection. Two challenges here: it can be fragile unless you set the delay to a value the request duration will never exceed, AND, frequently, only a small number of steps need that delay and you are increasing total test run time, creating excessive build times for a common build server delaying other pending builds.
Use https://postman-echo.com/delay/10 where the last URI element is number of seconds to wait. This is simple and concise and can be inserted as a single step after the long running request. The challenge is if the request duration varies widely, you may get false failures because you didn't wait long enough.
Retry the same step until success with postman.setNextRequest(request.name);. The challenge here is that Postman will execute the request as fast as it can which can DDoS your service, get you black-listed (and cause false failures), and chew up a lot of CPU if run on a common build server - slowing other builds.
Use setTimeout() in a Pre-request Script. The only downside I see in this approach is that if you have several steps needing this logic, you end up with some cut & paste code that you need to keep in sync
Note: there are minor variations on these - like setting them on a collection, a collection folder, a step, etc.
I like option 4 because it provides the right level of granularity for most of my cases. Note that this appears to be the only way to "sleep" in a Postman script. Now standard javascript sleep methods like a Promise with async and await are not supported and using the sandbox's lodash _.delay(function() {}, delay, args[...]) does not keep script execution on the Pre-request script.
In Postman standalone app v6.0.10, set your step Pre-request script to:
console.log('Waiting for job completion in step "' + request.name + '"');
// Construct our request URL from environment variables
var url = request['url'].replace('{{host}}', postman.getEnvironmentVariable('host'));
var retryDelay = 1000;
var retryLimit = 3;
function isProcessingComplete(retryCount) {
pm.sendRequest(url, function (err, response) {
if(err) {
// hmmm. Should I keep trying or fail this run? Just log it for now.
console.log(err);
} else {
// I could also check for response.json().results.length > 0, but that
// would omit SUCCESS with empty results which may be valid
if(response.json().meta.status !== 'SUCCESS') {
if (retryCount < retryLimit) {
console.log('Job is still PENDING. Retrying in ' + retryDelay + 'ms');
setTimeout(function() {
isProcessingComplete(++retryCount);
}, retryDelay);
} else {
console.log('Retry limit reached, giving up.');
postman.setNextRequest(null);
}
}
}
});
}
isProcessingComplete(1);
And you can do your standard tests in the same step.
Note: Standard caveats apply to making retryLimit large.
Try this:
var body = JSON.parse(responseBody);
if (body.meta.status !== "SUCCESS" && body.results.length === 0){
postman.setNextRequest("This_same_request_title");
} else {
postman.setNextRequest("Next_request_title");
/* you can also try postman.setNextRequest(null); */
}
I was searching for an answer to the same question and thought of a possible solution as I was reading your question.
Use postman workflow to rerun your request every time you don't get the response you're looking for. Anyway, that's what I'm gonna try.
postman.setNextRequest("request_name");
https://www.getpostman.com/docs/workflows
I didn't succeed to find the complete guidelines for this issue that's why I decided to invest some time and to describe all steps of the process from A to Z.
I will be observing an example where we will need to pass through transaction ids and in each iteration to change query param for next transaction id from the list.
Step 1. Prepare your request
https://some url/{{queryParam}}
Add {{queryParam}} variable for changing it from pre-request script.
If you need a token for request you should add it here, in Authorization tab.
Save request to collection (Save button in the right corner). For demonstration purpose I will use "Transactions Request" name. We will need to use this name later on.
Step 2. Prepare pre-request script
In postman use tab Pre-request Script to change transactionId variable from query param to actual transaction id.
let ids = pm.collectionVariables.get("TransactionIds");
ids = JSON.parse(ids);
const id = ids.shift();
console.log('id', id)
postman.setEnvironmentVariable("transactionId", id);
pm.collectionVariables.set("TransactionIds", JSON.stringify(ids));
pm.collectionVariables.get - gets array of transaction ids from collection variables. We will set it up in Step 4.
ids.shift() - we use it to remove id that we will use from our ids list (to prevent running twice on the same id)
postman.setEnvironmentVariable("transactionId", id) - change transaction id from query param to actual transaction id
pm.collectionVariables.set("TransactionIds", JSON.stringify(ids)) - we are setting up a new collection of variables that now does not include the id that was handled.
Step 3. Prepare Tests
In postman use tab Tests to create a loop logic. Tests will be executed after the request execution, so we can use it to make next request.
let ids = pm.collectionVariables.get("TransactionIds");
ids = JSON.parse(ids);
if (ids && ids.length > 0){
console.log('length', ids.length);
postman.setNextRequest("Transactions Request");
} else {
postman.setNextRequest(null);
}
postman.setNextRequest("Transactions Request") - calls a new request, in this case it will call the "Transactions Request" request
Step 4. Run Collections
In Postman from the left side bar you should choose Collections (click on it) and then choose a tab Variables.
This is the collection variables. In our example we used TransactionIds as a variable, so put in Current Value the array of transaction ids on which you want to loop.
Now you can click on Run (the button from right corner, near Save button) to run our loop requests.
You will be proposed to choose on which request you want to perform an action. Choose the request that we’ve created "Transactions Request".
It will run our request with pre-request script and with logic that we’ve set in Tests. In the end postman will open a new window with summary of our run.

Send a WS request for each URL in a list and map the responses to a new list

I'm developing a REST server in Play with Scala, that at some point needs to request data at one or more other web services. Based on the responses from these services the server must compose a unified result to use later on.
Example:
Event C on www.someplace.com needs to be executed. In order to execute Event C, Event A on www.anotherplace.com and Event B on www.athirdplace.com must also be executed.
Event C has a Seq(www.anotherplace.com, www.athirdplace.com) from which I would like to iterate and send a WS request to each URL respectively to check wether B and C are executed.
It is assumed that a GET to these URLs returns either true or false
How do I collect the responses from each request (preferably combined to a list) and assert that each response is equal to true?
EDIT: An event may contain an arbitrary number of URL's. So I cant know beforehand how many WS requests i need to send.
Short Answer
You can use sequence method available on Future object.
For example:
import scala.concurrent.Future
val urls = Seq("www.anotherplace.com", "www.athirdplace.com")
val requests = urls.map(WS.url)
val futureResponses = Future.sequence(requests.map(_.get()))
Aggregated Future
Note that the type of futureResponses will be Future[Seq[WSResponse]]. Now you can work on the results:
futureResponses.map { responses =>
responses.map { response =>
val body = response.body
// do something with response body
}
}
More Details
From ScalaDocs of sequence method:
Transforms a TraversableOnce[Future[A]] into a
Future[TraversableOnce[A]]. Useful for reducing many Futures into a
single Future.
Note that if any of the Futures you pass to sequence fails, the resulting Future will be failed as well. Only when all Futures are completed successfully the result will complete successfully. This is good for some purposes, especially if you want to send requests at the same time, no one after another.
Have a look at the documentation and see if you can get their example to work.
Try something like this:
val futureResponse: Future[WSResponse] = for {
responseOne <- WS.url(urlOne).get()
responseTwo <- WS.url(responseOne.body).get()
responseThree <- WS.url(responseTwo.body).get()
} yield responseOne && responseTwo && responseThree
You probably need to parse the response of your WebService since they (probably) won't return booleans, but you'll get the idea.