I want to strip out values from a url in varnish, so I can take different actions based on the url, for example:
URL: /product/123/price/available.
I'd like to convert this to /products?id=123&sort=price&available=true
I would also like to be able to (if this is possible), set values on the request header, so instead of passing all the params on the URL, I could do the following:
/products?id=123&sort=price
with header: x-show-available-only: true
I appreciate the second example appears a bit odd, but this way we could pass new params back to our legacy application, and ensure none of the new params interfere with current params - we would just read new params via the header, until we migrate all our functionality to our new platform.
I'm sure it's a regex thing, but can't work out how to do it.
This is NOT tested, but should be close enough that you can tweak it to get it right.
Based on your URL:
/product/123/price/available
You'll need to specify two backends (change the IPs for your own ones):
backend old_app {
.host = "127.0.0.1";
.port = "80";
}
backend new_app {
.host = "127.0.0.2";
.port = "80";
}
And the code in your vcl_recv:
if (req.url ~ "^/product/") {
set req.url = regsub(req.url, "/product/([0-9]/)/([a-zA-Z])", "/products?id=\1&sort=\2");
set req.backend = old_app;
} else {
set req.backend = new_app.
}
if (req.url ~ "/available") {
set req.http.x-show-available-only = "true";
}
You can add more regex rules as if/else blocks.
Good luck!
Related
I have an application that uses NextJS as a wrapper, and I make use of NextJS's dynamic routing feature. I had a problem when deploying it to CloudFront due to dns.com/path/page not being rendered, instead CloudFront expected it to be dns.com/path/page.html. I worked it around by applying this lambda-edge-nice-url solution. It works properly now. However, there's still one issue left: NextJS's dynamic routes. dsn.com/path/subpath/123 should work, since 123 is a dynamic parameter. However, that does no work. In only returns the page when I access dns.com/path/subpath/[id], which of course is not correct, since [id] is not a parameter I want to load.
The strangest thing is: if I try to access the URL as I stated above directly, it fails. However, inside the application I have buttons and links that redirect the user, and that works properly.
Navigating from inside the application (button with router.push inside its callback):
Trying to access the url directly:
Can anyone help me to properly route the requests?
I use a CloudFront Lambda#Edge origin request function to handle re-writing both my dynamic routes as well as static routes to the appropriate HTML file so that CloudFront can serve the intended file for any paths.
My lambda function looks like
export const handler: CloudFrontRequestHandler = async (event) => {
const eventRecord = event.Records[0];
const request = eventRecord.cf.request;
const uri = request.uri;
// handle /posts/[id] dynamic route
if (uri === '/posts' || uri.startsWith('/posts/')) {
request.uri = "/posts/[id].html";
return request;
}
// if URI includes ".", indicates file extension, return early and don't modify URI
if (uri.includes('.')) {
return request;
}
// if URI ends with "/" slash, then we need to remove the slash first before appending .html
if (uri.endsWith('/')) {
request.uri = request.uri.substring(0, request.uri.length - 1);
}
request.uri += '.html';
return request;
};
After trying a lot of different code, I finally came up with a Lambda edge expression that fixed two issues in one:
The need to insert .html in the end of the URL
NextJS dynamic routes that were't working on refresh or when accessed directly to URL.
The code below basically takes care of dynamic routes first. It uses a regex expression to understand the current URL and redirect the request to the proper [id].html file. After that, if the none of the regex are match, and the URL does not contain .html extension, it adds the extension and retrieves the correct file.
const config = {
suffix: '.html',
appendToDirs: 'index.html',
removeTrailingSlash: false,
};
const regexSuffixless = /\/[^/.]+$/; // e.g. "/some/page" but not "/", "/some/" or "/some.jpg"
const regexTrailingSlash = /.+\/$/; // e.g. "/some/" or "/some/page/" but not root "/"
const dynamicRouteRegex = /\/subpath\/\b[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}\b/; // e.g /urs/some-uuid; // e.g. '/subpath/uuid'
exports.handler = function handler(event, context, callback) {
const { request } = event.Records[0].cf;
const { uri } = request;
const { suffix, appendToDirs, removeTrailingSlash } = config;
//Checks for dynamic route and retrieves the proper [id].html file
if (uri.match(dynamicRouteRegex)) {
request.uri = "/subpath/[id].html";
callback(null, request);
return;
}
// Append ".html" to origin request
if (suffix && uri.match(regexSuffixless)) {
request.uri = uri + suffix;
callback(null, request);
return;
}
// Append "index.html" to origin request
if (appendToDirs && uri.match(regexTrailingSlash)) {
request.uri = uri + appendToDirs;
callback(null, request);
return;
}
// Redirect (301) non-root requests ending in "/" to URI without trailing slash
if (removeTrailingSlash && uri.match(/.+\/$/)) {
const response = {
// body: '',
// bodyEncoding: 'text',
headers: {
'location': [{
key: 'Location',
value: uri.slice(0, -1)
}]
},
status: '301',
statusDescription: 'Moved Permanently'
};
callback(null, response);
return;
}
// If nothing matches, return request unchanged
callback(null, request);
};
Many thanks to #LongZheng for his answer. For some reason his code did not work for me, but it might for some, so check his answer out. Also, big shoutout to Manc, the creator of this lambda-edge-nice-urls repo. My code is basically a mix of them both.
Consider my requested url is www.example.com/foo/emplooyee?names = test1;test2.
and varnish stores this entire URL along with query parameters to uniquely identify the cache.
now, in my backend, I'm running one service and which I'm supposed to configure as whenever there are changes in names (i.e. test1 or test2) is should fire an HTTP ban with an older name (single name at a time in ban expression) to invalidate all the cached URL which entered with similar names.
Questions:
My client request url could be like this,
www.example.com/foo/emplooyee?names = test1;test2
www.example.com/foo/emplooyee?names = test1;
www.example.com/foo/emplooyee?names = test2;test1;test3;test4
www.example.com/foo/emplooyee?names = test1;test4.
How to write a VCL code and in Ban expression to invalidate all object which has query parameter as test1?
This is the VCL code you need for banning:
vcl 4.1;
acl purge {
"localhost";
"192.168.55.0"/24;
}
sub vcl_recv {
if (req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405));
}
if(!req.http.x-invalidate-pattern) {
return(purge);
}
ban("obj.http.x-url ~ " + req.http.x-invalidate-pattern
+ " && obj.http.x-host == " + req.http.host);
return (synth(200,"Ban added"));
}
}
sub vcl_backend_response {
set beresp.http.x-url = bereq.url;
set beresp.http.x-host = bereq.http.host;
}
sub vcl_deliver {
unset resp.http.x-url;
unset resp.http.x-host;
}
Here's an example HTTP request to invalidate all requests that contain a test1 value in the query string:
PURGE / HTTP/1.1
Host: www.example.com
X-Invalidate-Pattern: ^/foo/emplooyee\?names=([^;]+;)*test1(;[^;]+)*$
Here's the same request via curl:
curl -XPURGE -H'X-Invalidate-Pattern: ^/foo/emplooyee\?names=([^;]+;)*test1(;[^;]+)*$' http://www.example.com
This VCL snippet has the flexibility to remove multiple items from cache through banning, but if you don't set the pattern through the X-Invalidate-Pattern header, it will just remove the URL itself from cache.
Here's an example where we just remove http://www.example.com/foo/emplooyee?names=test1 from the cache:
curl -XPURGE 'http://www.example.com/foo/emplooyee?names=test1'
Don't forget to modify the acl block in the VCL code and add the right IP addresses or IP ranges.
So I am using OpenId Connect and the AddOpenIdConnect configuration method to setup Authentication with Auth0. I have fully tested it locally using localhost as the domain of my cookies and everything works great! However, as soon as I release this to a production environment the cookies no longer seem to persist. The crazy thing is I am logging the context.Response.Headers directly after the context.Response.Cookies.Append(...) method and locally they are printed as expected, but in production they do not.
OnTokenValidated = (context) =>
{
var accessToken = context.TokenEndpointResponse.AccessToken;
var domain = HostingEnvironment.IsProduction() ? $"{cookieOptions.Domain}.{cookieOptions.Extension}" : "localhost";
context.Response.Cookies.Append("access_token", accessToken, new CookieOptions
{
Domain = domain,
SameSite = Microsoft.AspNetCore.Http.SameSiteMode.Lax,
Expires = DateTimeOffset.Now.AddHours(10),
HttpOnly = true,
Secure = HostingEnvironment.IsProduction(),
Path = "/"
});
var idToken = context.TokenEndpointResponse.IdToken;
context.Response.Cookies.Append("id_token", idToken, new CookieOptions
{
Domain = domain,
SameSite = Microsoft.AspNetCore.Http.SameSiteMode.Lax,
Expires = DateTimeOffset.Now.AddHours(10),
HttpOnly = false,
Secure = HostingEnvironment.IsProduction(),
Path = "/"
});
// Right here is where I can see the difference. Even though I am printing out the headers directly after appending them, they never are set in production.
context.HttpContext.RequestServices.GetService<ILogger<Startup>>().LogWarning($"[HEADERS] {JsonConvert.SerializeObject(context.Response.Headers)}");
return Task.CompletedTask;
},
I even went as far as looked at the implementation of IResponseCookies.Append(...) to see what could be going on but its very simple and I don't see any way it could be failing to add these to the Headers.
https://github.com/aspnet/HttpAbstractions/blob/bc7092a32b1943c7f17439e419d3f66cd94ce9bd/src/Microsoft.AspNetCore.Http/Internal/ResponseCookies.cs#L50
public void Append(string key, string value, CookieOptions options)
{
if (options == null)
{
throw new ArgumentNullException(nameof(options));
}
var setCookieHeaderValue = new SetCookieHeaderValue(
Uri.EscapeDataString(key),
Uri.EscapeDataString(value))
{
Domain = options.Domain,
Path = options.Path,
Expires = options.Expires,
MaxAge = options.MaxAge,
Secure = options.Secure,
SameSite = (Net.Http.Headers.SameSiteMode)options.SameSite,
HttpOnly = options.HttpOnly
};
var cookieValue = setCookieHeaderValue.ToString();
Headers[HeaderNames.SetCookie] = StringValues.Concat(Headers[HeaderNames.SetCookie], cookieValue);
}
Ok so I finally figured it all out. The Auth0 Quickstart for ASP .NET Core 3.x contains the following lines of code:
services.Configure<CookiePolicyOptions>(options =>
{
// This lambda determines whether user consent for non-essential cookies is needed for a given request.
options.CheckConsentNeeded = context => true;
options.MinimumSameSitePolicy = SameSiteMode.None;
});
This enforces "consent" to be required for all Cookies. I have yet to figure out what this means exactly as the MS docs seem to be very limited. After changing this to false it all started working again. false is the default, it was just the Auth0 Quickstart that added it for some reason.
What further lead to my confusion was I tried to look at the implementation of Cookies.Append(...) and I found the ResponseCookies class that implements IResponseCookies and ended up copy/pasting that implementation into my code to get it to work. After digging, I realized its not this implementation but instead this one for ResponseCookiesWrapper that is actually being used. As you can sees this implementation uses all the extra "consent" logic to make sure you are allowed to add a cookie which apparently I wasn't.
UPDATE:
Sorry what I mentioned was that I had this working only on localhost, but you might ask why it wasn't failing on localhost and PROD. That is because even though the Quickstart shows it hardcoded to true, the actual example code on GitHub uses HostingEnvironment.IsProduction() which is what I had as well.
EDIT: See my post below for answer how to fix this.
I am building a client app that will store some data in the user dropbox app folder. So currently I am using implicit grant that will redirect the user to the given redirect uri with the parameters passed after the # in the url
Example:
localhost:1666/Dropbox#access_token=...&token_type=.......
Creating a http listener over the localhost url it detects the request however everything after # is ignored and is not passed as part of the request. Is there a way to make capture the data after the #, or is there any other library that allows me to do so?
I am using the cpprestsdk https://github.com/microsoft/cpprestsdk
web::http::experimental::listener::http_listener* l = new web::http::experimental::listener::http_listener(m_authConfig.redirect_uri());
l->support([this](web::http::http_request request) -> void
{
auto uri = request.request_uri();
auto requestPath = web::uri::split_path(uri.path());
auto queryObjects = web::uri::split_query(uri.query());
auto s = uri.fragment();
if (request.request_uri().path() == U("/Dropbox")) && request.request_uri().query() != U(""))
{
request.reply(web::http::status_codes::OK, U("ok.") + uri.query());
}
else
{
request.reply(web::http::status_codes::OK, U("error.") + uri.query());
}
});
l->open().wait();
Thanks!
So after researching a bit, it turns out that # (fragments) are not sent back in most browsers, so to be able to get the data i return the following java-script script:
<script> window.location.replace([location.protocol, '//', location.host, location.pathname, '?', location.hash.substring(1,location.hash.length )].join(''));</script>
This will convert the hash part to a query string and redirect it the user to it so that the listener detects it.
I need to remove specific cookies from the backend response in varnish.
My backend server sets a bunch of cookies that I don't need and unfortunately I can not control, so I want to delete them.
However I need some of the cookies, so I want to be able to remove cookies by their name.
For example I want to rename a cookie named bad_cookie, but at the same time keep a cookie named good_cookie.
I have found a lot of resources about removing specific request cookies, but none about removing backend response cookies.
Is this possible in Varnish?
If you want to rename I think it would be something like:
sub vcl_fetch {
#renamed after receiving the backend
set beresp.http.set-cookie = regsuball(beresp.http.set-cookie, "bad_cookie", "good_cookie");
set beresp.http.cookie = regsuball(beresp.http.cookie, "bad_cookie", "good_cookie"); }
}
sub vcl_deliver {
#renamed before sending the client
set resp.http.set-cookie = regsuball(beresp.http.set-cookie, "bad_cookie", "good_cookie");
set resp.http.cookie = regsuball(beresp.http.cookie, "bad_cookie", "good_cookie"); }
}
If you want to delete all cookies:
sub vcl_fetch {
#deleted after receiving the backend
remove beresp.http.set-cookie;
remove beresp.http.cookie;
}
sub vcl_deliver {
#deleted before sending the client
remove resp.http.set-cookie;
remove resp.http.cookie;
}
beresp.http.set-cookie reads only the first Set-Cookie header, If you want to delete some and keep others can use: github.com/varnish/libvmod-header**