In Varnish, how can I read the Set-Cookie response header? - cookies

I am trying to detect if my application has set a cookie that holds an "alert message" for the user on the next page, where the Javascript displays it if detected.
In my vcl_fetch(), I need to detect if the specific cookie value "alert_message" appears anywhere in the Set-Cookie header (presumably in the VCL variable beresp.http.Set-Cookie). If detected, then I do not want to cache that next page (since Varnish strips the Set-Cookie header by default, which would obliterate the alert message before it makes it back to the browser).
So here is my simple test:
if(beresp.http.Set-Cookie ~ "alert_message") {
set req.http.no_cache = 1;
}
Strangely, it fails to evaluate to true.
So I throw the variable into the Server header to see what it looks like:
set beresp.http.Server = " MyApp Varnish implementation - test reading set-cookie: "+beresp.http.Set-Cookie;
But for some reason this only displays the FIRST Set-Cookie line in the response headers.
Here are the relevant response headers:
Server: MyApp Varnish implementation - test reading cookie: elstats_session=7d7279kjmsnkel31lre3s0vu24; expires=Wed, 10-Oct-2012 00:03:32 GMT; path=/; HttpOnly
Set-Cookie:app_session=7d7279kjmsnkel31lre3s0vu24; expires=Wed, 10-Oct-2012 00:03:32 GMT; path=/; HttpOnly
Set-Cookie:alert_message=Too+many+results.; expires=Tue, 09-Oct-2012 20:13:32 GMT; path=/; domain=.my.app.com
Set-Cookie:alert_key=flash_error; expires=Tue, 09-Oct-2012 20:13:32 GMT; path=/; domain=.my.app.com
Vary:Accept-Encoding
How do I read and run string detection on ALL Set-Cookie header lines?

You can resolve it with header.get function from vmod header (Varnish version >= 3)
For example, I have simple PHP script and more than one Set-Cookie in it:
<?php
setcookie ("Foo", "test", time() + 3600);
setcookie ("Bar", "test", time() + 3600);
setcookie ("TestCookie", "test", time() + 3600);
?>
By default, only first Set-Cookie header will be parsed with ' if(beresp.http.Set-Cookie ~ "somedata" '.
Of course, we can use std.collect procedure from vmod std (already comes with Varnish 3 and not requires compilation) to collapse all our Set-Cookie headers to one, but it will break cookies - Bar and TestCookie will not set.
header.get avoids this defect: it will check all headers for regex match:
if (header.get(beresp.http.set-cookie,"TestCookie=") ~ "TestCookie")
{
set beresp.http.cookie-test = 1;
return(hit_for_pass);
}
So, with it I got in response headers on first and next requests:
cookie-test:1
Set-Cookie:Foo=test; expires=Tue, 09-Oct-2012 22:33:37 GMT
Set-Cookie:Bar=test; expires=Tue, 09-Oct-2012 22:33:37 GMT
Set-Cookie:TestCookie=test; expires=Tue, 09-Oct-2012 22:33:37 GMT
X-Cache:MISS
If I comment out setcookie for cookie TestCookie, then I will got HITs on next requests.

Related

Set more than one cookie in the 'Set-Cookie' header in a cloudflare worker header.set('Set-Cookie' function

I'm trying to use a cloudfare worker to add 2 cookie key/value pairs to the response before sending it to the client.
Unfortunately all documentation for the cloudflare workers says to use the response.headers.set('Set-Cookie',xxx) function to set the cookie value:
let response = await fetch(request);
response = new Response(response.body, response);
response.headers.set('Set-Cookie', "val1=x; Expires=Wed, 21 Oct 2020 07:28:00 GMT; Path='/';");
return response;
This only allows you to set one cookie header, and if called twice just overwrites the existing header.
I have tried calling the function twice, only the last value comes in:
response.headers.set('Set-Cookie', "val1=1; Expires=Wed, 21 Oct 2020 07:28:00 GMT; Path='/';");
response.headers.set('Set-Cookie', "val2=2; Expires=Wed, 21 Oct 2020 07:28:00 GMT; Path='/';");
I have tried passing 2 cookies in the one header, separated with a comma, but only one comes in:
response.headers.set('Set-Cookie', "val1=1; Expires=Wed, 21 Oct 2020 07:28:00 GMT; Path='/';, val2=2; Expires=Wed, 21 Oct 2020 07:28:00 GMT; Path='/';");
I have tried passing 2 cookie key/value pairs, but the first key value is set to "1, val2=2":
response.headers.set('Set-Cookie', "val1=1, val2=2; Expires=Wed, 21 Oct 2020 07:28:00 GMT; Path='/';");
None of these work.
The only work around I have found is to bundle the vars up into one variable, and then use JS on the client side to unpack and apply the variable:
response.headers.set('Set-Cookie', "jsVal={val1:1, val2:2}; Expires=Wed, 21 Oct 2020 07:28:00 GMT; Path='/';");
.. and then in a js file apply the 2 cookie values. Obviously this is not ideal.
Has anyone had any luck applying 2 separate cookies in one response header via a cloudflare worker? Thanks.
There is Headers.append(): https://developer.mozilla.org/en-US/docs/Web/API/Headers/append
The difference between set() and append() is that if the specified
header already exists and accepts multiple values, set() will
overwrite the existing value with the new one, whereas append() will
append the new value onto the end of the set of values.

how to use varnish cache with set-cookie named mp3list and phpsessionid

i am new to php and i am interested to use varnish to improve site performance..
i installed varnish latest version : 4.0.2 varnish
HTTP/1.1 200 OK
Date: Sat, 06 Dec 2014 07:24:47 GMT
Server: Apache/2.2.29 (Unix) mod_ssl/2.2.29 OpenSSL/1.0.1e-fips mod_bwlimited/1.4
X-Powered-By: PHP/5.4.34
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=86dc704d405a80d0c012de043cd9408b; path=/
Set-Cookie: Mp3List=WDVMG2G4; expires=Tue, 03-Dec-2024 07:24:47 GMT
Vary: Accept-Encoding
Content-Type: text/html
X-Varnish: 2
Age: 0
Via: 1.1 varnish-v4
Connection: keep-alive
i use cookie named ( mp3list and phpsessionid) so i cant cache my pages,,
i used vcl with
$sub vcl_recv {
call normalize_req_url;
set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
# If the requested URL starts like "/cart.asp" then immediately pass it to the given
# backend and DO NOT cache the result ("pass" basically means "bypass the cache").
if (req.url ~ "^/playlist\.php$") {
return (pass);
}
}
sub vcl_backend_response {
# Happens after we have read the response headers from the backend.
#
# Here you clean the response headers, removing silly Set-Cookie headers
# and other mistakes your backend does.
if (beresp.http.Set-Cookie)
{
set beresp.http.Set-Cookie = regsub(beresp.http.Set-Cookie, "^php","");
if (beresp.http.Set-Cookie == "")
{
unset beresp.http.Set-Cookie;
}
}
unset beresp.http.Cache-Control;
set beresp.http.Cache-Control = "public";
}
sub vcl_deliver {
# Happens when we have all the pieces we need, and are about to send the
# response to the client.
#
# You can do accounting or modifying the final object here.
# Was a HIT or a MISS?
if ( obj.hits > 0 )
{
set resp.http.X-Cache = "HIT";
}
else
{
set resp.http.X-Cache = "MISS";
}
# And add the number of hits in the header:
set resp.http.X-Cache-Hits = obj.hits;
}
sub normalize_req_url {
# Strip out Google Analytics campaign variables. They are only needed
# by the javascript running on the page
# utm_source, utm_medium, utm_campaign, gclid, ...
if(req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|utm_[a-z]+|mr:[A-z]+)=") {
set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|utm_[a-z]+|mr:[A- z]+)=[%.-_A-z0-9]+&?", "");
}
set req.url = regsub(req.url, "(\?&?)$", "");
}
sub vcl_hash {
hash_data(req.url);
if (req.http.host) {
hash_data(req.http.host);
} else {
hash_data(server.ip);
}
hash_data(req.http.Cookie);
}
Now i removed mp3list using this,, still phpsessid is there,, how to remove phpsessid
HTTP/1.1 200 OK
Date: Sat, 06 Dec 2014 07:44:46 GMT
Server: Apache/2.2.29 (Unix) mod_ssl/2.2.29 OpenSSL/1.0.1e-fips mod_bwlimited/1.4
X-Powered-By: PHP/5.4.34
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html
Set-Cookie: PHPSESSID=ce934af9a97bd7d0fd14304bd49f8fe2; path=/
Cache-Control: public
X-Varnish: 163843
Age: 0
Via: 1.1 varnish-v4
X-Cache: MISS
X-Cache-Hits: 0
Connection: keep-alive
can anyone plz guide me how to bypass phpsessid and edit VCL and use varnish effectively
advance thanks for you ........
As you are noticing PHP sessions and Varnish do not intermix very nicely, ie the phpsessions destroys the cachebility of the urls within varnish as it makes them all unique.
Options:
1. disable php sessions altogether (doubt thats a real option for you).
2. Only enable php sessions on the pages you absolutely need it, so the rest can be cached.
3. split the session-related php logic into separate php files and include those using ESI into your main pages. that will allow you to partially cache the pages (the non-ESI bits).

java.net.ProtocolException: Bad Set-Cookie header

The block of code for your reference is given below:
String hostname = "Hostname Value";
URI uri = new URI(hostname + "/solr/add/story/" + story.getId() + ".html");
final HTTPConnection con = new HTTPConnection(uri);
con.setAllowUserInteraction(false);
final HTTPResponse response = con.Get(uri.getPathAndQuery());
Here, while accessing the response i am getting below exception:
[ WARN] [com.thestreet.cms.integration.solr.SolrService] 12/02/2013 22:52:54-Unable
update front end search engine index with story 10446446
java.net.ProtocolException: Bad Set-Cookie header: FV=OID-|PID-|MID-|PUC-|DATE-
529D5595; path=/; expires=Tue, 01-Jan-2035 00:00:00 GMT; domain=.thestreet.com;,
BRIS=C0.A8.41.91|529D55951FB74EF; path=/; expires=Tue, 01-Jan-2035 00:00:00 GMT;
domain=.thestreet.com;,
RGIS=-1386042773,192.168.65.145,BA42A8C0,1076F795713A21E010941898- 0-1386042773-;
path=/; expires=Tue, 01-Jan-2035 00:00:00 GMT; domain=.thestreet.com;,
JSESSIONID=8A8A377CF937F6184D3F4774CC6F4CBA; Version=1; Path="/"; HttpOnly
No '=' found for token starting at position 432
at HTTPClient.Cookie.parse(Cookie.java:243)
at HTTPClient.CookieModule.handleCookie(CookieModule.java:454)
at HTTPClient.CookieModule.responsePhase1Handler(CookieModule.java:403)
at HTTPClient.HTTPResponse.handleResponse(HTTPResponse.java:724)
at HTTPClient.HTTPResponse.getStatusCode(HTTPResponse.java:190)
at com.thestreet.cms.integration.solr.SolrService$1.run(SolrService.java:450)
at java.lang.Thread.run(Thread.java:722)
This seems to be caused by Httponly bit in the cookie header since its not in the form of key-value pair. Is there any way to avoid a cookie header or cookie check while reading the response? Please help.
Thanks in advance.
The problem is your http request/response contains "HttpOnly" in its header. It appears to be that this value is not supported by application servers anymore. In order to fix this I wrote a workaround where "HttpOnly" is removed from the response on a server side.
String header = resp.getHeader("Set-Cookie");
if (header != null && header.endsWith("HttpOnly")) {
resp.setHeader("Set-Cookie", header.substring(0, header.length() - 8));
}
But the best solution is to remove "HttpOnly" from the header on the http client side.

Using a TCP connection to GET data from a HTTP server, on rare occasion there's garbage data

I'm writing a networked game in C++ using Winsocks 2.2, using Visual Studio 2010, and decided it would be a good idea to use my web-server to store a list of active servers for the game. When a server starts up, it will register itself with my web server, on exit un-register; and the server itself will attempt to clean the list when someone accesses the server list depending (this behavior I'm still working on designing to not involve too much work on the server; but I figured when the game server tries to add itself, my php file will use fsockopen to detect if it can actually access the server from an outside network, if not, the server won't get added until it can properly setup port forwarding or somehow resolve the issue).
Okay, so after some research, I figured out how to get something from the server using a TCP connection from formatting a specialized message for the HTTP server. Here's what I have:
if(FAIL == Connection::Get_Connection(&m_Connection, networkSettings.ServerListAddress, 80))
{
return FAIL;
}
m_Connection.SendMsg("GET /servers.php HTTP/1.1\r\nHost: cyclotron.leetnightshade.com\r\nUser-Agent: CycloTron\r\n\r\n");
I'm expecting back properly formatted data, which I'm not exactly getting. Here's what I'm getting:
2f
Server Count:1
129.21.138.1,40000,Depth of Hell
0
Here's another output of some garbage with all the header information:
HTTP/1.1 200 OK
Date: Tue, 12 Apr 2011 23:23:11 GMT
Server: Apache
X-Powered-By: PHP/5.2.17
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=8254688ee345202bd177d57e4ba339b2; path=/
Set-Cookie: PHPSESSID=73eae89f61e7268f433af9bdfe299173; path=/
Set-Cookie: PHPSESSID=8fb5d6fd9f1023bb00290b4daa3c7952; path=/
Connection: close
Transfer-Encoding: chunked
Content-Type: text; charset=us-ascii
e
Server Count:1
21
129.21.138.1,40000,Depth of Hell
0
This is what my output is supposed to look like, and I do get this on occasion, but not all the time:
HTTP/1.1 200 OK
Date: Tue, 12 Apr 2011 23:32:13 GMT
Server: Apache
X-Powered-By: PHP/5.2.17
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: PHPSESSID=a3c88c2d96d45c6f6d3b029e095c429a; path=/
Set-Cookie: PHPSESSID=bf19734ff60813d6d0a5ba944410356a; path=/
Set-Cookie: PHPSESSID=c36a2d9e12c81d4a19a7f41dc5522b4e; path=/
Content-Length: 47
Connection: close
Content-Type: text; charset=us-ascii
Server Count:1
129.21.138.1,40000,Depth of Hell
I don't think this matters too much, but this is my PHP code on the web server:
$num = mysql_num_rows($result);
echo 'Server Count:'.$num;
while ($row = mysql_fetch_assoc($result))
{
// TODO: check date of entry, if it's really old, remove it.
echo PHP_EOL.$row['address'].','.$row['port'].','.$row['displayName'];
}
And here's some of the code involving receiving the string (yes it's a little bare bones at the moment, and I realize I could use a cstring function to look for the two new lines so I don't have to do the string copy, I'm just trying to stick to using strings to make things easier):
memset(m_MsgBuffer, 0, sizeof (char) * M_BufferSize);
m_Received = recv(m_Connection.M_Socket, m_MsgBuffer, M_BufferSize, 0);
m_MsgBuffer[m_Received] = '\0';
string str = string(m_MsgBuffer);
size_t index = str.find("\r\n\r\n");
str.erase(0,index);
std::cout << "Received message: " << str << std::endl;
So, do any of you have an idea where this garbage data is coming from?
EDIT: After looking at the correct header information, the one with garbage has "Transfer-Encoding: chunked" and doens't have "content-length." ...what is going on?
The so-called "garbage" is in fact the chunked data from the server. An HTTP/1.1 server is free to send data back in chunked format, if it prefers, and the HTTP/1.1 spec is pretty clear: "All HTTP/1.1 applications MUST be able to receive and decode the "chunked" transfer-coding".
The details of chunked encoding are described in the HTTP/1.1 spec:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
If you are writing an HTTP client, you need to read the HTTP spec...

Setting a Compact Privacy Policy with Django

How do I set a P3P compact privacy policy from Django so that IE accepts cookies from my site when the security settings are on HIGH - i.e. no cookies accepted unless there's a Compact Privacy Policy.
Cheers
Guy
Middleware is the preferred way to do things like this on an "every request" basis. For instance, here is a simple bit of middleware to add the same (example) P3P header to every response Django generates:
In settings.py:
P3P_COMPACT = 'policyref="http://www.example.com/p3p.xml", CP="NON DSP COR CURa TIA"'
MIDDLEWARE_CLASSES += ('myapp.middleware.P3PHeaderMiddleware',)
In myapp/middleware.py:
from django.conf import settings
class P3PHeaderMiddleware(object):
def process_response(self, request, response):
response['P3P'] = getattr(settings, 'P3P_COMPACT', None)
return response
You could also get a similar effect in a single view by setting the P3P header in the response:
def my_view(request):
response = render_to_response('my_template.html')
response['P3P'] = 'CP="NON DSP COR CURa TIA"'
return response
To expand on the topic a little bit, cookies and headers such as the P3P header are both sent at the same time, as part of the response; in fact, under the hood, cookies are set with another response header. You can see the cookie header using curl:
$ curl --head http://www.google.com/
HTTP/1.1 200 OK
Date: Wed, 13 Jan 2010 00:04:59 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=d2c09762c479f94e:TM=1263341099:LM=1263341099:S=oJby3NpU4RsRfuYa; expires=Fri, 13-Jan-2012 00:04:59 GMT; path=/; domain=.google.com
Set-Cookie: NID=30=kdKrd5e-u6Xs7cUe3p4eaNDtv6SO88uBL5v6_M1XMTSRmkh7okxrWLOm-l_uZdN37PxQIe4dBlekFFVCpTFXGyIDlUrz1hEwhgVLvXfIik_VeVWGmWzKbA5qu_Zq0sOi; expires=Thu, 15-Jul-2010 00:04:59 GMT; path=/; domain=.google.com; HttpOnly
Server: gws
X-XSS-Protection: 0
Transfer-Encoding: chunked
I don't know terribly much about p3p but I did a little digging and found this:
http://www.w3.org/TR/P3P11/#Well_Known_Location
You put the file at /w3c/p3p.xml
It looks as though p3p policies are similar to robots.txt files.
Additionally you can set p3p headers on all your pages if the robots.txt method isn't the way you want to go. That's a side-note, however, since you want the compact version which I'm assuming is the p3p.xml file.
Hope this helps get you on the right track.