Currently our organization is using the Akamai Fast Purge v3 API to invalidate cache records by cache tag. The problem I'm running into is that some of our lower environments are configured with a TTL of 0 seconds, apparently to facilitate business user testing.
As a result, I'm finding it strangely difficult to manually test the new purge system we have in place because Akamai isn't actually caching anything. We're working with the business to set this to closer match production environments, but in the meantime I'm wondering if there are any debug headers that can be used to figure out if and when an invalidation / deletion occurred.
I have of course googled and found this somewhat useful if not incomplete and IMO strangely half-baked article from Akamai themselves that discuss debug cache headers and their meaning, but it seems very incomplete.
As far as I can tell, X-Cache is my best option from this article and as mentioned because of the 0 second TTL I will always get a MISS.
Are there any additional debug headers that could be useful in determining if my purge logic is effective, despite the 0 second TTL?
The better resource for Akamai's Pragma headers is the documentation site.
That said, there is no response header that will tell you when a cache purge was issued to the platform. Instead, you would have to look at the Control Center Event Viewer where you see and filter all events including just Fast Purge calls.
Related
Whenever we purge cache in Cloudflare, we somehow lose the personalization on the Sitecore SXA page. This happens regularly and also on the subpages especially (basically the parent page shows everything correctly). This happens also most of the time inside the footer page design, as it has the personalization activated for certain component inside the footer.
Does anyone knows what the issue can be? Like I said the problem happens whenever we purge the cache inside the Cloudflare.
Also, this always happens inside the en-us language culture, nowhere else.
With no more info is not so easy to understand where to fix this issue.
In your application you should try to disable cache for static assets, you will get load if under real heavy traffic then take care.
This way you will make resources not cached forcing clients to retrieve all data (css, js and so on).
When you will find the bunch of files which makes you hungry (aka the files which cause your issue) you can restore default expire headers.
You can try this if you are not on free plan (if are on free plan min TTL on edge is 2h):
Cloudflare cache rule
set edge cache TTL to 30s
set browser cache TTL to respect existing
headers
set origin cache control to ON
set performance to OFF
I have a script that I install on a page and it will load some more JS and CSS from an S3 bucket.
I have versions, so when I do a release on Github for say 1.1.9 it will get deployed to /my-bucket/1.1.9/ on S3.
Question, if I want to have something like a symbolic link /my-bucket/v1 -> /my-bucket/1.1.9, how can I achieve this with AWS or CloudFlare?
The idea is that I want to release a new version by deploying it, to my bucket or whatever CDN, and than when I am ready I want to switch v1 to the latest 1.x.y version released. I want all websites to point to /v1 and get the latest when there is new release.
Is there a CDN or AWS service or configuration that will allow me to create a sort of a linux-like symbolic link like that?
A simple solution with CloudFront requires a slight change in your path design:
Bucket:
/1.1.9/v1/foo
Browser:
/v1/foo
CloudFront Origin Path (on the Origin tab)
/1.1.9
Whatever you configure as the Origin Path is added to the beginning of whatever the browser requested before sending the request to the Origin server.
Note that changing this means you also need to do a cache invalidation, because responses are cached based on what was requested, not what was fetched.
There is a potential race condition here, between the time you change the config and invalidate -- there is no correlation in the order of operations between configuration changes and invalidation requests -- a config change followed by an invalidation may be completed after,¹ so will probably need to invalidate, update config, invalidate, verify that the distribution had progressed a stable state, then invalidate once more. You don't need to invalidate objects individually, just /* or /v1*. It would be best if only the resource directly requested is subject to the rewrite, and not it's dependencies. Remember, also, that browser caching is a big cost-saver that you can't leverage as fully if you use the same request URI to represent a different object over time.
More complicated path rewriting in CloudFront requires a Lambda#Edge Origin Request trigger (or you could use Viewer Request, but these run more often and thus cost more and add to overall latency).
¹ Invalidation requests -- though this is not documented and is strictly anecdotal -- appear to involve a bit of time travel. Invalidations are timestamped, and it appears that they invalidate anything cached before their timestamp, rather than before the time they propagate to the edge locations. Architecturally, it would make sense if CloudFront is designed such that invalidations don't actively purge content, but only serve as directives for the cache to consider any cached object as stale if it pre-dates the timestamp on the invalidation request, allowing the actual purge to take place in the background. Invalidations seem to complete too rapidly for any other explanation. This means creating an invalidation request after the distribution returns to the stable Deployed state would assure that everything old is really purged, and that another invalidation request when the change is initially submitted would catch most of the stragglers that might be served from cache before the change is propagated. Changes and invalidations do appear to propagate to the edges via independent pipelines, based on observed completion timing.
I will be using AppFabric caching for an application we are going to build.
We will be maintaining two different caches using the technology.
So there should be proper communication if something happens in one or the other cache.
I have read about notification based caching which notifies on adding, removing or replacing items or regions. I have also tried the same and it is working fine.
What i am looking for is How to notify on Cache Invalidation. ?
Read from MSDN that there are two ways for invalidation cache - 1. TimeOut based and 2. Notification based.
I am looking for some same codes for the same.
Can any one help me out on this.
I don't think there is any way of notification which helps invalidation in the main cache. However, the same are available for local cache invalidation.
We have had a PCI scan on one of our websites passed on to us by one of our clients. There are a number of reports of vulnerabilities that look something like this:
Network service: 80/443 Application URL:
http://www.oursite.com/signup.php
The response contains SQL Server
errors. This suggests that the
hazardous characters inserted by the
test penetrated the application and
reached the SQL query itself (i.e.
that the application is vulnerable to
SQL Injection).
Summary test information: header: header X-Forwarded-For=%2527
I'm not sure how they are saying they have injected code here?
another example they provide for a different URL with supposedly the same issue has this as the exploit:
Summary test information: header:
header X-Forwarded-For='
EDIT
I've had a look into this header and it seems its only set by Proxy's or Load Balancers (which we dont use anyway). Either way, i've spoofed it myself and there is no vulnerability at our end at all so i'm not sure what they are highlighting. Since we make no use of this header i'm not sure what the supposed point of attack would be anyway?
Another example we have of a so-called vulnerability is this:
Network service: 80/443 Application URL:
http://www.oursite.com/products/product-na-here/370
The test successfully embedded a
script in the response, and it will be
executed once the page is loaded in
the user's browser. This means the
application is vulnerable to
Cross-Site Scripting.
Summary test
information:
path: path
/products/product-na-here/370 ->
/products/product-na-here/370,parameter:
header >'">alert(957652)
Again, i'm not sure what is being flagged here at all?
Thanks.
The scans are automated and can generate false positives. It is to alert you to possibilities of vulnerabilities, and you need to either explain how you aren't vulnerable or close the vulnerabilities. (Assuming you're doing this for PCI compliance audit....if not, then you just try to justify/close them internally.)
The scans are based on the OWASP top 10 vulnerabilities (http://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project) as mandated by PCI DSS. Take a look there; there are a lot of nice examples and really in-depth explanations of the vulnerabilities.
Another option is to use an ASV that does not provide solely automated results. There are some good ASVs around that take a blended approach to security results. They manually vet to confirm or deny each and every automatically found vulnerability, as well as providing manual testing to find things that only a human reliably can, such as SQL injection, cross-site scripting and sensitive information leakage, amongst many others, always providing clear examples of the attack vectors required.
Full disclosure: I work for an ASV that provides a service similar to what i describe.
As mentioned by other user most PCI scan results seem to flag either false positives or changing practices. I saw one once recommending that we didnt use bind and that allowing FTP access was a major security hole. I would suggest you challenge their findings where you see fit.
We're hosting a django service for some clients using really really poor and intermittent connectivity. Satellite and GPRS connectivity in parts of Africa that haven't benefited from the recent fiber cables making landfall.
I've consolidated the javascripts and used minificatied versions, tried to clean up the stylesheets, and what not...
Like a good django implementer, I'm letting apache serve up all the static information like css and JS and other static media. I've enabled apache modules deflate (for gzip) and expired to try to minimize retransmission of the javascript packages (mainly jQuery's huge cost). I've also enabled django's gzip middleware (but that doesn't seem to do much in combination with apache's deflate).
Main question - what else is there to do to optimize bandwidth utilization?
Are there django optimizations in headers or what not to make sure that "already seen data" will not travel over the network?
The django caching framework seems to be tailored towards server optimization (minimize hitting the database) - how does that translate to actual bandwidth utilization?
what other tweaks on apache are there to make sure that the browser won't try to get data it already has?
Some of your optimizations are important for wringing better performance out of your server, but don't confuse them with optimizing bandwidth utilization. In other words gzip/deflate are relevant but Apache serving static content is not (even though it is important).
Now, for your problem you need to look at three things: how much data is being sent, how many connections are required to get the data, and how good are the connections.
You mostly have the first area covered by using deflate/gzip, expires, minimization of javascript etc. so I can only add one or two things you might not know about. First, you should upgrade to Django 1.1, if you haven't already, because it has better support for ETags/Expires headers for your Django views. You probably already have those headers working properly for static data from Apache but if you're using older Django they (probably) aren't being set properly on your dynamic views.
For the next area, number of connections, you need to consolidate your javascript and css files into as few files as possible to reduce the number of connections. Also very helpful can be consolidating your image files into a single "sprite" image. There are a few Django projects to handle this aspect: django-compress, django-media-bundler (which is the only one that will create image sprites), and you can also see this SO answer.
For the last area of how good are the connections you should look at global CDN as suggested by Alex, or at the very least host your site at an ISP closer to your users. This could be tough for Africa, which in my experience can't even get decent connectivity into European ISP's (at least southern Africa... northern Africa might be better).
You could delegate jQuery to a CDN which may have better connectivity with Africa, e.g., google (and, it's a free service!-). Beyond that I recommend anything every written (or spoken on video, there's a lot of that!-) by Steve Souders -- while his talks and books and essays are invaluable to EVERY web developer I think they're particularly precious to ones serving a low-bandwidth audience (e.g., one of his tips in his latest books and talks is about a substantial fraction of the world's browsers NOT getting compression benefits from deflate or gzip -- it's not so much about the browsers themselves, but about proxies and firewalls doing things wrong, so "manual compression" is STILL important then!).
This is definitely not an area I've had a lot of experience in, but looking into Django's ConditionalGetMiddleware might prove useful. I think it might help you solve the first of your bullet points.
EDIT: This might be a good place to start: http://docs.djangoproject.com/en/dev/topics/conditional-view-processing/