Has anyone come across this issue where Amazon Cloudfront seems to refuse to cache a certain small number of static objects?
I've tried invaliding the cache (root path) several times to no avail.
I had a look at the file permissions of the objects in question, and they seemed all ok.
I've also gone into the Amazon Console and there are no errors logged.
You can see more details of this here :
http://www.webpagetest.org/performance_optimization.php?test=171106_A4_be80c122489ae6fabf5e2caadcac8123&run=1#use_of_cdn
My website is using Processwire 3 running Apache and a PW caching product called Procache.
One of your issues is that you are not taking advantage of cache control headers on your objects. This is why you are seeing the message No max-age or expires. Look at this link to learn more about Cache-Control and Expires. Note: You should be using these headers even if you do not use CloudFront as the browser will cache certain objects also.
Using Headers to Control Cache Duration for Individual Objects
You do not indicate what web server that you are using. I have included a link for setting up Apache mod_expires to add cache control headers to your objects.
Apache Module mod_expires
For static assests such as css, js, images, etc. I would setup S3 and serve those objects from S3 via CloudFront. You can control the headers for S3 objects.
The above steps will improve caching of your objects in CloudFront and in the users' browser cache.
Related
I'm building an API and for some responses it will stream the content of S3 objects back to the requester. I would prefer to serve the content directly rather than redirect to send a 302 (e.g. to redirect to a cloudfront distro).
The default is that I read the file into the application and then stream it back out.
If I were using apache or nginx with a local file system I could ask the reverse proxy to stream the content directly from disk with X-Sendfile or X-Accel-Redirect.
Is there an AWS-native mechanism for doing this, so I can avoid loading the file into the application and serving back out again?
I’m not entirely sure I understand your scenario correctly, but I’m thinking in the following direction:
Generally, Cloudfront works like a reverse proxy with a cache attached. (Unlike other vendor’s products where you would “deploy on” the CDN.)
You can attach different types of origins to Cloudfront, it has native support for S3 buckets, but basically everything that speaks HTTP can be attached as a custom origin.
So, in the most trivial scenario, you would place your S3 bucket behind the Cloudfront, add an Origin Access Policy (OAI) and a bucket policy which permits the OAI to access your content.
In order to benefit from caching on the Cloudfront edge, you will need to configure it appropriately, otherwise it will just be a proxy. Make sure to set the Cloudfront TTLs for your content. Check how min/max/default TTL work.
But also don’t forget to set headers for your clients to cache (Cache-Control etc); this may save you a lot of money if the same clients need the same content over and over again.
As we know, caching and cach invalidation in particular, are tricky. Make sure to understand how Cloudfront handles caching to not run into problems. For example: cache busting with query parameters does work, but you need to make Cloudfront aware that the query sting is significant.
Now here comes the exciting part: If you need to react dynamically to the request of the client, you have Lambda#Edge and Cloudfront Functions at your disposal.
Lambda#Edge is basically what it says; Lambda functions on the edge. They can work in four modes: Client request, origin request, origin response, client response. Depends what you need to modify; incoming vs. outgoing data and client-Cloudfront vs. Cloudfront-origin communication.
CF Functions are pretty limited (ES5 only, no XHR or anything, only works on viewer request/response) but very cheap at the same time. Check the AWS docs to determine what you need.
FWIW, Cloudfront also supports signed cookies and signed URLs in case you need to restrict the content to particular viewers.
I am trying to understand how Cloudfront works. Assume static website is static.com and dynamic website is dynamic.com. static.com has thousands of html files containing img tags referencing images coming from static.com.
dynamic.com is Java based dynamically generating HTML and img tags and images comes from dynamic.com
Assume images are not manually copied to s3. No modifications are made in both sites in regards to Cloudfront other than DNS settings.
Assume Cloudfront url setup for static.com is mystaticxyzz.cloudfront.net and for dynamic.com it is mydynamicxyz.cloudfront.net
CloudFront works as a CDN sitting in front of what are called Origins.
These origins are the endpoints that CloudFront forwards traffic to, to retrieve the response and content. This could be a single server, a load balancer or any other resolvable hostname that is publicly accessible.
If you want to split between static and dynamic content you would create an origin for each type of content within the same distribution. One would be the default origin whilst the other would be matched based on a file path (/css or /images).
Each of these origins can include their own cache behaviours which enable you to define whether they should be cached and how long.
When a user accesses the CloudFront domain dependant on the path it will route to the appropriate origin or retrieve a response from the edge cache where possible.
I know this is rather late, but I am just going to add this here for those struggling to cache dynamic and static content.
Firstly, you need to understand your application your application.
Client Side Rendering
if you have a reactjs you don't need to worry too much about your caching behavior as you will be rendering , the data which will be fetched from an api client side.
none of the static files/content will be changing which are being delivered to the end user.
Since the APIs requests will be coming from a different domain , that data won't be cached by the cdn . Moreover, the data being rendered will update the html via javascript. If your javascript files are continously updating then you can use invalidations for them.
If you have content that is not stored on the origin and your CSR app is fetching the content from using a separate domain from your website domain, you will need to set up a separate cdn and point the domain name to that cdn. You wont need to make any changes to your application as the domain name stays the same for that.
However, if you static content that exists in the same origin (e.g. s3) then you would just request the content using the domain name of the cdn from which the request will come from client to cdn to origin (if not cached / expired)
lastly, assume we have separate origins like an s3 bucket for react app and s3 bucket for images . We can set up a single cdn with multiple origins . This means we can use cloudfront as an aggregator , you will then be able to cache content from different origins by using special paths.
This means , where ever you make calls to those origins previously. i.e. using the the s3 domain names, you would need to update them to that single domain name as the caching behaviors will handle the requests to the respective domains
example:
www.example.com(react-app react s3 bucket)
www.example.com/images (some s3 image bucket)
<img source={{url:"www.example.com/images/example.jpg"}} />
cloud front will make a request to that server based on that origin for the behavior configured on "/images"
Server Side Rendered
for serverside rendered apps , ideally the default cahcing behavior on the origin should allow all the different http methods , because you will have post and put http requests which you will want cloudfront to forward to the origin .
Make sure that you forward all query strings and cookies to the origin using a request policy. You can fine tune it with white listing query strings or cookies but this will make life easier. Also, the default caching behavior should use a caching policy that disables cache i.e. min,default,max ttls = 0secs . this is because the content is dynamic in nature and gets rendered on the server and not client side thus you will encounter unexpected behaviors in your application depending on how it is set up.
if you have static content on different paths like "/img", "/css" , or "/web/pages/information" cache those independently from the default behavior the respective ttls on them.
you could do some cool stuff using the cache-control header which can by pass the cache if you dont want to configure a 101 behaviors.
https://aws.amazon.com/blogs/networking-and-content-delivery/improve-your-website-performance-with-amazon-cloudfront/
Just understand your application and you will be able to leverage cdn properly
if you have a webserver that does a mixture of server side and client side rendering
just identify which paths are client-side rendered and cache those static files.
Any thing that is dynamic in nature that requires the application to make requests to the origin , make use of the caching disabled policy within a behavior.
Moreover, any of those patterns(of using a single cdn with a single/multiple origins or multiple cdns with differing origins ) mentioned earlier is applicable to serverside rendering if some content gets rendered clients side such as images
I have set up a CloudFront on top of multiple S3 buckets (in different regions) to provide a fast stable version of my webapp. This webapp is implemented with React which means it's all one single HTML file and one single Javascript file.
Using the routing mechanism of React, all the paths in the URL are handled within the code. This means if I click on a link like www.example.com/users, there won't be a request sent to the server. Instead, the client code will render the appropriate page without any consultation with the server (I'm just talking about the HTML and not considering the data). This means that if some user types in the given URL, the server should return the index.html (the only HTML file I have) which then will take care of the URL on the client-side. In other words, all the requests sent to the server should either return the HTML file or the Javascript file I mentioned earlier. Even the requests that are pointing to none-existing files.
In order to implement this requirement, I asked this question and I got an answer like this:
I need to set up an error page for my distribution on CloudFront and
redirect all the 403 (Forbidden) requests to /index.html file. This
is because when the request is pointing to a nonexisting file on S3,
S3 will return 403 to CloudFront due to the lack of listing
permission. Or I can grant the listing permission and instead handle
the 404 error (I didn't test this latter option).
Anyways, I set this up and it works perfectly - for a few hours. But then, for some unknown reason, the request to the Javascript file also returns the HTML file. And of course, all I'm getting back is actually coming from CloudFront's cache which means, no matter how many times I send the request, it will keep returning the same value. That is until I invalidate the cache on CloudFront which will solve the problem for few more hours. And we go around and around.
Even though I'm not sure why this happens but my guess is that at some point the S3 buck is inaccessible to CloudFront which will result in CloudFront caching the index.html. What can I do about this?
I think I found the problem:
MAKE SURE YOUR STATIC CONTENT ON ALL THE S3 BUCKETS ARE IDENTICAL!!!
In my case, the Javascript filename is automatically generated by Webpack which means it's random. And since different regions were "compiled" separated, their filenames differed.
I have a URL, that is backed by cloudfront to host images. I've since then moved to a new image hosting solution, and need to redirect traffic from the old URL to the new URL, efficiently and without degrading user experience.
For example:
When my website loads an image from www.images.companyName.com/bucket/itemGroup1/itemId, I need traffic to go instead to www.someWebsite.com/xx/bucket/itemGroup1/itemId (notice the path has also slightly changed)
How can this be done leveraging AWS, if it cannot, what other options do I have. I've been thinking Lambda#Edge, but I am not sure if this is efficient.
If this is a one time Migration where you are expecting to achieve zero down time for the end users, you can set up a syncing mechanism from old bucket to new (keeping the old files also for the moment) and update the Cloudfront distribution with both Origins and Behaviors as necessary. This is assuming some clients will have the old paths and some will access the latest. Then later analyzing the Cloudfront logs you can decide to remove old Origin and Behaviors.
If you are planning for an URL rewrite, then you need to use Lambda#Edge for this, specially if there is no way of informing the end users that the URL is updated.
Let's say I have two files: one for safari and one for Firefox.
I want to check User-Agent and return file based on the User-Agent.
How do I do this without adding external server?
You can't do this without adding an extra server.
S3 supports static content. It does not¹ vary its response based on request headers.
CloudFront relies on the origin server if content needs to vary based on request headers. Note that by default, CloudFront doesn't forward most headers to the origin, but this can be changed in the cache behavior configuration. If you forward the User-Agent header to the origin, your cache hit rate drops dramatically, since CloudFront has no choice but to assume any and every change in the user agent string could trigger a change in the response, so an object in the cache that was requested by a specific user agent string will only be served to a future browser with an identical user agent string. It will cache each different copy, but this still hurts your hit rate. If you only want to know the general type of browser, CloudFront can inject special headers to tell the origin whether the user agent is desktop, smart-tv, mobile, or tablet, without actually forwarding the user agent string and causing the same negative impact on the cache hit ratio.
So CloudFront will correctly cache the appropriate version of a page for each unique user agent... but the origin server must implement the actual content selection logic. And when the origin is S3, that isn't supported -- unless you have a server between CloudFront and S3. This is a perfectly valid configuration -- I have such a setup, with a server that rewrites the request path received from CloudFront before sending the request to S3, then returns the content from S3 back to CloudFront, which returns the content to the browser.
AWS Lambda would be a potential candidate for an application like this, acting as the necessary server (a serverless server, if you will) between CloudFront and S3... but it does not yet suport binary data, so for anything other than text, that isn't an option, either.
¹At least, not in any sense that is relevant, here. Exceptions exist for CORS and when access is granted or denied based on a limited subset of request headers.