I need to Enable "gzip" compression on the Hapi.js API endpoints to improve the performance on API data downtime and download data size.
How should I enable compression in hapi.js ?
Any idea pls...
gzip is enabled by default, so look no further, inspect your headers
Related
Can anyone please help me before I go crazy?
I have been searching for any documentation/sample-code (in JavaScript) for uploading files to S3 via CloudFront but I can't find a proper guide.
I know I could use Tranfer Acceleration feature for faster uploads and yeah, Transfer Acceleration essentially does the job through CloudFront Edge Points but as long as I searched, it is possible to make the POST/PUT request via AWS.CloudFront...
Also read an article posted in 2013 says that AWS just added a functionality to make POST/PUT requests but says not a single thing about how to do it!?
CloudFront documentation for JavaScript sucks, it does not even show any sample codes. All they do is assuming that we already know all the things about the subject. If I knew, why would I dive into documentation in the first place.
I believe there is some confusion here about adding these requests. This feature was added simply to allow POST/PUT requests to be supported for your origin so that functionality in your application such as form submissions or API requests would now function.
The recommended approach as you pointed out is to make use of S3 transfer acceleration, which actually makes use of the CloudFront edge locations.
Transfer Acceleration takes advantage of Amazon CloudFront’s globally distributed edge locations. As the data arrives at an edge location, data is routed to Amazon S3 over an optimized network path.
AWS CloudFront has the option to compress files (see here)
What are the pros and cons of using AWS CloudFront gzip compression vs using the compression-webpack-plugin?
CloudFront compresses content on the fly and it has certain limitation.
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html
The common problem is
CloudFront is busy
In rare cases, when a CloudFront edge location is unusually busy, some files might not be compressed.
This is most common problem because if this happens CF caches a non compressed copy and serve it until you clear the cache.
compression-webpack-plugin on the other hand compresses at the disk which is more helpful.
I am running a cityscape and nature photography website that contains a lot of images which range from 50kb-2mb in size. I have already shrunk them down in size using a batch photo editor so I can't lose any more quality in the images without them getting too grainy.
Google page insights recommends lossless compression and I am trying to figure out how to solve this. These specific images are in s3 buckets and being served by AWS cloudfront
Losslessly compressing https://d339oe4gm47j4m.cloudfront.net/bw107.jpg could save 57.6KiB (38% reduction).
Losslessly compressing https://luminoto-misc.s3-us-west-2.amazonaws.com/bob_horsch.jpg could save 40.6KiB (42% reduction). ...... and a hundred more of the same.
Can Cloudfront do the compression before the image is server to the client? Or do I have to do some other type of compression and then reupload each file to a new s3 bucket. I am looking for a solution where cloudfront will do it.
I have searched around but haven't found a definitive answer.
Thanks,
Jeff
Update
As implicitly pointed out by Ryan Parman (+1), there are two different layers at play when it comes to compression (and/or optimization), which seem to get mixed a bit in this discussion so far:
My initial answer below has addressed lossless compression using Cloudfront as per your question title, which is concerned with the HTTP compression layer:
HTTP compression is a capability that can be built into web servers and web clients to make better use of available bandwidth, and provide greater transmission speeds between both.
[...] data is compressed before it is sent from the server: compliant browsers will announce what methods are supported to the server before downloading the correct format; browsers that do not support compliant compression method will download uncompressed data. [...]
That is, the compress/decompress operation is usually automatically handled by the server and the client to optimize bandwidth usage and transmission performance - the difference with CloudFront is, that its server implementation does indeed not handle compression automatically like most web servers, which is why you need to prepare a compressed representation yourself if desired.
This kind of compression works best with text files like HTML, CSS and JavaScript etc., but isn't useful (or even detrimental) with binary data formats that are already compressed by themselves like ZIP and other prepacked archives and esp. image formats like PNG and JPEG.
Now, your question body talks about a different compression/optimization layer all together, namely lossy JPEG_compression and specifically Lossless_editing as well as optimization via jpegoptim - this has nothing to do with how files are handled by HTTP servers and clients, rather just compressing/optimizing the files themselves to better match the performance constraints within specific use cases like web or mobile browsing, where the transmission of a digital photo in its original size wouldn't make any sense when it is simply to be viewed on a web page for example.
This kind of compression/optimization is one that is rarely offered by web servers themselves so far, even though notable efforts like Google's mod_pagespeed are available these days - usually it is the responsibility of the web designer to prepare appropriate assets, ideally optimized for and selectively delivered to the expected target audience via CSS Media queries.
Initial Answer
AWS CloudFront is capable of Serving Compressed Files, however, this is to be taken literally:
Amazon CloudFront can serve both compressed and uncompressed files
from an origin server. CloudFront relies on the origin server either
to compress the files or to have compressed and uncompressed versions
of files available; CloudFront does not perform the compression on
behalf of the origin server. With some qualifications, CloudFront can
also serve compressed content from Amazon S3. For more information,
see Choosing the File Types to Compress. [emphasis mine]
That is, you'll need to provide compressed versions yourself, but once you've set this up, this is transparent for clients - please note that the content must be compressed using gzip; other compression algorithms are not supported:
[...] If the request header includes additional content encodings, for example, deflate or sdch, CloudFront removes them before forwarding the request to the origin server. If gzip is missing from the Accept-Encoding field, CloudFront serves only the uncompressed version of the file. [...]
Details regarding the requirements and process are outlined in How CloudFront Serves Compressed Content from a Custom Origin and Serving Compressed Files from Amazon S3.
JPEGOptim doesn't do any compression -- it does optimization.
The short answer is, yes, you should always use JPEGOptim on your .jpg files to optimize them before uploading them to S3 (or whatever your source storage is). This has been a good idea since forever.
If you're talking about files which are plain text-based (e.g., CSS, JavaScript, HTML), then gzip-compression is the appropriate solution, and Steffen Opel would have had the 100% correct answer.
The only compression amazon really supports is zip or gzip. You are able to load those compressions into S3, and then do things like loads directly into resources like Redshift. So in short, no amazon does not provide you with the service you are looking for. This would be something you would have to leverage yourself...
We currently have some files stored on an S3 server. The files are log files (.log extension but plain text content) that have been gzipped to reduce disk space.
But gzip isn't splittable and now we are looking for a few good alternatives to store/process our files on Amazon EMR.
So what is the best compression or file format to use on log files? I came across avro and SequenceFile, bzip2, LZO and snappy. It's a bit much and I am a bit overwhelmed.
So I would appreciate any insights in this matter.
Data is to be used for pig jobs (map/reduce jobs)
Kind regards
If you check the Best Practices for Amazon EMR there's a section talking about compressing the outputs:
Compress mapper outputs–Compression means less data written to disk,
which improves disk I/O. You can monitor how much data written to disk
by looking at FILE_BYTES_WRITTEN Hadoop metric. Compression can also
help with the shuffle phase where reducers pull data. Compression can
benefit your cluster HDFS data replication as well. Enable compression
by setting mapred.compress.map.output to true. When you enable
compression, you can also choose the compression algorithm. LZO has
better performance and is faster to compress and decompress.
Hi We can use following algorithms as per our use cases.
GZIP(Algorithm) : Splittable(No), Compression Ratio(High),Compress and Decompress Speed(Medium)
SNAPPY(Algorithm) : Splittable(No), Compression Ratio(LOW),Compress and Decompress Speed(Very Fast)
BZIP2(Algorithm) : Splittable(Yes), Compression Ratio(Very High),Compress and Decompress Speed(Slow)
LZO(Algorithm) : Splittable(Yes), Compression Ratio(LOW),Compress and Decompress Speed(FAST)
I am using the amazon services S3 and CloudFront for a web application and I would like to have various statistics about accessing the data that I am providing through the logs of those services (there is logging activated in both services).
I did a bit of googling and the only thing I could find is how to manage my S3 storage. I also noticed that newrelic offers monitoring for many amazon services but not for those 2.
Is there something that you use? A service that could read my logs periodically and provide me with some nice analytics that would make developers and managers happy?
e.g.
I am trying to avoid writing my own log parsers.
I believe Piwik supports the Amazon S3 log format. Take a look at their demo site to see some example reports.
Well, this may not be what you expect but I use qloudstat for my cloudfront distributions.
The $5 plan covers my needs thats less than a burrito here where I live.
Best regards.
Well, we have a SaaS product Cloudlytics which offers you many reports including, Geo, IP tracking, SPAM, CloudFront cost analysis. You can try it for free for upto 25 MB of logs.
I might be answering this very late. But I have worked on a golang library that can run analysis of CDN and S3 usages and store them in a backend of your choice varying from influxdb, MongoDB or Cassandra for later time series evaluations. The project is hosted at http://github.com/meson10/cdnlysis
See if this fits.
Popular 3rd party analytics packages include S3stat, Cloudlytics and Qloudstat. They all run around $10/month for low traffic sites.
Several stand-alone analytics packages support Amazon's logfile format if you want to download logs each night and feed them in directly. Others might need pre-processing to transform to Combined Logfile Format (CLF) first.
I've written about how to do that here:
https://www.expatsoftware.com/articles/2007/11/roll-your-own-web-stats-for-amazon-s3.html