Please advise the best way to compress satellite Image. Details
Uncompressed size - 60 gb
Uncompressed format - IMG
4 Bands (To be retained after compression)
Preferred compression format - JPEG2000
Lossy enough to aid in Visual analysis.
Thanks
Monika
If I'm interpreting your question correctly, you'll already need to look at the file format defined in JPEG2000 Part 2 ("JPX") because of the multiple bands. Beyond that, because you are asking for "visually lossless" (e.g. lossy compression but tuned to the point where you can't see it), you'll need to experiment with various settings using your own files until you achieve what you want. For a small discussion on how the Internet Archive did this with print materials, see A Status Report on JPEG 2000 Implementation for Still Images: The UConn Survey.
You should check out the Kakadu JPEG 2000 software. It's really fast.
The summary of one of their time tests, which sounds in line with our observed results:
'"There is an example on the spreadsheet involving a 13.3K x 13.3K RGB image (531 MBytes), being compressed to 2 bits/pixel, at 0.145 pictures/second, using the speedpack on a standard 2.4 GB core-2 duo machine. From this, it can be inferred that compressing 60 MB down to 5 MB (12:1 is 2 bits/pixel for RGB, although colour properties were not specified in the original request) should occur at a rate of 1.2 pictures/second. Compressing to the slightly lower target size of 4 MB will be a little bit faster."
http://www.kakadusoftware.com/
Related
I am using a Basler camera and want to write images that I grabbed at 1000x1000px to disk as fast as possible.
Keeping a movie container open and saving as .avi doesn't really help (slows down 2x) and saving single images is even slower.
I am currently using openCV cvVideoCreator and imwrite functions.
My computer should be fast enough (Intel Xeon 3.6 GHz, 64GB RAM, SSD).
Any ideas?
There are multiple factors (steps) you may want to consider in order to reach your goal (saving color images (1K x 1K x 3) with 50 FPS or 150 MB/s).
Image encoding: most of well-known image formats such as png, tif, jpg takes time to encode image (e.g., 5.7 ms for png and 11.5 ms for tif in OpenCV with 3.6 GHz 4-core CPU) to the specific format even before saving the encoded format data to a disk.
File opening: Independent of file size, this may take time (e.g., 1.5 ms on 3.5" WD Black)
File writing: Dependent of file size, this may take time (e.g., 2.0 ms on 3.5" WD Black)
File closing: Dependent of file size, this may take a lot of time (e.g., 14.0 ms on 3.5" WD Black)
This means you have to finish all of the steps in 20 ms per image for your goal, and as I gave examples of timing, you may not be able to achieve your goal with OpenCV imwrite in JPG because the library sequentially does all the steps above in a single thread.
I think you have a couple of options
imwrite into BMP format on SSD as its encoding time is virtually zero (e.g., less than 0.3 ms)
do some of steps above (e.g., encoding or file closing) in a separate thread.
Especially, file closing is a good candidate to be run in a separate thread because it can be asynchronously done with the other steps. I was able to reach 400 MB/s bandwidth of saving with the second option, BMP file format, and a better hard disk.
Hope this helps you and others with similar goals.
The specs you state in your question are related to your ability to process and buffer the data, but not about the speed you can dump to disk.
You're trying to write (some numbers assumed, just replace with your own)
1000*1000 (size) * 4 (data/pixel) * 25 (frame rate) bytes per second.
(or 100M/s)
This is around abouts the limit of a traditional HDD, but if the disk is fragmented or full at all it's unlikely to keep up.
As a result you must find a way to either speed up your write time (switch to SSD for example); reduce the data being written (compress, reduction in colour depth / quality / frame rate) or buffer what you want to write while a background thread saves to disk.
The question you must ask is how long do you plan to record for. If it's not long enough to fill up your RAM, then you have all the options available to you. If however you plan to record for extended periods of time then you will have to pick one of the other 2.
I am making my own version of teamviewer, where you can show your screen to other people so that they can help you with issues and provide support. However, I have ran into a slight issue.
Firstly, when I take a screenshot the .BMP is about 1 MB in memory. After using EncodingParams and converting it to a JPEG with ulong quality = 20, it drops to 92 kb. However, I still feel like this might be a little too big to constantly send over a stream.
Is there any way I can reduce the size of an image or any kind of way I can make it less network intensive? Every single byte I remove would help speed it up for slower connections, and use less bandwidth.
I would appreciate if someone could give me some advice.
Thanks
Either use a lower quality than 20, or or reduce the size of the image. Both will of course degrade the quality. But there aren't any obvious compression techniques that you can use on a JPEG compressed image - it's about as small as it will get (even high quality JPEG's will compress rather poorly, because JPEG does a good job at compressing the actual data in the image, as well as "destroying" pixels when compression levels are quite high). You can prove this by taking a number of JPEG images and compress them with your favourite compression program (ZIP, GZIP, BZIP, etc).
You can also reduce the number of colours in the image - a 256 (8-bit) or 64K (16-bit) image will compress much better than a 16M (24/32-bit) image.
Other obvious answer is to only send the DIFFERENCE between one frame and another - this can give you pretty decent benefits as long as the picture isn't changing very much - if someone is playing a "shoot-em up" type game, where things explode and the scene changes at 50 fps, it's probably not going to help much, but for regular office type applications, the number of pixels that change each frame is probably minimal.
I am building an application which takes a great many number of screenshots during the process of "recording" operations performed by the user on the windows desktop.
For obvious reasons I'd like to store this data in as efficient a manner as possible.
At first I thought about using the PNG format to get this done. But I stumbled upon this: http://www.olegkikin.com/png_optimizers/
The best algorithms only managed a 3 to 5 percent improvement on an image of GUI icons. This is highly discouraging and reveals that I'm going to need to do better because just using PNG will not allow me to use previous frames to help the compression ratio. The filesize will continue to grow linearly with time.
I thought about solving this with a bit of a hack: Just save the frames in groups of some number, side by side. For example I could just store the content of 10 1280x1024 captures in a single 1280x10240 image, then the compression should be able to take advantage of repetitions across adjacent images.
But the problem with this is that the algorithms used to compress PNG are not designed for this. I am arbitrarily placing images at 1024 pixel intervals from each other, and only 10 of them can be grouped together at a time. From what I have gathered after a few minutes scanning the PNG spec, the compression operates on individual scanlines (which are filtered) and then chunked together, so there is actually no way that info from 1024 pixels above could be referenced from down below.
So I've found the MNG format which extends PNG to allow animations. This is much more appropriate for what I am doing.
One thing that I am worried about is how much support there is for "extending" an image/animation with new frames. The nature of the data generation in my application is that new frames get added to a list periodically. But I do have a simple semi-solution to this problem, which is to cache a chunk of recently generated data and incrementally produce an "animation", say, every 10 frames. This will allow me to tie up only 10 frames' worth of uncompressed image data in RAM, not as good as offloading it to the filesystem immediately, but it's not terrible. After the entire process is complete (or even using free cycles in a free thread, during execution) I can easily go back and stitch the groups of 10 together, if it's even worth the effort to do it.
Here is my actual question that everything has been leading up to. Is MNG the best format for my requirements? Those reqs are: 1. C/C++ implementation available with a permissive license, 2. 24/32 bit color, 4+ megapixel (some folks run 30 inch monitors) resolution, 3. lossless or near-lossless (retains text clarity) compression with provisions to reference previous frames to aid that compression.
For example, here is another option that I have thought about: video codecs. I'd like to have lossless quality, but I have seen examples of h.264/x264 reproducing remarkably sharp stills, and its performance is such that I can capture at a much faster interval. I suspect that I will just need to implement both of these and do my own benchmarking to adequately satisfy my curiosity.
If you have access to a PNG compression implementation, you could easily optimize the compression without having to use the MNG format by just preprocessing the "next" image as a difference with the previous one. This is naive but effective if the screenshots don't change much, and compression of "almost empty" PNGs will decrease a lot the storage space required.
Is there a table that gives the compression ratio of a jpeg image at a given quality?
Something like the table given on the wiki page, except for more values.
A formula could also do the trick.
Bonus: Are the [compression ratio] values on the wiki page roughly true for all images? Does the ratio depend on what the image is and the size of the image?
Purpose of these questions: I am trying to determine the upper bound of the size of a compressed image for a given quality.
Note: I am not looking to make a table myself(I already have). I am looking for other data to check with my own.
I had exactly the same question and I was disappointed that no one created such table (studies based on a single classic Lena image or JPEG tombstone are looking ridiculous). That's why I made my own study. I cannot say that it is perfect, but it is definitely better than others.
I took 60 real life photos from different devices with different dimensions. I created a script which compress them with different JPEG quality values (it uses our company imaging library, but it is based on libjpeg, so it should be fine for other software as well) and saved results to CSV file. After some Excel magic, I came to the following values (note, I did not calculated anything for JPEG quality lower than 55 as they seem to be useless to me):
Q=55 43.27
Q=60 36.90
Q=65 34.24
Q=70 31.50
Q=75 26.00
Q=80 25.06
Q=85 19.08
Q=90 14.30
Q=95 9.88
Q=100 5.27
To tell the truth, the dispersion of the values is significant (e.g. for Q=55 min compression ratio is 22.91 while max value is 116.55) and the distribution is not normal. So it is not so easy to understand what value should be taken as typical for a specific JPEG quality. But I think these values are good as a rough estimate.
I wrote a blog post which explains how I received these numbers.
http://www.graphicsmill.com/blog/2014/11/06/Compression-ratio-for-different-JPEG-quality-values
Hopefully anyone will find it useful.
Browsing Wikipedia a little more led to http://en.wikipedia.org/wiki/Standard_test_image and Kodak's test suite. Although they're a little outdated and small, you could make your own table.
Alternately, pictures of stars and galaxies from NASA.gov should stress the compressor well, being large, almost exclusively composed of tiny speckled detail, and distributed in uncompressed format. In other words, HUBBLE GOTCHOO!
The compression you get will depend on what the image is of as well as the size. Obviously a larger image will produce a larger file even if it's of the same scene.
As an example, a random set of photos from my digital camera (a Canon EOS 450) range from 1.8MB to 3.6MB. Another set has even more variation - 1.5MB to 4.6MB.
If I understand correctly, one of the key mechanisms for attaining compression in JPEG is using frequency analysis on every 8x8 pixel block of the image and scaling the resulting amplitudes with a "quantization matrix" that varies with the specified compression quality.
The scaling of high frequency components often result in the block containing many zeros, which can be encoded at negligible cost.
From this we can deduce that in principle there is no relation between the quality and the final compression ratio that will be independent of the image. The number of frequency components that can be dropped from a block without perceptually altering its content significantly will necessarily depend on the intensity of those components, i.e. whether the block contains a sharp edge, highly variable content, noise, etc.
Be patient since I haven't worked with compression algorithms much so this may be obvious to some of you. Something I've always noticed when some streaming video starts to lag. I only realized I was curious when looking over this question:
Twitter image encoding challenge
I'm not talking about the pixels themselves but rather the grid like layout that results from the compression. What sort of algorithm or technique is this indicative of? What can you tell me about it?
Take a look at this Wikipedia article on MPEG-2. To quote a part of it:
Briefly, the raw frame is divided into 8 pixel by 8 pixel blocks. The data in each block is transformed by a discrete cosine transform. The result is an 8 by 8 matrix of coefficients. The transform converts spatial variations into frequency variations, but it does not change the information in the block; the original block can be recreated exactly by applying the inverse cosine transform.
In other words, the grid-like structure you see is a direct effect of this DCT being applied to the 8x8 blocks of pixels.
The rationale for blocks is linked to the location/frequency trade off. The image is divided into blocks before the compression in the spectral domain (DCT) so that the artefacts due to the compression are more localized. In standard JPEG, the blocks are of constant size on the whole picture. For more recent formats like JPEG2000, the blocks are adapted to the picture, using wavelets. I am not familiar with video formats details, but the rationale is the same.
This is the same phenomenon for audio coding (mp3): instead of computing the spectrum on the whole audio file, you split the file into some sections of a few samples (a few hundred generally for 44.1 kHz signals). And similarly, if there is corruption of the compressed data (network, corrupted file), you will hear noises which are due to missing windows.
It's called Macroblocking.