How to use ZLIB deflate method? - compression

Question

I am trying to use zlib to compress a text file. It seems to kinda work except I pretty sure my calculation of the number of bytes to write to the output is wrong. My code (guided by http://zlib.net/zlib_how.html) is below:
int
deflateFile(
char *infile,
char *outfile)
{
#define CHUNKSIZE 1000
int n,nr,nw,towrite;
z_stream strm;
FILE *fin,*fout;
BYTE *inbuf,*outbuf;
int ntot=0;
printf( "Start doDeflateFile:\n" );
// ALLOC BUFFERS
inbuf = malloc( CHUNKSIZE+1 );
outbuf = malloc( CHUNKSIZE+1 );
// OPEN FILES
fin = fopen( infile, "rb" );
fout = fopen( outfile, "wb" );
// SETUP Z STREAM
strm.zalloc = Z_NULL;
strm.zfree = Z_NULL;
strm.opaque = Z_NULL;
strm.avail_in = CHUNKSIZE; // size of input
strm.next_in = inbuf; // input buffer
strm.avail_out = CHUNKSIZE; // size of output
strm.next_out = outbuf; // output buffer
deflateInit( &strm, Z_BEST_COMPRESSION ); // init stream level
while( TRUE ) { // loop til EOF on input file
// READ NEXT INPUT CHUNK
nr = fread( inbuf, 1, CHUNKSIZE, fin );
if( nr <= 0 ) {
printf( "End of input\n" );
break;
}
printf( "\nread chunk of %6d bytes\n", nr );
printf( "calling deflate...\n" );
n = deflate(&strm, Z_FINISH); // call ZLIB deflate
towrite = CHUNKSIZE - strm.avail_out; // calc # bytes to write (FIXME???)
printf( "#bytes to write %6d bytes\n", towrite );
nw = fwrite( outbuf, 1, towrite, fout );
if( nw != towrite ) break;
printf( "wrote chunk of %6d bytes\n", nw );
ntot += nw;
}
deflateEnd(&strm); // end deflate
printf( "wrote total of %d bytes\n", ntot );
printf( "End deflateFile.\n" );
return( 0 );
}
The output for a 1010-byte input file with a CHUNKSIZE of 1000 is:
Start deflateFile:
read chunk of 1000 bytes
calling deflate...
#bytes to write 200 bytes
wrote chunk of 200 bytes
read chunk of 10 bytes
calling deflate...
#bytes to write 200 bytes
wrote chunk of 200 bytes
End of input
wrote total of 400 bytes
End deflateFile.
SO #4538586 sort of addressed this but not quite and it's very old..
Can anybody point out my problem?

Answer 1

You should read that page again. Much more carefully this time.
You are not setting avail_in properly at the start, and you are not resetting next_in, avail_in, next_out, and avail_out in the loop. The only thing you are doing correctly is the thing you think is wrong, which is the calculation of how many bytes to write out. What you have will not even "kinda work".
First off, avail_in must always be set to the amount of available input at next_in. Hence the name avail_in. You are setting it to CHUNKSIZE and calling inflateInit(), even though there is no available input in that buffer yet.
Then after you read data into the input buffer, you ignore nr! You need to set avail_in to nr, to indicate how much data is actually in the buffer. It might be less than CHUNKSIZE.
You should read data into the input buffer only if you have processed all of the data that was there from the last read, indicated by avail_in being zero.
When a call of deflate() completes inside the loop, it has updated next_in, avail_in, next_out, and avail_out. To use the inbuf and outbuf buffers again, you need reset the values of next_in, next_out, and avail_out to the values you did initially. avail_in will be set at the top of the loop from nr.
You are calling deflate() with Z_FINISH every time! The way this works is that you call deflate() with Z_NO_FLUSH until the last of the input is provided, and then you use Z_FINISH, to let it know to finish. (That's why it's called that.)
Your loop will exit prematurely, since you need to finish compressing and writing the output, not just finish reading the input.
You are not checking the return code of deflate(). Always check return codes. That's why they're there.
Good luck.