adapt zlib zpipe for char arrays - c++

Question

I have a char array (char* dataToInflate) obtained from a .gz file I would like to inflate into another char array.
I don't know the original decompressed size, so I believe this means I can't use the uncompress function that is within the zlib library, since per the manual:
The size of the uncompressed data must have been saved previously by the compressor and transmitted to the decompressor by some mechanism outside the scope of this compression library.
I have looked at the zpipe.C example (https://zlib.net/zpipe.c), and the inf function here looks suitable but I'm not sure how to adapt it from FILEs to char arrays.
Does anyone know how or have any other ideas for inflating a char array into another char array?
Update:
I read here: Uncompress() of 'zlib' returns Z_DATA_ERROR
that for arrays obtained through gzip files, uncompress isn't suitable.
I found that I could decompress the file in full using gzopen, gzread and gzclose like so:
gzFile in_file_gz = gzopen(gz_char_array, "rb");
char unzip_buffer[8192];
int unzipped_bytes;
std::vector<char> unzipped_data;
while (true) {
unzipped_bytes = gzread(in_file_gz, unzip_buffer, 8192);
if (unzipped_bytes > 0) {
unzipped_data.insert(unzipped_data.end(), unzip_buffer, unzip_buffer + unzipped_bytes);
} else {
break;
}
}
gzclose(in_file_gz)
but I would also like to be able to decompress the char array. I tried with the following method:
void test_inflate(Byte *compr, uLong comprLen, Byte *uncompr, uLong *uncomprLen) {
int err;
z_stream d_stream; /* decompression stream */
d_stream.zalloc = NULL;
d_stream.zfree = NULL;
d_stream.opaque = NULL;
d_stream.next_in = compr;
d_stream.avail_in = 0;
d_stream.next_out = uncompr;
err = inflateInit2(&d_stream, MAX_WBITS + 16);
CHECK_ERR(err, "inflateInit");
while (d_stream.total_out < *uncomprLen && d_stream.total_in < comprLen) {
d_stream.avail_in = d_stream.avail_out = 1; /* force small buffers */
err = inflate(&d_stream, Z_NO_FLUSH);
if (err == Z_STREAM_END)
break;
CHECK_ERR(err, "inflate");
}
err = inflateEnd(&d_stream);
*uncomprLen = d_stream.total_out;
}
but in the while loop, the inflate method returns Z_STREAM_END before the file has decompressed in full.
The method returns successfully, but only a partial buffer has been written.
I put a minimum working example here:
https://github.com/alanjtaylor/zlibExample
if anyone has time to look.
Thanks a lot!

Answer 1

The example you have on github, "zippedFile.gz" is a concatenation of seven independent gzip members. This is permitted by the gzip standard (RFC 1952), and the zlib gz* file functions automatically process all of the members.
pigz will show all of the members:
% pigz -lvt zippedFile.gz
method check timestamp compressed original reduced name
gzip 8 e323586d ------ ----- 616431 1543643 60.1% zippedFile
gzip 8 7efd928a ------ ----- 369231 921600 59.9% <...>
gzip 8 7ebd8b2a ------ ----- 919565 2319970 60.4% <...>
gzip 8 3dd6e2ba ------ ----- 619670 1549236 60.0% <...>
gzip 8 c1cb922e ------ ----- 600367 1533151 60.8% <...>
gzip 8 a9fef06c ------ ----- 620250 1541785 59.8% <...>
gzip 8 43b57506 ------ ----- 623081 1555203 59.9% <...>
The inflate* functions will only process one member at a time, in order to let you know with Z_STREAM_END that the member decompressed successfully and that the CRC checked out ok.
All you need to do is put your inflator in a loop and run it until the input is exhausted, or you run into an error. (This is noted in the documentation for inflateInit2 in zlib.h.)
There are a few issues with your inflator, but I understand that it is just an initial attempt to get things working, so I won't comment.

Answer 2

uncompress is indeed designed for where you have all that information ready. It's a utility function.
It probably wraps inflate, which is what you want to use. You have to run it in a loop and manage the "stream" parameters yourself by repeatedly pointing to the next chunk of buffered data until it's all been eaten.
There's an annotated example in the documentation.