How to compress/decompress buffer using Fast-LZMA2 - c++

Question

I want to compress/decompress a unsigned char buffer using fast-LZMA2 by 7Zip : https://github.com/conor42/fast-lzma2
In the sample there's two function :
static int compress_file(FL2_CStream *fcs)
{
unsigned char in_buffer[8 * 1024];
unsigned char out_buffer[4 * 1024];
FL2_inBuffer in_buf = { in_buffer, sizeof(in_buffer), sizeof(in_buffer) };
FL2_outBuffer out_buf = { out_buffer, sizeof(out_buffer), 0 };
size_t res = 0;
size_t in_size = 0;
size_t out_size = 0;
do {
if (in_buf.pos == in_buf.size) {
in_buf.size = fread(in_buffer, 1, sizeof(in_buffer), fin);
in_size += in_buf.size;
in_buf.pos = 0;
}
res = FL2_compressStream(fcs, &out_buf, &in_buf);
if (FL2_isError(res))
goto error_out;
fwrite(out_buf.dst, 1, out_buf.pos, fout);
out_size += out_buf.pos;
out_buf.pos = 0;
} while (in_buf.size == sizeof(in_buffer));
do {
res = FL2_endStream(fcs, &out_buf);
if (FL2_isError(res))
goto error_out;
fwrite(out_buf.dst, 1, out_buf.pos, fout);
out_size += out_buf.pos;
out_buf.pos = 0;
} while (res);
fprintf(stdout, "\t%ld -> %ld\n", in_size, out_size);
return 0;
error_out:
fprintf(stderr, "Error: %s\n", FL2_getErrorName(res));
return 1;
}
static int decompress_file(FL2_DStream *fds)
{
unsigned char in_buffer[4 * 1024];
unsigned char out_buffer[8 * 1024];
FL2_inBuffer in_buf = { in_buffer, sizeof(in_buffer), sizeof(in_buffer) };
FL2_outBuffer out_buf = { out_buffer, sizeof(out_buffer), 0 };
size_t res;
size_t in_size = 0;
size_t out_size = 0;
do {
if (in_buf.pos == in_buf.size) {
in_buf.size = fread(in_buffer, 1, sizeof(in_buffer), fout);
in_size += in_buf.size;
in_buf.pos = 0;
}
res = FL2_decompressStream(fds, &out_buf, &in_buf);
if (FL2_isError(res))
goto error_out;
/* Discard the output. XXhash will verify the integrity. */
out_size += out_buf.pos;
out_buf.pos = 0;
} while (res && in_buf.size);
fprintf(stdout, "\t%ld -> %ld\n", in_size, out_size);
return 0;
error_out:
fprintf(stderr, "Error: %s\n", FL2_getErrorName(res));
return 1;
}
But I have no idea how to make it work with a buffer and also without size limit like 8*1024
like zlib deflate compression.
I want something like
LZMA2_Compress(void* buffer,size_t bufferSize);
and LZMA2_Decompress(void* buffer,size_t bufferSize);
I want to use this algorithm on some heavy files and Fast LZMA2 is the fastest high ratio compression I found, Please don't suggest me using other methods.
Here's my test code, It's working but just need to correct information:
https://gist.github.com/Bit00009/3241bb66301f8aaba16074537d094e61

Answer 1

Check the header file for all of the functions available. This one looks like the one you need. You will need to cast your buffers as (void *).
High level functions
fast-lzma2.h
...
/*! FL2_compress() :
* Compresses `src` content as a single LZMA2 compressed stream into already allocated `dst`.
* Call FL2_compressMt() to use > 1 thread. Specify nbThreads = 0 to use all cores.
* #return : compressed size written into `dst` (<= `dstCapacity),
* or an error code if it fails (which can be tested using FL2_isError()). */
FL2LIB_API size_t FL2LIB_CALL FL2_compress(void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
int compressionLevel);
...
Management of memory and options
To do explicit memory management (set dictionary size, buffer size, etc.) you need to create a context:
fast-lzma2.h
/*= Compression context
* When compressing many times, it is recommended to allocate a context just once,
* and re-use it for each successive compression operation. This will make workload
* friendlier for system's memory. The context may not use the number of threads requested
* if the library is compiled for single-threaded compression or nbThreads > FL2_MAXTHREADS.
* Call FL2_getCCtxThreadCount to obtain the actual number allocated. */
typedef struct FL2_CCtx_s FL2_CCtx;
FL2LIB_API FL2_CCtx* FL2LIB_CALL FL2_createCCtx(void);
than you can use FL2_CCtx_setParameter() to set the parameters in the context. The possible values for the paramters are listed in FL2_cParameter , and the value FL2_p_dictionarySize will allow you to set the dictionary size.
/*! FL2_CCtx_setParameter() :
* Set one compression parameter, selected by enum FL2_cParameter.
* #result : informational value (typically, the one being set, possibly corrected),
* or an error code (which can be tested with FL2_isError()). */
FL2LIB_API size_t FL2LIB_CALL FL2_CCtx_setParameter(FL2_CCtx* cctx, FL2_cParameter param, size_t value);
Finally you can compress the buffer by calling FL2_compressCCtx()
/*! FL2_compressCCtx() :
* Same as FL2_compress(), but requires an allocated FL2_CCtx (see FL2_createCCtx()). */
FL2LIB_API size_t FL2LIB_CALL FL2_compressCCtx(FL2_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
int compressionLevel);