in-memory decompression of binary data in gzip format

[aryan_]

Hi All,

I have a C++ stringstream variable that stores some compressed binary data in gzip format.

I want to decompress this stringstream variable in memory.

First of all, for in-memory decompression of binary data in gzip format, what third party library do you suggest to use ?

I noticed zlib library for compression/decompression of gzip and deflate formats.

However, the two functions handling decompression that zlip provides do not seem to meet my needs exactly:

int uncompress (Bytef *dest, uLongf *destLen, const Bytef *source, uLong sourceLen); 
int gzread (gzFile file, voidp buf, unsigned len);

The first one (uncompress) requires me to know the length of the decompressed data in advance to properly allocate enough memory for storage. In my case, it is unknown.

On the other hand, the second one (gzread) takes a file as input, not a memory buffer.

What do you suggest for an "efficient" in-memory decompression using zlip or some other library ?

Thanks.

[Rubberman]

These should be as efficient as anything for gzip'd data. Since you can get the pointer to the raw data in your stringstream object, then uncompress() would be the preferred choice I would think. You can allocate a decompression buffer to a reasonable maximum size that you expect to need. If the uncompress() function returns Z_BUF_ERROR indicating the output buffer was too small then you can reallocate the buffer to a bigger size and call uncompress again. If your initial buffer size is sensible, then this should not occur very often so as to create a performance problem.

[aryan_]

These should be as efficient as anything for gzip'd data. Since you can get the pointer to the raw data in your stringstream object, then uncompress() would be the preferred choice I would think. You can allocate a decompression buffer to a reasonable maximum size that you expect to need. If the uncompress() function returns Z_BUF_ERROR indicating the output buffer was too small then you can reallocate the buffer to a bigger size and call uncompress again. If your initial buffer size is sensible, then this should not occur very often so as to create a performance problem.

In my case, initial buffer size is not sensible, which may create a performance problem.

Instead, I noticed following set of functions in zlib which should be used in combination:

int inflateInit(z_streamp strm);
int inflate(z_streamp strm, int flush);
int inflateEnd(z_streamp strm);

They seem to get the exact content size from the gzip header itself, and allocate all data structures accordingly.

Do you have enough experience with them to suggest their use considering the performance issues ?

Thanks.

[Rubberman]

In my case, initial buffer size is not sensible, which may create a performance problem.

Instead, I noticed following set of functions in zlib which should be used in combination:

int inflateInit(z_streamp strm);
int inflate(z_streamp strm, int flush);
int inflateEnd(z_streamp strm);

They seem to get the exact content size from the gzip header itself, and allocate all data structures accordingly.

Do you have enough experience with them to suggest their use considering the performance issues ?

Thanks.

No. I'm not personally familiar with the zlib API's or C++ class interfaces. This sounds like a good starting point, however. Also, since the source code is easily available, you can see what they do to get the inflated size of the compressed data if you want to go more directly to the data.