Friday, 7 April 2017

Node zlib module inflate, and knowing how much data was read (total_in)?

I'm trying to parse out a stream of data which contains sequences of Zlib compressed data. To do this, I need to know how much data is read for decompressing, so I know where to continue reading from for the next decompression.

In the native zlib library, this value would be exposed through the total_in member of the z_stream_s struct.

I can do this with the JavaScript port of zlib pako, but I would prefer to use the native Node module, as it would avoid the extra dependency and has asyncronous support.

Here's a code sample that shows this dilemma in action.

'use strict';

const zlib = require('zlib');
const pako = require('pako');

const str = 'testing';
const extra = new Uint8Array([2, 4, 6, 8, 10]);

const data = new Uint8Array(zlib.deflateSync(str));

const dataExtra = new Uint8Array(data.length + extra.length);
dataExtra.set(data);
dataExtra.set(extra, data.length);

console.log(data);
console.log(dataExtra);


// Inflate with pako.
// Accessing the amount read is easy.
const pakoInflate = new pako.Inflate();
pakoInflate.push(dataExtra, true);
const pakoInflated = new Buffer(pakoInflate.result.buffer);
console.log(pakoInflated, pakoInflated.toString());
console.log(pakoInflate.strm.total_in);


// Inflate with zlib module.
// Any way to know how much data was actually read?
const zlibInflated = zlib.inflateSync(new Buffer(dataExtra));
console.log(zlibInflated, zlibInflated.toString());

Sample output (the 15 being the length of the compressed data read):

Uint8Array [ 120, 156, 43, 73, 45, 46, 201, 204, 75, 7, 0, 12, 30, 2, 255 ]
Uint8Array [ 120, 156, 43, 73, 45, 46, 201, 204, 75, 7, 0, 12, 30, 2, 255, 2, 4, 6, 8, 10 ]
<Buffer 74 65 73 74 69 6e 67> 'testing'
15
<Buffer 74 65 73 74 69 6e 67> 'testing'

It appears that zlib.inflate* methods do not expose this information, but is there another way to do this? Perhaps one of the other methods enables this? Or is this information completely unavailable to JavaScript code?



via Alexander O'Mara

No comments:

Post a Comment