title: Sharing DEFLATE-compressed data chunks between streaming HTTP clients
body: <p>I want to serve some streaming data over HTTP. Additionally, I'd like to compress the data to save bandwidth. I can trivially compress each streaming response separately, but in my case, the data stream is largely the same for every single client (fanout), so it seems like a waste of CPU time to compress the same data for each connection. My plan is to compress each chunk of streaming data preemptively, so clients who connect at any time can start reading the next chunk (this comes at the cost of reduced compression efficiency, but as long as the individual chunks of data are big enough, this should be fine).</p>
  
  <p>Compliant HTTP clients are apparently supposed to be able to accept multiple gzipped files in a <code>Content-Encoding: gzip</code> response, <a href="https://stackoverflow.com/questions/16740034/http-how-to-send-multiple-pre-cached-gzipped-chunks">but the answers to this question indicate web browsers do not</a>. However, from my understanding of DEFLATE/zlib, you can instead send <a href="http://zlib.net/manual.html#Advanced" rel="nofollow noreferrer"><code>Z_FULL_FLUSH</code> <code>0x0000FFFF</code></a> bytes to reset the stream which should serve the same effect of individually decompressible chunks.</p>
  
  <p>I've set up a simple POC in node.js that streams a message as a Server Sent Events stream, but I can't get a web-browser to read the data; it will open the connection, but will never flush the data. I'm using <code>Z_NO_COMPRESSION</code> for simplicity.</p>
  
  <pre><code>var http, zlib, gzip, numClients;
  http = require('http');
  zlib = require('zlib');
  gzip = zlib.createDeflateRaw({
    flush: zlib.Z_SYNC_FLUSH,
    level: zlib.Z_NO_COMPRESSION
  });
  numClients = 0;
  setInterval(function(){
    if (numClients &gt; 0) {
      gzip.write("data: hi\n\n");
    }
  }, 1000);
  http.createServer(function(req, res){
    res.socket.setTimeout(Infinity);
    res.writeHead(200, {
      'Content-Type': 'text/event-stream',
      'Content-Encoding': 'deflate',
      'Transfer-Encoding': 'identity',
      'Access-Control-Allow-Origin': '*'
    });
  
    res.write('\x78\x01'); // write zlib magic number
    gzip.pipe(res);
    numClients++;
    res.on('close', function(){
      numClients--;
      gzip.unpipe(res);
    });
    res.on('error', function(){
      numClients--;
      gzip.unpipe(res);
    });
  }).listen(8080);
  numClients++;
  gzip.pipe(process.stdout);
  </code></pre>
  
  <p>And a simple client:</p>
  
  <pre><code>&lt;!DOCTYPE html5&gt;
  &lt;html lang=en&gt;
  &lt;meta charset=utf-8&gt;
  &lt;title&gt;hi&lt;/title&gt;
  
  &lt;script&gt;
  var es = new EventSource("http://localhost:8080/");
  es.addEventListener('data', console.log);
  es.addEventListener('open', console.log);
  es.addEventListener('error', console.log);
  &lt;/script&gt;
  </code></pre>
  
  <p>Bytes look like this, (<code>curl -N localhost:8080</code> piped through <code>xxd</code>):</p>
  
  <pre><code>0000000: 7801 000a 00f5 ff64 6174 613a 2068 690a  x......data: hi.
  0000010: 0a00 0000 ffff 000a 00f5 ff64 6174 613a  ...........data:
  0000020: 2068 690a 0a00 0000 ffff 000a 00f5 ff64   hi............d
  0000030: 6174 613a 2068 690a 0a00 0000 ffff 000a  ata: hi.........
  0000040: 00f5 ff64 6174 613a 2068 690a 0a00 0000  ...data: hi.....
  0000050: ffff 000a 00f5 ff64 6174 613a 2068 690a  .......data: hi.
  0000060: 0a00 0000 ffff 000a 00f5 ff64 6174 613a  ...........data:
  </code></pre>
  
  <p>Do I need to add extra framing for a DEFLATE decompressor to detect the flush points?</p>
  
  <p>EDIT: I added the <a href="https://stackoverflow.com/questions/9050260/what-does-a-zlib-header-look-like">zlib magic number</a> to make the http stream a valid DEFLATE stream, but
  the web browser still won't flush the blocks. However, piping the <code>gzip</code> stream back through <code>zlib.createInflate()</code> works fine if the magic number is added. I also know that the http stream isn't being buffered because <code>curl -N localhost:8080</code> will display the raw bytes.</p>