title: Node.js proxy, dealing with gzip DEcompression
body: <p>I'm currently working on a proxy server where we in this case have to modify the data (by using regexp) that we push through it.</p>
  
  <p>In most cases it works fine except for websites that use gzip as content-encoding (I think), I've come across a module called <a href="https://github.com/egorich239/node-compress" rel="nofollow">compress</a> and tried to push the chunks that I receive through a decompress / gunzip stream but it isn't really turning out as I expected. (see below for code)</p>
  
  <p>figured i'd post some code to support my prob, this is the proxy that gets loaded with mvc (express):</p>
  
  <pre><code>module.exports = {
  index: function(request, response){
      var iframe_url = "www.nu.nl"; // site with gzip encoding    
  
      var http = require('http');     
      var httpClient = http.createClient(80, iframe_url);
      var headers = request.headers;
      headers.host = iframe_url;
  
      var remoteRequest = httpClient.request(request.method, request.url, headers);
  
      request.on('data', function(chunk) {
          remoteRequest.write(chunk);
      });
  
      request.on('end', function() {
          remoteRequest.end();
      });
  
      remoteRequest.on('response', function (remoteResponse){         
          var body_regexp = new RegExp("&lt;head&gt;"); // regex to find first head tag
          var href_regexp = new RegExp('\&lt;a href="(.*)"', 'g'); // regex to find hrefs
  
          response.writeHead(remoteResponse.statusCode, remoteResponse.headers);
  
          remoteResponse.on('data', function (chunk) {
      var body = doDecompress(new compress.GunzipStream(), chunk);
              body = body.replace(body_regexp, "&lt;head&gt;&lt;base href=\"http://"+ iframe_url +"/\"&gt;");
              body = body.replace(href_regexp, '&lt;a href="#" onclick="javascript:return false;"');             
  
              response.write(body, 'binary');
          });
  
          remoteResponse.on('end', function() {
  
              response.end();
              });
          });
      }
  };
  </code></pre>
  
  <p>at the var body part i want to read the body and for example in this case remove all hrefs by replacing them with an #. The problem here of course is when we have an site which is gzip encoded/ compressed it's all jibberish and we can't apply the regexps.</p>
  
  <p>now I've already tired to mess around with the node-compress module:</p>
  
  <pre><code> doDecompress(new compress.GunzipStream(), chunk);
  </code></pre>
  
  <p>which refers to </p>
  
  <pre><code>function doDecompress(decompressor, input) {
    var d1 = input.substr(0, 25);
    var d2 = input.substr(25);
  
    sys.puts('Making decompression requests...');
    var output = '';
    decompressor.setInputEncoding('binary');
    decompressor.setEncoding('utf8');
    decompressor.addListener('data', function(data) {
      output += data;
    }).addListener('error', function(err) {
      throw err;
    }).addListener('end', function() {
      sys.puts('Decompressed length: ' + output.length);
      sys.puts('Raw data: ' + output);
    });
    decompressor.write(d1);
    decompressor.write(d2);
    decompressor.close();
    sys.puts('Requests done.');
  }
  </code></pre>
  
  <p>But it fails on it since the chunk input is an object, so i tried supplying it as an chunk.toString() which also fails with invalid input data.</p>
  
  <p>I was wondering if I am at all heading in the right direction?</p>