title: How to properly handle a gzipped page when using curl?
body: <p>I wrote a bash script that gets output from a website using curl and does a bunch of string manipulation on the html output. The problem is when I run it against a site that is returning its output gzipped. Going to the site in a browser works fine. </p>
  
  <p>When I run curl by hand, I get gzipped output:</p>
  
  <pre><code>$ curl "http://example.com"
  </code></pre>
  
  <p>Here's the header from that particular site:</p>
  
  <pre><code>HTTP/1.1 200 OK
  Server: nginx
  Content-Type: text/html; charset=utf-8
  X-Powered-By: PHP/5.2.17
  Last-Modified: Sat, 03 Dec 2011 00:07:57 GMT
  ETag: "6c38e1154f32dbd9ba211db8ad189b27"
  Expires: Sun, 19 Nov 1978 05:00:00 GMT
  Cache-Control: must-revalidate
  Content-Encoding: gzip
  Content-Length: 7796
  Date: Sat, 03 Dec 2011 00:46:22 GMT
  X-Varnish: 1509870407 1509810501
  Age: 504
  Via: 1.1 varnish
  Connection: keep-alive
  X-Cache-Svr: p2137050.pubip.peer1.net
  X-Cache: HIT
  X-Cache-Hits: 425
  </code></pre>
  
  <p>I know the returned data is gzipped, because this returns html, as expected:</p>
  
  <pre><code>$ curl "http://example.com" | gunzip
  </code></pre>
  
  <p>I don't want to pipe the output through gunzip, because the script works as-is on other sites, and piping through gzip would break that functionality.</p>
  
  <h1>What I've tried</h1>
  
  <ol>
  <li>changing the user-agent (I tried the same string my browser sends, "Mozilla/4.0", etc)</li>
  <li>man curl</li>
  <li><a href="http://xkcd.com/979/">google search</a></li>
  <li>searching stackoverflow</li>
  </ol>
  
  <p>Everything came up empty</p>
  
  <p>Any ideas?</p>