title: Error after zgrep system call from python for large files only
body: <p>I am using a python script to make a system call to <code>zgrep</code> and only print the first result using the <code>-m1</code> option.</p>
  
  <h3>Script:</h3>
  
  <pre><code>#! /usr/bin/env python2.7
  
  import subprocess
  
  print subprocess.check_output("zgrep -m1 'a' test.txt.gz", shell=True)
  </code></pre>
  
  <h3>Error:</h3>
  
  <p>When running the script on large files (+2MB), the following error is generated.</p>
  
  
  
  <pre class="lang-none prettyprint-override"><code>&gt; ./broken-zgrep.py
  
  gzip: stdout: Broken pipe
  Traceback (most recent call last):
    File "./broken-zgrep.py", line 25, in &lt;module&gt;
      print subprocess.check_output("zgrep -m1 'a' test.txt.gz", shell=True)
    File "/usr/intel/pkgs/python/2.7/lib/python2.7/subprocess.py", line 537, in check_output
      raise CalledProcessError(retcode, cmd, output=output)
  subprocess.CalledProcessError: Command 'zgrep -m1 'a' test.txt.gz' returned non-zero exit status 2
  </code></pre>
  
  <p>However, if I copy the command that python complains about and run it in the shell directly, it works fine. </p>
  
  <pre class="lang-none prettyprint-override"><code>&gt; zgrep -m1 'a' test.txt.gz
  0000000 8c82 524d 67a4 c37d 0595 a457 b110 3192
  </code></pre>
  
  <p>The command's exit status is <code>0</code> after manually running in the shell, which indicates a success. Python says the command exits with error code <code>2</code>.</p>
  
  <pre class="lang-none prettyprint-override"><code>&gt; echo $?
  0
  </code></pre>
  
  <p>Here is how to make an example test file to reproduce the error. It creates a 100000 line hex file of random vaules and uses <code>gzip</code> to compress it. </p>
  
  <pre class="lang-none prettyprint-override"><code>cat /dev/urandom | hexdump | head -n 100000 | gzip &gt; test.txt.gz
  </code></pre>
  
  <hr>
  
  <h3>Seemingly irrelevant changes that will prevent the error:</h3>
  
  <ul>
  <li><p>Make a smaller test file</p>
  
  <p><code>cat /dev/urandom | hexdump | head -n 100 | gzip &gt; test.txt.gz</code></p></li>
  <li><p>Running without the <code>-m1</code> option (warning: will spam terminal)</p>
  
  <p><code>print subprocess.check_output("zgrep 'a' test.txt.gz", shell=True)</code></p></li>
  <li><p>Using <code>grep</code> instead of <code>zgrep</code> on an uncompressed file</p>
  
  <p><code>cat /dev/urandom | hexdump | head -n 100000 &gt; test.txt</code></p>
  
  <p><code>print subprocess.check_output("grep -m1 'a' test.txt", shell=True)</code></p></li>
  <li><p>Running the equivalent command in <code>perl</code></p>
  
  <p><code>perl -e 'print `zgrep -m1 'a' test.txt.gz`'</code></p></li>
  </ul>
  
  <hr>
  
  <p>I don't know why the combination of <code>python</code>, <code>zgrep</code>, <code>-m</code> option, and large files produces this error. If any of these factors is eliminated, then there is no error. </p>
  
  <p>My best guess about the cause is from reading the <code>grep</code> <code>man</code> page about the <code>-m</code> option.</p>
  
  <pre class="lang-none prettyprint-override"><code>   -m NUM, --max-count=NUM
            Stop reading a file after NUM matching lines.  If the  input  is
            standard  input  from a regular file, and NUM matching lines are
            output, grep ensures that the standard input  is  positioned  to
            just  after the last matching line before exiting, regardless of
            the presence of trailing context lines.  This enables a  calling
            process  to resume a search.  When grep stops after NUM matching
            lines, it outputs any trailing context lines.
  </code></pre>
  
  <p>I originally assumed that the <code>-m</code> option would simply cause <code>grep</code> to exit after finding NUM matches. But maybe there is something funny going on with <code>grep</code> and standard input. This still doesn't explain why the error only occours for large compressed files though.</p>
  
  <p>I ended up porting my script from python to perl to get around this problem, so there is not any immediate need for a solution. But I would really like to gain a better understanding of why this perfect storm of circumstances fails.</p>