title: Why does IPC::Open3 get deadlocked?
body: <p>I went through the documentation of <a href="http://perldoc.perl.org/IPC/Open3.html" rel="noreferrer">open3</a> and here is the portion that I could not understand:</p>
  <blockquote>
  <p>If you try to read from the child's stdout writer and their stderr
  writer, you'll have problems with blocking, which means you'll want to
  use select() or the IO::Select, which means you'd best use sysread()
  instead of readline() for normal stuff.</p>
  <p>This is very dangerous, as you may block forever. It assumes it's
  going to talk to something like bc, both writing to it and reading
  from it. This is presumably safe because you &quot;know&quot; that commands like
  bc will read a line at a time and output a line at a time. Programs
  like sort that read their entire input stream first, however, are
  quite apt to cause deadlock.</p>
  </blockquote>
  <p>So I tried out <code>open3</code> hoping to know it better. Here is the first attempt:</p>
  <pre><code>sub hung_execute {
      my($cmd) = @_;
      print &quot;[COMMAND]: $cmd\n&quot;;
      my $pid = open3(my $in, my $out, my $err = gensym(), $cmd);
      print &quot;[PID]: $pid\n&quot;;
      waitpid($pid, 0);
      if(&lt;$err&gt;) {
          print &quot;[ERROR] : $_&quot; while(&lt;$err&gt;);
          die;
      }
      print &quot;[OUTPUT]: $_&quot; while (&lt;$out&gt;);
  }
  </code></pre>
  <p>It's interesting to note that I must initialize <code>$err</code> here.</p>
  <p>Anyway, this just hangs when I <code>execute(&quot;sort $some_file&quot;);</code> given that <code>$some_file</code> is a text file containing more than 4096 chars (limits for my machine).</p>
  <p>I then looked into <a href="http://perldoc.perl.org/perlfaq8.html#How-can-I-capture-STDERR-from-an-external-command%3f" rel="noreferrer">this</a> FAQ, and below was my new version of execute:</p>
  <pre><code>sub good_execute {
      my($cmd) = @_;
      print &quot;[COMMAND]: $cmd\n&quot;;
      my $in = gensym();
      #---------------------------------------------------
      # using $in, $out doesn't work. it expects a glob?
      local *OUT = IO::File-&gt;new_tmpfile;
      local *ERR = IO::File-&gt;new_tmpfile;
      my $pid = open3($in, &quot;&gt;&amp;OUT&quot;, &quot;&gt;&amp;ERR&quot;, $cmd);
      print &quot;[PID]: $pid\n&quot;;
      waitpid($pid, 0);
      seek $_, 0, 0 for \*OUT, \*ERR;
      if(&lt;ERR&gt;) {
          print &quot;[ERROR] : $_&quot; while(&lt;ERR&gt;);
          die;
      }
      print &quot;[OUTPUT]: $_&quot; while (&lt;OUT&gt;);
  }
  </code></pre>
  <p>The <code>sort</code> command executes fine now, but I can't figure out why.</p>
  <p><strong>[Update]</strong> After reading @tchrist's answer, I read <code>IO::Select</code>, and after some more googling, have come up with this version of <code>execute</code>:</p>
  <pre><code>sub good_execute {
      my($cmd) = @_;
      print &quot;[COMMAND]: $cmd\n&quot;;
      my $pid = open3(my $in, my $out, my $err = gensym(), $cmd);
      print &quot;[PID]: $pid\n&quot;;
      my $sel = new IO::Select;
      $sel-&gt;add($out, $err);
      while(my @fhs = $sel-&gt;can_read) {
          foreach my $fh (@fhs) {
              my $line = &lt;$fh&gt;;
              unless(defined $line) {
                  $sel-&gt;remove($fh);
                  next;
              }
              if($fh == $out) {
                  print &quot;[OUTPUT]: $line&quot;;
              }elsif($fh == $err) {
                  print &quot;[ERROR] : $line&quot;;
              }else{
                  die &quot;[ERROR]: This should never execute!&quot;;
              }
          }
      }
      waitpid($pid, 0);
  }
  </code></pre>
  <p>This is working fine, and a few things have become clearer now. But the overall picture is still a little hazy.</p>
  <p>So my questions are:</p>
  <ol>
  <li>What's wrong with <code>hung_execute</code>?</li>
  <li>I guess <code>good_execute</code> works because of the <code>&gt;&amp;</code> in the open3 call. But why and how?</li>
  <li>Also, <code>good_execute</code> did not work when I used lexical variables (<code>my $out</code> instead of <code>OUT</code>) for filehandles. It gave this error: <code>open3: open(GLOB(0x610920), &gt;&amp;main::OUT) failed: Invalid argument</code>. Why so?</li>
  <li>It seems that only one of the file-handles can write at a given time, and if I discard the handle that's holding on to the resource, the other handles keep on waiting. I used to think that STDERR and STDOUT were independent streams, and did not share any resource. I guess my understanding is a bit flawed here. Kindly give me some pointers on this too.</li>
  </ol>