title: Why are hanging SSH commands waiting for output from a pipe with both ends open in 'sshd' on the server?
body: <p>This is on StackOverflow as opposed to SuperUser/ServerFault since it has to do with the syscalls and OS interactions being performed by sshd, not the problem I'm having <em>using</em> SSH (though assistance with that is appreciated, too :p).</p>
  
  <p><strong>Context:</strong></p>
  
  <p>I invoke a complex series of scripts via SSH, e.g. <code>ssh user@host -- /my/command</code>. The remote command does a lot of complex forking and execcing and eventually results in a backgrounded daemon process running on the remote host. Occasionally (I'm slowly going mad trying to find out reliable reproduction conditions), the <code>ssh</code> command will never return control to the client shell. In those situations, I can go onto the target host and see an <code>sshd: user@notty</code> process with no children hanging indefinitely.</p>
  
  <p>Fixing that issue is not what this question is about. This question is about what that <code>sshd</code> process is <em>doing</em>.</p>
  
  <p>The SSH implementation is OpenSSH, and the version version is 5.3p1-112.el6_7.</p>
  
  <p><strong>The problem:</strong></p>
  
  <p>If I find one of those stuck <code>sshd</code>s and <code>strace</code> it, I can see it's doing a select on two handles, e.g. <code>select(12, [3 6], [], NULL, NULL</code> or similar. <code>lsof</code> tells me that one of those handles is the TCP socket connecting back to the SSH client. The other is a pipe, <em>the other end of which is only open in the same <code>sshd</code> process</em>. If I search for that pipe by ID using the answer to <a href="https://superuser.com/questions/401066/find-what-process-is-on-the-other-end-of-a-pipe">this SuperUser question</a>, the only process that contains references to that pipe is the same process. <code>lsof</code> confirms this: both the read and write ends of the pipe are open in the same process, e.g. (for pipe 788422703 and <code>sshd</code> PID 22744):</p>
  
  <pre><code>sshd    22744 user    6r  FIFO                0,8      0t0 788422703 pipe
  sshd    22744 user    7w  FIFO                0,8      0t0 788422703 pipe 
  </code></pre>
  
  <p><strong>Questions:</strong></p>
  
  <p>What is SSH waiting for? If the pipe isn't connected to anything and there are no child processes, I can't imagine what event it could be expecting.</p>
  
  <p>What is that "looped" pipe/what does it represent? My only theory is that maybe if STDIN isn't supplied to the SSH client, the target host <code>sshd</code> opens a dummy STDIN pipe so some of its internal child-management code can be more uniform? But that seems pretty tenuous.</p>
  
  <p>How does SSH get into this situation?</p>
  
  <p><strong>What I've Tried/Additional Info:</strong></p>
  
  <ul>
  <li>Initially, I thought this was a handle leak to a daemon. It's possible to create a waiting, child-less <code>sshd</code> process by issuing a command that backgrounds itself, e.g. <code>ssh user@host -- 'sleep 60 &amp;'</code>; <code>sshd</code> will wait for the streams to be closed to the daemonized process; not just the exit of its immediate child. Since the scripts in question eventually result (way down the process tree) in a daemon being started, it initially seemed possible that the daemon was holding onto a handle. However, that doesn't seem to hold up--using the <code>sleep 60 &amp;</code> command as an example, <code>sshd</code> processes communicating with daemons hold and select on <em>four</em> open pipes, not just two, and at least two of the pipes are connected from <code>sshd</code> to the daemon process, not looped. Unless there's a method of tracking/pointing to a pipe I don't know about (and there likely is--for example, I have no idea how <code>dup</code>ed filehandles play into <code>close()</code> semaphore waiting or piping), I don't think the pipe-to-self situation represents a waiting-on-daemon case.</li>
  <li><code>sshd</code> periodically receives communication on the TCP socket/ssh connection itself, which wakes it up out of the <code>select</code>s for a brief period of communication (during which <code>strace</code> shows it blocking SIGCHLD), and then it goes back to waiting on the same FDs.</li>
  <li>It's possible that I'm being affected by <a href="http://www.snailbook.com/faq/background-jobs.auto.html" rel="nofollow noreferrer">this race condition</a> (SIGCHLD getting delivered before the kernel makes data available in the pipe). However, that seems unlikely, both given the rate at which this condition manifests, and the fact that the processes being run on the target host are Perl scripts, and the <a href="http://perldoc.perl.org/5.8.9/functions/close.html" rel="nofollow noreferrer">Perl runtime closes and flushes open file descriptors on shutdown</a>.</li>
  </ul>