title: Python Paramiko timeout with long execution, need full output
body: <p>There's lots of topics touching on part of the title, but nothing that quite satisfies the whole thing.  I'm pushing a command on a remote server and need the full output after a long execution time, say 5 minutes or so.  Using channel I was able to set a timeout, but when I read back stdout I got only a small portion of output.  The solution seemed to be to wait for channel.exit_status_ready().  This worked on a successful call, but a failed call would never trigger the channel timeout.  Having reviewed the docs, I theorize that's because the timeout only works on a read operation, and waiting for exit status doesn't qualify.  Here's that attempt:</p>
  
  <pre><code>channel = ssh.get_transport().open_session()
  channel.settimeout(timeout)
  channel.exec_command(cmd)  # return on this is not reliable
  while True:
      try:
          if channel.exit_status_ready():
              if channel.recv_ready():  # so use recv instead...
                  output = channel.recv(1048576)
                  break
          if channel.recv_stderr_ready():  # then check error
              error = channel.recv_stderr(1048576)
              break
      except socket.timeout:
          print("SSH channel timeout exceeded.")
          break
      except Exception:
          traceback.print_exc()
          break
  </code></pre>
  
  <p>Pretty, ain't it?  Wish it worked.</p>
  
  <p>My first attempt at a solution was to use time.time() to get a start, then check start - time.time() > timeout.  This seems straightforward, but in my present version, I output start - time.time() with a fixed timeout that should trigger a break...and see differences that double and triple the timeout with no break occurring.  To save space, I'll mention my third attempt, which I've rolled up with this one.  I read on here about using select.select to wait for output, and noted in the documentation that there's a timeout there as well.  As you'll see from the code below, I've mixed all three methods -- channel timeout, time.time timeout, and select timeout -- yet still have to kill the process.  Here's the frankencode:</p>
  
  <pre><code>channel = ssh.get_transport().open_session()
  channel.settimeout(timeout)
  channel.exec_command(cmd)  # return on this is not reliable
  print("{0}".format(cmd))
  start = time.time()
  while True:
      try:
          rlist, wlist, elist = select([channel], [], [],
              float(timeout))
          print("{0}, {1}, {2}".format(rlist, wlist, elist))
          if rlist is not None and len(rlist) &gt; 0:
              if channel.exit_status_ready():
                  if channel.recv_ready():  # so use recv instead...
                      output = channel.recv(1048576)
                      break
          elif elist is not None and len(elist) &gt; 0:
              if channel.recv_stderr_ready():  # then check error
                  error = channel.recv_stderr(1048576)
                  break
          print("{0} - {1} = {2}".format(
              time.time(), start, time.time() - start))
          if time.time() - start &gt; timeout:
              break
      except socket.timeout:
          print("SSH channel timeout exceeded.")
          break
      except Exception:
          traceback.print_exc()
          break
  </code></pre>
  
  <p>Here's some typical output:</p>
  
  <pre><code>[&lt;paramiko.Channel 3 (open) window=515488 -&gt; &lt;paramiko.Transport at 0x888414cL (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))&gt;&gt;], [], []
  1352494558.42 - 1352494554.69 = 3.73274183273
  </code></pre>
  
  <p>The top line is [rlist, wlist, elist] from select, the bottom line is time.time() - start = (time.time() - start).  I got this run to break by counting the iterations and breaking at the bottom of the try after looping 1000 times.  timeout was set to 3 on the sample run.  Which proves that we get through the try, but obviously, none of the three ways that should be timing out works.</p>
  
  <p>Feel free to rip into the code if I've fundamentally misunderstood something.  I'd like for this to be uber-Pythonic and am still learning.</p>