title: python pipe subprocess i/o over socket
body: <p>I know there are similar questions out there, but I'm having trouble with this concrete example, and haven't found a good answer.  I'm trying to set up a remote backup server for dar, <a href="http://dar.linux.free.fr/doc/usage_notes.html#dar_remote" rel="nofollow noreferrer">along these lines</a>.  I've asked a <a href="https://stackoverflow.com/questions/7922477/">separate question</a> about doing this by invoking netcat with subprocess.Popen, but I'd prefer to set up the sockets and do all the piping with python if possible.  There will be several gigs transferred, so I can't just read all the input first and then pass it on.  </p>
  
  <p>The problem is that the server doesn't seem to be reading the data.</p>
  
  <p>At the moment, I've got the following code:</p>
  
  <pre><code>from socket import socket, AF_INET, SOCK_STREAM
  import sys
  import SocketServer
  import subprocess
  
  class DarHandler(SocketServer.BaseRequestHandler):
      def handle(self):
          print('entering handler')
          data = self.request.recv(1024).strip()
          print('got: ' + data)
          if data == 'xform':
              s = socket(AF_INET, SOCK_STREAM)
              s.bind(('',0))
              myaddr, myport = s.getsockname()
              print('bound new socket to {0}:{1}'.format(myaddr, myport))
              self.request.send(str(myport))
              s.listen(1)
              conn, remoteaddr = s.accept()
              print('accepted connection from {0}:{1}'.format(*remoteaddr))
              xform_input = conn.makefile('rb',0)
              proc = subprocess.Popen(
                  ['/usr/bin/dar_xform', '-s', '10k', '-', 'archives/sockbackup',],
                   stdin=xform_input
              )
              return_code = proc.wait()
              print('dar_xform returned {0}'.format(return_code))
              conn.close()
              self.request.send(str(return_code))
          else:
              self.request.send('bad request')
          print('send result, exiting handler')
  
  server_address = ('localhost', 18010)
  def server():
      server = SocketServer.TCPServer(server_address, DarHandler)
      print('listening')
      server.serve_forever()
  
  def client():
      sock = socket(AF_INET, SOCK_STREAM)
      print('connecting to server')
      sock.connect(('localhost', 18010))
      print('connected, sending request')
      sock.send('xform')
      print('waiting for response')
      port = sock.recv(1024)
      print('got: ' + port)
      s = socket(AF_INET, SOCK_STREAM)
      s.connect(('localhost', int(port)))
      print('connected to dar_xform port')
      dar_output = s.makefile('wb',0)
      return_code = subprocess.call(
            ['/usr/bin/dar', '-B', 'config/test.dcf', '-c', '-',], 
            stdout=dar_output
      )
      print('dar returned {0}'.format(return_code))
      s.close()
      result = sock.recv(1024)
      print('received: ' + result)
      sock.close()
      print('socket closed, exiting')
  
  if __name__ == "__main__":
      if sys.argv[1].startswith('serv'):
          server()
      else:
          client()
  </code></pre>
  
  <p>I get the following output from the server side when I run the client:</p>
  
  <pre>
  listening
  entering handler
  got: xform
  bound new socket to 0.0.0.0:41658
  accepted connection from 127.0.0.1:42440
  </pre>
  
  <p>Then it just sits there.  Meanwhile, <code>dar</code> runs on the client and the client is stuck waiting for a response from the server:</p>
  
  <pre>
  connecting to server
  connected, sending request
  waiting for response
  got: 41300
  connected to dar_xform port
  
  
   --------------------------------------------
   53 inode(s) saved
   with 0 hard link(s) recorded
   0 inode(s) changed at the moment of the backup
   0 inode(s) not saved (no inode/file change)
   0 inode(s) failed to save (filesystem error)
   1 inode(s) ignored (excluded by filters)
   0 inode(s) recorded as deleted from reference backup
   --------------------------------------------
   Total number of inodes considered: 54
   --------------------------------------------
   EA saved for 0 inode(s)
   --------------------------------------------
  dar returned 0
  </pre>