title: Flushing out data from Popen buffer
body: <p>I am working on a python script that reads data by tailing a file and then puts in a different file. The script works in a time bound manner and eventually flushes out the data from the buffer when the ENDTIME is reached. However there has been a mismatch in the source and target file in terms of size.</p>
  
  <p>Following is a snippet:</p>
  
  <pre><code>    self.read_size = 2048
      self.tail_buffer = 2048
  
      # start the file tail
      cmd = '%s -f -o %s %s' % (self.jtailcmd, offset, self.source_journal)
      self.logger.debug('[%s] starting FILE Tail' % self.getName())
      try:
          self.jtail = popen2.Popen3(cmd, bufsize = self.tail_buffer)
          self.jtail.tochild.close()
          out = self.jtail.fromchild
          outfd = self.jtail.fromchild.fileno()
          flags = fcntl.fcntl(outfd, fcntl.F_GETFL)
          fcntl.fcntl(outfd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
      except:
          message = '[%s] error reading file' % self.getName()
          self.logger.error(message)
          self.logger.error('[%s] %s: %s' % \
              (self.getName(), sys.exc_info()[0], sys.exc_info()[1]))
          send_alert('AE', message)
          self.sleep(60)
          self.close_tail()
          self.close_ssh()
  </code></pre>
  
  <p>And then eventually it flushes out the data:</p>
  
  <pre><code>        try:
              [i, o, e] = select.select([outfd], [], [], 1)
              if i:
                  data = out.read(self.read_size)
              else:
                  data = None
          except:
              message = '[%s] error reading file' % self.getName()
              self.logger.error(message)
              self.logger.error('[%s] %s: %s' % \
                  (self.getName(), sys.exc_info()[0], sys.exc_info()[1]))
              send_alert('AE', message)
              self.close_tail()
              self.close_ssh()
              self.sleep(60)
              break
          if data:
              if self.sshcat.poll() != -1:
                  self.logger.error('[%s] connection error' % self.getName())
                  self.close_tail()
                  self.close_ssh()
                  break
              try:
                  self.sshcat.tochild.writelines(data)
                  self.sshcat.tochild.flush()
              except:
                  message = '[%s] error writing remote file' % self.getName()
  </code></pre>
  
  <p>While troubleshooting, I narrowed out the problem to <code>tail_buffer</code> size! By reducing the <code>tail_buffer</code> size , the script worked fine. </p>
  
  <p>I do not want to rely on tail_buffer size. Ideally the script should be independent of it! </p>
  
  <p>Is there a way to flush data from the POPEN buffer ? </p>
  
  <p>Please help!</p>