title: Continuously capture output from subprocess.Popen
body: <p>I want to capture the output of a process called with</p>
  
  <pre><code>cmd = "doit input={input} conf={conf} output_dir={dir} gpu={gpu} --db-file .doit_gpu{gpu_id}.db".format(**kwargs)
  process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
  </code></pre>
  
  <p>continuously.
  This process calls functions that print their result in the console as dictionary strings. E.g.:</p>
  
  <pre><code>{'subprocess_1': [&lt;some data&gt;]}
  # then after some time
  {'subprocess_2': [&lt;some data&gt;]}
  # and some seconds later
  {'subprocess_3': [&lt;some data&gt;]}
  </code></pre>
  
  <p>I want to be able to continuously capture the output and redirect it <strong>immediately</strong> to another dictionary that I initialized before.</p>
  
  <pre><code>process_table = {}
  </code></pre>
  
  <p>So this is how the dictionary would have to look like after the whole process finished:</p>
  
  <pre><code>process_table = {
      {'subprocess_1': [&lt;some data&gt;]},
      {'subprocess_2': [&lt;some data&gt;]},
      {'subprocess_3': [&lt;some data&gt;]}
  }
  </code></pre>
  
  <p>The key point here (which is why I don't seem to find any related questions) is that <code>process_table</code> should be updated immediately after a subprocess finishes, not only after the entire process finished. This is why this answer <a href="https://stackoverflow.com/questions/2804543/read-subprocess-stdout-line-by-line">here</a> <strong>is not enough</strong>:</p>
  
  <pre><code>for line in process.stdout:
      print("this is the output:", line,)
  </code></pre>