title: Parsing the output of a subprocess while executing and clearing the memory (Python 2.7)
body: <p>I need to parse the output produced by an external program (third party, I have no control over it) which produces large amounts of data. Since the size of the output greatly exceeds the available memory, I would like to parse the output while the process is running
  and remove from the memory the data that have already been processed.</p>
  
  <p>So far I do something like this:</p>
  
  <pre><code>import subprocess
  
  p_pre = subprocess.Popen("preprocessor",stdout = subprocess.PIPE) 
  # preprocessor is an external bash script that produces the input for the third-party software
  p_3party = subprocess.Popen("thirdparty",stdin = p_pre.stdout, stdout = subprocess.PIPE)
  
  (data_to_parse,can_be_thrown) = p_3party.communicate()
  
  parsed_data = myparser(data_to_parse)
  </code></pre>
  
  <p>When "thirdparty" output is small enough, this approach works. But as stated in the Python documentation: </p>
  
  <blockquote>
    <p>The data read is buffered in memory, so do not use this method if the data size is large or unlimited.</p>
  </blockquote>
  
  <p>I think a better approach (that could actually make me save some time),
  would be to start processing <code>data_to_parse</code> while it is being produces,
  and when the parsing has been done correctly "clear" <code>data_to_parse</code> removing
  the data that have already been parsed.</p>
  
  <p>I have also tried to use a for cycle like:</p>
  
  <pre><code>parsed_data=[]
  for i in p_3party.stdout:
      parsed_data.append(myparser(i))
  </code></pre>
  
  <p>but it gets stuck and can't understand why.</p>
  
  <p>So I would like to know what it is the best approach to accomplish this? What are the issues to be aware of?</p>