Python popen running out of memory with large output
<p>I am using the subprocess.popen() function to run an external tool that reads &amp; writes a lot of data (>GB) to stdout. However, I'm finding that the kernel is killing the python process when it runs out of memory:</p>

<pre><code>Out of memory: Kill process 8221 (python) score 971 or sacrifice child
Killed process 8221 (python) total-vm:8532708kB, anon-rss:3703912kB, file-rss:48kB
</code></pre>

<p>Since I know I'm handling a large amount of data I've set up popen to write stdout and stderr to files so I'm not using pipes. My code looks something like this:</p>

<pre><code>errorFile = open(errorFilePath, "w")
outFile = open(outFilePath, "w")
#Use Popen to run the command
try:                
    procExecCommand = subprocess.Popen(commandToExecute, shell=False, stderr=errorFile, stdout=outFile)
    exitCode = procExecCommand.wait()

except Exception, e:
    #Write exception to error log       
    errorFile.write(str(e))     

errorFile.close()
outFile.close()        
</code></pre>

<p>I've tried changing the shell parameter to True and setting the bufsize parameter = -1 also with no luck.</p>

<p>I've profiled the memory running this script and via bash and I see big spike in the memory usage when running via Python than compared to bash. </p>

<p>I'm not sure what exactly Python is doing to consume so much more memory than the just using bash unless it has something to with trying to write the output to the file? The bash script just pipes the output to a file.</p>

<p>I initially found that my swap space was quite low so I increased it and that helped initially, but as the volume of data grows then I start running out of memory again. </p>

<p>So is there anything with Python I can do to try and handle these data volumes better, or is it just a case of recommending more memory with plenty of swap space. That or jettison Python altogether.</p>

<p>System details:</p>

<ul>
<li>Ubuntu 12.04</li>
<li>Python 2.7.3</li>
<li>The tool I'm running is mpileup from samtools.</li>
</ul>
