title: Process XML by chunks in Python
body: <p>I have a series of large XML files (~3GB each) that I'm trying to process. The rough format of the XML is </p>
  
  <pre><code>&lt;FILE&gt;
  &lt;DOC&gt;
      &lt;FIELD1&gt;
          Some text.
      &lt;/FIELD1&gt;
      &lt;FIELD2&gt;
          Some text. Probably some more fields nested within this one.
      &lt;/FIELD2&gt;
      &lt;FIELD3&gt;
          Some text.
      &lt;/FIELD3&gt;
      &lt;FIELD4&gt;
          Some text. Etc.
      &lt;/FIELD4&gt;
  &lt;/DOC&gt;
  &lt;DOC&gt;
      &lt;FIELD1&gt;
          Some text.
      &lt;/FIELD1&gt;
      &lt;FIELD2&gt;
          Some text. Probably some more fields nested within this one.
      &lt;/FIELD2&gt;
      &lt;FIELD3&gt;
          Some text.
      &lt;/FIELD3&gt;
      &lt;FIELD4&gt;
          Some text. Etc.
      &lt;/FIELD4&gt;
  &lt;/DOC&gt;
  &lt;/FILE&gt;
  </code></pre>
  
  <p>My current approach is (mimicking the code seen at <a href="http://effbot.org/zone/element-iterparse.htm#incremental-parsing" rel="nofollow">http://effbot.org/zone/element-iterparse.htm#incremental-parsing</a>): </p>
  
  <pre><code>#Added this in the edit.
  import xml.etree.ElementTree as ET
  
  tree = ET.iterparse(xml_file)
  tree = iter(tree)
  event, root = tree.next()
  
  for event, elem in tree:
      #Need to find the &lt;DOC&gt; elements
      if event == "end" and elem.tag == "DOC":
          #Code to process the fields within the &lt;DOC&gt; element. 
          #The code here mainly just iterates through the inner 
          #elements and extracts what I need
          root.clear()
  </code></pre>
  
  <p>This blows up, though, and uses all of my system memory (16GB). At first I thought it was the position of the <code>root.clear()</code> so I tried moving that out to after the if-statement, but that didn't seem to have any effect. Given this, I'm note quite sure how to proceed other than "get more memory."</p>
  
  <p><strong>EDIT</strong>:</p>
  
  <p>Deleted the previous edit because it was wrong.</p>