title: Cleaning up large XML files in Python (stream parse)
body: <p>I tried to use Python to cleanup some messy XML files, which does three things:</p>
  
  <ol>
  <li>Converting 40%-50% tag names from upper case to lower case</li>
  <li>Removing NULL between tags</li>
  <li>Removing empty rows between tags</li>
  </ol>
  
  <p>I did this in using <code>BeautifulSoup</code>, however, I ran into memory issues since some of my XML files are over 1GB. Instead, I looked into some stream method like <code>xml.sax</code>, but I did not quite get the approach. So can anyone give me some suggestions?</p>
  
  <pre><code>xml_str = """
  &lt;DATA&gt;
  
      &lt;ROW&gt;
          &lt;assmtid&gt;1&lt;/assmtid&gt;
          &lt;Year&gt;1988&lt;/Year&gt;
      &lt;/ROW&gt;
  
      &lt;ROW&gt;
          &lt;assmtid&gt;2&lt;/assmtid&gt;
          &lt;Year&gt;NULL&lt;/Year&gt;
      &lt;/ROW&gt;
  
      &lt;ROW&gt;
          &lt;assmtid&gt;2&lt;/assmtid&gt;
          &lt;Year&gt;1990&lt;/Year&gt;
      &lt;/ROW&gt;
  
  &lt;/DATA&gt;
  """
  
  xml_str_update = re.sub(r"&gt;NULL", "&gt;", xml_str)
  soup = BeautifulSoup(xml_str_update, "lxml")
  print soup.data.prettify().encode('utf-8').strip()
  </code></pre>
  
  <h2>Update</h2>
  
  <p>After some testing and taking suggestions from Jarrod Roberson, below is one possible solution.</p>
  
  <pre><code>import os
  import xml.etree.cElementTree as etree
  from cStringIO import StringIO
  
  def getelements(xml_str):
      context = iter(etree.iterparse(StringIO(xml_str), events=('start', 'end')))
      event, root = next(context)
  
      for event, elem in context:
          if event == 'end' and elem.tag == "ROW":
              elem.tag = elem.tag.lower()
              elem.text = "\n\t\t"
              elem.tail = "\n\t"
  
              for child in elem:
                  child.tag = child.tag.lower()
                  if child.text == "NULL":
                      # if do not like self-closing tag, 
                      # add &amp;#x200B;, which is a zero width space
                      child.text = ""  
                  if child.text == None:
                      child.text = ""
                  # print event, elem.tag
              yield elem
              root.clear()
  
  with open(pth_to_output_xml, 'wb') as file:
      # start root
      file.write('&lt;data&gt;\n\t')
      for page in getelements(xml_str):
          file.write(etree.tostring(page, encoding='utf-8'))
      # close root
      file.write('&lt;/data&gt;')
  </code></pre>