title: Iteratively parsing HTML (with lxml?)
body: <p>I'm currently trying to iteratively parse a very large HTML document (I know.. yuck) to reduce the amount of memory used. The problem I'm having is that I'm getting XML syntax errors such as:</p>
  
  <p><code>lxml.etree.XMLSyntaxError: Attribute name redefined, line 134, column 59</code></p>
  
  <p>This then causes everything to stop.</p>
  
  <p>Is there a way to iteratively parse HTML without choking on syntax errors?</p>
  
  <p>At the moment I'm extracting the line number from the XML syntax error exception, removing that line from the document, and then restarting the process. Seems like a pretty disgusting solution. Is there a better way?</p>
  
  <p><strong>Edit:</strong></p>
  
  <p>This is what I'm currently doing:</p>
  
  <pre><code>context = etree.iterparse(tfile, events=('start', 'end'), html=True)
  in_table = False
  header_row = True
  while context:
      try:
          event, el = context.next()
  
          # do something
  
          # remove old elements
          while el.getprevious() is not None:
              del el.getparent()[0]
  
      except etree.XMLSyntaxError, e:
          print e.msg
          lineno = int(re.search(r'line (\d+),', e.msg).group(1))
          remove_line(tfilename, lineno)
          tfile = open(tfilename)
          context = etree.iterparse(tfile, events=('start', 'end'), html=True)
      except KeyError:
          print 'oops keyerror'
  </code></pre>