Stream Parse Wiki Xml dump
Question: I'm not sure if this question has been asked before but it has been I couldn't
find it and please pardon my ignorance. I want to parse (stream parse)
wikipedia xml dumps that are about 40 Gigs. I'm trying to use lxml iterparse
aka stream parser to do this job for me but the code I wrote isn't working for
me for some reason. Before I explain what I want to do let's consider this xml
format

    
    
    <root>
      <page>
        <title> A  </title>
        <text> ..........blah blah...... </text>
      </page>
    
      <page>
      <title> B </title>
      <text> This is a line of sample text in title B  </text>
      </page>
    
      <page>
      <title> C </title>
      <text> ............blah blah........ </text>
      </page>
    </root>
    

In the above xml structure, which is very much what wiki dump has, what I want
to do is read each and every thing inside "page" tag (which basically
separates different articles in the dump) using a python stream parser such as
lxml (without loading the entire xml tree into memory which is impossible) and
do a regular expression stuff on it. Precisely what I want to do is, if in the
current "page" element look for inside the text tag "Category:Living People",
if such a matching regex is found, then write out the entire content inside
"text" tag into a text file. But before the regex part, I'm having issues with
the lxml parser itself. Sample code that I tried below. Thanks in advance :)

    
    
    #LXML parser
    from lxml import etree
    
    def fast_iter(context, func):
        for event, elem in context:
            func(elem)
            elem.clear()
            while elem.getprevious() is not None:
                del elem.getparent()[0]
        del context
    
    def process_element(elem):
        print elem.xpath( 'description/text( )' )
    
    context = etree.iterparse( MYFILE, tag='item' )
    fast_iter(context,process_element)
    

Feel free to completely alter the code and give your own version, I really
wouldn't mind as long as I get my problem sorted!

Someone help please!


Answer: 
    from xml.sax import ContentHandler, parseString
    from StringIO import StringIO
    from lxml import etree
    
    CONTENT = """
    <root>
      <page>
        <title> A  </title>
        <text> ..........blah blah...... </text>
      </page>
    
      <page>
      <title> B </title>
      <text> This is a line of sample text in title B  </text>
      </page>
    
      <page>
      <title> C </title>
      <text> ............blah blah........ </text>
      </page>
    </root>
    """
    
    def fast_iter(context, func):
        for action, elem in context:
            func(elem)
            elem.clear()
            while elem.getprevious() is not None:
                del elem.getparent()[0]
        del context
    
    def process_element(elem):
        print elem.xpath( './text/text( )' )
    
    class WikiContentHandler(ContentHandler):
    
        def startDocument(self):
            self.character_buffer = None
    
        def startElement(self, name, attrs):
            if name == 'text':
                self.character_buffer = StringIO()
    
        def endElement(self, name):
            if name == 'text':
                print self.character_buffer.getvalue()
                self.character_buffer.close()
                self.character_buffer = None
    
        def characters(self, content):
            if self.character_buffer != None:
                self.character_buffer.write(content)
    
    def parse_wiki():
        parseString(CONTENT, WikiContentHandler())
    
    if __name__ == '__main__':
        parse_wiki()
    
        context = etree.iterparse( StringIO(CONTENT), tag='page' )
        fast_iter(context, process_element)
    

Above are two ways of parsing the XML, one using lxml, the other using the
Python standard library. Both of them print out all the information in the
`<text>` tags. Just look where the print statements are and do your additional
processing.

