Parsing large xml data using python's elementtree
Question: I'm currently learning how to parse xml data using elementtree. I got an error
that say:ParseError: not well-formed (invalid token): line 1, column 2.

My code is right below, and a bit of the xml data is after my code.

    
    
    import xml.etree.ElementTree as ET
    
    tree = ET.fromstring("C:\pbc.xml")
    root = tree.getroot()
    
    
    for article in root.findall('article'):
        print ' '.join([t.text for t in pub.findall('title')])
        for author in article.findall('author'):
            print 'Author name: {}'.format(author.text)
        for journal in article.findall('journal'):  # all venue tags with id attribute
            print 'journal'
    
    
    
    <?xml version="1.0" encoding="ISO-8859-1"?>
    <!DOCTYPE dblp SYSTEM "dblp.dtd">
    <dblp>
    <article mdate="2002-01-03" key="persons/Codd71a">
    <author>E. F. Codd</author>
    <title>Further Normalization of the Data Base Relational Model.</title>
    <journal>IBM Research Report, San Jose, California</journal>
    <volume>RJ909</volume>
    <month>August</month>
    <year>1971</year>
    <cdrom>ibmTR/rj909.pdf</cdrom>
    <ee>db/labs/ibm/RJ909.html</ee>
    </article>
    
    <article mdate="2002-01-03" key="persons/Hall74">
    <author>Patrick A. V. Hall</author>
    <title>Common Subexpression Identification in General Algebraic Systems.</title>
    <journal>Technical Rep. UKSC 0060, IBM United Kingdom Scientific Centre</journal>
    <month>November</month>
    <year>1974</year>
    </article>
    

Any ideas will be appreciated. Thanks in advance.


Answer: 
    with open("C:\pbc.xml", 'rb') as f:
        root = ET.fromstring(f.read().strip())
    

* * *

Unlike `ET.parse`, `ET.fromstring` expects a string with XML content, not the
name of a file.

Also in contrast to `ET.parse`, `ET.fromstring` returns a root Element, not a
Tree. So you should omit

    
    
    root = tree.getroot()
    

* * *

Also, the XML snippet you posted needs a closing `</dblp>` to be parsable. I
assume your real data has that closing tag...

* * *

The iterparse provided by `xml.etree.ElementTree` does not have a `tag`
argument, although `lxml.etree.iterparse` does have a `tag` argument.

Try:

    
    
    import xml.etree.ElementTree as ET
    import htmlentitydefs
    
    filename = "test.xml"
    # http://stackoverflow.com/a/10792473/190597 (lambacck)
    parser = ET.XMLParser()
    parser.entity.update((x, unichr(i)) for x, i in htmlentitydefs.name2codepoint.iteritems())
    context = ET.iterparse(filename, events = ('end', ), parser=parser)
    for event, elem in context:
        if elem.tag == 'article':
            for author in elem.findall('author'):
                print 'Author name: {}'.format(author.text)
            for journal in elem.findall('journal'):  # all venue tags with id attribute
                print(journal.text)
            elem.clear()
    

Note: To use `iterparse` your XML must be valid, which means among other
things that there can not be empty lines at the beginning of the file.

