Trouble parsing html files (to csv) using ElementTree xpath in python
Question: I am trying to parse a few thousand html files and dump the variables into a
csv file (excel spreadsheet). I've come up against several roadblocks--the
first one which was (thankfully) solve here, a few days ago. The (hopefully)
final roadblock is this: I can not get it to properly parse the file using
xpath. Below is a brief explanation, the python code and example of the html
code.

The trouble starts here:

    
    
    for node in tree.iter():
                name = node.attrib.get('/html/body/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td[1]/p/span')
                if category =='/html/body/center/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td[1]/font':
                category=node.text
    

It runs, but does not parse. I do not get any traceback errors.

I think I am misunderstanding the logic of parsing with ElementTree.

There are several headers that are the same--it is therefor difficult to find
a unique id/header. Here is an example of the html:

    
    
    <span class="s1">Business: Give Back to the Community and Save Money 
    on Equipment, Technology, Promotional Products, and Market<span 
    class="Apple-converted-space">&nbsp;</span></span>
    

For which the xpath is:

    
    
        /html/body/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]
    /table/tbody/tr[1]/td[1]/p/span
    

I would like to scrape the text from this span (among others) and put it in
the excel spreadsheet.

You can see an example of a similar page
[**HERE**](http://www.usprwire.com/Detailed/Banking_Finance_Investment/Confused.com_reveals_that_Life_Insurance_is_more_than_a_form_of_future_protection_284764.shtml)

At any rate, because many spans/headers are no uniquely identified, I think I
should use xpath. However, I have yet to be able to figure out how to
successfully use xpath commands with ElementTree. In searching the
documentation, the answer to this question (as well as the logic) eludes me. I
have read up on <http://lxml.de/parsing.html> as well as on this site and have
yet to find something that works.

So far, the code iterates through all the files (in dropbox) nicely. It also
creates the csv file and creates the headers (though not in separate columns,
only as one line separated by semicolons-- but that should be easy to fix).

In sum, I would like it to parse the text from different lines on in each file
(webpage) and dump it into the excel file.

Any input would be greatly appreciated.

The python code:

    
    
    import xml.etree.ElementTree as ET
    import csv, codecs, os
    from cStringIO import StringIO
    # Note: you need to download and install this..
    import unicodecsv
    import lxml.html
    # TODO: make into command line params (instead of constant)
    CSV_FILE='output.csv'
    HTML_PATH='/Users/C/data/Folder_NS'
    f = open(CSV_FILE, 'wb')
    w = unicodecsv.writer(f, encoding='utf-8', delimiter=';')
    w.writerow(['file', 'category', 'about', 'title', 'subtitle', 'date', 'bodyarticle'])
    
     # redundant declarations:
    category=''
    about=''
    title=''
    subtitle=''
    date=''
    bodyarticle=''
    print "headers created"
    
    allFiles = os.listdir(HTML_PATH)
    #with open(CSV_FILE, 'wb') as csvfile:
    print "all defined"
    
    for file in allFiles:
        #print allFiles
        if '.html' in file:
            print "in html loop"
            tree = lxml.html.parse(HTML_PATH+"/"+file)
            print '===================='
            print 'Parsing file: '+file
            print '===================='
            for node in tree.iter():
                name = node.attrib.get('/html/body/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td[1]/p/span')
    
                if category =='/html/body/center/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td[1]/font':
                print 'Category:'
                category=node.text
    
    f.close()
    

14 June 2015 (most recent change); I have just changed this section

    
    
            for node in tree.iter():
                name = node.attrib.get('/html/body/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td[1]/p/span')
    
                if category =='/html/body/center/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td[1]/font':
                print 'Category:'
                category=node.text
    

to this:

    
    
        for node in tree.iter():
                row = dict.fromkeys(cols)
                Category_name = tree.xpath('/html/body/table/tbody/tr/td/table/tbody/tr[3]/td/table/tbody/tr[2]/td[2]/table/tbody/tr[1]/td[1]/p/span')
                row['category'] = Category_name[0].text_content().encode('utf-8')
    

It still runs, but does not parse.


Answer: Try following code:

    
    
    from lxml import etree 
    import requests
    from StringIO import StringIO
    
    data = requests.get('http://www.usprwire.com/Detailed/Banking_Finance_Investment/Confused.com_reveals_that_Life_Insurance_is_more_than_a_form_of_future_protection_284764.shtml').content
    parser = etree.HTMLParser()
    root = etree.parse(StringIO(data), parser)
    category = root.xpath('//table/td/font/text()')
    print category[0]
    

It uses `requests` library to download the `html` code of the page. You can
choose whatever method that fits your needs. The important part is the `xpath`
that searches any `<table>` followed by `<td>` followed by `<font>`, and it
returns a list with two elements. The second one are blank characters and the
first one contains the text.

Run it and yields just the sentence you are looking for:

    
    
    Banking, Finance & Investment: Confused.com reveals that Life Insurance is more than a form of future protection
    

