Clean up .html reports and export as .txt files
Question: I've been searching all night but I'm still not sure how to get the job done.
I'm new to python, so please forgive me first if I'm asking some simple
questions.

I've three thousands .html files (all new product description downloaded from
a trusted website) stored in one folder, now I would like to clean up these
files one by one (i.e., only keeping the content/product description and
removing tags and so on) and then store each content as a single .txt file.

After reading a few Q&As posted here, I think I need to use lxml package
instead of beautiful soup because all the .html files are from a highly
trusted source. However, I don't know which command/option within lxml I
should use, could you please kindly let me know?


Answer: lxml is a good choice not only because it's a good source but for its speed.
Just take into consideration that a trusted source doesn't mean properly
formatted markup which is what matters between libraries.

If all the pages have the same structure, xpath will do the job. First you'll
need to get the [xpath](http://www.w3schools.com/xsl/) which Chrome can do for
you, just do 'Inspect Element' -> right click on the html element you need to
parse -> select 'Copy xpath'.

In your python code once you request the page. Take the html and convert it:

    
    
    from lxml import html
    tree = html.fromstring("htmlString") #you can switch this with the path of the html file
    name = tree.xpath('XPATH GOES HERE') 
    

This will return a list object in most cases. To get just the text from an
attribute add '/text()' to the end of the xpath. Chrome's xpath versions
sometimes differ from what python reads (I had this issue with tables) so if
it doesn't return anything, play with the xpath a little to make sure to
works.

Alternatively,

You can "navigate" the structure of the html file instead of using xpath with
the

    
    
    .find_class('css_class_Name') .getnext() .getchildren()
    

for example and then use

    
    
    .text_content() 
    

to extract the text from the html element. I encourage you to read these
[docs](http://lxml.de/api/lxml.etree._Element-class.html) for this to see
exactly which one you need to use.

