python - pull pdfs from webpage and convert to html
Question: My goal is to have a python script that will access particular webpages,
extract all pdf files on each page that have a certain word in their filename,
convert them into html/xml, then go through the html files to read data from
the pdfs' tables.

So far I have imported mechanize (for browsing the pages/finding the pdf
files) and I have pdfminer, however I'm not sure how to use it in a script to
perform the same functionality it does on the command line.

What is the most effective group of libraries for accomplishing my task, and
how would you recommend approaching each step? I apologize if this is too
specific for stackoverflow, but I'm having trouble using google searches and
sparse documentation to piece together how to code this. Thanks!

  
EDIT: So I've decided to go with Scrapy on this one. I'm really liking it so
far, but now I have a new question. I've defined a PDFItem() class to use with
my spider with fields title and url. I have a selector thats grabbing all the
links I want, and I want to go through these links and create a PDFItem for
each one. Here's the code I have below:

    
    
    links = sel.xpath('//a[contains(@href, "enforcementactions.pdf") and contains(@class, "titlelink")]')
    item = PDFItem()
    for link in links:
            item['title'] = link.xpath('/text()')
            item['url'] = URL + link.xpath('@href').extract()[0]
    

The url line works well, but I don't really know how to do the same for title.
I guess I could just perform the query at the top, but adding '/text()' to the
end of the selector, but this seems excessive. Is there a better way to just
go through each link object in the links array and grab the text and href
value?


Answer: I would use [Scrapy](http://scrapy.org/). Scrapy is the best tool for crawling
an entire website and generating a list of all PDF links. A spider like this
would be very easy to write. You definitely don't need Mechanize.

After that, I would use [Poppler](http://poppler.freedesktop.org/) to convert
each PDF to HTML. It's not a Python module, but you can use the command
`pdftohtml`. In my experience, I've had better results with Poppler than
PDFMiner.

**Edit:**

    
    
    links = sel.xpath('//a[contains(@href, "enforcementactions.pdf") and contains(@class, "titlelink")]')
    for link in links:
        item = PDFItem()
        item['title'] = link.xpath('text()').extract()[0]
        item['url'] = URL + link.xpath('@href').extract()[0]
    

