Using scrapy to save files on a web page by extension type
Question: I am very new to Python and I am trying to use scrapy to download and save the
pdf files in this website:
<http://www.legco.gov.hk/general/chinese/counmtg/yr04-08/mtg_0708.htm#hansard>

The following is my code:

    
    
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector
    
    
    class legco(BaseSpider):
      name = "legco"
      allowed_domains = ["http://www.legco.gov.hk/"]
      start_urls = ["http://www.legco.gov.hk/general/chinese/counmtg/yr04-08/mtg_0708.htm#hansard"]
      rules =(
        Rule(SgmlLinkExtractor(allow=r"\.pdf"), callback="save_pdf")
              )
    
    def parse_listing(self, response):
        hxs = HtmlXPathSelector(response)
        pdf_urls=hxs.select("a/@href").extract()
        for url in pdf_urls:
            yield Request(url, callback=self.save_pdf)
    
    def save_pdf(self, response):
        path = self.get_path(response.url)
        with open(path, "wb") as f:
            f.write(response.body)
    

Basically I tried to restrict the search to just links with ".pdf" and then
select by "a/@hfref".

From the output, I see this error:

> 2015-03-09 11:00:22-0700 [legco] ERROR: Spider error processing
> http://www.legco.gov.hk/general/chinese/counmtg/yr04-08/mtg_0708.htm#hansard>

Can anyone advise how can I fix my code? Thanks a lot!


Answer: First of all, _you need to use a`CrawlSpider`_ if you want the `rules` to
work. Also, the `rules` should be defined as an iterable, usually it is a
tuple (there was a missing comma).

Anyway, instead of taking this approach, I would use a normal `BaseSpider`,
loop over the links and check the `href` to end with `.pdf`, then, in the
callback, save a pdf to a file:

    
    
    import urlparse
    
    from scrapy.http import Request
    from scrapy.spider import BaseSpider
    
    
    class legco(BaseSpider):
        name = "legco"
    
        allowed_domains = ["www.legco.gov.hk"]
        start_urls = ["http://www.legco.gov.hk/general/chinese/counmtg/yr04-08/mtg_0708.htm#hansard"]
    
        def parse(self, response):
            base_url = 'http://www.legco.gov.hk/general/chinese/counmtg/yr04-08/'
            for a in response.xpath('//a[@href]/@href'):
                link = a.extract()
                if link.endswith('.pdf'):
                    link = urlparse.urljoin(base_url, link)
                    yield Request(link, callback=self.save_pdf)
    
        def save_pdf(self, response):
            path = response.url.split('/')[-1]
            with open(path, 'wb') as f:
                f.write(response.body)
    

(worked for me)

