title: Using Scrapy to to find and download pdf files from a website
body: <p>I've been tasked with pulling pdf files from websites using Scrapy. I'm not new to Python, but Scrapy is a very new to me. I've been experimenting with the console and a few rudimentary spiders. I've found and modified this code:</p>
  
  <pre><code>import urlparse
  import scrapy
  
  from scrapy.http import Request
  
  class pwc_tax(scrapy.Spider):
      name = "pwc_tax"
  
      allowed_domains = ["www.pwc.com"]
      start_urls = ["http://www.pwc.com/us/en/tax-services/publications/research-and-insights.html"]
  
      def parse(self, response):
          base_url = "http://www.pwc.com/us/en/tax-services/publications/research-and-insights.html"
          for a in response.xpath('//a[@href]/@href'):
              link = a.extract()
              if link.endswith('.pdf'):
                  link = urlparse.urljoin(base_url, link)
                  yield Request(link, callback=self.save_pdf)
  
      def save_pdf(self, response):
          path = response.url.split('/')[-1]
          with open(path, 'wb') as f:
              f.write(response.body)
  </code></pre>
  
  <p>I run this code at the command line with</p>
  
  <pre><code>scrapy crawl mySpider
  </code></pre>
  
  <p>and I get nothing back. I didn't create a scrapy item because I want to crawl and download the file, no meta data. I would appreciate any help on this.</p>