title: Creating a simple python crawler with scrapy
body: <p>I am currently trying to make a simple crawler in python using Scrapey. What I want it to do is read a list of links and save the html of the websites that they link to. Right now, I am able to get all of the URLs, but I am unable to figure out how to download the page. Here is the code for my spider so far:</p>
  
  <pre><code>from scrapy.spider import BaseSpider
  from scrapy.selector import HtmlXPathSelector
  from tutorial.items import BookItem
  
  # Book scrappy spider
  
  class DmozSpider(BaseSpider):
      name = "book"
      allowed_domains = ["learnpythonthehardway.org"]
      start_urls = [
          "http://www.learnpythonthehardway.org/book/",
      ]
  
      def parse(self, response):
          filename = response.url.split("/")[-2]
          file = open(filename,'wb')
          file.write(response.body)
          file.close()
  
          hxs = HtmlXPathSelector(response)
          sites = hxs.select('//ul/li')
          items = []
          for site in sites:
              item = BookItem()
              item['title'] = site.select('a/text()').extract()
              item['link'] = site.select('a/@href').extract()
              items.append(item)
          return items
  </code></pre>