text: Spiders are classes which define how a certain site (or domain) will be scraped, including how to crawl the site and how to extract scraped items from their pages. In other words, Spiders are the place where you define the custom behaviour for crawling and parsing pages for a particular site.
  For spiders, the scraping cycle goes through something like this:
  You start by generating the initial Requests to crawl the first URLs, and specify a callback function to be called with the response downloaded from those requests.
  The first requests to perform are obtained by calling the start_requests() method which (by default) generates Request for the URLs specified in the start_urls and the parse method as callback function for the Requests.
  In the callback function, you parse the response (web page) and return either Item objects, Request objects, or an iterable of both. Those Requests will also contain a callback (maybe the same) and will then be downloaded by Scrapy and then their response handled by the specified callback.
  In callback functions, you parse the page contents, typically using XPath Selectors (but you can also use BeautifuSoup, lxml or whatever mechanism you prefer) and generate items with the parsed data.
  Finally, the items returned from the spider will be typically persisted in some Item pipeline.
  Even though this cycle applies (more or less) to any kind of spider, there are different kinds of default spiders bundled into Scrapy for different purposes. We will talk about those types here. each of the resulting responses.
  A string which defines the name for this spider. The spider name is how the spider is located (and instantiated) by Scrapy, so it must be unique. However, nothing prevents you from instantiating more than one instance of the same spider. This is the most important spider attribute and it’s required.
  Is recommended to name your spiders after the domain that their crawl..
  This method must return an iterable with the first Requests to crawl for this spider.
  This is the method called by Scrapy when the spider is opened for scraping when no particular URLs are specified. If particular URLs are specified, the make_requests_from_url() is used instead to create the Requests. This method is also called only once from Scrapy, so it’s safe to implement it as a generator.
  The default implementation uses make_requests_from_url() to generate Requests for each url in start_urls.
  If you want to change the Requests used to start scraping a domain, this is the method to override. For example, if you need to start by logging in using a POST request, you could do:
  def start_requests(self): return [FormRequest("", formdata={'user': 'john', 'pass': 'secret'}, callback=self.logged_in)] def logged_in(self, response): # here you would extract links to follow and return Requests for # each of them, with another callback pass)
  This is the most commonly used spider for crawling regular websites, as it provides a convenient mechanism for following links by defining a set of rules. It may not be the best suited for your particular web sites or project, but it’s generic enough for several cases, so you can start from it and override it as needed for more custom functionality, or just implement your own spider.
  Apart from the attributes inherited from BaseSpider (that you must specify), this class supports a new attribute: is a callable or a string (in which case a method from the spider object with that name will be used) to be called for each link extracted with the specified link_extractor. This callback receives a response as its first argument and must return a list containing Item).
  Let’s now take a look at an example CrawlSpider with rules: which defines the iterator to use. It can be either:
  - 'iternodes' - a fast iterator based on regular expressions
  - 'html' - an iterator which uses HtmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem for big feeds
  - 'xml' - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem for big feeds
  It defaults to: 'iternodes'.:
  A method that receives the response as soon as it arrives from the spider middleware, before the spider starts parsing it. It can be used to modify the response body before parsing it. This method receives a response and also returns a response (it could be the same or another one). method is called for each result (item or request) returned by the spider, and it’s intended to perform any last time processing required before returning the results to the framework core, for example setting the item IDs. It receives a list of results and the response which originated those results. It must return a list of results (Items or Requests).
  These spiders are pretty easy to use, let’s have a look at one example:
  from scrapy import log from scrapy.contrib.spiders import XMLFeedSpider from myproject.items import TestItem class MySpider(XMLFeedSpider): name = 'example.com' allowed_domains = ['example.com'] start_urls = [''] iterator = 'iternodes' # This is actually unnecesary, since it's the default value itertag = 'item' def parse_node(self, response, node): log.msg('Hi, this is a <%s> node!: %s' % (self.itertag, ''.join(node.extract()))) item = Item() item['id'] = node.select('@id').extract() item['name'] = node.select('name').extract() item['description'] = node.select('description').extract() return item
  Basically what we did up there was to create a spider that downloads a feed from the given start_urls, and then iterates through each of its item tags, prints them out, and stores some random data in an Item..
  Let’s see an example similar to the previous one, but using a CSVFeedSpider:
  from scrapy
  SitemapSpider allows you to crawl a site by discovering the URLs using Sitemaps.
  It supports nested sitemaps and discovering sitemap urls from robots.txt.
  A list of urls pointing to the sitemaps whose urls you want to crawl.
  You can also point to a robots.txt and it will be parsed to extract sitemap urls from it.
  A list of tuples (regex, callback) where:.spiders import SitemapSpider class MySpider(SitemapSpider): sitemap_urls = [''] sitemap_rules = [ ('/product/', 'parse_product'), ('/category/', 'parse_category'), ] def parse_product(self, response): pass # ... scrape product ... def parse_category(self, response): pass # ... scrape category ...
  Follow sitemaps defined in the robots.txt file and only follow sitemaps whose url contains /sitemap_shop:
  from scrapy.contrib.spiders import SitemapSpider class MySpider(SitemapSpider): sitemap_urls = [''] sitemap_rules = [ ('/shop/', 'parse_shop'), ] sitemap_follow = ['/sitemap_shops'] def parse_shop(self, response): pass # ... scrape shop here ...
  Combine SitemapSpider with other sources of urls:
  from scrapy ...