title: Scrapy get website with error "DNS lookup failed"
body: <p>I'm trying to use Scrapy to get all links on websites where the "DNS lookup failed".</p>
  
  <p>The problem is, every website without any errors are print on the <strong>parse_obj</strong> method but when an url return DNS lookup failed, the callback <strong>parse_obj is not call</strong>.</p>
  
  <p>I want to get all domain with the error "<strong>DNS lookup failed</strong>", how can I do that ?</p>
  
  <p><strong>Logs :</strong></p>
  
  <pre><code>2016-03-08 12:55:12 [scrapy] INFO: Spider opened
  2016-03-08 12:55:12 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
  2016-03-08 12:55:12 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
  2016-03-08 12:55:12 [scrapy] DEBUG: Crawled (200) &lt;GET http://domain.com&gt; (referer: None)
  2016-03-08 12:55:12 [scrapy] DEBUG: Retrying &lt;GET http://expired-domain.com/&gt; (failed 1 times): DNS lookup failed: address 'expired-domain.com' not found: [Errno 11001] getaddrinfo failed.
  </code></pre>
  
  <p><strong>Code :</strong></p>
  
  <pre><code>class MyItem(Item):
      url= Field()
  
  class someSpider(CrawlSpider):
      name = 'Crawler'        
      start_urls = ['http://domain.com']
      rules = (Rule(LxmlLinkExtractor(allow=()), callback='parse_obj', follow=True),)
  
      def parse_obj(self, response):
          item = MyItem()
          item['url'] = []
          for link in LxmlLinkExtractor(allow=()).extract_links(response):
              parsed_uri = urlparse(link.url)
              url = '{uri.scheme}://{uri.netloc}/'.format(uri=parsed_uri)
              print url
  </code></pre>