Scrapy crawl spider stopped working
Question: Prehistory: I'm running Scrapy version 0.16.2 on Python 2.7.2+ and it is on
Linux Mint. A few days ago [I had this
problem](http://stackoverflow.com/questions/20025427/scrapy-crawler-spider-
doesnt-follow-links) and with help I managed to overcome it. For a few moments
Crawler worked as it should:

    
    
    2013-11-23 01:02:51+0200 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
    2013-11-23 01:02:51+0200 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
    2013-11-23 01:02:51+0200 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
    2013-11-23 01:02:51+0200 [scrapy] DEBUG: Enabled item pipelines: 
    2013-11-23 01:02:51+0200 [basketsp17] INFO: Spider opened
    2013-11-23 01:02:51+0200 [basketsp17] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-11-23 01:02:51+0200 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6024
    2013-11-23 01:02:51+0200 [scrapy] DEBUG: Web service listening on 0.0.0.0:6081
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Redirecting (301) to <GET http://www.euroleague.net/main/results/by-date> from <GET http://www.euroleague.net/main/results/by-date/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Crawled (200) <GET http://www.euroleague.net/main/results/by-date> (referer: None)
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'www.euroleaguebasketball.net': <GET http://www.euroleaguebasketball.net/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'www.eurocupbasketball.com': <GET http://www.eurocupbasketball.com/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'www.euroleague.tv': <GET http://www.euroleague.tv/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'www.euroleaguestore.net': <GET http://www.euroleaguestore.net/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'fantasychallenge.euroleague.net': <GET http://fantasychallenge.euroleague.net/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'www.facebook.com': <GET http://www.facebook.com/TheEuroleague>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'www.youtube.com': <GET http://www.youtube.com/euroleague>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'euroleaguedevotion.ourtoolbar.com': <GET http://euroleaguedevotion.ourtoolbar.com/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'euroleague.synapticdigital.com': <GET http://euroleague.synapticdigital.com/>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'twitter.com': <GET http://twitter.com/Euroleague>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'kort.es': <GET http://kort.es/ulpGt>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Filtered offsite request to 'adserver.itsfogo.com': <GET http://adserver.itsfogo.com/click.aspx?zoneid=136145>
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Crawled (200) <GET http://www.euroleague.net/> (referer: http://www.euroleague.net/main/results/by-date)
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Crawled (200) <GET http://www.euroleague.net/devotion/home> (referer: http://www.euroleague.net/main/results/by-date)
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Crawled (200) <GET http://www.euroleague.net/euroleaguenews/transactions/2013-14-signings> (referer: http://www.euroleague.net/main/results/by-date)
    2013-11-23 01:02:51+0200 [basketsp17] DEBUG: Crawled (200) <GET http://www.euroleague.net/features/blog/2013-2014> (referer: http://www.euroleague.net/main/results/by-date)
    

But after several times it stopped crawling. I want to know where is the
problem. If I try code next day it works again for several moments and stops.
Well, it works but it doesn't crawl. If I change start_urls it starts to work
again and it stops again with the same code. What could be wrong here?

Here is what I see after it stops:

    
    
    scrapy crawl basketsp17
    2013-11-22 03:07:15+0200 [scrapy] INFO: Scrapy 0.20.0 started (bot: basketbase)
    2013-11-22 03:07:15+0200 [scrapy] DEBUG: Optional features available: ssl, http11, boto, django
    2013-11-22 03:07:15+0200 [scrapy] DEBUG: Overridden settings: {'NEWSPIDER_MODULE': 'basketbase.spiders', 'SPIDER_MODULES': ['basketbase.spiders'], 'BOT_NAME': 'basketbase'}
    2013-11-22 03:07:16+0200 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
    2013-11-22 03:07:16+0200 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
    2013-11-22 03:07:16+0200 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
    2013-11-22 03:07:16+0200 [scrapy] DEBUG: Enabled item pipelines: 
    2013-11-22 03:07:16+0200 [basketsp17] INFO: Spider opened
    2013-11-22 03:07:16+0200 [basketsp17] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
    2013-11-22 03:07:16+0200 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
    2013-11-22 03:07:16+0200 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
    2013-11-22 03:07:16+0200 [basketsp17] DEBUG: Redirecting (301) to <GET http://www.euroleague.net/main/results/by-date> from <GET http://www.euroleague.net/main/results/by-date/>
    2013-11-22 03:07:16+0200 [basketsp17] DEBUG: Crawled (200) <GET http://www.euroleague.net/main/results/by-date> (referer: None)
    2013-11-22 03:07:16+0200 [basketsp17] INFO: Closing spider (finished)
    2013-11-22 03:07:16+0200 [basketsp17] INFO: Dumping Scrapy stats:
        {'downloader/request_bytes': 489,
         'downloader/request_count': 2,
         'downloader/request_method_count/GET': 2,
         'downloader/response_bytes': 12181,
         'downloader/response_count': 2,
         'downloader/response_status_count/200': 1,
         'downloader/response_status_count/301': 1,
         'finish_reason': 'finished',
         'finish_time': datetime.datetime(2013, 11, 22, 1, 7, 16, 471690),
         'log_count/DEBUG': 8,
         'log_count/INFO': 3,
         'response_received_count': 1,
         'scheduler/dequeued': 2,
         'scheduler/dequeued/memory': 2,
         'scheduler/enqueued': 2,
         'scheduler/enqueued/memory': 2,
         'start_time': datetime.datetime(2013, 11, 22, 1, 7, 16, 172756)}
    2013-11-22 03:07:16+0200 [basketsp17] INFO: Spider closed (finished)
    

Here is a code that I am using:

    
    
    from basketbase.items import BasketbaseItem
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
    from scrapy.selector import HtmlXPathSelector
    from scrapy.http import Request
    from scrapy.http import TextResponse 
    from scrapy.http import HtmlResponse
    
    
    class Basketspider(CrawlSpider):
        name = "basketsp17"
        allowed_domains = ["www.euroleague.net"]
        start_urls = ["http://www.euroleague.net/main/results/by-date/"]
        rules = (
            Rule(SgmlLinkExtractor(allow=('main\/results\/showgame\?gamecode\=/\d$\&seasoncode\=E2013\#!boxscore')),follow=True),
            Rule(SgmlLinkExtractor(allow=()),callback='parse_item'),
        )
    
        def init_request(self):
            return HtmlResponse("http://www.euroleague.net/main/results/by-date/", body = body)
    
        def parse_item(self, response):
            sel = HtmlXPathSelector(response)
            items=[]
            item = BasketbaseItem()
            item['date'] = sel.select('//div[@class="gs-dates"]/text()').extract() # Game date
            item['time'] = sel.select('//div[@class="gs-dates"]/span[@class="GameScoreTimeContainer"]/text()').extract() # Game time
    
            items.append(item) 
            return items 
    


Answer: I modified the code of yours to make it work. The changes,

I don't see the purpose of init_request, at least I don't think anybody is
calling it.

Overriding the parse of the CrawlSpider and changing the response to
HtmlResponse before passing it to the base parse.

Again changing the response to HtmlResponse in the parse_item

Please understand that we are blindly converting response to HtmlResponse. At
least you should check that response is type "Response" and if possible check
for html tag in body before converting it to HtmlResponse.(Other checks scrapy
does, but fails). Also, may be this conversion may be neatly handled in
downloadmiddleware. If may try converting it Response in the process_response
method, subject to , that process_response is handled before the , call_back
of the spider.

    
    
    #from basketbase.items import BasketbaseItem
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
    from scrapy.selector import HtmlXPathSelector
    from scrapy.http import Request
    from scrapy.http import TextResponse
    from scrapy.http import HtmlResponse
    
    
    class Basketspider(CrawlSpider):
        name = "basketsp17"
        allowed_domains = ["www.euroleague.net"]
        start_urls = ["http://www.euroleague.net/main/results/by-date/"]
        rules = (
            Rule(SgmlLinkExtractor(allow=('main\/results\/showgame\?gamecode\=/\d$\&seasoncode\=E2013\#!boxscore')),follow=True),
            Rule(SgmlLinkExtractor(allow=()),callback='parse_item'),
        )  
    
        def init_request(self):
            print 'init request is called'
            return HtmlResponse("http://www.euroleague.net/main/results/by-date/", body = body)
    
        def parse(self,response):
            response = HtmlResponse(url=response.url, status=response.status, headers=response.headers, body=response.body)
            return super(Basketspider,self).parse(response)
    
        def parse_item(self, response):
            response = HtmlResponse(url=response.url, status=response.status, headers=response.headers, body=response.body)
            sel = HtmlXPathSelector(response)
            items=[]
            print 'parse item is called'
            #item = BasketbaseItem()
            #item['date'] = sel.select('//div[@class="gs-dates"]/text()').extract() # Game date
            #item['time'] = sel.select('//div[@class="gs-dates"]/span[@class="GameScoreTimeContainer"]/text()').extract() # Game time
    
            #items.append(item) 
            return items
    

I think this problem of yours is both a combination of the site not following
standard and scrapy not using body to build the reponse. I think we should
raise this issue with scrapy either as a enquiry or issue.

