Scrapy 4xx/5xx error handling
Question: We're building a distributed system that uses Amazon's SQS to dispatch
messages to workers that run scrapy spiders based on the messages' contents.

We (obviously) only want to remove a message from the queue if its
corresponding spider has been run successfully, i.e. without encountering
4xx/5xx responses.

What I'd like to do is hook into scrapy's `signals` API to fire a callback
that deletes the message from the queue when the spider's closed successfully,
but I'm not sure whether that's actually the semantics of
`signals.spider_closed` (as opposed to "this spider has closed for literally
any reason.")

It's also not clear (at least to me) whether `signals.spider_error` is fired
when encountering an HTTP error code, or only when a Python error is raised
from within the spider.

Any suggestions?


Answer: `signals.spider_error` is raised when a Python error occurs during the spider-
crawl process. If the error occurs in the `spider_closed` signal handler then
`spider_error` is not raised.

A basic approach would be to have a signal handler extension which will
register to the `spider_closed` and `spider_error` events to handle the
statuses -- do not remove the URL from the queue if it contains a status
higher than 399 for example.

Then in these handler methods you can utilize the stats collected by the
spider to see if it was OK or not:

    
    
    class SignalHandler(object):
    
        @classmethod
        def from_crawler(cls,crawler):
            ext = cls()
            crawler.signals.connect(ext.spider_error, signal=signals.spider_error)
            crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed)
            return ext
    
        def spider_error(self, failure, response, spider):
            print "Error on {0}, traceback: {1}".format(response.url, failure.getTraceback())
    
        def spider_closed(self, spider):
            if spider.crawler.stats.get_value('downloader/response_status_count/200') == spider.crawler.stats.get_value('downloader/response_count'):
                # OK, all went fine
            if spider.crawler.stats.get_value('downloader/response_status_count/404') != 0 or spider.crawler.stats.get_value('downloader/response_status_count/503') != 0:
                # something went wrong
    

And of course do not forget to add your `SignalHandler` in the `settings.py`:

    
    
    EXTENSIONS = {'myproject.extensions.signal_handler.SignalHandler': 599,}
    

There is another way of course which requires a bit more coding:

You could handle the status codes yourself with the `handle_httpstatus_list`
parameter of your spider. This allows your spider to handle a list of HTTP
status which would be ignored per default.

Summarizing an approach would be to handle the statuses you are interested in
in your spider and collect them into a `set`.

This would be the spider:

    
    
    class SomeSpider(scrapy.Spider):
        name = "somespider"
    
        start_urls = {"http://stackoverflow.com/questions/25308231/liferay-6-2-lar-import-no-journalfolder-exists-with-the-primary-key-2"}
    
        handle_httpstatus_list = [404, 503]
    
        encountered = set()
    
        def parse(self, response):
            self.encountered.add(response.status)
            # parse the response
    

This would be the extension's new method:

    
    
    def spider_closed(self, spider):
        if 404 in spider.encountered:
            # handle 404
    

