title: Scrapy not writing all outputs to CSV
body: <p>I am using Scrapy to ping a bunch of webpages and determine which of the homepages are no longer live. </p>
  
  <p>The full code of my spider is very simple, and reproduced below:</p>
  
  <pre><code>class BrokenLinksSpider(scrapy.Spider):
  name = 'broken-link-spider'
  
  def __init__(self, id_homepage_mapping, *args, **kwargs):
      self.org_homepages = id_homepage_mapping
      self.download_timeout = 10
  
  def start_requests(self):
      for qId, url in self.org_homepages.items():
          if url.endswith("/"):
              url = url[0:-1]
          yield scrapy.Request(url,
              callback=self.complete_callback,
              errback=self.error_callback,
              dont_filter=True,
              meta={ 'handle_httpstatus_all': True, 'id': id })
  
  def complete_callback(self, response):
      if str(response.status)[0] in ("4", "5"):
          yield BrokenLinkCheckerItem(
              url=response.request.url,
              status=response.status,
              qId=response.meta['id'])
  
  def error_callback(self, failure):
      yield BrokenLinkCheckerItem(
          url=failure.request.url,
          status=repr(failure),
          qId=failure.request.meta['id'])
  </code></pre>
  
  <p>In the logs of my app, I can see in the scrapy stats collector that there were exceptions with fetching about 40,000 links. However, the csv output file that I'm writing to only contains about 8k rows. Furthermore, the CSV file seems to contain all of the different types of exceptions reported by the StatsCollector, just less instances of each. </p>
  
  <p>So my question is, why is there such a discrepancy between what the statsCollector reports and what's actually written to the CSV? Given my understanding of the <code>errback</code> callback function, shouldn't all those exceptions end up being written to the CSV? </p>
  
  <p>Thanks! </p>