title: Scrapy skips image downloads
body: <p>I am trying to scrape image URL's from multiple pages. This is working fine and it outputs all the URL's in my command prompt. However there are only 40-70 images downloaded when I run the spider, this amount of images that should be scraped are 4000. </p>
  
  <p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
  <div class="snippet-code">
  <pre class="snippet-code-js lang-js prettyprint-override"><code>import scrapy
  from stscrape.items import ImageItem
  import logging
  import json
  from PIL import Image
  
  class StSpider(scrapy.Spider):
      name = "lst"
  
      def start_requests(self):
          urls = [
               'https://www.example.com/api/?subcategory=lts&amp;page=1'
          ]
          for url in urls:
              yield scrapy.Request(url=url, callback=self.parse)
  
      def parse(self, response):
          logging.warning('in page')
          jsonresponse = json.loads(response.body_as_unicode())
          pages = jsonresponse['data']['pagination']['total_pages']
          for page in range(pages):
              yield scrapy.Request(url='https://www.example.com/api/?subcategory=lts&amp;page=' + str(page), callback=self.page)
  
      def page(self, response):
          pageresponse = json.loads(response.body_as_unicode())
          items = pageresponse['data']['segments'][0]['segment_items']
          image_url = []
          for item in items:
              image_url.append(item['product_card']['image_url'])
          yield ImageItem(image_urls=image_url)</code></pre>
  </div>
  </div>
  </p>
  
  <p>I think the problem has to do with the process of the image downloads. I feel like it somehow blocks more images from being downloaded.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>This is what scrapy outputs</p>
  
  <pre><code>2018-01-23 21:43:20 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
  {'downloader/request_bytes': 91540,
   'downloader/request_count': 160,
   'downloader/request_method_count/GET': 160,
   'downloader/response_bytes': 1313499,
   'downloader/response_count': 160,
   'downloader/response_status_count/200': 160,
   'dupefilter/filtered': 1,
   'file_count': 55,
   'file_status_count/downloaded': 55,
   'finish_reason': 'finished',
   'finish_time': datetime.datetime(2018, 1, 23, 20, 43, 20, 693000),
   'item_scraped_count': 99,
   'log_count/DEBUG': 316,
   'log_count/INFO': 7,
   'request_depth_max': 1,
   'response_received_count': 160,
   'scheduler/dequeued': 100,
   'scheduler/dequeued/memory': 100,
   'scheduler/enqueued': 100,
   'scheduler/enqueued/memory': 100,
   'start_time': datetime.datetime(2018, 1, 23, 20, 43, 5, 45000)}
  2018-01-23 21:43:20 [scrapy.core.engine] INFO: Spider closed (finished)
  </code></pre>
  
  <p>I think the problem is that once 1 URL has been processed and the images have been placed in the image_urls, they will be processed with Yield. This somehow blocks other images from being processed.</p>