title: Scrapy - generating items outside of parse callback
body: <p>This might be a bit of an odd one. I have a Scrapy project with a few spiders that inherit from CrawlSpider. Aside for their normal execution (going through the intended website), I also want to be able to <em>push items outside of the scope of the original callback</em>. </p>
  
  <p>I have a thread defined that goes over files in a folder, and then passes them on to parse_files, as if it was content downloaded by Scrapy. Is there any way I can get the items generated from that through the pipelines and middlewares I have, as if it was just another downloaded page?</p>
  
  <p>I know that's not the architecture they had in mind, but I'm wondering if I can work around this. I'm familiar with Scrapy's architecture, and am basically looking for a good way to push items to the Engine.</p>
  
  <pre><code>class SomeSpider(CrawlSpider):
  name = "generic_spider"
  
  def __init__(self):
      CrawlSpider.__init__(self, instance_config)
      self.file_thread = Thread(target=self._file_thread_loop)
      self.file_thread.daemon = True
      self.file_thread.start()
  
      self.rules += (Rule(LxmlLinkExtractor(allow=['/somepath/'], deny=[], callback=self.parse_items, follow=True),)
  
  def _file_thread_loop(self):
      while True:
      #... read files...
          for file in files:
              response = HtmlResponse(url=file['url'], body=file['body'])
              for item in self.parse_items(response):
                  yield item # &lt;-- I want this to go to the pipelines and middlewares
  
          time.sleep(10)
  
  
  def parse_items(self, response):
      hxs = Selector(response)
  
      # ... parse page ...
      for item in resulting_items:
          yield item    
  </code></pre>