Running Scrapy tasks in Python
Question: My Scrapy script seems to work just fine when I run it in 'one off' scenarios
from the command line, but if I try running the code twice in the same python
session I get this error:

"ReactorNotRestartable"

Why?

The offending code (last line throws the error):

    
    
    crawler = CrawlerProcess(settings)
    crawler.install()
    crawler.configure()
    
    # schedule spider
    #crawler.crawl(MySpider())
    spider = MySpider()
    crawler.queue.append_spider(spider)
    
    # start engine scrapy/twisted
    crawler.start()
    


Answer: Close to Joël's answer, but I want to elaborate a bit more than is possible in
the comments. If you look at the [Crawler source
code](http://dev.scrapy.org/browser/scrapy/crawler.py), you see that the
`CrawlerProcess` class has a `start`, but also a `stop` function. This `stop`
function takes care of cleaning up the internals of the crawling so that the
system ends up in a state from which it can start again.

So, if you want to restart the crawling without leaving your process, call
`crawler.stop()` at the appropriate time. Later on, simply call
`crawler.start()` again to resume operations.

**Edit:** in retrospect, this is not possible (due to the Twisted reactor, as
mentioned in a different answer); the `stop` just takes care of a clean
termination. Looking back at my code, I happened to have a wrapper for the
Crawler processes. Below you can find some (redacted) code to make it work
using Python's multiprocessing module. In this way you can more easily restart
crawlers. (Note: I found the code online last month, but I didn't include the
source... so if someone knows where it came from, I'll update the credits for
the source.)

    
    
    from scrapy import project, signals
    from scrapy.conf import settings
    from scrapy.crawler import CrawlerProcess
    from scrapy.xlib.pydispatch import dispatcher
    from multiprocessing.queues import Queue
    from multiprocessing import Process
    
    class CrawlerWorker(Process):
        def __init__(self, spider, results):
            Process.__init__(self)
            self.results = results
    
            self.crawler = CrawlerProcess(settings)
            if not hasattr(project, 'crawler'):
                self.crawler.install()
            self.crawler.configure()
    
            self.items = []
            self.spider = spider
            dispatcher.connect(self._item_passed, signals.item_passed)
    
        def _item_passed(self, item):
            self.items.append(item)
    
        def run(self):
            self.crawler.crawl(self.spider)
            self.crawler.start()
            self.crawler.stop()
            self.results.put(self.items)
    
    # The part below can be called as often as you want
    results = Queue()
    crawler = CrawlerWorker(MySpider(myArgs), results)
    crawler.start()
    for item in results.get():
        pass # Do something with item
    

