Run Multiple Spider sequentially
Question: 
    Class Myspider1
    #do something....
    
    Class Myspider2
    #do something...
    

The above is the architecture of my spider.py file. and i am trying to run the
Myspider1 first and then run the Myspider2 multiples times depend on some
conditions. How Could I do that??? any tips?

    
    
    configure_logging()
    runner = CrawlerRunner()
    def crawl():
        yield runner.crawl(Myspider1,arg.....)
        yield runner.crawl(Myspider2,arg.....)
    crawl()
    reactor.run()
    

I am trying to use this way.but have no idea how to run it. Should I run the
cmd on the cmd(what commands?) or just run the python file??

thanks a lot!!!


Answer: **run the python file**  
for example: **test.py**

    
    
    import scrapy
    from twisted.internet import reactor, defer
    from scrapy.crawler import CrawlerRunner
    from scrapy.utils.log import configure_logging
    
    class MySpider1(scrapy.Spider):
        # Your first spider definition
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
                   "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"
        ]
    
        def parse(self, response):
            print "first spider"
    
    class MySpider2(scrapy.Spider):
        # Your second spider definition
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
                    "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]
    
        def parse(self, response):
            print "second spider"
    
    configure_logging()
    runner = CrawlerRunner()
    
    @defer.inlineCallbacks
    def crawl():
        yield runner.crawl(MySpider1)
        yield runner.crawl(MySpider2)
        reactor.stop()
    
    crawl()
    reactor.run() # the script will block here until the last crawl call is finished
    

Now run **python test.py > output.txt**  
You can observe from the output.txt that your spiders run sequentially.

