Scrapy 1.0 - Getting return value after running from python script
Question: I am using the below code to run my crawler from a python script:

    
    
    import scrapy
    from scrapy.crawler import CrawlerProcess
    
    process = CrawlerProcess()
    
    process.crawl(MySpider)
    process.start() # the script will block here until the crawling is finished
    

where my `MySpider` returns an object.

How can I get the return value from this? Please keep in mind this is Scrapy
1.0. All the examples I found are for older versions of Scrapy.

The above code is from :
<http://doc.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-
script>


Answer: Well, the solution would be to implement custom item pipeline to store your
results in a file -- and read the results later.

Alternatively you can do this with configuring the export in Scrapy too. For
example export the items in a file called `results.json`:

    
    
    settings.overrides['FEED_FORMAT'] = 'json'
    settings.overrides['FEED_URI'] = 'results.json'
    

To avoid the temporal file you should use a custom pipeline class and handle
your results on the `spider_closed` signal.

You can refer to this answer for more details:
<http://stackoverflow.com/a/23574703/3941341>

