prompt: We are currently solving the following issue within our repository. Here is the issue text:
  --- BEGIN ISSUE ---
  Scrapy does not use a non-zero exit code when pipeline's open_spider throws the exception
  <!--
  
  Thanks for taking an interest in Scrapy!
  
  If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
  The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.
  
  Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md
  
  The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs
  
  -->
  
  ### Description
  In our case, we execute command `scrapy crawl` in airflow task and the exit code would be used to judge this task success or failure. I agree that `scrapy crawl` ignores spider exceptions because it's unpredictable in the crawling process. 
  
  Back to our case, we export data to file or database in the pipeline and we create the directory or database connection in `open_spider(self, spider)`. I think if there is an exception happens during this function, it's reasonable to propagate a non-zero exit code. it because we normally do some initialization in this function.
  
  ### Steps to Reproduce
  
  - scrapy startproject test_spider
  - cd test_spider
  - scrapy genspider example example.com
  - modify spiders/example.py to  
  ```
  # -*- coding: utf-8 -*-
  import scrapy
  
  
  class ExampleSpider(scrapy.Spider):
      name = 'example'
      allowed_domains = ['example.com']
      start_urls = ['http://example.com/']
  
      custom_settings = {
          'ITEM_PIPELINES': {
              'test_spider.pipelines.TestSpiderPipeline': 300
          }
      }
  
      def parse(self, response):
          pass
  ```
  - modify pipelines.py to 
  ```
  # -*- coding: utf-8 -*-
  
  # Define your item pipelines here
  #
  # Don't forget to add your pipeline to the ITEM_PIPELINES setting
  # See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html
  
  
  class TestSpiderPipeline(object):
  
      def open_spider(self, spider):
          raise Exception('error')
  
      def process_item(self, item, spider):
          return item
  ```
  - scrapy crawl example
  - echo $?  
  
  **Expected behavior:** [What you expect to happen]
  return non-zero exit code
  
  **Actual behavior:** [What actually happens]
  return zero exit code
  
  **Reproduces how often:** [What percentage of the time does it reproduce?]
  100%
  
  ### Versions
  Scrapy       : 1.8.0
  lxml         : 4.3.3.0
  libxml2      : 2.9.9
  cssselect    : 1.0.3
  parsel       : 1.5.1
  w3lib        : 1.20.0
  Twisted      : 19.2.0
  Python       : 3.7.3 (default, Mar 27 2019, 09:23:39) - [Clang 10.0.0 (clang-1000.11.45.5)]
  pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1b  26 Feb 2019)
  cryptography : 2.6.1
  Platform     : Darwin-18.5.0-x86_64-i386-64bit
  
  ### Additional context
  
  I could get the expected behavior if I change `def run(self, args, opts)` in scrapy/commands/crawl.py  to 
  ```
      def run(self, args, opts):
          if len(args) < 1:
              raise UsageError()
          elif len(args) > 1:
              raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
          spname = args[0]
  
          res = self.crawler_process.crawl(spname, **opts.spargs)
  
          if hasattr(res, 'result') and res.result is not None and issubclass(res.result.type, Exception):
              self.exitcode = 1
          else:
              self.crawler_process.start()
  
              if self.crawler_process.bootstrap_failed:
                  self.exitcode = 1
  ```
  original `def run(self, args, opts)`
  ```
      def run(self, args, opts):
          if len(args) < 1:
              raise UsageError()
          elif len(args) > 1:
              raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
          spname = args[0]
  
          self.crawler_process.crawl(spname, **opts.spargs)
          self.crawler_process.start()
  
          if self.crawler_process.bootstrap_failed:
              self.exitcode = 1
  ```
  
  Is it the proper way to modify the code for achieving this purpose? if it is, could I create a PR request for this issue?
  
  
  --- END ISSUE ---
  
  Below are some code segments, each from a relevant file. One or more of these files may contain bugs.
  --- BEGIN FILES ---
  Path: `scrapy/commands/crawl.py`
  Content:
  ```
  1 import os
  2 from scrapy.commands import ScrapyCommand
  3 from scrapy.utils.conf import arglist_to_dict
  4 from scrapy.utils.python import without_none_values
  5 from scrapy.exceptions import UsageError
  6 
  7 
  8 class Command(ScrapyCommand):
  9 
  10     requires_project = True
  11 
  12     def syntax(self):
  13         return "[options] <spider>"
  14 
  15     def short_desc(self):
  16         return "Run a spider"
  17 
  18     def add_options(self, parser):
  19         ScrapyCommand.add_options(self, parser)
  20         parser.add_option("-a", dest="spargs", action="append", default=[], metavar="NAME=VALUE",
  21                           help="set spider argument (may be repeated)")
  22         parser.add_option("-o", "--output", metavar="FILE",
  23                           help="dump scraped items into FILE (use - for stdout)")
  24         parser.add_option("-t", "--output-format", metavar="FORMAT",
  25                           help="format to use for dumping items with -o")
  26 
  27     def process_options(self, args, opts):
  28         ScrapyCommand.process_options(self, args, opts)
  29         try:
  30             opts.spargs = arglist_to_dict(opts.spargs)
  31         except ValueError:
  32             raise UsageError("Invalid -a value, use -a NAME=VALUE", print_help=False)
  33         if opts.output:
  34             if opts.output == '-':
  35                 self.settings.set('FEED_URI', 'stdout:', priority='cmdline')
  36             else:
  37                 self.settings.set('FEED_URI', opts.output, priority='cmdline')
  38             feed_exporters = without_none_values(
  39                 self.settings.getwithbase('FEED_EXPORTERS'))
  40             valid_output_formats = feed_exporters.keys()
  41             if not opts.output_format:
  42                 opts.output_format = os.path.splitext(opts.output)[1].replace(".", "")
  43             if opts.output_format not in valid_output_formats:
  44                 raise UsageError("Unrecognized output format '%s', set one"
  45                                  " using the '-t' switch or as a file extension"
  46                                  " from the supported list %s" % (opts.output_format,
  47                                                                   tuple(valid_output_formats)))
  48             self.settings.set('FEED_FORMAT', opts.output_format, priority='cmdline')
  49 
  50     def run(self, args, opts):
  51         if len(args) < 1:
  52             raise UsageError()
  53         elif len(args) > 1:
  54             raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
  55         spname = args[0]
  56 
  57         self.crawler_process.crawl(spname, **opts.spargs)
  58         self.crawler_process.start()
  59 
  60         if self.crawler_process.bootstrap_failed:
  61             self.exitcode = 1
  62
  ```
  --- END FILES ---
  
  Please first localize the bug based on the issue statement, and then generate a patch according to the `git diff` format fenced by three backticks.
  
  Here is an example:
  
  ```diff
  diff --git a/examples/server_async.py b/examples/server_async.py
  --- a/examples/server_async.py
  +++ b/examples/server_async.py
  @@ -313,4 +313,4 @@
   
   
   if __name__ == "__main__":
  -    asyncio.run(run_async_server("."), debug=True)
  +    asyncio.run(run_async_server(), debug=True)
  diff --git a/examples/server_sync.py b/examples/server_sync.py
  --- a/examples/server_sync.py
  +++ b/examples/server_sync.py
  @@ -313,5 +313,5 @@
   
   
   if __name__ == "__main__":
  -    server = run_sync_server(".")
  +    server = run_sync_server()
       server.shutdown()
  
  ```
  
golden_diff: diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py
  --- a/scrapy/commands/crawl.py
  +++ b/scrapy/commands/crawl.py
  @@ -54,8 +54,13 @@
               raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
           spname = args[0]
   
  -        self.crawler_process.crawl(spname, **opts.spargs)
  -        self.crawler_process.start()
  +        crawl_defer = self.crawler_process.crawl(spname, **opts.spargs)
   
  -        if self.crawler_process.bootstrap_failed:
  +        if getattr(crawl_defer, 'result', None) is not None and issubclass(crawl_defer.result.type, Exception):
               self.exitcode = 1
  +        else:
  +            self.crawler_process.start()
  +
  +            if self.crawler_process.bootstrap_failed or \
  +                    (hasattr(self.crawler_process, 'has_exception') and self.crawler_process.has_exception):
  +                self.exitcode = 1
  
verification_info: {"golden_diff": "diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py\n--- a/scrapy/commands/crawl.py\n+++ b/scrapy/commands/crawl.py\n@@ -54,8 +54,13 @@\n             raise UsageError(\"running 'scrapy crawl' with more than one spider is no longer supported\")\n         spname = args[0]\n \n-        self.crawler_process.crawl(spname, **opts.spargs)\n-        self.crawler_process.start()\n+        crawl_defer = self.crawler_process.crawl(spname, **opts.spargs)\n \n-        if self.crawler_process.bootstrap_failed:\n+        if getattr(crawl_defer, 'result', None) is not None and issubclass(crawl_defer.result.type, Exception):\n             self.exitcode = 1\n+        else:\n+            self.crawler_process.start()\n+\n+            if self.crawler_process.bootstrap_failed or \\\n+                    (hasattr(self.crawler_process, 'has_exception') and self.crawler_process.has_exception):\n+                self.exitcode = 1\n", "issue": "Scrapy does not use a non-zero exit code when pipeline's open_spider throws the exception\n<!--\r\n\r\nThanks for taking an interest in Scrapy!\r\n\r\nIf you have a question that starts with \"How to...\", please see the Scrapy Community page: https://scrapy.org/community/.\r\nThe GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.\r\n\r\nKeep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md\r\n\r\nThe following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs\r\n\r\n-->\r\n\r\n### Description\r\nIn our case, we execute command `scrapy crawl` in airflow task and the exit code would be used to judge this task success or failure. I agree that `scrapy crawl` ignores spider exceptions because it's unpredictable in the crawling process. \r\n\r\nBack to our case, we export data to file or database in the pipeline and we create the directory or database connection in `open_spider(self, spider)`. I think if there is an exception happens during this function, it's reasonable to propagate a non-zero exit code. it because we normally do some initialization in this function.\r\n\r\n### Steps to Reproduce\r\n\r\n- scrapy startproject test_spider\r\n- cd test_spider\r\n- scrapy genspider example example.com\r\n- modify spiders/example.py to  \r\n```\r\n# -*- coding: utf-8 -*-\r\nimport scrapy\r\n\r\n\r\nclass ExampleSpider(scrapy.Spider):\r\n    name = 'example'\r\n    allowed_domains = ['example.com']\r\n    start_urls = ['http://example.com/']\r\n\r\n    custom_settings = {\r\n        'ITEM_PIPELINES': {\r\n            'test_spider.pipelines.TestSpiderPipeline': 300\r\n        }\r\n    }\r\n\r\n    def parse(self, response):\r\n        pass\r\n```\r\n- modify pipelines.py to \r\n```\r\n# -*- coding: utf-8 -*-\r\n\r\n# Define your item pipelines here\r\n#\r\n# Don't forget to add your pipeline to the ITEM_PIPELINES setting\r\n# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html\r\n\r\n\r\nclass TestSpiderPipeline(object):\r\n\r\n    def open_spider(self, spider):\r\n        raise Exception('error')\r\n\r\n    def process_item(self, item, spider):\r\n        return item\r\n```\r\n- scrapy crawl example\r\n- echo $?  \r\n\r\n**Expected behavior:** [What you expect to happen]\r\nreturn non-zero exit code\r\n\r\n**Actual behavior:** [What actually happens]\r\nreturn zero exit code\r\n\r\n**Reproduces how often:** [What percentage of the time does it reproduce?]\r\n100%\r\n\r\n### Versions\r\nScrapy       : 1.8.0\r\nlxml         : 4.3.3.0\r\nlibxml2      : 2.9.9\r\ncssselect    : 1.0.3\r\nparsel       : 1.5.1\r\nw3lib        : 1.20.0\r\nTwisted      : 19.2.0\r\nPython       : 3.7.3 (default, Mar 27 2019, 09:23:39) - [Clang 10.0.0 (clang-1000.11.45.5)]\r\npyOpenSSL    : 19.0.0 (OpenSSL 1.1.1b  26 Feb 2019)\r\ncryptography : 2.6.1\r\nPlatform     : Darwin-18.5.0-x86_64-i386-64bit\r\n\r\n### Additional context\r\n\r\nI could get the expected behavior if I change `def run(self, args, opts)` in scrapy/commands/crawl.py  to \r\n```\r\n    def run(self, args, opts):\r\n        if len(args) < 1:\r\n            raise UsageError()\r\n        elif len(args) > 1:\r\n            raise UsageError(\"running 'scrapy crawl' with more than one spider is no longer supported\")\r\n        spname = args[0]\r\n\r\n        res = self.crawler_process.crawl(spname, **opts.spargs)\r\n\r\n        if hasattr(res, 'result') and res.result is not None and issubclass(res.result.type, Exception):\r\n            self.exitcode = 1\r\n        else:\r\n            self.crawler_process.start()\r\n\r\n            if self.crawler_process.bootstrap_failed:\r\n                self.exitcode = 1\r\n```\r\noriginal `def run(self, args, opts)`\r\n```\r\n    def run(self, args, opts):\r\n        if len(args) < 1:\r\n            raise UsageError()\r\n        elif len(args) > 1:\r\n            raise UsageError(\"running 'scrapy crawl' with more than one spider is no longer supported\")\r\n        spname = args[0]\r\n\r\n        self.crawler_process.crawl(spname, **opts.spargs)\r\n        self.crawler_process.start()\r\n\r\n        if self.crawler_process.bootstrap_failed:\r\n            self.exitcode = 1\r\n```\r\n\r\nIs it the proper way to modify the code for achieving this purpose? if it is, could I create a PR request for this issue?\r\n\n", "before_files": [{"content": "import os\nfrom scrapy.commands import ScrapyCommand\nfrom scrapy.utils.conf import arglist_to_dict\nfrom scrapy.utils.python import without_none_values\nfrom scrapy.exceptions import UsageError\n\n\nclass Command(ScrapyCommand):\n\n    requires_project = True\n\n    def syntax(self):\n        return \"[options] <spider>\"\n\n    def short_desc(self):\n        return \"Run a spider\"\n\n    def add_options(self, parser):\n        ScrapyCommand.add_options(self, parser)\n        parser.add_option(\"-a\", dest=\"spargs\", action=\"append\", default=[], metavar=\"NAME=VALUE\",\n                          help=\"set spider argument (may be repeated)\")\n        parser.add_option(\"-o\", \"--output\", metavar=\"FILE\",\n                          help=\"dump scraped items into FILE (use - for stdout)\")\n        parser.add_option(\"-t\", \"--output-format\", metavar=\"FORMAT\",\n                          help=\"format to use for dumping items with -o\")\n\n    def process_options(self, args, opts):\n        ScrapyCommand.process_options(self, args, opts)\n        try:\n            opts.spargs = arglist_to_dict(opts.spargs)\n        except ValueError:\n            raise UsageError(\"Invalid -a value, use -a NAME=VALUE\", print_help=False)\n        if opts.output:\n            if opts.output == '-':\n                self.settings.set('FEED_URI', 'stdout:', priority='cmdline')\n            else:\n                self.settings.set('FEED_URI', opts.output, priority='cmdline')\n            feed_exporters = without_none_values(\n                self.settings.getwithbase('FEED_EXPORTERS'))\n            valid_output_formats = feed_exporters.keys()\n            if not opts.output_format:\n                opts.output_format = os.path.splitext(opts.output)[1].replace(\".\", \"\")\n            if opts.output_format not in valid_output_formats:\n                raise UsageError(\"Unrecognized output format '%s', set one\"\n                                 \" using the '-t' switch or as a file extension\"\n                                 \" from the supported list %s\" % (opts.output_format,\n                                                                  tuple(valid_output_formats)))\n            self.settings.set('FEED_FORMAT', opts.output_format, priority='cmdline')\n\n    def run(self, args, opts):\n        if len(args) < 1:\n            raise UsageError()\n        elif len(args) > 1:\n            raise UsageError(\"running 'scrapy crawl' with more than one spider is no longer supported\")\n        spname = args[0]\n\n        self.crawler_process.crawl(spname, **opts.spargs)\n        self.crawler_process.start()\n\n        if self.crawler_process.bootstrap_failed:\n            self.exitcode = 1\n", "path": "scrapy/commands/crawl.py"}], "after_files": [{"content": "import os\nfrom scrapy.commands import ScrapyCommand\nfrom scrapy.utils.conf import arglist_to_dict\nfrom scrapy.utils.python import without_none_values\nfrom scrapy.exceptions import UsageError\n\n\nclass Command(ScrapyCommand):\n\n    requires_project = True\n\n    def syntax(self):\n        return \"[options] <spider>\"\n\n    def short_desc(self):\n        return \"Run a spider\"\n\n    def add_options(self, parser):\n        ScrapyCommand.add_options(self, parser)\n        parser.add_option(\"-a\", dest=\"spargs\", action=\"append\", default=[], metavar=\"NAME=VALUE\",\n                          help=\"set spider argument (may be repeated)\")\n        parser.add_option(\"-o\", \"--output\", metavar=\"FILE\",\n                          help=\"dump scraped items into FILE (use - for stdout)\")\n        parser.add_option(\"-t\", \"--output-format\", metavar=\"FORMAT\",\n                          help=\"format to use for dumping items with -o\")\n\n    def process_options(self, args, opts):\n        ScrapyCommand.process_options(self, args, opts)\n        try:\n            opts.spargs = arglist_to_dict(opts.spargs)\n        except ValueError:\n            raise UsageError(\"Invalid -a value, use -a NAME=VALUE\", print_help=False)\n        if opts.output:\n            if opts.output == '-':\n                self.settings.set('FEED_URI', 'stdout:', priority='cmdline')\n            else:\n                self.settings.set('FEED_URI', opts.output, priority='cmdline')\n            feed_exporters = without_none_values(\n                self.settings.getwithbase('FEED_EXPORTERS'))\n            valid_output_formats = feed_exporters.keys()\n            if not opts.output_format:\n                opts.output_format = os.path.splitext(opts.output)[1].replace(\".\", \"\")\n            if opts.output_format not in valid_output_formats:\n                raise UsageError(\"Unrecognized output format '%s', set one\"\n                                 \" using the '-t' switch or as a file extension\"\n                                 \" from the supported list %s\" % (opts.output_format,\n                                                                  tuple(valid_output_formats)))\n            self.settings.set('FEED_FORMAT', opts.output_format, priority='cmdline')\n\n    def run(self, args, opts):\n        if len(args) < 1:\n            raise UsageError()\n        elif len(args) > 1:\n            raise UsageError(\"running 'scrapy crawl' with more than one spider is no longer supported\")\n        spname = args[0]\n\n        crawl_defer = self.crawler_process.crawl(spname, **opts.spargs)\n\n        if getattr(crawl_defer, 'result', None) is not None and issubclass(crawl_defer.result.type, Exception):\n            self.exitcode = 1\n        else:\n            self.crawler_process.start()\n\n            if self.crawler_process.bootstrap_failed or \\\n                    (hasattr(self.crawler_process, 'has_exception') and self.crawler_process.has_exception):\n                self.exitcode = 1\n", "path": "scrapy/commands/crawl.py"}]}