<h2 dir="auto">Checklist</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.10
            billiard:3.5.0.3 librabbitmq:1.6.1
platform -&gt; system:Linux arch:64bit, ELF imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:librabbitmq results:mongodb://cyberivy:**@127.0.0.1/"><pre class="notranslate"><code class="notranslate">software -&gt; celery:4.0.2 (latentcall) kombu:4.0.2 py:2.7.10
            billiard:3.5.0.3 librabbitmq:1.6.1
platform -&gt; system:Linux arch:64bit, ELF imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:librabbitmq results:mongodb://cyberivy:**@127.0.0.1/
</code></pre></div>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
</ul>
<h2 dir="auto">Steps to reproduce</h2>
<p dir="auto">It rarely happened. I can't reproduce</p>
<h2 dir="auto">Expected behavior</h2>
<h2 dir="auto">Actual behavior</h2>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Traceback (most recent call last):
  File &quot;/opt/python2.7.10-customized/lib/python2.7/site-packages/celery/app/trace.py&quot;, line 362, in trace_task
    request=task_request,
  File &quot;/opt/python2.7.10-customized/lib/python2.7/site-packages/celery/backends/base.py&quot;, line 309, in store_result
    request=request, **kwargs)
  File &quot;/opt/python2.7.10-customized/lib/python2.7/site-packages/celery/backends/mongodb.py&quot;, line 175, in _store_result
    self.collection.save(meta)
  File &quot;/opt/python2.7.10-customized/lib/python2.7/site-packages/pymongo/collection.py&quot;, line 2185, in save
    check_keys, False, manipulate, write_concern)
  File &quot;/opt/python2.7.10-customized/lib/python2.7/site-packages/pymongo/collection.py&quot;, line 710, in _update
    _check_write_command_response([(0, result)])
  File &quot;/opt/python2.7.10-customized/lib/python2.7/site-packages/pymongo/helpers.py&quot;, line 300, in _check_write_command_response
    raise DuplicateKeyError(error.get(&quot;errmsg&quot;), 11000, error)
DuplicateKeyError: E11000 duplicate key error collection: celery.celery_taskmeta index: _id_ dup key: { : &quot;671ea5ba-9eff-447e-b6c5-329477da3cb1&quot; }"><pre class="notranslate"><code class="notranslate">Traceback (most recent call last):
  File "/opt/python2.7.10-customized/lib/python2.7/site-packages/celery/app/trace.py", line 362, in trace_task
    request=task_request,
  File "/opt/python2.7.10-customized/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result
    request=request, **kwargs)
  File "/opt/python2.7.10-customized/lib/python2.7/site-packages/celery/backends/mongodb.py", line 175, in _store_result
    self.collection.save(meta)
  File "/opt/python2.7.10-customized/lib/python2.7/site-packages/pymongo/collection.py", line 2185, in save
    check_keys, False, manipulate, write_concern)
  File "/opt/python2.7.10-customized/lib/python2.7/site-packages/pymongo/collection.py", line 710, in _update
    _check_write_command_response([(0, result)])
  File "/opt/python2.7.10-customized/lib/python2.7/site-packages/pymongo/helpers.py", line 300, in _check_write_command_response
    raise DuplicateKeyError(error.get("errmsg"), 11000, error)
DuplicateKeyError: E11000 duplicate key error collection: celery.celery_taskmeta index: _id_ dup key: { : "671ea5ba-9eff-447e-b6c5-329477da3cb1" }
</code></pre></div>
<p dir="auto">Thanks</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://github.com/celery/celery/discussions">discussions forum</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate">
</code></pre></div>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 5.2.3 (dawn-chorus)</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:5.2.3 (dawn-chorus) kombu:5.2.3 py:3.9.10
            billiard:3.6.4.0 py-amqp:5.0.9
platform -&gt; system:Darwin arch:64bit
            kernel version:20.6.0 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:django-db"><pre class="notranslate"><code class="notranslate">software -&gt; celery:5.2.3 (dawn-chorus) kombu:5.2.3 py:3.9.10
            billiard:3.6.4.0 py-amqp:5.0.9
platform -&gt; system:Darwin arch:64bit
            kernel version:20.6.0 imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:django-db
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Celery Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate"></code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">Expected worker to exit with a non-zero code</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">Ran celery with the following settings</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CELERY_BROKER_CONNECTION_RETRY = False
CELERY_BROKER_CONNECTION_TIMEOUT = 2 # seconds"><pre class="notranslate"><code class="notranslate">CELERY_BROKER_CONNECTION_RETRY = False
CELERY_BROKER_CONNECTION_TIMEOUT = 2 # seconds
</code></pre></div>
<p dir="auto">because for kubernetes related reasons it's better for us if the workers just die immediately if the config is wrong.</p>
<p dir="auto">The worker does exit fast enough, but the exit code is 0 despite the following backtrace</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2022-03-18 17:33:30,670: CRITICAL/MainProcess] Unrecoverable error: ConnectionRefusedError(61, 'Connection refused')
Traceback (most recent call last):
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/worker.py&quot;, line 203, in start
    self.blueprint.start(self)
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/bootsteps.py&quot;, line 116, in start
    step.start(parent)
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/bootsteps.py&quot;, line 365, in start
    return self.obj.start()
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py&quot;, line 326, in start
    blueprint.start(self)
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/bootsteps.py&quot;, line 116, in start
    step.start(parent)
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/connection.py&quot;, line 21, in start
    c.connection = c.connect()
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py&quot;, line 422, in connect
    conn = self.connection_for_read(heartbeat=self.amqheartbeat)
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py&quot;, line 428, in connection_for_read
    return self.ensure_connected(
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py&quot;, line 451, in ensure_connected
    conn.connect()
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py&quot;, line 275, in connect
    return self._ensure_connection(
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py&quot;, line 434, in _ensure_connection
    return retry_over_time(
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/utils/functional.py&quot;, line 312, in retry_over_time
    return fun(*args, **kwargs)
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py&quot;, line 878, in _connection_factory
    self._connection = self._establish_connection()
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py&quot;, line 813, in _establish_connection
    conn = self.transport.establish_connection()
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/transport/pyamqp.py&quot;, line 201, in establish_connection
    conn.connect()
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/amqp/connection.py&quot;, line 323, in connect
    self.transport.connect()
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/amqp/transport.py&quot;, line 113, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File &quot;/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/amqp/transport.py&quot;, line 197, in _connect
    self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused"><pre class="notranslate"><code class="notranslate">[2022-03-18 17:33:30,670: CRITICAL/MainProcess] Unrecoverable error: ConnectionRefusedError(61, 'Connection refused')
Traceback (most recent call last):
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 326, in start
    blueprint.start(self)
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/connection.py", line 21, in start
    c.connection = c.connect()
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 422, in connect
    conn = self.connection_for_read(heartbeat=self.amqheartbeat)
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 428, in connection_for_read
    return self.ensure_connected(
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 451, in ensure_connected
    conn.connect()
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py", line 275, in connect
    return self._ensure_connection(
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py", line 434, in _ensure_connection
    return retry_over_time(
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
    return fun(*args, **kwargs)
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py", line 878, in _connection_factory
    self._connection = self._establish_connection()
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/connection.py", line 813, in _establish_connection
    conn = self.transport.establish_connection()
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/kombu/transport/pyamqp.py", line 201, in establish_connection
    conn.connect()
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/amqp/connection.py", line 323, in connect
    self.transport.connect()
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/amqp/transport.py", line 113, in connect
    self._connect(self.host, self.port, self.connect_timeout)
  File "/Users/tomparker-shemilt/.virtualenvs/case-cards/lib/python3.9/site-packages/amqp/transport.py", line 197, in _connect
    self.sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused
</code></pre></div>
<p dir="auto">I tried messing around with locally editing <a href="https://github.com/celery/celery/blob/master/celery/bin/worker.py">https://github.com/celery/celery/blob/master/celery/bin/worker.py</a> and noted the</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="        worker.start()
        return worker.exitcode
    except SecurityError as e:
        ctx.obj.error(e.args[0])
        ctx.exit(1)"><pre class="notranslate"><code class="notranslate">        worker.start()
        return worker.exitcode
    except SecurityError as e:
        ctx.obj.error(e.args[0])
        ctx.exit(1)
</code></pre></div>
<p dir="auto">block at the end. If I replaced <code class="notranslate">return worker.exitcode</code> with <code class="notranslate">ctx.exit(worker.exitcode)</code> it seems to work as I was expecting i.e. exit code of 1</p>