<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery. (not against latest commit specifically but against latest version 4.4.5)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.4.5 (cliffs)</p>

<p dir="auto">Celery Report:<br>
<a href="https://pastebin.com/0x13Xjiq" rel="nofollow">https://pastebin.com/0x13Xjiq</a></p>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.6.8</li>
<li><strong>Minimal Celery Version</strong>: 4.4.1</li>
<li><strong>Minimal Kombu Version</strong>: 4.6.8</li>
<li><strong>Minimal Broker Version</strong>: Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: Ubuntu 18</li>
<li><strong>Minimal Broker Client Version</strong>: Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Only relevant packages shown below as 15five is not open-source:
amqp==2.6.0
billiard==3.6.3.0
celery==4.4.5
Django==2.2.13
django-celery-results==1.2.1
future==0.18.2
importlib-metadata==1.6.1
kombu==4.6.10
vine==1.3.0
pytz==2020.1
zipp==3.1.0"><pre class="notranslate"><code class="notranslate">Only relevant packages shown below as 15five is not open-source:
amqp==2.6.0
billiard==3.6.3.0
celery==4.4.5
Django==2.2.13
django-celery-results==1.2.1
future==0.18.2
importlib-metadata==1.6.1
kombu==4.6.10
vine==1.3.0
pytz==2020.1
zipp==3.1.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
broker: AWS elasticache redis
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<p dir="auto"><a href="https://github.com/caleb15/celeryPickleBugReproduction">https://github.com/caleb15/celeryPickleBugReproduction</a></p>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">Error to not happen. If error is a result of our faulty user code, then I would expect the celery queue to log a error and continue operating.</p>
<h1 dir="auto">Actual Behavior</h1>

<p dir="auto">We get a AttributeError and the queue dies. Upon celery restart the queue continues to process events, but eventually hit the same error and dies again.</p>
<p dir="auto">I can send a link to the sentry error event directly to a celery engineer if one is interested. Otherwise the relevant info is below:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="AttributeError billiard.reduction in loadbuf
Error on stopping Pool: &quot;AttributeError(\&quot;can't set attribute\&quot;)&quot;
billiard/reduction.py in loadbuf at line 61
billiard/connection.py in recv at line 279
celery/concurrency/asynpool.py in _flush_outqueue at line 412
celery/concurrency/asynpool.py in iterate_file_descriptors_safely at line 240
celery/concurrency/asynpool.py in on_stop_not_started at line 387
billiard/pool.py in stop at line 526
billiard/pool.py in stop_if_not_current at line 145
billiard/pool.py in join at line 1622
celery/concurrency/prefork.py in on_stop at line 145
celery/concurrency/base.py in stop at line 123
celery/bootsteps.py in stop at line 373
celery/bootsteps.py in send_all at line 151"><pre class="notranslate"><code class="notranslate">AttributeError billiard.reduction in loadbuf
Error on stopping Pool: "AttributeError(\"can't set attribute\")"
billiard/reduction.py in loadbuf at line 61
billiard/connection.py in recv at line 279
celery/concurrency/asynpool.py in _flush_outqueue at line 412
celery/concurrency/asynpool.py in iterate_file_descriptors_safely at line 240
celery/concurrency/asynpool.py in on_stop_not_started at line 387
billiard/pool.py in stop at line 526
billiard/pool.py in stop_if_not_current at line 145
billiard/pool.py in join at line 1622
celery/concurrency/prefork.py in on_stop at line 145
celery/concurrency/base.py in stop at line 123
celery/bootsteps.py in stop at line 373
celery/bootsteps.py in send_all at line 151
</code></pre></div>
<p dir="auto">Another stacktrace:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="celery/concurrency/asynpool.py in _recv_message at line 332
celery/concurrency/asynpool.py in on_result_readable at line 351
kombu/asynchronous/hub.py in create_loop at line 364
celery/worker/loops.py in asynloop at line 83
celery/worker/consumer/consumer.py in start at line 599
celery/bootsteps.py in start at line 119
celery/worker/consumer/consumer.py in start at line 318
celery/bootsteps.py in start at line 369
celery/bootsteps.py in start at line 119
celery/worker/worker.py in start at line 208"><pre class="notranslate"><code class="notranslate">celery/concurrency/asynpool.py in _recv_message at line 332
celery/concurrency/asynpool.py in on_result_readable at line 351
kombu/asynchronous/hub.py in create_loop at line 364
celery/worker/loops.py in asynloop at line 83
celery/worker/consumer/consumer.py in start at line 599
celery/bootsteps.py in start at line 119
celery/worker/consumer/consumer.py in start at line 318
celery/bootsteps.py in start at line 369
celery/bootsteps.py in start at line 119
celery/worker/worker.py in start at line 208
</code></pre></div>
<p dir="auto">sys.argv:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[/srv/venvs/py3/lib/python3.7/site-packages/celery/__main__.py, worker, -A, ff, --loglevel=INFO, -Q, celery, --logfile=/var/log/celery/default%I.log, --pidfile=/var/run/celery/default.pid, --hostname=default@celery0]"><pre class="notranslate"><code class="notranslate">[/srv/venvs/py3/lib/python3.7/site-packages/celery/__main__.py, worker, -A, ff, --loglevel=INFO, -Q, celery, --logfile=/var/log/celery/default%I.log, --pidfile=/var/run/celery/default.pid, --hostname=default@celery0]
</code></pre></div>
<h2 dir="auto">What we have tried so far:</h2>
<p dir="auto">We have tried the following without success:</p>
<ul dir="auto">
<li>Updating python version from 3.6.8 to 3.7.7</li>
<li>Upgrading celery</li>
<li>Moving most tasks out of the default queue into other queues</li>
<li>pinpointing which task caused the error by looking through logs (we have been unable to pinpoint any specific task, the last task processed before error varies)</li>
</ul>
<h2 dir="auto">[update] identification &amp; cause of error:</h2>
<p dir="auto">Our staff engineer noticed the task responsible in the sentry breadcrumbs and I figured out that the task was raising a ValidationError that I suspected was not pickleable. We ignored the error and so far the queue hasn't died on us. My suspicion was confirmed when a CI failure revealed that the error was unpickleable:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="test_update_billing_add_ons_ignores_simultaneous_request_error (ff.mods.billing.tests.test_subscription_logic.UpdateBillingAddOnsTestCase) failed:

    ValidationError(b'&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;\n&lt;error&gt;\n
    &lt;symbol&gt;simultaneous_request&lt;/symbol&gt;\n  &lt;description&gt;A change for
    subscription 456ee8a3881160536ad68346fe91eede is already in
    progress.&lt;/description&gt;\n&lt;/error&gt;')

Unfortunately, the exception it raised cannot be pickled, making it impossible
for the parallel test runner to handle it cleanly.

Here's the error encountered while trying to pickle the exception:

    AttributeError(&quot;can't set attribute&quot;)

You should re-run this test with the --parallel=1 option to reproduce the
failure and get a correct traceback.

Destroying test database for alias 'default'...
TOTAL RUNTIME: 67.09s
TOTAL RUNTIME: 67.09s
multiprocessing.pool.RemoteTraceback: 
&quot;&quot;&quot;
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.7/multiprocessing/pool.py&quot;, line 121, in worker
    result = (True, func(*args, **kwds))
  File &quot;/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py&quot;, line 309, in _run_subsuite
    result = runner.run(subsuite)
  File &quot;/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py&quot;, line 256, in run
    test(result)
  File &quot;/usr/local/lib/python3.7/unittest/suite.py&quot;, line 84, in __call__
    return self.run(*args, **kwds)
  File &quot;/usr/local/lib/python3.7/unittest/suite.py&quot;, line 122, in run
    test(result)
  File &quot;/home/circleci/project/venv/lib/python3.7/site-packages/django/test/testcases.py&quot;, line 271, in __call__
    super().__call__(result)
  File &quot;/usr/local/lib/python3.7/unittest/case.py&quot;, line 676, in __call__
    return self.run(*args, **kwds)
  File &quot;/usr/local/lib/python3.7/unittest/case.py&quot;, line 636, in run
    self._feedErrorsToResult(result, outcome.errors)
  File &quot;/usr/local/lib/python3.7/unittest/case.py&quot;, line 566, in _feedErrorsToResult
    result.addError(test, exc_info)
  File &quot;/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py&quot;, line 197, in addError
    self.check_picklable(test, err)
  File &quot;/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py&quot;, line 130, in check_picklable
    self._confirm_picklable(err)
  File &quot;/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py&quot;, line 104, in _confirm_picklable
    pickle.loads(pickle.dumps(obj))
AttributeError: can't set attribute"><pre class="notranslate"><code class="notranslate">test_update_billing_add_ons_ignores_simultaneous_request_error (ff.mods.billing.tests.test_subscription_logic.UpdateBillingAddOnsTestCase) failed:

    ValidationError(b'&lt;?xml version="1.0" encoding="UTF-8"?&gt;\n&lt;error&gt;\n
    &lt;symbol&gt;simultaneous_request&lt;/symbol&gt;\n  &lt;description&gt;A change for
    subscription 456ee8a3881160536ad68346fe91eede is already in
    progress.&lt;/description&gt;\n&lt;/error&gt;')

Unfortunately, the exception it raised cannot be pickled, making it impossible
for the parallel test runner to handle it cleanly.

Here's the error encountered while trying to pickle the exception:

    AttributeError("can't set attribute")

You should re-run this test with the --parallel=1 option to reproduce the
failure and get a correct traceback.

Destroying test database for alias 'default'...
TOTAL RUNTIME: 67.09s
TOTAL RUNTIME: 67.09s
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 309, in _run_subsuite
    result = runner.run(subsuite)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 256, in run
    test(result)
  File "/usr/local/lib/python3.7/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/local/lib/python3.7/unittest/suite.py", line 122, in run
    test(result)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/testcases.py", line 271, in __call__
    super().__call__(result)
  File "/usr/local/lib/python3.7/unittest/case.py", line 676, in __call__
    return self.run(*args, **kwds)
  File "/usr/local/lib/python3.7/unittest/case.py", line 636, in run
    self._feedErrorsToResult(result, outcome.errors)
  File "/usr/local/lib/python3.7/unittest/case.py", line 566, in _feedErrorsToResult
    result.addError(test, exc_info)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 197, in addError
    self.check_picklable(test, err)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 130, in check_picklable
    self._confirm_picklable(err)
  File "/home/circleci/project/venv/lib/python3.7/site-packages/django/test/runner.py", line 104, in _confirm_picklable
    pickle.loads(pickle.dumps(obj))
AttributeError: can't set attribute
</code></pre></div>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>:</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b>
</summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="
software -&gt; celery:5.2.0rc1 (dawn-chorus) kombu:5.2.0rc1 py:3.7.12
            billiard:3.6.4.0 py-amqp:5.0.6
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.10.47-linuxkit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:pyamqp results:disabled

broker_url: 'amqp://guest:********@broker:5672//'
deprecated_settings: None
"><pre class="notranslate"><code class="notranslate">
software -&gt; celery:5.2.0rc1 (dawn-chorus) kombu:5.2.0rc1 py:3.7.12
            billiard:3.6.4.0 py-amqp:5.0.6
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.10.47-linuxkit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:pyamqp results:disabled

broker_url: 'amqp://guest:********@broker:5672//'
deprecated_settings: None

</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.7</li>
<li><strong>Minimal Celery Version</strong>: 5.1.2</li>
<li><strong>Minimal Kombu Version</strong>: N/A</li>
<li><strong>Minimal Broker Version</strong>: N/A</li>
<li><strong>Minimal Result Backend Version</strong>: N/A</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A</li>
<li><strong>Minimal Broker Client Version</strong>: N/A</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="amqp==5.0.6
billiard==3.6.4.0
cached-property==1.5.2
celery @ git+https://github.com/celery/celery@49452916f94d5ec60af246cea600855e6d976b48
click==8.0.1
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
importlib-metadata==4.8.1
kombu==5.2.0rc1
prompt-toolkit==3.0.20
pytz==2021.3
six==1.16.0
typing-extensions==3.10.0.2
vine==5.0.0
wcwidth==0.2.5
zipp==3.6.0"><pre class="notranslate"><code class="notranslate">amqp==5.0.6
billiard==3.6.4.0
cached-property==1.5.2
celery @ git+https://github.com/celery/celery@49452916f94d5ec60af246cea600855e6d976b48
click==8.0.1
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.2.0
importlib-metadata==4.8.1
kombu==5.2.0rc1
prompt-toolkit==3.0.20
pytz==2021.3
six==1.16.0
typing-extensions==3.10.0.2
vine==5.0.0
wcwidth==0.2.5
zipp==3.6.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from celery import Celery

app = Celery(&quot;tasks&quot;, broker=&quot;pyamqp://guest@broker//&quot;)


class BrokenException(Exception):
    &quot;&quot;&quot;
    This Exception can't be unpickled by default
    &quot;&quot;&quot;
    def __init__(self, msg, other_arg):
        super().__init__(msg)
        self.other_arg = other_arg

@app.task(bind=True)
def unsafe_exception(self):
    self.retry(exc=BrokenException(&quot;msg&quot;, &quot;other_arg&quot;))


def main():
    unsafe_exception.delay()

if __name__ == &quot;__main__&quot;:
    main()"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-v">Celery</span>

<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(<span class="pl-s">"tasks"</span>, <span class="pl-s1">broker</span><span class="pl-c1">=</span><span class="pl-s">"pyamqp://guest@broker//"</span>)


<span class="pl-k">class</span> <span class="pl-v">BrokenException</span>(<span class="pl-v">Exception</span>):
    <span class="pl-s">"""</span>
<span class="pl-s">    This Exception can't be unpickled by default</span>
<span class="pl-s">    """</span>
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">msg</span>, <span class="pl-s1">other_arg</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init__</span>(<span class="pl-s1">msg</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">other_arg</span> <span class="pl-c1">=</span> <span class="pl-s1">other_arg</span>

<span class="pl-en">@<span class="pl-s1">app</span>.<span class="pl-en">task</span>(<span class="pl-s1">bind</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">unsafe_exception</span>(<span class="pl-s1">self</span>):
    <span class="pl-s1">self</span>.<span class="pl-en">retry</span>(<span class="pl-s1">exc</span><span class="pl-c1">=</span><span class="pl-v">BrokenException</span>(<span class="pl-s">"msg"</span>, <span class="pl-s">"other_arg"</span>))


<span class="pl-k">def</span> <span class="pl-en">main</span>():
    <span class="pl-s1">unsafe_exception</span>.<span class="pl-en">delay</span>()

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">"__main__"</span>:
    <span class="pl-en">main</span>()</pre></div>
<p dir="auto">Then run with <code class="notranslate">python tasks.py</code></p>
<p dir="auto">Repo here:<br>
<a href="https://github.com/amcgavin/celery-issue/">https://github.com/amcgavin/celery-issue/</a></p>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">Exception occurs in the worker handler, but handled gracefully.</p>
<p dir="auto"><em>note</em>: this was the behaviour in celery==5.0.2</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">worker process crashes, with exit code 0.</p>
<p dir="auto">Note that raising the exception directly doesn't crash (see the repo for detailed example).</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="celery_1  | [2021-10-08 08:34:10,200: INFO/MainProcess] Task tasks.unsafe_exception[38b08c18-451b-4645-a4ac-ada2e6b43e5e] received
celery_1  | [2021-10-08 08:34:10,201: CRITICAL/MainProcess] Unrecoverable error: TypeError(&quot;__init__() missing 1 required positional argument: 'other_arg'&quot;)
celery_1  | Traceback (most recent call last):
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/worker.py&quot;, line 203, in start
celery_1  |     self.blueprint.start(self)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 365, in start
celery_1  |     return self.obj.start()
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 326, in start
celery_1  |     blueprint.start(self)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 618, in start
celery_1  |     c.loop(*c.loop_args())
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/loops.py&quot;, line 81, in asynloop
celery_1  |     next(loop)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py&quot;, line 361, in create_loop
celery_1  |     cb(*cbargs)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py&quot;, line 325, in on_result_readable
celery_1  |     next(it)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py&quot;, line 306, in _recv_message
celery_1  |     message = load(bufv)
celery_1  | TypeError: __init__() missing 1 required positional argument: 'other_arg'
celery_1  | [2021-10-08 08:34:41,378: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 0 Job: 2.')
celery_1  | Traceback (most recent call last):
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/worker.py&quot;, line 203, in start
celery_1  |     self.blueprint.start(self)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 365, in start
celery_1  |     return self.obj.start()
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 326, in start
celery_1  |     blueprint.start(self)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/bootsteps.py&quot;, line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py&quot;, line 618, in start
celery_1  |     c.loop(*c.loop_args())
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/worker/loops.py&quot;, line 81, in asynloop
celery_1  |     next(loop)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py&quot;, line 361, in create_loop
celery_1  |     cb(*cbargs)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py&quot;, line 325, in on_result_readable
celery_1  |     next(it)
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py&quot;, line 306, in _recv_message
celery_1  |     message = load(bufv)
celery_1  | TypeError: __init__() missing 1 required positional argument: 'other_arg'
celery_1  | 
celery_1  | During handling of the above exception, another exception occurred:
celery_1  | 
celery_1  | Traceback (most recent call last):
celery_1  |   File &quot;/usr/local/lib/python3.7/site-packages/billiard/pool.py&quot;, line 1267, in mark_as_worker_lost
celery_1  |     human_status(exitcode), job._job),
celery_1  | billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 0 Job: 2.
celery-example_celery_1 exited with code 0"><pre class="notranslate"><code class="notranslate">celery_1  | [2021-10-08 08:34:10,200: INFO/MainProcess] Task tasks.unsafe_exception[38b08c18-451b-4645-a4ac-ada2e6b43e5e] received
celery_1  | [2021-10-08 08:34:10,201: CRITICAL/MainProcess] Unrecoverable error: TypeError("__init__() missing 1 required positional argument: 'other_arg'")
celery_1  | Traceback (most recent call last):
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
celery_1  |     self.blueprint.start(self)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
celery_1  |     return self.obj.start()
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
celery_1  |     blueprint.start(self)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
celery_1  |     c.loop(*c.loop_args())
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/loops.py", line 81, in asynloop
celery_1  |     next(loop)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 361, in create_loop
celery_1  |     cb(*cbargs)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 325, in on_result_readable
celery_1  |     next(it)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 306, in _recv_message
celery_1  |     message = load(bufv)
celery_1  | TypeError: __init__() missing 1 required positional argument: 'other_arg'
celery_1  | [2021-10-08 08:34:41,378: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 0 Job: 2.')
celery_1  | Traceback (most recent call last):
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/worker.py", line 203, in start
celery_1  |     self.blueprint.start(self)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 365, in start
celery_1  |     return self.obj.start()
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 326, in start
celery_1  |     blueprint.start(self)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/bootsteps.py", line 116, in start
celery_1  |     step.start(parent)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 618, in start
celery_1  |     c.loop(*c.loop_args())
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/worker/loops.py", line 81, in asynloop
celery_1  |     next(loop)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/kombu/asynchronous/hub.py", line 361, in create_loop
celery_1  |     cb(*cbargs)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 325, in on_result_readable
celery_1  |     next(it)
celery_1  |   File "/usr/local/lib/python3.7/site-packages/celery/concurrency/asynpool.py", line 306, in _recv_message
celery_1  |     message = load(bufv)
celery_1  | TypeError: __init__() missing 1 required positional argument: 'other_arg'
celery_1  | 
celery_1  | During handling of the above exception, another exception occurred:
celery_1  | 
celery_1  | Traceback (most recent call last):
celery_1  |   File "/usr/local/lib/python3.7/site-packages/billiard/pool.py", line 1267, in mark_as_worker_lost
celery_1  |     human_status(exitcode), job._job),
celery_1  | billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 0 Job: 2.
celery-example_celery_1 exited with code 0
</code></pre></div>