<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 4.4.7</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')"><pre class="notranslate"><code class="notranslate">Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.7</li>
<li><strong>Minimal Celery Version</strong>: 4.4.7</li>
<li><strong>Minimal Kombu Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Version</strong>: RabbitMQ v3.8.21</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="amqp==5.0.6
arabic-reshaper==2.1.3
asgiref==3.4.1
autopep8==1.5.7
bcrypt==3.2.0
billiard==3.6.4.0
bitstring==3.1.9
blurhash-python==1.1.1
boto3==1.18.24
botocore==1.21.24
CacheControl==0.12.6
cachetools==4.2.2
celery==5.1.2
celery-prometheus-exporter==1.7.0
certifi==2020.12.5
cffi==1.14.6
chardet==4.0.0
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
cryptography==3.4.7
defusedxml==0.7.1
Deprecated==1.2.12
diff-match-patch==20200713
Django==3.2.6
django-admin-tools==0.9.1
django-admintool-command==0.1.1
django-celery-beat==2.2.1
django-celery-results==2.2.0
django-compat==1.0.15
django-extensions==3.1.3
django-import-export==2.5.0
django-ipware==3.0.7
django-oauth-toolkit==1.5.0
django-prometheus==2.1.0
django-shortuuidfield==0.1.3
django-silk==4.1.0
django-storages==1.11.1
django-sympycharfield==0.4.1
django-timezone-field==4.2.1
djangorestframework==3.12.4
drf-recaptcha==2.0.4
ecdsa==0.17.0
esptool==3.1
et-xmlfile==1.1.0
fcm-django==1.0.5
firebase-admin==5.0.2
future==0.18.2
google-api-core==1.31.2
google-api-python-client==2.17.0
google-auth==2.0.0
google-auth-httplib2==0.1.0
google-cloud-core==1.7.2
google-cloud-firestore==2.2.0
google-cloud-storage==1.42.0
google-crc32c==1.1.2
google-resumable-media==1.3.3
googleapis-common-protos==1.53.0
gprof2dot==2021.2.21
grpcio==1.39.0
html5lib==1.1
httplib2==0.19.1
idna==2.10
Jinja2==3.0.1
jmespath==0.10.0
jwcrypto==1.0
kombu==5.1.0
MarkupPy==1.14
MarkupSafe==2.0.1
mpmath==1.2.1
msgpack==1.0.2
oauthlib==3.1.1
odfpy==1.4.1
openpyxl==3.0.7
packaging==21.0
paho-mqtt==1.5.1
phonenumberslite==8.12.30
Pillow==8.3.1
pip-chill==1.0.1
pkg-resources==0.0.0
prometheus-client==0.11.0
prompt-toolkit==3.0.19
proto-plus==1.19.0
protobuf==3.17.3
psycopg2-binary==2.9.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.7.0
pycparser==2.20
pycryptodome==3.10.1
Pygments==2.10.0
pyparsing==2.4.7
PyPDF2==1.26.0
pyserial==3.5
python-bidi==0.4.2
python-crontab==2.5.1
python-dateutil==2.8.2
pytz==2021.1
PyYAML==5.4.1
qrcode==7.2
reedsolo==1.5.4
reportlab==3.6.1
requests==2.25.1
rsa==4.7.2
s3transfer==0.5.0
sentry-sdk==1.3.1
shortuuid==1.0.1
six==1.16.0
sqlparse==0.4.1
sympy==1.8
tablib==3.0.0
termcolor==1.1.0
toml==0.10.2
uritemplate==3.0.1
urllib3==1.26.2
uWSGI==2.0.19.1
vine==5.0.0
wcwidth==0.2.5
webencodings==0.5.1
wrapt==1.12.1
xhtml2pdf==0.2.5
xlrd==2.0.1
xlwt==1.3.0"><pre class="notranslate"><code class="notranslate">amqp==5.0.6
arabic-reshaper==2.1.3
asgiref==3.4.1
autopep8==1.5.7
bcrypt==3.2.0
billiard==3.6.4.0
bitstring==3.1.9
blurhash-python==1.1.1
boto3==1.18.24
botocore==1.21.24
CacheControl==0.12.6
cachetools==4.2.2
celery==5.1.2
celery-prometheus-exporter==1.7.0
certifi==2020.12.5
cffi==1.14.6
chardet==4.0.0
click==7.1.2
click-didyoumean==0.0.3
click-plugins==1.1.1
click-repl==0.2.0
cryptography==3.4.7
defusedxml==0.7.1
Deprecated==1.2.12
diff-match-patch==20200713
Django==3.2.6
django-admin-tools==0.9.1
django-admintool-command==0.1.1
django-celery-beat==2.2.1
django-celery-results==2.2.0
django-compat==1.0.15
django-extensions==3.1.3
django-import-export==2.5.0
django-ipware==3.0.7
django-oauth-toolkit==1.5.0
django-prometheus==2.1.0
django-shortuuidfield==0.1.3
django-silk==4.1.0
django-storages==1.11.1
django-sympycharfield==0.4.1
django-timezone-field==4.2.1
djangorestframework==3.12.4
drf-recaptcha==2.0.4
ecdsa==0.17.0
esptool==3.1
et-xmlfile==1.1.0
fcm-django==1.0.5
firebase-admin==5.0.2
future==0.18.2
google-api-core==1.31.2
google-api-python-client==2.17.0
google-auth==2.0.0
google-auth-httplib2==0.1.0
google-cloud-core==1.7.2
google-cloud-firestore==2.2.0
google-cloud-storage==1.42.0
google-crc32c==1.1.2
google-resumable-media==1.3.3
googleapis-common-protos==1.53.0
gprof2dot==2021.2.21
grpcio==1.39.0
html5lib==1.1
httplib2==0.19.1
idna==2.10
Jinja2==3.0.1
jmespath==0.10.0
jwcrypto==1.0
kombu==5.1.0
MarkupPy==1.14
MarkupSafe==2.0.1
mpmath==1.2.1
msgpack==1.0.2
oauthlib==3.1.1
odfpy==1.4.1
openpyxl==3.0.7
packaging==21.0
paho-mqtt==1.5.1
phonenumberslite==8.12.30
Pillow==8.3.1
pip-chill==1.0.1
pkg-resources==0.0.0
prometheus-client==0.11.0
prompt-toolkit==3.0.19
proto-plus==1.19.0
protobuf==3.17.3
psycopg2-binary==2.9.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.7.0
pycparser==2.20
pycryptodome==3.10.1
Pygments==2.10.0
pyparsing==2.4.7
PyPDF2==1.26.0
pyserial==3.5
python-bidi==0.4.2
python-crontab==2.5.1
python-dateutil==2.8.2
pytz==2021.1
PyYAML==5.4.1
qrcode==7.2
reedsolo==1.5.4
reportlab==3.6.1
requests==2.25.1
rsa==4.7.2
s3transfer==0.5.0
sentry-sdk==1.3.1
shortuuid==1.0.1
six==1.16.0
sqlparse==0.4.1
sympy==1.8
tablib==3.0.0
termcolor==1.1.0
toml==0.10.2
uritemplate==3.0.1
urllib3==1.26.2
uWSGI==2.0.19.1
vine==5.0.0
wcwidth==0.2.5
webencodings==0.5.1
wrapt==1.12.1
xhtml2pdf==0.2.5
xlrd==2.0.1
xlwt==1.3.0
</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">I am using latest version of Celery 4 with RabbitMQ broker.</p>
<p dir="auto">Recently I upgrade my RabbitMQ broker and started getting crashes, with following critical error in log:<br>
<code class="notranslate">Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')</code></p>
<p dir="auto">Doing some google searching lead me to <a href="https://groups.google.com/g/rabbitmq-users/c/X-x3DKVJb8k" rel="nofollow">a thread on RabbitMQ user group</a>, this mentions that "ack timeout" was added from v3.8.15. This is new (unexpected) behaviour with the broker upgrade, earlier with  <code class="notranslate">v3.8.6</code> it did not happen.</p>
<p dir="auto">Now is this known issue and fixed in Celery5 or is it new? I do not see any issues in the list regarding it.</p>
<p dir="auto">Already posted this in <a href="https://groups.google.com/g/celery-users/c/fsSACtCz8U4" rel="nofollow">celery-user group</a>, but no response.</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">Proper ackhnowledgments should be sent before timeout that is causing crash.</p>
<h1 dir="auto">Checklist</h1>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>

<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li>Retry not working properly when using chunks <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="876328567" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6755" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6755/hovercard" href="https://github.com/celery/celery/issues/6755">#6755</a></li>
<li>Pickling Retry instance is incorrect <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="870723267" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/6748" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/6748/hovercard" href="https://github.com/celery/celery/issues/6748">#6748</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>

<p dir="auto"><strong>Celery version</strong>: 5.0.5</p>

<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"><code class="notranslate">
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<p dir="auto">Create a django application with celery with default settings using pyamqp and rabbitmq 3.8.16 (current version) as broker, and run it in docker using docker-compose.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="services:
  rabbit:
    container_name: rabbit
    image: rabbitmq:3.8.16-management
    hostname: myapp-rabbit
  celery:
    command: [
      &quot;bash&quot;, &quot;-c&quot;, &quot;date &amp;&amp; celery -A myapp worker -l debug --scheduler django_celery_beat.schedulers:DatabaseScheduler -B&quot;
    ]
    image: &lt;image based on ubuntu:20.04&gt;
    container_name: celery
    depends_on:
      - rabbit
      - db
   db:
    image: postgres:10"><pre class="notranslate"><code class="notranslate">services:
  rabbit:
    container_name: rabbit
    image: rabbitmq:3.8.16-management
    hostname: myapp-rabbit
  celery:
    command: [
      "bash", "-c", "date &amp;&amp; celery -A myapp worker -l debug --scheduler django_celery_beat.schedulers:DatabaseScheduler -B"
    ]
    image: &lt;image based on ubuntu:20.04&gt;
    container_name: celery
    depends_on:
      - rabbit
      - db
   db:
    image: postgres:10
</code></pre></div>
<p dir="auto">Implement a task that will call retry, and then start everything and set up the scheduler to run the task.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="@app.task(bind=True)
def debug_task(self):
    print(&quot;Running debug task&quot;)
    self.retry(countdown=60 * 60, max_retries=1)"><pre class="notranslate"><code class="notranslate">@app.task(bind=True)
def debug_task(self):
    print("Running debug task")
    self.retry(countdown=60 * 60, max_retries=1)
</code></pre></div>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.8</li>
<li><strong>Minimal Celery Version</strong>: 5.0.5</li>
<li><strong>Minimal Kombu Version</strong>: 5.0.2</li>
<li><strong>Minimal Broker Version</strong>: rabbit 3.8.16</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: ubuntu 20.04</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Django==3.1.5
amqp==5.0.6
celery==5.0.5
django-celery-beat==2.1.0
django-celery-results==2.0.1
kombu==5.0.2&lt;/summary&gt;"><pre class="notranslate"><code class="notranslate">Django==3.1.5
amqp==5.0.6
celery==5.0.5
django-celery-beat==2.1.0
django-celery-results==2.0.1
kombu==5.0.2&lt;/summary&gt;
</code></pre></div>
<p dir="auto"></p>
</summary></details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>

<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=""><pre class="notranslate"></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">Expected the task to retry after one hour</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">After 15 minutes, the entire celery container crashes with the following error</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="07/05/2021 06:51:38 MainProcess    : celery.worker  : CRITICAL Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - consumer ack timed out on channel 1', (0, 0), '')
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.8/dist-packages/celery/worker/worker.py&quot;, line 203, in start
    self.blueprint.start(self)
  File &quot;/usr/local/lib/python3.8/dist-packages/celery/bootsteps.py&quot;, line 116, in start
    step.start(parent)
  File &quot;/usr/local/lib/python3.8/dist-packages/celery/bootsteps.py&quot;, line 365, in start
    return self.obj.start()
  File &quot;/usr/local/lib/python3.8/dist-packages/celery/worker/consumer/consumer.py&quot;, line 311, in start
    blueprint.start(self)
  File &quot;/usr/local/lib/python3.8/dist-packages/celery/bootsteps.py&quot;, line 116, in start
    step.start(parent)
  File &quot;/usr/local/lib/python3.8/dist-packages/celery/worker/consumer/consumer.py&quot;, line 592, in start
    c.loop(*c.loop_args())
  File &quot;/usr/local/lib/python3.8/dist-packages/celery/worker/loops.py&quot;, line 81, in asynloop
    next(loop)
  File &quot;/usr/local/lib/python3.8/dist-packages/kombu/asynchronous/hub.py&quot;, line 361, in create_loop
    cb(*cbargs)
  File &quot;/usr/local/lib/python3.8/dist-packages/kombu/transport/base.py&quot;, line 235, in on_readable
    reader(loop)
  File &quot;/usr/local/lib/python3.8/dist-packages/kombu/transport/base.py&quot;, line 217, in _read
    drain_events(timeout=0)
  File &quot;/usr/local/lib/python3.8/dist-packages/amqp/connection.py&quot;, line 523, in drain_events
    while not self.blocking_read(timeout):
  File &quot;/usr/local/lib/python3.8/dist-packages/amqp/connection.py&quot;, line 529, in blocking_read
    return self.on_inbound_frame(frame)
  File &quot;/usr/local/lib/python3.8/dist-packages/amqp/method_framing.py&quot;, line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File &quot;/usr/local/lib/python3.8/dist-packages/amqp/connection.py&quot;, line 535, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
  File &quot;/usr/local/lib/python3.8/dist-packages/amqp/abstract_channel.py&quot;, line 143, in dispatch_method
    listener(*args)
  File &quot;/usr/local/lib/python3.8/dist-packages/amqp/channel.py&quot;, line 277, in _on_close
    raise error_for_code(
amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - consumer ack timed out on channel 1"><pre class="notranslate"><code class="notranslate">07/05/2021 06:51:38 MainProcess    : celery.worker  : CRITICAL Unrecoverable error: PreconditionFailed(406, 'PRECONDITION_FAILED - consumer ack timed out on channel 1', (0, 0), '')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.8/dist-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.8/dist-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/usr/local/lib/python3.8/dist-packages/celery/worker/consumer/consumer.py", line 311, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.8/dist-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.8/dist-packages/celery/worker/consumer/consumer.py", line 592, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.8/dist-packages/celery/worker/loops.py", line 81, in asynloop
    next(loop)
  File "/usr/local/lib/python3.8/dist-packages/kombu/asynchronous/hub.py", line 361, in create_loop
    cb(*cbargs)
  File "/usr/local/lib/python3.8/dist-packages/kombu/transport/base.py", line 235, in on_readable
    reader(loop)
  File "/usr/local/lib/python3.8/dist-packages/kombu/transport/base.py", line 217, in _read
    drain_events(timeout=0)
  File "/usr/local/lib/python3.8/dist-packages/amqp/connection.py", line 523, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/lib/python3.8/dist-packages/amqp/connection.py", line 529, in blocking_read
    return self.on_inbound_frame(frame)
  File "/usr/local/lib/python3.8/dist-packages/amqp/method_framing.py", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File "/usr/local/lib/python3.8/dist-packages/amqp/connection.py", line 535, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
  File "/usr/local/lib/python3.8/dist-packages/amqp/abstract_channel.py", line 143, in dispatch_method
    listener(*args)
  File "/usr/local/lib/python3.8/dist-packages/amqp/channel.py", line 277, in _on_close
    raise error_for_code(
amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - consumer ack timed out on channel 1
</code></pre></div>
<p dir="auto">Rabbit logs the following:<br>
<code class="notranslate">[warning] &lt;0.10196.1&gt; Consumer None4 on channel 1 has timed out waiting on consumer acknowledgement. Timeout used: 900000 ms</code></p>
<p dir="auto">Note that by downgrading rabbitmq to version 3.8.14 this doesn't happen anymore. (This might be a bug in rabbit instead.)</p>