<h1 dir="auto">Checklist</h1>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> This has already been asked to the <a href="https://groups.google.com/forum/#!forum/celery-users" rel="nofollow">discussion group</a> first.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have read the relevant section in the<br>
<a href="http://docs.celeryproject.org/en/latest/contributing.html#other-bugs" rel="nofollow">contribution guide</a><br>
on reporting bugs.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22">issues list</a><br>
for similar or identical bug reports.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have checked the <a href="https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22">pull requests list</a><br>
for existing proposed fixes.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have checked the <a href="https://github.com/celery/celery/commits/master">commit log</a><br>
to find out if the bug was already fixed in the master branch.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all related issues and possible duplicate issues<br>
in this issue (If there are none, check this box anyway).</li>
</ul>
<h2 dir="auto">Mandatory Debugging Information</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the contents of <code class="notranslate">pip freeze</code> in the issue.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have included all the versions of all the external dependencies required<br>
to reproduce this bug.</li>
</ul>
<h2 dir="auto">Optional Debugging Information</h2>

<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one Python version<br>
and/or implementation.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one message broker and/or<br>
result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one version of the message<br>
broker and/or result backend.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one operating system.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue on more than one workers pool.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue with autoscaling, retries,<br>
ETA/Countdown &amp; rate limits disabled.</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> I have tried reproducing the issue after downgrading<br>
and/or upgrading Celery and its dependencies.</li>
</ul>
<h2 dir="auto">Related Issues and Possible Duplicates</h2>
<h4 dir="auto">Related Issues</h4>
<ul dir="auto">
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="554728891" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/5935" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/5935/hovercard" href="https://github.com/celery/celery/issues/5935">#5935</a></li>
</ul>
<h4 dir="auto">Possible Duplicates</h4>
<ul dir="auto">
<li>None</li>
</ul>
<h2 dir="auto">Environment &amp; Settings</h2>
<p dir="auto"><strong>Celery version</strong>: 4.4.3</p>
<details>
<summary><b><code class="notranslate">celery report</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:4.4.3 (cliffs) kombu:4.6.10 py:3.8.0
            billiard:3.6.4.0 redis:3.5.3
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.3.0-40-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:redis://worker-wm-01.local:6379/

broker_url: 'redis://worker-wm-01.local:6379//'
result_backend: 'redis://worker-wm-01.local:6379/'
include: ['tasks']
broker: 'redis://worker-wm-01.local:6379'
broker_transport_options: {
 'interval_max': 5, 'interval_start': 0, 'interval_step': 0.4, 'max_retries': 5}
broker_pool_limit: None
backend: 'redis://worker-wm-01.local:6379/'
redis_backend_health_check_interval: 120
task_acks_late: True
worker_prefetch_multiplier: 1"><pre class="notranslate"><code class="notranslate">software -&gt; celery:4.4.3 (cliffs) kombu:4.6.10 py:3.8.0
            billiard:3.6.4.0 redis:3.5.3
platform -&gt; system:Linux arch:64bit, ELF
            kernel version:5.3.0-40-generic imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:redis://worker-wm-01.local:6379/

broker_url: 'redis://worker-wm-01.local:6379//'
result_backend: 'redis://worker-wm-01.local:6379/'
include: ['tasks']
broker: 'redis://worker-wm-01.local:6379'
broker_transport_options: {
 'interval_max': 5, 'interval_start': 0, 'interval_step': 0.4, 'max_retries': 5}
broker_pool_limit: None
backend: 'redis://worker-wm-01.local:6379/'
redis_backend_health_check_interval: 120
task_acks_late: True
worker_prefetch_multiplier: 1
</code></pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Steps to Reproduce</h1>
<h2 dir="auto">Required Dependencies</h2>

<ul dir="auto">
<li><strong>Minimal Python Version</strong>: 3.8.0</li>
<li><strong>Minimal Celery Version</strong>: 4.4.3</li>
<li><strong>Minimal Kombu Version</strong>: 4.6.10</li>
<li><strong>Minimal Broker Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Version</strong>: N/A or Unknown</li>
<li><strong>Minimal OS and/or Kernel Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Broker Client Version</strong>: N/A or Unknown</li>
<li><strong>Minimal Result Backend Client Version</strong>: N/A or Unknown</li>
</ul>
<h3 dir="auto">Python Packages</h3>

<details>
<summary><b><code class="notranslate">pip freeze</code> Output:</b></summary>
<p dir="auto">
</p><div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="absl-py==0.12.0         
alembic==1.4.1   
amqp==2.6.1 
ase==3.21.1       
astroid==2.5.4   
attrs==20.3.0    
billiard==3.6.4.0
cachetools==4.2.1 
celery==4.4.3                
certifi==2020.12.5  
chardet==3.0.4
click==7.1.2 
cloudpickle==1.6.0  
coverage==5.5         
cycler==0.10.0      
databricks-cli==0.14.3
dataclasses==0.6        
decorator==4.4.2
docker==5.0.0             
entrypoints==0.3                 
flake8==3.9.1   
Flask==1.1.2
Flask-PyMongo==2.3.0    
future==0.18.2 
gitdb==4.0.7   
GitPython==3.1.14
google-auth==1.29.0
google-auth-oauthlib==0.4.4
googledrivedownloader==0.4
greenlet==1.0.0       
grpcio==1.37.0       
gunicorn==20.1.0    
h11==0.8.1          
h2==3.2.0               
h5py==3.2.1 
hpack==3.0.0 
http3==0.6.7             
hyperframe==5.2.0
idna==2.10  
iniconfig==1.1.1
isodate==0.6.0       
isort==5.8.0            
itsdangerous==1.1.0
Jinja2==2.11.3
joblib==1.0.1       
kiwisolver==1.3.1
kombu==4.6.10  
lazy-object-proxy==1.6.0
llvmlite==0.36.0 
Mako==1.1.4 
Markdown==3.3.4   
MarkupSafe==1.1.1
matplotlib==3.4.1
mccabe==0.6.1  
minio==7.0.3      
mlflow==1.15.0               
networkx==2.5.1     
numba==0.53.1
numpy==1.20.2
oauthlib==3.1.0     
packaging==20.9       
pandas==1.2.4       
pefile==2019.4.18  
Pillow==8.2.0           
pluggy==0.13.1
prometheus-client==0.10.1 
prometheus-flask-exporter==0.18.1
protobuf==3.15.8
py==1.10.0 
pyasn1==0.4.8           
pyasn1-modules==0.2.8
pycodestyle==2.7.0
pyflakes==2.3.1
pylint==2.7.4
pymongo==3.11.3
pyparsing==2.4.7
pytest==6.2.3
python-dateutil==2.8.1
python-dotenv==0.17.0
python-editor==1.0.4
python-louvain==0.15
pytorch-lightning==0.9.0
pytz==2021.1
PyYAML==5.4.1
querystring-parser==1.2.4
rdflib==5.0.0
redis==3.5.3
requests==2.25.1
requests-async==0.6.2
requests-oauthlib==1.3.0
rfc3986==1.4.0
rsa==4.7.2
scikit-learn==0.24.1
scipy==1.6.2
seaborn==0.11.1
six==1.15.0
smart-open==5.0.0
smmap==4.0.0
SQLAlchemy==1.4.11
sqlparse==0.4.1
structlog==21.1.0
tabulate==0.8.9
tensorboard==2.2.0
tensorboard-plugin-wit==1.8.0
threadpoolctl==2.1.0
toml==0.10.2
torch==1.7.0
torch-cluster==1.5.8
torch-geometric==1.6.3
torch-scatter==2.0.5
torch-sparse==0.6.8
torch-spline-conv==1.2.0
tqdm==4.60.0
typing-extensions==3.7.4.3
urllib3==1.26.4
uWSGI==2.0.19.1
vine==1.3.0
websocket-client==0.58.0
Werkzeug==1.0.1
wrapt==1.12.1
"><pre class="notranslate"><code class="notranslate">absl-py==0.12.0         
alembic==1.4.1   
amqp==2.6.1 
ase==3.21.1       
astroid==2.5.4   
attrs==20.3.0    
billiard==3.6.4.0
cachetools==4.2.1 
celery==4.4.3                
certifi==2020.12.5  
chardet==3.0.4
click==7.1.2 
cloudpickle==1.6.0  
coverage==5.5         
cycler==0.10.0      
databricks-cli==0.14.3
dataclasses==0.6        
decorator==4.4.2
docker==5.0.0             
entrypoints==0.3                 
flake8==3.9.1   
Flask==1.1.2
Flask-PyMongo==2.3.0    
future==0.18.2 
gitdb==4.0.7   
GitPython==3.1.14
google-auth==1.29.0
google-auth-oauthlib==0.4.4
googledrivedownloader==0.4
greenlet==1.0.0       
grpcio==1.37.0       
gunicorn==20.1.0    
h11==0.8.1          
h2==3.2.0               
h5py==3.2.1 
hpack==3.0.0 
http3==0.6.7             
hyperframe==5.2.0
idna==2.10  
iniconfig==1.1.1
isodate==0.6.0       
isort==5.8.0            
itsdangerous==1.1.0
Jinja2==2.11.3
joblib==1.0.1       
kiwisolver==1.3.1
kombu==4.6.10  
lazy-object-proxy==1.6.0
llvmlite==0.36.0 
Mako==1.1.4 
Markdown==3.3.4   
MarkupSafe==1.1.1
matplotlib==3.4.1
mccabe==0.6.1  
minio==7.0.3      
mlflow==1.15.0               
networkx==2.5.1     
numba==0.53.1
numpy==1.20.2
oauthlib==3.1.0     
packaging==20.9       
pandas==1.2.4       
pefile==2019.4.18  
Pillow==8.2.0           
pluggy==0.13.1
prometheus-client==0.10.1 
prometheus-flask-exporter==0.18.1
protobuf==3.15.8
py==1.10.0 
pyasn1==0.4.8           
pyasn1-modules==0.2.8
pycodestyle==2.7.0
pyflakes==2.3.1
pylint==2.7.4
pymongo==3.11.3
pyparsing==2.4.7
pytest==6.2.3
python-dateutil==2.8.1
python-dotenv==0.17.0
python-editor==1.0.4
python-louvain==0.15
pytorch-lightning==0.9.0
pytz==2021.1
PyYAML==5.4.1
querystring-parser==1.2.4
rdflib==5.0.0
redis==3.5.3
requests==2.25.1
requests-async==0.6.2
requests-oauthlib==1.3.0
rfc3986==1.4.0
rsa==4.7.2
scikit-learn==0.24.1
scipy==1.6.2
seaborn==0.11.1
six==1.15.0
smart-open==5.0.0
smmap==4.0.0
SQLAlchemy==1.4.11
sqlparse==0.4.1
structlog==21.1.0
tabulate==0.8.9
tensorboard==2.2.0
tensorboard-plugin-wit==1.8.0
threadpoolctl==2.1.0
toml==0.10.2
torch==1.7.0
torch-cluster==1.5.8
torch-geometric==1.6.3
torch-scatter==2.0.5
torch-sparse==0.6.8
torch-spline-conv==1.2.0
tqdm==4.60.0
typing-extensions==3.7.4.3
urllib3==1.26.4
uWSGI==2.0.19.1
vine==1.3.0
websocket-client==0.58.0
Werkzeug==1.0.1
wrapt==1.12.1

</code></pre></div>
<p dir="auto"></p>
</details>
<h3 dir="auto">Other Dependencies</h3>

<details>
<p dir="auto">
N/A
</p>
</details>
<h2 dir="auto">Minimally Reproducible Test Case</h2>
<details>
<p dir="auto">
</p><div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="@shared_task(bind=True)
def process(self, filename):
    run_long_process()  # more than 2 hours
    return filename

celery_app.send_task(&quot;tasks.process&quot;, args=(filename,), queue=queue)
"><pre class="notranslate"><span class="pl-en">@<span class="pl-en">shared_task</span>(<span class="pl-s1">bind</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">process</span>(<span class="pl-s1">self</span>, <span class="pl-s1">filename</span>):
    <span class="pl-en">run_long_process</span>()  <span class="pl-c"># more than 2 hours</span>
    <span class="pl-k">return</span> <span class="pl-s1">filename</span>

<span class="pl-s1">celery_app</span>.<span class="pl-en">send_task</span>(<span class="pl-s">"tasks.process"</span>, <span class="pl-s1">args</span><span class="pl-c1">=</span>(<span class="pl-s1">filename</span>,), <span class="pl-s1">queue</span><span class="pl-c1">=</span><span class="pl-s1">queue</span>)</pre></div>
<p dir="auto"></p>
</details>
<h1 dir="auto">Expected Behavior</h1>
<p dir="auto">Task is executed only once.</p>
<h1 dir="auto">Actual Behavior</h1>
<p dir="auto">the task is keep executing after having got state SUCCESS once:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="&gt;&gt;&gt; from app import celery_app
&gt;&gt;&gt; ar = celery_app.AsyncResult('46906783-4351-4397-8893-dfc69c8421cf')
&gt;&gt;&gt; ar.state
'SUCCESS'"><pre class="notranslate"><code class="notranslate">&gt;&gt;&gt; from app import celery_app
&gt;&gt;&gt; ar = celery_app.AsyncResult('46906783-4351-4397-8893-dfc69c8421cf')
&gt;&gt;&gt; ar.state
'SUCCESS'
</code></pre></div>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2021-07-22 21:03:05,141: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  
[2021-07-22 23:22:02,694: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  
[2021-07-23 01:41:35,151: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  
[2021-07-23 04:02:10,839: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  "><pre class="notranslate"><code class="notranslate">[2021-07-22 21:03:05,141: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  
[2021-07-22 23:22:02,694: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  
[2021-07-23 01:41:35,151: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  
[2021-07-23 04:02:10,839: INFO/MainProcess] Received task: tasks.process[46906783-4351-4397-8893-dfc69c8421cf]  
</code></pre></div>
<p dir="auto">I've noticed that unlike the similar issue, the default value of 1 hour visibility timeout doesn't affect me. Tasks between 1 and 2 hours long are performed only once, but this one takes about 140 minutes and keeps reappearing. I also read that actually visibility timeout doesn't affect anything and this kind of bugs happens mostly with Redis backend. Like in other cases, change to Rabbit is not an option.</p>
<p dir="auto">I would like to discover more about this issue, but i'm not sure where to head at. and whether it is similar to other cases or a new one. and what exactly is the nature of getting tasks to a worker (is it only that <a href="https://github.com/celery/celery/blob/f462a437e3371acb867e94b52c2595b6d0a742d8/celery/worker/request.py#L587">this method</a> must be called for a task to never be picked again or it doesn't matter in some cases)?</p>
<p dir="auto">I am using celery 4.3 + Redis + flower. I have a few long-running jobs with acks_late=True and task_reject_on_worker_lost=True. I am using celery grouping to group jobs run parallelly and append result and use in parent job.</p>
<p dir="auto">In this scenario, my few jobs will run more than an hour, after every one hour the same child jobs are redelivering again to the worker.</p>
<p dir="auto">The sample jobs as below.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="@app.task(queue='q1', bind=True, acks_late=True, task_reject_on_worker_lost=True, max_retries=3)
def job_1(self):
  do_something()
  task_group = group(job_2.s(batch) for batch in range(0, len([1,2,3,4,5,6]), 3))
  result_group = task_group.apply_async()

@app.task(queue='q1', bind=True, acks_late=True, task_reject_on_worker_lost=True, max_retries=3)
def job_2(self, batch):
  do_something()
  return result"><pre class="notranslate"><code class="notranslate">@app.task(queue='q1', bind=True, acks_late=True, task_reject_on_worker_lost=True, max_retries=3)
def job_1(self):
  do_something()
  task_group = group(job_2.s(batch) for batch in range(0, len([1,2,3,4,5,6]), 3))
  result_group = task_group.apply_async()

@app.task(queue='q1', bind=True, acks_late=True, task_reject_on_worker_lost=True, max_retries=3)
def job_2(self, batch):
  do_something()
  return result
</code></pre></div>
<p dir="auto">The above job_2 will run more than hour and after one hour the same is redelivering again to the worker.</p>
<p dir="auto">My celery setup and config as shown below:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="c = Celery(app.import_name,
           backend=app.config['CELERY_RESULT_BACKEND'],
           broker=app.config['CELERY_BROKER_URL'])
config.py

CELERY_BROKER_URL = os.environ['CELERY_BROKER_URL']
CELERY_RESULT_BACKEND = os.environ['CELERY_RESULT_BACKEND']
CELERY_BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 36000}"><pre class="notranslate"><code class="notranslate">c = Celery(app.import_name,
           backend=app.config['CELERY_RESULT_BACKEND'],
           broker=app.config['CELERY_BROKER_URL'])
config.py

CELERY_BROKER_URL = os.environ['CELERY_BROKER_URL']
CELERY_RESULT_BACKEND = os.environ['CELERY_RESULT_BACKEND']
CELERY_BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 36000}
</code></pre></div>
<p dir="auto">I tried to increase visibility timeout to 10 hours after the redelivering issue like above in configuration. but it looks like not working.</p>
<p dir="auto">Please help with this issue and let me know if any solution.</p>