<h2 dir="auto">Checklist</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ celery -A app report

software -&gt; celery:4.0.2 (latentcall) kombu:4.0.2 py:3.4.3
            billiard:3.5.0.2 redis:2.10.5
platform -&gt; system:Darwin arch:64bit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:disabled

broker_url: 'redis://127.0.0.1:6379/3'"><pre class="notranslate"><code class="notranslate">$ celery -A app report

software -&gt; celery:4.0.2 (latentcall) kombu:4.0.2 py:3.4.3
            billiard:3.5.0.2 redis:2.10.5
platform -&gt; system:Darwin arch:64bit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:redis results:disabled

broker_url: 'redis://127.0.0.1:6379/3'
</code></pre></div>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.</li>
</ul>
<h2 dir="auto">Steps to reproduce</h2>
<p dir="auto">Tasks that finish and succeed after the warm shutdown signal are not acked if <code class="notranslate">acks_late=True</code>. To reproduce:</p>
<ol dir="auto">
<li>Create <code class="notranslate">app.py</code>:</li>
</ol>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from time import sleep

from celery import Celery

app = Celery('foo', broker='redis://127.0.0.1:6379/3')

@app.task(acks_late=True)
def foo():
    sleep(5)
    return 'foo'"><pre class="notranslate"><span class="pl-k">from</span> <span class="pl-s1">time</span> <span class="pl-k">import</span> <span class="pl-s1">sleep</span>

<span class="pl-k">from</span> <span class="pl-s1">celery</span> <span class="pl-k">import</span> <span class="pl-v">Celery</span>

<span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-v">Celery</span>(<span class="pl-s">'foo'</span>, <span class="pl-s1">broker</span><span class="pl-c1">=</span><span class="pl-s">'redis://127.0.0.1:6379/3'</span>)

<span class="pl-en">@<span class="pl-s1">app</span>.<span class="pl-en">task</span>(<span class="pl-s1">acks_late</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">foo</span>():
    <span class="pl-en">sleep</span>(<span class="pl-c1">5</span>)
    <span class="pl-k">return</span> <span class="pl-s">'foo'</span></pre></div>
<ol start="2" dir="auto">
<li>Queue up a task:</li>
</ol>
<div class="highlight highlight-source-shell notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="$ python -c &quot;from app import foo; foo.apply_async()&quot;"><pre class="notranslate">$ python -c <span class="pl-s"><span class="pl-pds">"</span>from app import foo; foo.apply_async()<span class="pl-pds">"</span></span></pre></div>
<ol start="3" dir="auto">
<li>Start the worker and issue a warm shutdown (CTRL+C) before the task finishes.</li>
</ol>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ celery -A app worker --loglevel=debug
[2017-01-30 16:23:46,092: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-01-30 16:23:46,094: DEBUG/MainProcess] | Worker: Building graph...
[2017-01-30 16:23:46,094: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2017-01-30 16:23:46,108: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-01-30 16:23:46,109: DEBUG/MainProcess] | Consumer: Building graph...
[2017-01-30 16:23:46,124: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Agent, Tasks, Control, Heart, event loop}



celery@foobar v4.0.2 (latentcall)

Darwin-16.3.0-x86_64-i386-64bit 2017-01-30 16:23:46

[config]
.&gt; app:         foo:0x1034ddc88
.&gt; transport:   redis://127.0.0.1:6379/3
.&gt; results:     disabled://
.&gt; concurrency: 4 (prefork)
.&gt; task events: OFF (enable -E to monitor tasks in this worker)

[queues]
.&gt; celery           exchange=celery(direct) key=celery


[tasks]
  . app.foo
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap

[2017-01-30 16:23:46,139: DEBUG/MainProcess] | Worker: Starting Hub
[2017-01-30 16:23:46,139: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,140: DEBUG/MainProcess] | Worker: Starting Pool
[2017-01-30 16:23:46,248: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,249: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-01-30 16:23:46,250: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-01-30 16:23:46,268: INFO/MainProcess] Connected to redis://127.0.0.1:6379/3
[2017-01-30 16:23:46,268: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,269: DEBUG/MainProcess] | Consumer: Starting Events
[2017-01-30 16:23:46,278: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,278: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-01-30 16:23:46,278: INFO/MainProcess] mingle: searching for neighbors
[2017-01-30 16:23:47,309: INFO/MainProcess] mingle: all alone
[2017-01-30 16:23:47,309: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,309: DEBUG/MainProcess] | Consumer: Starting Gossip
[2017-01-30 16:23:47,316: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,316: DEBUG/MainProcess] | Consumer: Starting Tasks
[2017-01-30 16:23:47,320: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,320: DEBUG/MainProcess] | Consumer: Starting Control
[2017-01-30 16:23:47,325: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,325: DEBUG/MainProcess] | Consumer: Starting Heart
[2017-01-30 16:23:47,329: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,329: DEBUG/MainProcess] | Consumer: Starting event loop
[2017-01-30 16:23:47,329: INFO/MainProcess] celery@foobar ready.
[2017-01-30 16:23:47,330: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2017-01-30 16:23:47,330: DEBUG/MainProcess] basic.qos: prefetch_count-&gt;16
[2017-01-30 16:23:48,093: INFO/MainProcess] Received task: app.foo[a1d83951-0e24-488b-b400-0ddbd4161254]
[2017-01-30 16:23:48,093: DEBUG/MainProcess] TaskPool: Apply &lt;function _fast_trace_task at 0x10357c598&gt; (args:('app.foo', 'a1d83951-0e24-488b-b400-0ddbd4161254', {'kwargsrepr': '{}', 'id': 'a1d83951-0e24-488b-b400-0ddbd4161254', 'task': 'app.foo', 'expires': None, 'reply_to': '6de20f21-2460-3189-b534-29ae63c15d1c', 'lang': 'py', 'group': None, 'argsrepr': '()', 'eta': None, 'retries': 0, 'delivery_info': {'exchange': '', 'redelivered': None, 'routing_key': 'celery', 'priority': 0}, 'root_id': 'a1d83951-0e24-488b-b400-0ddbd4161254', 'correlation_id': 'a1d83951-0e24-488b-b400-0ddbd4161254', 'origin': 'gen52076@foobar', 'timelimit': [None, None], 'parent_id': None}, b'[[], {}, {&quot;chord&quot;: null, &quot;callbacks&quot;: null, &quot;chain&quot;: null, &quot;errbacks&quot;: null}]', 'application/json', 'utf-8') kwargs:{})
[2017-01-30 16:23:48,095: DEBUG/MainProcess] Task accepted: app.foo[a1d83951-0e24-488b-b400-0ddbd4161254] pid:52112
^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)
[2017-01-30 16:23:49,613: DEBUG/MainProcess] | Worker: Closing Hub...
[2017-01-30 16:23:49,613: DEBUG/MainProcess] | Worker: Closing Pool...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Worker: Closing Consumer...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Connection...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Events...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Gossip...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Closing Control...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Closing Heart...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Closing event loop...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Stopping event loop...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Stopping Heart...
[2017-01-30 16:23:49,617: DEBUG/MainProcess] | Consumer: Stopping Control...
[2017-01-30 16:23:49,620: DEBUG/MainProcess] | Consumer: Stopping Tasks...
[2017-01-30 16:23:49,620: DEBUG/MainProcess] Canceling task consumer...
[2017-01-30 16:23:49,621: DEBUG/MainProcess] | Consumer: Stopping Gossip...
[2017-01-30 16:23:49,624: DEBUG/MainProcess] | Consumer: Stopping Mingle...
[2017-01-30 16:23:49,624: DEBUG/MainProcess] | Consumer: Stopping Events...
[2017-01-30 16:23:49,624: DEBUG/MainProcess] | Consumer: Stopping Connection...
[2017-01-30 16:23:49,625: DEBUG/MainProcess] | Worker: Stopping Pool...
[2017-01-30 16:23:53,099: INFO/PoolWorker-2] Task app.foo[a1d83951-0e24-488b-b400-0ddbd4161254] succeeded in 5.002987815998495s: 'foo'
[2017-01-30 16:23:54,103: DEBUG/MainProcess] | Worker: Stopping Hub...
[2017-01-30 16:23:54,103: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] Canceling task consumer...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] Closing consumer channel...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2017-01-30 16:23:54,105: DEBUG/MainProcess] | Consumer: Shutdown Connection...
[2017-01-30 16:23:54,106: WARNING/MainProcess] Restoring 1 unacknowledged message(s)
[2017-01-30 16:23:54,111: DEBUG/MainProcess] removing tasks from inqueue until task handler finished"><pre class="notranslate"><code class="notranslate">$ celery -A app worker --loglevel=debug
[2017-01-30 16:23:46,092: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-01-30 16:23:46,094: DEBUG/MainProcess] | Worker: Building graph...
[2017-01-30 16:23:46,094: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Pool, Autoscaler, StateDB, Beat, Consumer}
[2017-01-30 16:23:46,108: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-01-30 16:23:46,109: DEBUG/MainProcess] | Consumer: Building graph...
[2017-01-30 16:23:46,124: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Gossip, Agent, Tasks, Control, Heart, event loop}



celery@foobar v4.0.2 (latentcall)

Darwin-16.3.0-x86_64-i386-64bit 2017-01-30 16:23:46

[config]
.&gt; app:         foo:0x1034ddc88
.&gt; transport:   redis://127.0.0.1:6379/3
.&gt; results:     disabled://
.&gt; concurrency: 4 (prefork)
.&gt; task events: OFF (enable -E to monitor tasks in this worker)

[queues]
.&gt; celery           exchange=celery(direct) key=celery


[tasks]
  . app.foo
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap

[2017-01-30 16:23:46,139: DEBUG/MainProcess] | Worker: Starting Hub
[2017-01-30 16:23:46,139: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,140: DEBUG/MainProcess] | Worker: Starting Pool
[2017-01-30 16:23:46,248: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,249: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-01-30 16:23:46,250: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-01-30 16:23:46,268: INFO/MainProcess] Connected to redis://127.0.0.1:6379/3
[2017-01-30 16:23:46,268: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,269: DEBUG/MainProcess] | Consumer: Starting Events
[2017-01-30 16:23:46,278: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:46,278: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-01-30 16:23:46,278: INFO/MainProcess] mingle: searching for neighbors
[2017-01-30 16:23:47,309: INFO/MainProcess] mingle: all alone
[2017-01-30 16:23:47,309: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,309: DEBUG/MainProcess] | Consumer: Starting Gossip
[2017-01-30 16:23:47,316: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,316: DEBUG/MainProcess] | Consumer: Starting Tasks
[2017-01-30 16:23:47,320: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,320: DEBUG/MainProcess] | Consumer: Starting Control
[2017-01-30 16:23:47,325: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,325: DEBUG/MainProcess] | Consumer: Starting Heart
[2017-01-30 16:23:47,329: DEBUG/MainProcess] ^-- substep ok
[2017-01-30 16:23:47,329: DEBUG/MainProcess] | Consumer: Starting event loop
[2017-01-30 16:23:47,329: INFO/MainProcess] celery@foobar ready.
[2017-01-30 16:23:47,330: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2017-01-30 16:23:47,330: DEBUG/MainProcess] basic.qos: prefetch_count-&gt;16
[2017-01-30 16:23:48,093: INFO/MainProcess] Received task: app.foo[a1d83951-0e24-488b-b400-0ddbd4161254]
[2017-01-30 16:23:48,093: DEBUG/MainProcess] TaskPool: Apply &lt;function _fast_trace_task at 0x10357c598&gt; (args:('app.foo', 'a1d83951-0e24-488b-b400-0ddbd4161254', {'kwargsrepr': '{}', 'id': 'a1d83951-0e24-488b-b400-0ddbd4161254', 'task': 'app.foo', 'expires': None, 'reply_to': '6de20f21-2460-3189-b534-29ae63c15d1c', 'lang': 'py', 'group': None, 'argsrepr': '()', 'eta': None, 'retries': 0, 'delivery_info': {'exchange': '', 'redelivered': None, 'routing_key': 'celery', 'priority': 0}, 'root_id': 'a1d83951-0e24-488b-b400-0ddbd4161254', 'correlation_id': 'a1d83951-0e24-488b-b400-0ddbd4161254', 'origin': 'gen52076@foobar', 'timelimit': [None, None], 'parent_id': None}, b'[[], {}, {"chord": null, "callbacks": null, "chain": null, "errbacks": null}]', 'application/json', 'utf-8') kwargs:{})
[2017-01-30 16:23:48,095: DEBUG/MainProcess] Task accepted: app.foo[a1d83951-0e24-488b-b400-0ddbd4161254] pid:52112
^C
worker: Hitting Ctrl+C again will terminate all running tasks!

worker: Warm shutdown (MainProcess)
[2017-01-30 16:23:49,613: DEBUG/MainProcess] | Worker: Closing Hub...
[2017-01-30 16:23:49,613: DEBUG/MainProcess] | Worker: Closing Pool...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Worker: Closing Consumer...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Connection...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Events...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Mingle...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Gossip...
[2017-01-30 16:23:49,614: DEBUG/MainProcess] | Consumer: Closing Tasks...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Closing Control...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Closing Heart...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Closing event loop...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Stopping event loop...
[2017-01-30 16:23:49,615: DEBUG/MainProcess] | Consumer: Stopping Heart...
[2017-01-30 16:23:49,617: DEBUG/MainProcess] | Consumer: Stopping Control...
[2017-01-30 16:23:49,620: DEBUG/MainProcess] | Consumer: Stopping Tasks...
[2017-01-30 16:23:49,620: DEBUG/MainProcess] Canceling task consumer...
[2017-01-30 16:23:49,621: DEBUG/MainProcess] | Consumer: Stopping Gossip...
[2017-01-30 16:23:49,624: DEBUG/MainProcess] | Consumer: Stopping Mingle...
[2017-01-30 16:23:49,624: DEBUG/MainProcess] | Consumer: Stopping Events...
[2017-01-30 16:23:49,624: DEBUG/MainProcess] | Consumer: Stopping Connection...
[2017-01-30 16:23:49,625: DEBUG/MainProcess] | Worker: Stopping Pool...
[2017-01-30 16:23:53,099: INFO/PoolWorker-2] Task app.foo[a1d83951-0e24-488b-b400-0ddbd4161254] succeeded in 5.002987815998495s: 'foo'
[2017-01-30 16:23:54,103: DEBUG/MainProcess] | Worker: Stopping Hub...
[2017-01-30 16:23:54,103: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] Canceling task consumer...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] Closing consumer channel...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2017-01-30 16:23:54,104: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2017-01-30 16:23:54,105: DEBUG/MainProcess] | Consumer: Shutdown Connection...
[2017-01-30 16:23:54,106: WARNING/MainProcess] Restoring 1 unacknowledged message(s)
[2017-01-30 16:23:54,111: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
</code></pre></div>
<p dir="auto">(Note the second to last line even after the log says the task succeded.)</p>
<h2 dir="auto">Expected behavior</h2>
<p dir="auto">The task is acked before exit.</p>
<h2 dir="auto">Actual behavior</h2>
<p dir="auto">The task is not acked and restored. The next time the worker starts up, the same task will be executed again.</p>
<p dir="auto">I want to use celery in my project and I need that the celery tasks (subprocesses) will run in a diffrent python virtualenv.<br>
I noticed that there is an <code class="notranslate">--executable</code> flag but it will change the executable for all celery processes (main and tasks).<br>
If this feature is not exeists I'm willing to put some work to add this but need some guidance on where to begin with.</p>