<p dir="auto">I have a host with a local-only (everything on the same host) Celery setup for some continuous data processing. Recently the load got much higher, all the CPU cores often are under 100% load and I've started to get <code class="notranslate">WorkerLostError: Worker exited prematurely: exitcode 155</code> out of the blue multiple times per day. A worker works and then suddenly</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ERROR/MainProcess] Task some_task[some_id] raised unexpected: WorkerLostError('Worker exited prematurely: exitcode 155.',)
Traceback (most recent call last):
  File &quot;...venv/local/lib/python2.7/site-packages/billiard/pool.py&quot;, line 1175, in mark_as_worker_lost
    human_status(exitcode)),
WorkerLostError: Worker exited prematurely: exitcode 155."><pre class="notranslate"><code class="notranslate">ERROR/MainProcess] Task some_task[some_id] raised unexpected: WorkerLostError('Worker exited prematurely: exitcode 155.',)
Traceback (most recent call last):
  File "...venv/local/lib/python2.7/site-packages/billiard/pool.py", line 1175, in mark_as_worker_lost
    human_status(exitcode)),
WorkerLostError: Worker exited prematurely: exitcode 155.
</code></pre></div>
<p dir="auto">I had Celery 3.1.19 but upgrading to the latest version changed nothing. Right now the versions are:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="celery==3.1.20
amqp==1.4.9
billiard==3.3.0.22
kombu==3.0.33"><pre class="notranslate"><code class="notranslate">celery==3.1.20
amqp==1.4.9
billiard==3.3.0.22
kombu==3.0.33
</code></pre></div>
<p dir="auto">I use Redis as the broker (tuned to save each message on the disk), and maybe these Celery settings will tell you something:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="CELERY_ACKS_LATE = True
CELERYD_MAX_TASKS_PER_CHILD = 1
CELERYD_PREFETCH_MULTIPLIER = 1
#-Ofair
#CELERYD_CONCURRENCY is &gt; 1
BROKER_CONNECTION_TIMEOUT = 4
BROKER_CONNECTION_MAX_RETRIES = 3
BROKER_TRANSPORT_OPTIONS = {
    'fanout_prefix': True,
    'fanout_patterns': True,
    'visibility_timeout': CELERYD_TASK_TIME_LIMIT,
}"><pre class="notranslate"><code class="notranslate">CELERY_ACKS_LATE = True
CELERYD_MAX_TASKS_PER_CHILD = 1
CELERYD_PREFETCH_MULTIPLIER = 1
#-Ofair
#CELERYD_CONCURRENCY is &gt; 1
BROKER_CONNECTION_TIMEOUT = 4
BROKER_CONNECTION_MAX_RETRIES = 3
BROKER_TRANSPORT_OPTIONS = {
    'fanout_prefix': True,
    'fanout_patterns': True,
    'visibility_timeout': CELERYD_TASK_TIME_LIMIT,
}
</code></pre></div>
<p dir="auto">Under high load we encountered duplicate job execution, as well as <code class="notranslate">WorkerLostError: Worker exited prematurely: exitcode 155</code> exceptions.</p>
<p dir="auto">We use <code class="notranslate">--max-memory-per-child</code> which leads to workers recycling (exit code 155) between jobs, and sometime the last job seems to be re-executed, as if the MainProcess didn't get the job result from the WorkerProcess.</p>
<h2 dir="auto">Checklist</h2>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have included the output of <code class="notranslate">celery -A proj report</code> in the issue.<br>
(if you are not able to do this, then at least specify the Celery<br>
version affected).</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="root@fe29479f29e2:/app# celery -A tasks report

software -&gt; celery:4.2.0 (windowlicker) kombu:4.2.1 py:3.6.6
            billiard:3.5.0.4 py-amqp:2.3.2
platform -&gt; system:Linux arch:64bit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:db+postgres://postgres:**@db:5432/postgres"><pre class="notranslate"><code class="notranslate">root@fe29479f29e2:/app# celery -A tasks report

software -&gt; celery:4.2.0 (windowlicker) kombu:4.2.1 py:3.6.6
            billiard:3.5.0.4 py-amqp:2.3.2
platform -&gt; system:Linux arch:64bit imp:CPython
loader   -&gt; celery.loaders.app.AppLoader
settings -&gt; transport:amqp results:db+postgres://postgres:**@db:5432/postgres
</code></pre></div>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> I have verified that the issue exists against the <code class="notranslate">master</code> branch of Celery.<br>
Using current master HEAD <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/celery/commit/21baef53c39bc1909fd6eee9a2a20e6ce851e88c/hovercard" href="https://github.com/celery/celery/commit/21baef53c39bc1909fd6eee9a2a20e6ce851e88c"><tt>21baef5</tt></a>:</li>
</ul>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="pip install 'git+https://github.com/celery/celery.git@21baef5#egg=celery [sqlalchemy]'"><pre class="notranslate"><code class="notranslate">pip install 'git+https://github.com/celery/celery.git@21baef5#egg=celery [sqlalchemy]'
</code></pre></div>
<h2 dir="auto">Steps to reproduce</h2>
<h3 dir="auto">Global scenario</h3>
<ul dir="auto">
<li>with <code class="notranslate">rabbitmq</code> as broker, <code class="notranslate">postgresql</code> as backend using <code class="notranslate">sqlalchemy</code></li>
<li>start <code class="notranslate">celery</code> with:
<ul dir="auto">
<li>lot's of children worker processes to simulate high load: e.g. 64 workers</li>
<li><code class="notranslate">--max-tasks-per-child=1</code> to simulate workers process high churn</li>
</ul>
</li>
<li>send lots of tasks</li>
<li>watch <code class="notranslate">celery</code> output for duplicate job execution:
<ul dir="auto">
<li><code class="notranslate">[2018-10-17 09:46:15,694: INFO/MainProcess] Received task: work[fc23b1f5-a363-4c22-a1e3-28052811c237]  </code></li>
<li><code class="notranslate">[2018-10-17 09:46:15,695: WARNING/ForkPoolWorker-5526] do_work:5570</code></li>
<li>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2018-10-17 09:46:12,491: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.',)
Traceback (most recent call last):
  File &quot;/usr/local/lib/python3.6/dist-packages/billiard/pool.py&quot;, line 1223, in mark_as_worker_lost
    human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 155."><pre class="notranslate"><code class="notranslate">[2018-10-17 09:46:12,491: ERROR/MainProcess] Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.',)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/billiard/pool.py", line 1223, in mark_as_worker_lost
    human_status(exitcode)),
billiard.exceptions.WorkerLostError: Worker exited prematurely: exitcode 155.
</code></pre></div>
</li>
</ul>
</li>
</ul>
<h3 dir="auto">Scripted scenario</h3>
<p dir="auto">I made a reproducible scenario using docker and docker-compose:<br>
<a href="https://gist.github.com/thomas-riccardi/6295ca2b39e16619141d349e7b189d75">https://gist.github.com/thomas-riccardi/6295ca2b39e16619141d349e7b189d75</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="# get scenario
git clone https://gist.github.com/6295ca2b39e16619141d349e7b189d75.git celery-duplicate-jobs
cd celery-duplicate-jobs

# start everything
docker-compose up --build; docker-compose down

# in a second terminal, in same directory: watch `celery` output
while sleep 1; do
  echo &quot;&quot;
  log=$(docker-compose logs worker)
  echo &quot;MainProcess receved task count uniqueness:&quot;
  echo &quot;${log}&quot; | grep &quot;Received task:&quot; | awk '{print $8}' | sort | uniq -c | sort -g | awk '{print $1}' | uniq -c
  echo -n &quot;WorkerLostError: &quot;
  echo &quot;${log}&quot; | grep &quot;raised error: WorkerLostError&quot; | wc -l
  echo &quot;Job execution uniqueness (job count per job execution count):&quot;
  echo &quot;${log}&quot; | grep do_work: | awk '{print $6}' | sort | uniq -c | sort -g | awk '{print $1}' | uniq -c
done

# to stop everything: just Ctrl-C on `docker-compose` and the bash script"><pre class="notranslate"><code class="notranslate"># get scenario
git clone https://gist.github.com/6295ca2b39e16619141d349e7b189d75.git celery-duplicate-jobs
cd celery-duplicate-jobs

# start everything
docker-compose up --build; docker-compose down

# in a second terminal, in same directory: watch `celery` output
while sleep 1; do
  echo ""
  log=$(docker-compose logs worker)
  echo "MainProcess receved task count uniqueness:"
  echo "${log}" | grep "Received task:" | awk '{print $8}' | sort | uniq -c | sort -g | awk '{print $1}' | uniq -c
  echo -n "WorkerLostError: "
  echo "${log}" | grep "raised error: WorkerLostError" | wc -l
  echo "Job execution uniqueness (job count per job execution count):"
  echo "${log}" | grep do_work: | awk '{print $6}' | sort | uniq -c | sort -g | awk '{print $1}' | uniq -c
done

# to stop everything: just Ctrl-C on `docker-compose` and the bash script
</code></pre></div>
<h2 dir="auto">Expected behavior</h2>
<ul dir="auto">
<li>No (<code class="notranslate">WorkerLostError</code>) error logger.</li>
<li>Jobs are executed exactly once.</li>
</ul>
<h3 dir="auto">Scripted scenario</h3>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="MainProcess received task count uniqueness:
   1649 1
WorkerLostError: 0
Job execution uniqueness (job count per job execution count):
   1534 1"><pre class="notranslate"><code class="notranslate">MainProcess received task count uniqueness:
   1649 1
WorkerLostError: 0
Job execution uniqueness (job count per job execution count):
   1534 1
</code></pre></div>
<ul dir="auto">
<li>The MainProcess received 1649 tasks, all unique.</li>
<li>The <code class="notranslate">WorkerLostError</code> exception is never logged.</li>
<li>There is no job that was executed multiple times.</li>
</ul>
<h2 dir="auto">Actual behavior</h2>
<ul dir="auto">
<li>Many <code class="notranslate">WorkerLostError</code> exceptions are logged as error.</li>
<li>Some jobs are executed multiple times.</li>
</ul>
<h3 dir="auto">Scripted scenario</h3>
<p dir="auto">After some time (~1min on my machine) we start to get <code class="notranslate">WorkerLostError</code> exceptions and jobs executed multiple times:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="MainProcess received task count uniqueness:
   1649 1
WorkerLostError: 125
Job execution uniqueness (job count per job execution count):
   1523 1
     11 2"><pre class="notranslate"><code class="notranslate">MainProcess received task count uniqueness:
   1649 1
WorkerLostError: 125
Job execution uniqueness (job count per job execution count):
   1523 1
     11 2
</code></pre></div>
<ul dir="auto">
<li>The MainProcess received 1649 tasks, all unique (so it's not a broker-level issue).</li>
<li>The WorkerLostError exception is logged 125 times.</li>
<li>There are 11 jobs that were executed 2 times.</li>
</ul>
<h2 dir="auto">More tests</h2>
<h3 dir="auto">No backend</h3>
<p dir="auto">When disabling the celery backend I could not reproduce the issues:</p>
<ul dir="auto">
<li>no <code class="notranslate">WorkerLostError</code> exception</li>
<li>no duplicated job execution</li>
</ul>
<h3 dir="auto">Redis backend</h3>
<p dir="auto">With <code class="notranslate">redis</code> as backend instead of <code class="notranslate">postgresql</code>, I quickly have a burst of <code class="notranslate">WorkerLostError</code> but it never increases afterward, and no duplicate job execution:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="MainProcess receved task count uniqueness:
    223 1
WorkerLostError: 0
Job execution uniqueness (job count per job execution count):
     64 1

MainProcess receved task count uniqueness:
    278 1
WorkerLostError: 10
Job execution uniqueness (job count per job execution count):
    128 1

...

MainProcess receved task count uniqueness:
  10000 1
WorkerLostError: 10
Job execution uniqueness (job count per job execution count):
  10000 1"><pre class="notranslate"><code class="notranslate">MainProcess receved task count uniqueness:
    223 1
WorkerLostError: 0
Job execution uniqueness (job count per job execution count):
     64 1

MainProcess receved task count uniqueness:
    278 1
WorkerLostError: 10
Job execution uniqueness (job count per job execution count):
    128 1

...

MainProcess receved task count uniqueness:
  10000 1
WorkerLostError: 10
Job execution uniqueness (job count per job execution count):
  10000 1
</code></pre></div>
<p dir="auto">I believe the issue is also there, but the race condition isn't triggered somehow (is redis too fast?).</p>
<h3 dir="auto">PostgreSQL backend, Redis broker</h3>
<p dir="auto">With <code class="notranslate">redis</code> as broker instead of <code class="notranslate">rabbitmq</code> (and still <code class="notranslate">postgresql</code> as backend) I could not reproduce the issues either:</p>
<ul dir="auto">
<li>no <code class="notranslate">WorkerLostError</code> exception</li>
<li>no duplicated job execution</li>
</ul>
<p dir="auto">Same conclusion as before.</p>
<h2 dir="auto">More analysis</h2>
<p dir="auto">I tried to analyze the issue by adding lots of <code class="notranslate">debug</code> log in <code class="notranslate">billiard/pool.py</code> and <code class="notranslate">celery/concurrency/asynpool.py</code>.<br>
What I understood happens:<br>
...</p>
<ol dir="auto">
<li><code class="notranslate">MainProcess</code> receives a job from the broker</li>
<li><code class="notranslate">MainProcess</code> sends the job to one of its <code class="notranslate">Worker</code></li>
<li><code class="notranslate">Worker</code> receives the job</li>
<li><code class="notranslate">Worker</code> <code class="notranslate">put</code>s <code class="notranslate">ACK</code> for the job in its out queue</li>
<li><code class="notranslate">Worker</code> executes the job and finishes it</li>
<li><code class="notranslate">Worker</code> <code class="notranslate">put</code>s the job result it in its out queue</li>
<li><code class="notranslate">Worker</code> finds it must stop: (max memory, max task..): it exits with code 155</li>
<li><code class="notranslate">MainProcess</code> sees its child <code class="notranslate">Worker</code> process die with code 155</li>
<li><code class="notranslate">MainProcess</code> checks pending job passed to this <code class="notranslate">Worker</code>: seems to re-queue them on another <code class="notranslate">Worker</code> =&gt; duplicate job execution</li>
<li><code class="notranslate">MainProcess</code> logs exception: <code class="notranslate">2018-10-12 14:08:35,781 celery.worker.request ERROR    Task handler raised error: WorkerLostError('Worker exited prematurely: exitcode 155.',)</code></li>
<li>another <code class="notranslate">Worker</code> gets the job</li>
</ol>
<p dir="auto">...</p>
<p dir="auto">What I understand should happen:<br>
Different behavior starting at step <code class="notranslate">9.</code>:<br>
9. <code class="notranslate">MainProcess</code> should handle all remaining messages in <code class="notranslate">Worker</code> out queue: mark job as finished<br>
10. <code class="notranslate">MainProcess</code> doesn't logs exception as there should be no job currently handled by the <code class="notranslate">Worker</code><br>
11. <code class="notranslate">MainProcess</code> acknowledges the job to the broker/backend</p>
<h3 dir="auto">Debug logs</h3>
<h4 dir="auto"><code class="notranslate">billiard.ResultHandler on_state_change</code></h4>
<p dir="auto"><a href="https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L812-L814">https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L812-L814</a><br>
Logging here reveals that the <code class="notranslate">MainProcess</code> never processes the job response (state <code class="notranslate">READY = 1</code>), and most of the time not even the <code class="notranslate">ACK</code> (state <code class="notranslate">ACK = 0</code>). This explains why the job is re-executed by another worker.<br>
(During my tests I saw the <code class="notranslate">ACK</code> always processed, but it was with a modified celery/billiard, see later hacks.)</p>
<h3 dir="auto">Strange things in celery/billiard code/tried some fix/hacks</h3>
<h4 dir="auto"><code class="notranslate">billiard.Pool._join_exited_workers</code>, <code class="notranslate">LOST_WORKER_TIMEOUT</code></h4>
<p dir="auto">In billiard there seems to be a feature to wait some time after the <code class="notranslate">MainProcess</code> sees a child Worker process die, to have some time to process the remaining messages from it.<br>
It's controlled by <code class="notranslate">LOST_WORKER_TIMEOUT</code>.</p>
<p dir="auto"><code class="notranslate">billiard.Pool</code>:</p>
<ul dir="auto">
<li>when a worker dies, we tag each of its jobs with current timestamp with <code class="notranslate">on_job_process_lost</code>
<ul dir="auto">
<li><a href="https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1185-L1187">https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1185-L1187</a></li>
<li><a href="https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1216-L1217">https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1216-L1217</a></li>
</ul>
</li>
<li>then later we check for such tagged jobs if timeout was reached, in which case we really fail these jobs (with <code class="notranslate">mark_as_worker_lost</code>)<br>
<a href="https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1131-L1140">https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1131-L1140</a></li>
</ul>
<p dir="auto">Celery <code class="notranslate">AsynPool</code> child class:</p>
<ul dir="auto">
<li>it overrides <code class="notranslate">on_job_process_lost</code> to directly call <code class="notranslate">mark_as_worker_lost</code>:<br>
<a href="https://github.com/celery/celery/blame/v4.2.1/celery/concurrency/asynpool.py#L1085-L1092">https://github.com/celery/celery/blame/v4.2.1/celery/concurrency/asynpool.py#L1085-L1092</a></li>
</ul>
<p dir="auto">=&gt; Celery <code class="notranslate">AsynPool</code> child class bypasses the whole <code class="notranslate">LOST_WORKER_TIMEOUT</code> timeout.</p>
<h5 dir="auto">Code timeline</h5>
<ul dir="auto">
<li>
<p dir="auto">2013-05-01 billiard: code refactoring: notably adds <code class="notranslate">on_job_worker_lost</code> hook and <code class="notranslate">mark_as_worker_lost</code>: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/billiard/commit/06bc65e0696acc433c68ff2bb5a64445b17a4c70/hovercard" href="https://github.com/celery/billiard/commit/06bc65e0696acc433c68ff2bb5a64445b17a4c70">celery/billiard@<tt>06bc65e</tt></a></p>
</li>
<li>
<p dir="auto">2013-05-01 celery: adapt code for child class AsynPool: <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/celery/commit/43fda6fa13b466eb377704424f2ed1426c7b75a6/hovercard" href="https://github.com/celery/celery/commit/43fda6fa13b466eb377704424f2ed1426c7b75a6"><tt>43fda6f</tt></a></p>
<ul dir="auto">
<li>by implementing the new <code class="notranslate">on_job_worker_lost</code> hook by directly calling <code class="notranslate">mark_as_worker_lost</code>, it bypasses the whole mechanism</li>
</ul>
</li>
<li>
<p dir="auto">2013-07-26 first <code class="notranslate">WorkerLostError 155</code> bug report: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="17275382" data-permission-text="Title is private" data-url="https://github.com/celery/billiard/issues/67" data-hovercard-type="issue" data-hovercard-url="/celery/billiard/issues/67/hovercard" href="https://github.com/celery/billiard/issues/67">celery/billiard#67</a></p>
</li>
<li>
<p dir="auto">2013-11-28 second bug report: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="23424810" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/1703" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/1703/hovercard" href="https://github.com/celery/celery/issues/1703">#1703</a></p>
</li>
<li>
<p dir="auto">2013-11-28 wrong? fix: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="23464467" data-permission-text="Title is private" data-url="https://github.com/celery/billiard/issues/87" data-hovercard-type="pull_request" data-hovercard-url="/celery/billiard/pull/87/hovercard" href="https://github.com/celery/billiard/pull/87">celery/billiard#87</a></p>
</li>
</ul>
<h5 dir="auto">Questions</h5>
<ul dir="auto">
<li>Is it an issue that this feature was bypassed ?</li>
</ul>
<h5 dir="auto">Hack/Fix</h5>
<h6 dir="auto">Hack 1</h6>
<p dir="auto">I tried to restore the feature by removing the Celery <code class="notranslate">AsynPool</code> <code class="notranslate">on_job_process_lost</code> override: it doesn't fix the issue.</p>
<h6 dir="auto">Hack 2</h6>
<p dir="auto">In fact there seem to be more breakage of this feature:</p>
<ul dir="auto">
<li>the timeout stored in <code class="notranslate">job._lost_worker</code> is reset each time <code class="notranslate">_join_exited_workers</code> is called, instead of being set only the first time we check this job when its worker is dead.</li>
<li>we also lose the <code class="notranslate">exitcode</code> when checking again a job</li>
</ul>
<p dir="auto"><code class="notranslate">on_job_process_lost</code> is called here: <a href="https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1180-L1187">https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L1180-L1187</a></p>
<p dir="auto">I'm not sure how to fix this, either call only once, or make <code class="notranslate">on_job_process_lost</code> idempotent?</p>
<p dir="auto">Result: it did not fix the issue either... We just wait for 10 seconds now before calling <code class="notranslate">mark_as_worker_lost</code>: last out queue messages are still not processed.</p>
<h4 dir="auto"><code class="notranslate">billiard.Worker.__call__</code>, <code class="notranslate">_do_exit</code></h4>
<p dir="auto">Following the previous analysis, I checked the Worker exit code:<br>
<a href="https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L274-L310">https://github.com/celery/billiard/blame/v3.5.0.4/billiard/pool.py#L274-L310</a></p>
<h5 dir="auto">Questions</h5>
<ul dir="auto">
<li>Why do we override <code class="notranslate">sys.exit</code> ?</li>
<li>How come we still end up in <code class="notranslate">_do_exit</code> since we seem to call <code class="notranslate">sys.exit</code> when <code class="notranslate">workloop</code> returns with <code class="notranslate">EX_RECYCLE</code> (the infamous <code class="notranslate">155</code>).</li>
<li>Why do we do extra work for non win32 platform?
<ul dir="auto">
<li>Why <code class="notranslate">put</code> <code class="notranslate">DEATH</code> (<code class="notranslate">4</code>) then <code class="notranslate">sleep 1</code>?
<ul dir="auto">
<li>This comes from <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/celery/billiard/commit/896fc0d344bb18249a472153484a26328b6ebbe6/hovercard" href="https://github.com/celery/billiard/commit/896fc0d344bb18249a472153484a26328b6ebbe6">celery/billiard@<tt>896fc0d</tt></a> with no related issue
<ul dir="auto">
<li>The <code class="notranslate">MainProcess</code> <code class="notranslate">on_death</code> then kills the worker with <code class="notranslate">SIGTERM</code>, this may risk breaking the explicit <code class="notranslate">155</code> recycle exit code mechanism...</li>
</ul>
</li>
<li>This reveals there is clearly a race condition in the system, sleeping won't help</li>
</ul>
</li>
</ul>
</li>
<li>Why <code class="notranslate">os._exit</code> instead of <code class="notranslate">sys.exit</code> ?
<ul dir="auto">
<li><code class="notranslate">os._exit</code> doesn't seem to close/flush the fd/socket/Pipe, it may explain why the <code class="notranslate">MainProcess</code> misses messages.</li>
<li><code class="notranslate">sys.exit</code> may risk closing resources still used by the <code class="notranslate">MainProcess</code>...</li>
</ul>
</li>
</ul>
<h5 dir="auto">Hack/Fix</h5>
<h6 dir="auto">Hack 3</h6>
<p dir="auto">(Cumulative with hack 1 and 2)</p>
<ul dir="auto">
<li>Stopped to override <code class="notranslate">sys.exit</code></li>
<li>Directly called <code class="notranslate">exit</code></li>
<li>In <code class="notranslate">_do_exit</code> called <code class="notranslate">sys.exit</code>
<ul dir="auto">
<li>Even tried explicit flush: <code class="notranslate">self.outq._writer.close()</code></li>
</ul>
</li>
</ul>
<p dir="auto">Result: still some <code class="notranslate">WorkerLostError</code>.</p>
<p dir="auto">I also tried cumulatively to increase the <code class="notranslate">sleep</code>: with <code class="notranslate">sleep(10)</code> it seems to stop reproducing the issues, but it's still an ugly workaround...</p>
<h3 dir="auto">Remaining questions</h3>
<p dir="auto">I don't have much more time to invest in this without help.</p>
<ul dir="auto">
<li>I did not dig into the whole "receiving and handling messages from workers" and how it interacts with the "worker is dead" management code.</li>
<li>Maybe we should try to guarantee working on the last messages from a worker when it dies, instead of just waiting 10s (that was even disabled/broken in 2013)</li>
<li>And/or maybe we should use a proper shutdown sequence, with a <code class="notranslate">DEATH_ACK</code> ? Something like:
<ul dir="auto">
<li><code class="notranslate">Worker</code>: send <code class="notranslate">DEATH</code> at exit, after having sent everything else. Wait for DEATH_ACK</li>
<li><code class="notranslate">MainProcess</code>: receive <code class="notranslate">DEATH</code>, read order is guaranteed sequential (is it?): we are sure we already handled all application messages. Send <code class="notranslate">DEATH_ACK</code></li>
<li><code class="notranslate">Worker</code>: receive <code class="notranslate">DEATH_ACK</code>: <code class="notranslate">sys.exit(exit_code)</code></li>
<li><code class="notranslate">MainProcess</code>: wait for worker process death</li>
<li>maybe some retry? maybe some later force-kill?</li>
</ul>
</li>
</ul>
<h2 dir="auto">Related issues</h2>
<ul dir="auto">
<li>2013-07-26 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="17275382" data-permission-text="Title is private" data-url="https://github.com/celery/billiard/issues/67" data-hovercard-type="issue" data-hovercard-url="/celery/billiard/issues/67/hovercard" href="https://github.com/celery/billiard/issues/67">celery/billiard#67</a></li>
<li>2013-11-28 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="23424810" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/1703" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/1703/hovercard" href="https://github.com/celery/celery/issues/1703">#1703</a></li>
<li>2016-03-02 <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="137939038" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/3085" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/3085/hovercard" href="https://github.com/celery/celery/issues/3085">#3085</a></li>
</ul>