<p dir="auto">Hello,</p>
<p dir="auto">I was investigating strange error which we have for one of our tasks. We received regular errors about billard losing workers:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Task (...)[5832c7bd-fe9d-45cd-865e-b6beb4c09e47] raised unexpected: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)"><pre class="notranslate"><code class="notranslate">Task (...)[5832c7bd-fe9d-45cd-865e-b6beb4c09e47] raised unexpected: WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
</code></pre></div>
<p dir="auto">I was searching for some other processes which could killed our workers but system logs was empty and when I was going to give up we turned off autoscale due to some other issues and these errors dissapered completely.</p>
<p dir="auto">Celery 3.1.18<br>
Django 1.7.8<br>
librabbitmq 1.6.1</p>
<p dir="auto">I get the following traceback from my worker and then the worker stops consuming tasks. I have confirmed that there are several thousand messages in the RabbitMQ celery queue, the worker is not consuming them.</p>
<p dir="auto">I started my worker with the following command:</p>
<p dir="auto"><code class="notranslate">celery worker -A project -l info -n worker1.%h -Ofair --autoscale=50,1 --maxtasksperchild=100  --beat</code></p>
<p dir="auto">I am using the <code class="notranslate">-Ofair</code> flag because I was also experiencing another breaking bug (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="26829016" data-permission-text="Title is private" data-url="https://github.com/celery/celery/issues/1845" data-hovercard-type="issue" data-hovercard-url="/celery/celery/issues/1845/hovercard" href="https://github.com/celery/celery/issues/1845">#1845</a>) and found <code class="notranslate">-Ofair</code> to be a workaround.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="2015-02-02 23:08:08,442 - ERROR - 15875:MainProcess - Unrecoverable error: AssertionError()
Traceback (most recent call last):
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/__init__.py&quot;, line 206, in start
    self.blueprint.start(self)
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/bootsteps.py&quot;, line 123, in start
    step.start(parent)
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/bootsteps.py&quot;, line 374, in start
    return self.obj.start()
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/consumer.py&quot;, line 278, in start
    blueprint.start(self)
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/bootsteps.py&quot;, line 123, in start
    step.start(parent)
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/consumer.py&quot;, line 821, in start
    c.loop(*c.loop_args())
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/loops.py&quot;, line 70, in asynloop
    next(loop)
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/kombu/async/hub.py&quot;, line 274, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/kombu/async/hub.py&quot;, line 136, in fire_timers
    entry()
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/kombu/async/timer.py&quot;, line 64, in __call__
    return self.fun(*self.args, **self.kwargs)
  File &quot;/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/concurrency/asynpool.py&quot;, line 521, in verify_process_alive
    assert proc.outqR_fd in hub.readers
AssertionError"><pre class="notranslate"><code class="notranslate">2015-02-02 23:08:08,442 - ERROR - 15875:MainProcess - Unrecoverable error: AssertionError()
Traceback (most recent call last):
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/__init__.py", line 206, in start
    self.blueprint.start(self)
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/bootsteps.py", line 374, in start
    return self.obj.start()
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 278, in start
    blueprint.start(self)
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/bootsteps.py", line 123, in start
    step.start(parent)
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/consumer.py", line 821, in start
    c.loop(*c.loop_args())
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/worker/loops.py", line 70, in asynloop
    next(loop)
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/kombu/async/hub.py", line 274, in create_loop
    poll_timeout = fire_timers(propagate=propagate) if scheduled else 1
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/kombu/async/hub.py", line 136, in fire_timers
    entry()
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/kombu/async/timer.py", line 64, in __call__
    return self.fun(*self.args, **self.kwargs)
  File "/home/djacobs/.virtualenvs/workflow/local/lib/python2.7/site-packages/celery/concurrency/asynpool.py", line 521, in verify_process_alive
    assert proc.outqR_fd in hub.readers
AssertionError
</code></pre></div>