<p dir="auto">Running Celery with MongoDB as broker and results backend we're seeing these errors happening regularly:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="[2012-09-07 21:15:25,643: ERROR/MainProcess] Consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File &quot;/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py&quot;, line 369, in start
    self.consume_messages()
  File &quot;/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py&quot;, line 842, in consume_messages
    self.connection.drain_events(timeout=10.0)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/connection.py&quot;, line 197, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py&quot;, line 743, in drain_events
    item, channel = get(timeout=timeout)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/scheduling.py&quot;, line 42, in get
    return self.fun(resource, **kwargs), resource
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py&quot;, line 763, in _drain_channel
    return channel.drain_events(timeout=timeout)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py&quot;, line 560, in drain_events
    return self._poll(self.cycle, timeout=timeout)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py&quot;, line 290, in _poll
    return cycle.get()
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/scheduling.py&quot;, line 42, in get
    return self.fun(resource, **kwargs), resource
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/mongodb.py&quot;, line 52, in _get
    msg = self._queue_cursors[queue].next()
  File &quot;/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py&quot;, line 703, in next
    if len(self.__data) or self._refresh():
  File &quot;/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py&quot;, line 679, in _refresh
    limit, self.__id))
  File &quot;/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py&quot;, line 628, in __send_message
    self.__tz_aware)
  File &quot;/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/helpers.py&quot;, line 95, in _unpack_response
    cursor_id)
OperationFailure: cursor id '1323433598770136089' not valid at server
[2012-09-07 21:15:25,703: ERROR/MainProcess] Unrecoverable error: KeyError('ip-10-248-69-149.celery.pidbox',)
Traceback (most recent call last):
  File &quot;/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/__init__.py&quot;, line 353, in start
    component.start()
  File &quot;/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py&quot;, line 368, in start
    self.reset_connection()
  File &quot;/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py&quot;, line 712, in reset_connection
    self.reset_pidbox_node()
  File &quot;/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py&quot;, line 656, in reset_pidbox_node
    callback=self.on_control)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/pidbox.py&quot;, line 68, in listen
    callbacks=[callback or self.handle_message])
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/pidbox.py&quot;, line 59, in Consumer
    **options)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/messaging.py&quot;, line 285, in __init__
    self.revive(self.channel)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/messaging.py&quot;, line 297, in revive
    self.declare()
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/messaging.py&quot;, line 307, in declare
    queue.declare()
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/entity.py&quot;, line 387, in declare
    self.queue_declare(nowait, passive=False)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/entity.py&quot;, line 409, in queue_declare
    nowait=nowait)
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py&quot;, line 401, in queue_declare
    return queue, self._size(queue), 0
  File &quot;/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/mongodb.py&quot;, line 73, in _size
    return (self._queue_cursors[queue].count() -
KeyError: 'ip-10-248-69-149.celery.pidbox'"><pre class="notranslate"><code class="notranslate">[2012-09-07 21:15:25,643: ERROR/MainProcess] Consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py", line 369, in start
    self.consume_messages()
  File "/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py", line 842, in consume_messages
    self.connection.drain_events(timeout=10.0)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/connection.py", line 197, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py", line 743, in drain_events
    item, channel = get(timeout=timeout)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/scheduling.py", line 42, in get
    return self.fun(resource, **kwargs), resource
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py", line 763, in _drain_channel
    return channel.drain_events(timeout=timeout)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py", line 560, in drain_events
    return self._poll(self.cycle, timeout=timeout)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py", line 290, in _poll
    return cycle.get()
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/scheduling.py", line 42, in get
    return self.fun(resource, **kwargs), resource
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/mongodb.py", line 52, in _get
    msg = self._queue_cursors[queue].next()
  File "/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 703, in next
    if len(self.__data) or self._refresh():
  File "/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 679, in _refresh
    limit, self.__id))
  File "/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/cursor.py", line 628, in __send_message
    self.__tz_aware)
  File "/mnt/nimble/eggs/pymongo-2.1.1-py2.7-linux-x86_64.egg/pymongo/helpers.py", line 95, in _unpack_response
    cursor_id)
OperationFailure: cursor id '1323433598770136089' not valid at server
[2012-09-07 21:15:25,703: ERROR/MainProcess] Unrecoverable error: KeyError('ip-10-248-69-149.celery.pidbox',)
Traceback (most recent call last):
  File "/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/__init__.py", line 353, in start
    component.start()
  File "/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py", line 368, in start
    self.reset_connection()
  File "/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py", line 712, in reset_connection
    self.reset_pidbox_node()
  File "/mnt/nimble/eggs/celery-3.0.9-py2.7.egg/celery/worker/consumer.py", line 656, in reset_pidbox_node
    callback=self.on_control)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/pidbox.py", line 68, in listen
    callbacks=[callback or self.handle_message])
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/pidbox.py", line 59, in Consumer
    **options)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/messaging.py", line 285, in __init__
    self.revive(self.channel)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/messaging.py", line 297, in revive
    self.declare()
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/messaging.py", line 307, in declare
    queue.declare()
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/entity.py", line 387, in declare
    self.queue_declare(nowait, passive=False)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/entity.py", line 409, in queue_declare
    nowait=nowait)
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/virtual/__init__.py", line 401, in queue_declare
    return queue, self._size(queue), 0
  File "/mnt/nimble/eggs/kombu-2.4.5-py2.7.egg/kombu/transport/mongodb.py", line 73, in _size
    return (self._queue_cursors[queue].count() -
KeyError: 'ip-10-248-69-149.celery.pidbox'
</code></pre></div>
<p dir="auto">That's followed by Celery exiting with error code 0. MongoDB database is sharded cluster and Celery connects to it via mongos instance run on localhost. The error itself looks similar to timeouted cursor, but I believe that timeouts are not applicable to tailable cursors.</p>
<p dir="auto">Celery report for our machine is here:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="software -&gt; celery:3.0.9 (Chiastic Slide) kombu:2.4.5 py:2.7.3
            billiard:2.7.3.12 pymongo:2.1.1
platform -&gt; system:Linux arch:64bit, ELF imp:CPython
loader   -&gt; celery.loaders.default.Loader
settings -&gt; transport:mongodb results:mongodb"><pre class="notranslate"><code class="notranslate">software -&gt; celery:3.0.9 (Chiastic Slide) kombu:2.4.5 py:2.7.3
            billiard:2.7.3.12 pymongo:2.1.1
platform -&gt; system:Linux arch:64bit, ELF imp:CPython
loader   -&gt; celery.loaders.default.Loader
settings -&gt; transport:mongodb results:mongodb
</code></pre></div>
<p dir="auto">We've started observing this problem in our production systems  following major increase in traffic on the mongo backend.</p>
<p dir="auto">I am cross posting this issue with kombu (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="5974033" data-permission-text="Title is private" data-url="https://github.com/celery/kombu/issues/150" data-hovercard-type="issue" data-hovercard-url="/celery/kombu/issues/150/hovercard" href="https://github.com/celery/kombu/issues/150">celery/kombu#150</a>), as it's not clear who is misbehaving.</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="[2012-08-01 00:30:27,163: ERROR/MainProcess] Unrecoverable error: KeyError('awfe1.example.com.celeryd.pidbox',)
Traceback (most recent call last):
  File &quot;/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/__init__.py&quot;, line 268, in start
    component.start()
  File &quot;/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/consumer.py&quot;, line 309, in start
    self.reset_connection()
  File &quot;/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/consumer.py&quot;, line 592, in reset_connection
    self.reset_pidbox_node()
  File &quot;/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/consumer.py&quot;, line 531, in reset_pidbox_node
    callback=self.on_control)
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/pidbox.py&quot;, line 65, in listen
    callbacks=[callback or self.handle_message])
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/pidbox.py&quot;, line 56, in Consumer
    **options)
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/messaging.py&quot;, line 285, in __init__
    self.declare()
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/messaging.py&quot;, line 295, in declare
    queue.declare()
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/entity.py&quot;, line 388, in declare
    self.queue_declare(nowait, passive=False)
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/entity.py&quot;, line 408, in queue_declare
    nowait=nowait)
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/transport/virtual/__init__.py&quot;, line 380, in queue_declare
    return queue, self._size(queue), 0
  File &quot;/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/transport/mongodb.py&quot;, line 71, in _size
    return (self._queue_cursors[queue].count() -
KeyError: 'awfe1.example.com.celeryd.pidbox'"><pre class="notranslate">[<span class="pl-c1">2012</span><span class="pl-c1">-</span><span class="pl-c1">08</span><span class="pl-c1">-</span><span class="pl-c1">01</span> <span class="pl-c1">00</span>:<span class="pl-c1">30</span>:<span class="pl-c1">27</span>,<span class="pl-c1">163</span>: <span class="pl-v">ERROR</span><span class="pl-c1">/</span><span class="pl-v">MainProcess</span>] <span class="pl-v">Unrecoverable</span> <span class="pl-s1">error</span>: <span class="pl-v">KeyError</span>(<span class="pl-s">'awfe1.example.com.celeryd.pidbox'</span>,)
<span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/__init__.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">268</span>, <span class="pl-s1">in</span> <span class="pl-s1">start</span>
    <span class="pl-s1">component</span>.<span class="pl-en">start</span>()
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/consumer.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">309</span>, <span class="pl-s1">in</span> <span class="pl-s1">start</span>
    <span class="pl-s1">self</span>.<span class="pl-en">reset_connection</span>()
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/consumer.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">592</span>, <span class="pl-s1">in</span> <span class="pl-s1">reset_connection</span>
    <span class="pl-s1">self</span>.<span class="pl-en">reset_pidbox_node</span>()
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/celery-2.4.6-py2.7.egg/celery/worker/consumer.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">531</span>, <span class="pl-s1">in</span> <span class="pl-s1">reset_pidbox_node</span>
    <span class="pl-s1">callback</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-s1">on_control</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/pidbox.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">65</span>, <span class="pl-s1">in</span> <span class="pl-s1">listen</span>
    <span class="pl-s1">callbacks</span><span class="pl-c1">=</span>[<span class="pl-s1">callback</span> <span class="pl-c1">or</span> <span class="pl-s1">self</span>.<span class="pl-s1">handle_message</span>])
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/pidbox.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">56</span>, <span class="pl-s1">in</span> <span class="pl-v">Consumer</span>
    <span class="pl-c1">**</span><span class="pl-s1">options</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/messaging.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">285</span>, <span class="pl-s1">in</span> <span class="pl-s1">__init__</span>
    <span class="pl-s1">self</span>.<span class="pl-en">declare</span>()
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/messaging.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">295</span>, <span class="pl-s1">in</span> <span class="pl-s1">declare</span>
    <span class="pl-s1">queue</span>.<span class="pl-en">declare</span>()
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/entity.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">388</span>, <span class="pl-s1">in</span> <span class="pl-s1">declare</span>
    <span class="pl-s1">self</span>.<span class="pl-en">queue_declare</span>(<span class="pl-s1">nowait</span>, <span class="pl-s1">passive</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/entity.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">408</span>, <span class="pl-s1">in</span> <span class="pl-s1">queue_declare</span>
    <span class="pl-s1">nowait</span><span class="pl-c1">=</span><span class="pl-s1">nowait</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/transport/virtual/__init__.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">380</span>, <span class="pl-s1">in</span> <span class="pl-s1">queue_declare</span>
    <span class="pl-k">return</span> <span class="pl-s1">queue</span>, <span class="pl-s1">self</span>.<span class="pl-en">_size</span>(<span class="pl-s1">queue</span>), <span class="pl-c1">0</span>
  <span class="pl-v">File</span> <span class="pl-s">"/opt/lfsp/eggs/kombu-2.1.1-py2.7.egg/kombu/transport/mongodb.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">71</span>, <span class="pl-s1">in</span> <span class="pl-s1">_size</span>
    <span class="pl-k">return</span> (<span class="pl-s1">self</span>.<span class="pl-s1">_queue_cursors</span>[<span class="pl-s1">queue</span>].<span class="pl-en">count</span>() <span class="pl-c1">-</span>
<span class="pl-v">KeyError</span>: <span class="pl-s">'awfe1.example.com.celeryd.pidbox'</span></pre></div>