title: Mongodb replica set auto reconect don't work after down and up for nginx + uwsgi with several processes
body: <p>Hi everyone I have next envirement for <code>python2.7.5</code>:</p>
  
  <pre><code>flask==0.10.1
  flask-wtf==0.8.4
  jinja2==2.7
  werkzeug==0.9.1
  flask-mongoengine==0.7.0
  mongoengine==0.8.2
  pymongo==2.5.2
  uwsgi==1.9.13
  </code></pre>
  
  <p>and have next application 'app.py':</p>
  
  <pre><code>from flask import Flask
  from flask.ext.mongoengine import Document, MongoEngine
  from mongoengine import StringField  
  
  class Config(object):
      DEBUG = True
      MONGODB_HOST = ('mongodb://localhost:27017,localhost:27018/'
                      'test?replicaSet=rs0')
      MONGODB_DB = True
  
  app = Flask(__name__)
  app.config.from_object(Config)
  MongoEngine(app)
  
  class Test(Document):
      test = StringField(default='test')
  
      meta = {
          'allow_inheritance': False,
      }
  
      def __unicode__(self):
          return self.test
  
  Test(test='test1').save()
  
  @app.route('/')
  def hello_world():
      return unicode(Test.objects.first())
  
  if __name__ == '__main__':
      app.run('0.0.0.0', 8080, True)
  </code></pre>
  
  <p>I have next nginx config:</p>
  
  <pre><code>server {
      listen       80;
      server_name  localhost;
      location / {
          include uwsgi_params;
          uwsgi_pass unix:/tmp/uwsgi.sock;
      }
  
  }
  </code></pre>
  
  <p>I start uwsgi as:</p>
  
  <pre><code>/path/to/env/bin/uwsgi \
    --module app:app \
    --env /path/to/env/ \
    --pythonpath /path/to/app/ \
    --socket /tmp/uwsgi.sock \
    --pidfile /tmp/uwsgi.pid \
    --daemonize /tmp/uwsgi.log \
    --processes 2 \
    --threads 2 \
    --master
  </code></pre>
  
  <p>I have two mongodb instances:</p>
  
  <pre><code>mongod --port 27017 --dbpath /path/to/mongo/data/rs0-0 --replSet rs0 \
    --smallfiles --oplogSize 128
  </code></pre>
  
  <p>and</p>
  
  <pre><code>mongod --port 27018 --dbpath /path/to/mongo/data/rs0-1 --replSet rs0 \
    --smallfiles --oplogSize 128
  </code></pre>
  
  <p>And configured replica set in mongo console as:</p>
  
  <pre><code>rsconf = {
      _id: "rs0",
      members: [{_id: 0, host: "127.0.0.1:27017"}]
  };
  rs.initiate(rsconf);
  rs.add("127.0.0.1:27018");
  </code></pre>
  
  <p>So it's working good. But when I down and up primary or secondary mongo instance then my application can't restore connection and I have next exceptions every time after:</p>
  
  <pre><code>...
    File "/path/to/app/replica.py", line 33, in hello_world
      return unicode(Test.objects.first())
    File "/path/to/env/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.py", line 325, in first
      result = queryset[0]
    File "/path/to/env/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.py", line 211, in __getitem__
      return queryset._document._from_son(queryset._cursor[key],
    File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 470, in __getitem__
      for doc in clone:
    File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 814, in next
      if len(self.__data) or self._refresh():
    File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 763, in _refresh
      self.__uuid_subtype))
    File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 700, in __send_message
      **kwargs)
    File "/path/to/env/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1546, in _send_message_with_response
      raise AutoReconnect(msg, errors)
  pymongo.errors.AutoReconnect: No replica set primary available for query with ReadPreference PRIMARY
  </code></pre>
  
  <p>When I use <code>mongoengie==0.7.10</code> which use <code>ReplicaSetConnection</code> instead <code>MongoReplicaSetClient</code> in <code>mongoengine==0.8.2</code> then I have next exceptions:</p>
  
  <ol>
  <li><p>Down secondary, get requests, up secondary, get requests:</p>
  
  <p>I have at first time:</p>
  
  <pre><code>pymongo.errors.AutoReconnect: 127.0.0.1:27017: [Errno 104] Connection reset by peer
  </code></pre>
  
  <p>after:</p>
  
  <pre><code>pymongo.errors.AutoReconnect: No replica set primary available for query with ReadPreference PRIMARY
  </code></pre></li>
  <li><p>Down primary, get requests, up primary, get requests:</p>
  
  <p>I have at first time:</p>
  
  <pre><code>pymongo.errors.AutoReconnect: 127.0.0.1:27017: [Errno 111] Connection refused
  </code></pre>
  
  <p>after:</p>
  
  <pre><code>pymongo.errors.AutoReconnect: No replica set primary available for query with ReadPreference PRIMARY
  </code></pre></li>
  <li><p>Down primary or secondary, up primary or secondary, get requests I have always:</p>
  
  <pre><code>pymongo.errors.AutoReconnect: not master and slaveOk=false
  </code></pre></li>
  </ol>
  
  <p>So two mongo instances just simple example. If I add one more instance (total 3), then:</p>
  
  <ol>
  <li><p>If I down any secondary then all work fine. If I down and up one secondary and then down or down and up second secondary then all work fine too.</p></li>
  <li><p>If I down and up two secondaries - some problem.</p></li>
  <li><p>If I down and up primary or just down one primary (two secondaries availible) - some problem, despite that mongo elect new primary!!!</p></li>
  </ol>
  
  <p>If I start one uwsgi process (without <code>--master</code> for two or three mongo instances) as:</p>
  
  <pre><code>/path/to/env/bin/uwsgi \
    --module app:app \
    --env /path/to/env/ \
    --pythonpath /path/to/app/ \
    --socket /tmp/uwsgi.sock \
    --pidfile /tmp/uwsgi.pid \
    --daemonize /tmp/uwsgi.log \
    --processes 1 \
    --threads 2
  </code></pre>
  
  <p>or run application with dev server:</p>
  
  <pre><code>/path/to/env/bin/python app.py
  </code></pre>
  
  <p>then application restore connection without problems after down and up mongo instances.</p>
  
  <p>I have some deployment in production and sometimes connection to mongo instances can disapire (to several seconds). After that my application don't work normal before uwsgi will be restarted.</p>
  
  <p>So I have two questions:</p>
  
  <ol>
  <li>Why it's happenes with several uwsgi process?</li>
  <li>How fix normal application work after down and up mongo instances?</li>
  </ol>
  
  <hr>
  
  <p><strong>UPD1</strong>:
  I trying understand problem and now I have different behaviour for <a href="https://github.com/mongodb/mongo-python-driver/blob/master/pymongo/mongo_replica_set_client.py#L1449" rel="nofollow"><code>self.__schedule_refresh()</code></a> when get connection exception when I down one mongo node:</p>
  
  <ol>
  <li><p>For one process:</p>
  
  <ol>
  <li>Before this statement: <code>rs_state</code> have two members: active with <code>up == True</code>, downed with <code>up == False</code>.</li>
  <li>After this statement: <code>rs_state</code> have one active member with <code>up == True</code>.</li>
  </ol></li>
  <li><p>For two processes:</p>
  
  <ol>
  <li>Before this statement: <code>rs_state</code> have two members: active with <code>up == True</code>, downed with <code>up == False</code>.</li>
  <li>After this statement: <code>rs_state</code> have two members: active with <code>up == True</code>, downed with <code>up == False</code> (no changed).</li>
  </ol></li>
  </ol>
  
  <p>When I up mongo node then <a href="https://github.com/mongodb/mongo-python-driver/blob/master/pymongo/mongo_replica_set_client.py#L1481" rel="nofollow"><code>self.__schedule_refresh(sync=sync)</code></a> also have different behaviour:</p>
  
  <ol>
  <li><p>For one process:</p>
  
  <ol>
  <li>Before this statement: <code>rs_state</code> have one active member with <code>up == True</code>.</li>
  <li>After this statement: <code>rs_state</code> have two members active with <code>up == True</code>, upped with <code>up == True</code>.</li>
  </ol></li>
  <li><p>For two processes:</p>
  
  <ol>
  <li>Before this statement: <code>rs_state</code> have two members active with <code>up == True</code>, upped with <code>up == False</code>.</li>
  <li>After this statement: <code>rs_state</code> have two members active with <code>up == True</code>, upped with <code>up == False</code> (no changed).</li>
  </ol></li>
  </ol>
  
  <p>So look like mongo can't update replica set state (see <a href="https://github.com/mongodb/mongo-python-driver/blob/master/pymongo/mongo_replica_set_client.py#L1027" rel="nofollow"><code>__schedule_refresh</code></a>):</p>
  
  <pre><code>def __schedule_refresh(self, sync=False):
      """Awake the monitor to update our view of the replica set's state.
  
      If `sync` is True, block until the refresh completes.
  
      If multiple application threads call __schedule_refresh while refresh
      is in progress, the work of refreshing the state is only performed
      once.
      """
      self.__monitor.schedule_refresh()
      if sync:
          self.__monitor.wait_for_refresh(timeout_seconds=5)
  </code></pre>