title: MongoDB replica heartbeat request time exceeded
body: <p>I have replica set (hosted on amazon) which has:</p>
  
  <ul>
  <li>primary</li>
  <li>secondary</li>
  <li>arbiter</li>
  </ul>
  
  <p>All of them are version 3.2.6 and this replica is making one shard in my sharded cluster (if that is important although I think it is not).</p>
  
  <p>When I type <code>rs.status()</code> on primary it says that cannot reach secondary (the same thing is on arbiter):</p>
  
  <pre><code>        {
          "_id" : 1,
          "name" : "secondary-ip:27017",
          "health" : 0,
          "state" : 8,
          "stateStr" : "(not reachable/healthy)",
          "uptime" : 0,
          "optime" : {
              "ts" : Timestamp(0, 0),
              "t" : NumberLong(-1)
          },
          "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
          "lastHeartbeat" : ISODate("2016-07-20T15:40:50.479Z"),
          "lastHeartbeatRecv" : ISODate("2016-07-20T15:40:51.793Z"),
          "pingMs" : NumberLong(0),
          "lastHeartbeatMessage" : "Couldn't get a connection within the time limit",
          "configVersion" : -1
      }
  </code></pre>
  
  <p>(btw look at the <strong>optimeDate</strong> O.o)</p>
  
  <p>Error in my log is:</p>
  
  <pre><code>[ReplicationExecutor] Error in heartbeat request to secondary-ip:27017; ExceededTimeLimit: Couldn't get a connection within the time limit
  </code></pre>
  
  <p>Strange thing is that when I go on secondary and type <code>rs.status()</code> everything looks OK. Also I am able to connect to secondary from my primary instance (with <code>mongo --host secondary</code>) so I guess it is not network issue. Yesterday it was all working fine.</p>
  
  <p><strong>TL;DR</strong> my primary cannot see secondary and arbiter cannot see secondary and my secondary sees primary and it was all working fine just day ago and I am able manually connect to secondary from primary instance.</p>
  
  <p>Anyone has an idea what could go wrong?</p>
  
  <p>Tnx,
  Ivan</p>