title: data lost in mongodb replica set mode
body: <p>My replica set has two nodes:</p>
  
  <ul>
  <li>1: the master node</li>
  <li>2: a slave node with priority:0, votes:0</li>
  </ul>
  
  <p>The oplog size is 5000MB.</p>
  
  <p>run this for loop in master shell:</p>
  
  <pre><code>for (i=0;i&lt;1000000;i++)
    {
      db.getSiblingDB("ff").c.insert(
        { a:i,
          d:i+".@234"+(++i)+".234546"+(++i)+".568679"+(++i)+"31234."+(++i)+".12342354"+(++i)+"5346457."+(++i)+"33543465456."+(++i)+".6346456"+(++i)+"123235434."+(++i)+".2345345345"+(++i)
        }
     )
  }
  </code></pre>
  
  <p>Kill the slave node while the for loop is running: <code>kill -9 $(pidof slave_node)</code></p>
  
  <p>Stop the for loop after a second; then restart the slave node.</p>
  
  <p>Then run <code>db.getSiblingDB("ff").c.count()</code> to check data in both slave and master nodes, with the results:</p>
  
  <ul>
  <li>master:20w</li>
  <li>slave:15w</li>
  </ul>
  
  <p>The slave node can catch up with the primary, but there is a lot of data lost from the slave.</p>
  
  <p>Why is this?</p>
  
  <p>Here is the slave node's log as it restarts after being killed:</p>
  
  <pre><code>2017-11-27T05:53:53.873+0000 I NETWORK  [thread1] waiting for connections on port 28006
  2017-11-27T05:53:53.876+0000 I REPL     [replExecDBWorker-0] New replica set config in use: { _id: "cpconfig2", version: 2, protocolVersion: 1, members: [ { _id: 0, host: "127.0.0.1:28007", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 3.0, tags: {}, slaveDelay: 0, votes: 1 }, { _id: 1, host: "127.0.0.1:28006", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 0.0, tags: {}, slaveDelay: 0, votes: 0 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 60000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5a1ba5bbb0a652502a5f002a') } }
  2017-11-27T05:53:53.876+0000 I REPL     [replExecDBWorker-0] This node is 127.0.0.1:28006 in the config
  2017-11-27T05:53:53.876+0000 I REPL     [replExecDBWorker-0] transition to STARTUP2
  2017-11-27T05:53:53.876+0000 I REPL     [replExecDBWorker-0] Starting replication storage threads
  2017-11-27T05:53:53.877+0000 I REPL     [replExecDBWorker-0] Starting replication fetcher thread
  2017-11-27T05:53:53.877+0000 I REPL     [replExecDBWorker-0] Starting replication applier thread
  2017-11-27T05:53:53.877+0000 I REPL     [replExecDBWorker-0] Starting replication reporter thread
  2017-11-27T05:53:53.877+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to 127.0.0.1:28007
  2017-11-27T05:53:53.877+0000 I REPL     [rsSync] transition to RECOVERING
  2017-11-27T05:53:53.878+0000 I REPL     [rsSync] transition to SECONDARY
  2017-11-27T05:53:53.879+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Successfully connected to 127.0.0.1:28007, took 2ms (1 connections now open to 127.0.0.1:28007)
  2017-11-27T05:53:53.879+0000 I REPL     [ReplicationExecutor] Member 127.0.0.1:28007 is now in state PRIMARY
  2017-11-27T05:53:54.011+0000 I FTDC     [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
  2017-11-27T05:53:54.645+0000 I NETWORK  [thread1] connection accepted from 127.0.0.1:52404 #1 (1 connection now open)
  2017-11-27T05:53:54.645+0000 I NETWORK  [conn1] received client metadata from 127.0.0.1:52404 conn1: { driver: { name: "NetworkInterfaceASIO-Replication", version: "3.4.9" }, os: { type: "Linux", name: "PRETTY_NAME="Debian GNU/Linux 8 (jessie)"", architecture: "x86_64", version: "Kernel 3.10.0" } }
  2017-11-27T05:53:59.878+0000 I REPL     [rsBackgroundSync] sync source candidate: 127.0.0.1:28007
  </code></pre>