title: MongoDB Update with upsert and unique index atomicity when document is not in the working set
body: <p>In summary, we have ran into this weird behavior in doing concurrent updates on an existing document when the document is not part of the working set (not in resident memory).</p>
  
  <p>More details:</p>
  
  <p>Given a collection with a unique index and when running concurrent updates (3 threads) with upsert as true on a given existing document, 1 to 2 threads raise the following exception:</p>
  
  <pre><code>Processing failed (Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$key_1  dup key: { : 1008 }'):
  </code></pre>
  
  <p>According to the documentation, I would expect all of the three updates to succeed because the document I am trying to update already exists. Instead, it looks like it is trying to do an insert on few or all of the update requests and few fails due to the unique index.</p>
  
  <p>Repeating the same concurrent update on the document does not raise any exceptions. Also, using find() on a document to bring it to the working set, then running the concurrent updates on that document also runs as expected. 
  Also, using findAndModify with the same query and settings does not have the same problem.</p>
  
  <p>Is this working as expected or am I missing something?</p>
  
  <p>Setup:</p>
  
  <p>-mongodb java driver 3.0.1</p>
  
  <p>-3 node replica set running MongoDB version "2.6.3"</p>
  
  <p>Query:</p>
  
  <pre><code>BasicDBObject query = new BasicDBObject();  
  query.put("docId", 123L);
  collection.update (query, object, true, false);
  </code></pre>
  
  <p>Index:</p>
  
  <pre><code>name: docId_1
  unique: true
  key: {"docId":1}
  background: true
  </code></pre>
  
  <p>Updated on May 28 to include sample code to reproduce the issue.
  Run MongoDB locally as follow  (Note that the test will write about ~4 GB of data):
  ./mongodb-osx-x86_64-2.6.10/bin/mongod --dbpath /tmp/mongo
  Run the following code, restart the database, comment out "fillUpCollection(testMongoDB.col1, value, 0, 300);", then run the code again. Depending on the machine, you may need to tweak some of the numbers to be able to see the exceptions.</p>
  
  <pre><code>package test;
  
  import com.mongodb.BasicDBObject;
  import com.mongodb.DBCollection;
  import com.mongodb.DBObject;
  import com.mongodb.Mongo;
  import com.mongodb.MongoClient;
  
  import java.io.IOException;
  import java.util.ArrayList;
  import java.util.List;
  import java.util.Random;
  
  public class TestMongoDB {
      public static final String DOC_ID = "docId";
      public static final String VALUE = "value";
      public static final String DB_NAME = "db1";
      public static final String UNIQUE = "unique";
      public static final String BACKGROUND = "background";
      private DBCollection col1;
      private DBCollection col2;
  
      private static DBCollection getCollection(Mongo mongo, String collectionName) {
          DBCollection col =  mongo.getDB(DB_NAME).getCollection(collectionName);
          BasicDBObject index = new BasicDBObject();
          index.append(DOC_ID, 1);
          DBObject indexOptions = new BasicDBObject();
          indexOptions.put(UNIQUE, true);
          indexOptions.put(BACKGROUND, true);
          col.createIndex(index, indexOptions);
          return col;
      }
  
      private static void storeDoc(String docId, DBObject doc, DBCollection dbCollection) throws IOException {
          BasicDBObject query = new BasicDBObject();
          query.put(DOC_ID, docId);
          dbCollection.update(query, doc, true, false);
          //dbCollection.findAndModify(query, null, null, false, doc, false, true);
      }
  
      public static void main(String[] args) throws Exception{
          final String value = new String(new char[1000000]).replace('\0', 'a');
          Mongo mongo = new MongoClient("localhost:27017");
          final TestMongoDB testMongoDB = new TestMongoDB();
          testMongoDB.col1 = getCollection(mongo, "col1");
          testMongoDB.col2 = getCollection(mongo, "col2");
  
          fillUpCollection(testMongoDB.col1, value, 0, 300);
          //restart Database, comment out previous line, and run again
          fillUpCollection(testMongoDB.col2, value, 0, 2000);
          updateExistingDocuments(testMongoDB, value);
      }
  
      private static void updateExistingDocuments(TestMongoDB testMongoDB, String value) {
          List&lt;String&gt; docIds = new ArrayList&lt;String&gt;();
          for(int i = 0; i &lt; 10; i++) {
              docIds.add(new Random().nextInt(300) + "");
          }
          multiThreadUpdate(testMongoDB.col1, value, docIds);
      }
  
  
      private static void multiThreadUpdate(final DBCollection col, final String value, final List&lt;String&gt; docIds) {
          Runnable worker = new Runnable() {
              @Override
              public void run() {
                  try {
                      System.out.println("Started Thread");
                      for(String id : docIds) {
                          storeDoc(id, getDbObject(value, id), col);
                      }
                  } catch (Exception e) {
                      System.out.println(e);
                  } finally {
                      System.out.println("Completed");
                  }
              }
          };
  
          for(int i = 0; i &lt; 8; i++) {
              new Thread(worker).start();
          }
      }
  
      private static DBObject getDbObject(String value, String docId) {
          final DBObject object2 = new BasicDBObject();
          object2.put(DOC_ID, docId);
          object2.put(VALUE, value);
          return object2;
      }
  
      private static void fillUpCollection(DBCollection col, String value, int from, int to) throws IOException {
          for(int i = from ; i &lt;= to; i++) {
              storeDoc(i + "", getDbObject(value, i + ""), col);
          }
      }
  }
  </code></pre>
  
  <p>Sample Output on the second run:</p>
  
  <pre><code>Started Thread
  Started Thread
  Started Thread
  Started Thread
  Started Thread
  Started Thread
  Started Thread
  Started Thread
  com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "290" }'
  Completed
  com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "170" }'
  Completed
  com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "241" }'
  Completed
  com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "127" }'
  Completed
  com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "120" }'
  Completed
  com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "91" }'
  Completed
  com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'insertDocument :: caused by :: 11000 E11000 duplicate key error index: db1.col1.$docId_1  dup key: { : "136" }'
  Completed
  Completed
  </code></pre>