title: How to delete documents by query efficiently in mongo?
body: <p>I have a query, which selects documents to be removed. Right now, I remove them manually, like this (using python):</p>
  
  <pre><code>for id in mycoll.find(query, fields={}):
    mycoll.remove(id)
  </code></pre>
  
  <p>This does not seem to be very efficient. Is there a better way?</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>OK, I owe an apology for forgetting to mention the query details, because it matters. Here is the complete python code:</p>
  
  <pre><code>def reduce_duplicates(mydb, max_group_size):
    # 1. Count the group sizes
    res = mydb.static.map_reduce(jstrMeasureGroupMap, jstrMeasureGroupReduce, 'filter_scratch', full_response = True)
    # 2. For each entry from the filter scratch collection having count &gt; max_group_size
    deleteFindArgs = {'fields': {}, 'sort': [('test_date', ASCENDING)]}
    for entry in mydb.filter_scratch.find({'value': {'$gt': max_group_size}}):
      key = entry['_id']
      group_size = int(entry['value'])
      # 2b. query the original collection by the entry key, order it by test_date ascending, limit to the group size minus max_group_size.
      for id in mydb.static.find(key, limit = group_size - max_group_size, **deleteFindArgs):
        mydb.static.remove(id)
    return res['counts']['input']
  </code></pre>
  
  <p>So, what does it do? It reduces the number of duplicate keys to at most <code>max_group_size</code> per key value, <strong>leaving only the newest records</strong>. It works like this:</p>
  
  <ol>
  <li>MR the data to <code>(key, count)</code> pairs.</li>
  <li>Iterate over all the pairs with <code>count &gt; max_group_size</code></li>
  <li>Query the data by <code>key</code>, while sorting it ascending by the timestamp (the oldest first) and limiting the result to the <code>count - max_group_size</code> oldest records</li>
  <li>Delete each and every found record.</li>
  </ol>
  
  <p>As you can see, this accomplishes the task of reducing the duplicates to at most N newest records. So, the last two steps are <code>foreach-found-remove</code> and this is the important detail of my question, that changes everything and I had to be more specific about it - sorry.</p>
  
  <p>Now, about the collection remove command. It does accept query, but mine include sorting and limiting. Can I do it with remove? Well, I have tried:</p>
  
  <pre><code>mydb.static.find(key, limit = group_size - max_group_size, sort=[('test_date', ASCENDING)])
  </code></pre>
  
  <p>This attempt fails miserably. Moreover, it seems to screw mongo.Observe:</p>
  
  <pre><code>C:\dev\poc\SDR&gt;python FilterOoklaData.py
  bad offset:0 accessing file: /data/db/ookla.0 - consider repairing database
  </code></pre>
  
  <p>Needless to say, that the foreach-found-remove approach works and yields the expected results.</p>
  
  <p>Now, I hope I have provided enough context and (hopefully) have restored my lost honour.</p>