title: In MongoDB, how do I find distinct values of a large, sharded collection?
body: <p>I have a large mongodb collection:</p>
  
  <ul>
  <li>with 3 shards,   </li>
  <li>Totalling 300M records (at least)</li>
  <li>Shard key is (field1:1,field2:1)</li>
  <li>There are other non-indexed fields.    </li>
  <li>Field1 is a ~200-characters string  </li>
  <li>Field2 is an int.    </li>
  <li>There are about 10M distinct values of Field1, with more added all the time.</li>
  </ul>
  
  <p><strong>PART 1:  DISTINCT VALUES</strong></p>
  
  <p>I need to find all the distinct values of field1.  </p>
  
  <p>Calling db.myCollection.distinct("field1") fails because there are more than 16MB of data in the result set.</p>
  
  <p>Since the shard key is an index, this should be a covered query.</p>
  
  <p><strong>PART 2:  RETURN SORTED RESULTS</strong></p>
  
  <p>Presuming there's an answer for the above, I'd like to make this recover from failures, that is, return results in sorted order.  It's not vital.</p>
  
  <p>The goal of sorted output:  if the query fails halfway thru, I can resume from where I left off by adding the query specifier of field1:{$gt:lastGoodValue}.  </p>
  
  <p>So: Is this possible?  Easy?  Do I have to aggregate or map-reduce?  Currently, I'm iterating over all 300M records and it shoves a lot of data around unnecessarily.</p>