Finding unreferenced documents I have a database containing at least 2 collections datagroup and data.
Each data document have a reference to the _id of one of the datagroup documents.

I want to find all datagroup documents that have no data pointing to, the fastest as possible.

For this, I use a javascript mongo shell script.

I have these options for now (if you have one more, feel free to suggest):  


  Notes:  
  
  
  data.vid is the same value as datagroup._id (like external key, of kind ObjectId)  
  There is a large number of data (~100,000,000) and datagroup (~20,000)
  


Option 1: (123s)

/* 0s since beginning (experimented) */
// get id of all datagroups used in data
let withData = db.data.find({})           // get all data
                      .map(d=&gt;d.vid)      // to array of ObjectId
/* 1m40s since beginning (experimented) */

withData = Array.from(new Set(withData)); // remove duplicates in array
/* 1m52s since beginning (experimented) */

// find datagroup that ids not in ObjectId array
let withoutData = db.datagroup.find({_id: {$nin: withData}})
                              .map(dg=&gt;dg._id);  // get their ids
/* 2m03s since beginning (experimented) */


Option 2: (59s)

/* 0s since beginning (experimented) */
// get id of all datagroups used in data
let withData = db.data.aggregate([
    {$project:{vid:1, _id:0}},       // keep only references to datagroups
    {$group:{_id:"$vid"}}            // remove duplicates
]).map(d=&gt;d.vid.str);                // to array of ObjectId strings
/* 37s since beginning (experimented) */

let withoutData = db.datagroup.find({})      // find all datagroups
                              .map(dg=&gt;dg._id) // get theys ids
                                             // compare with array of ids
                              .filter(vid =&gt; withData.indexOf(vid.str) &lt; 0);
/* 59s since beginning (experimented) */


Option 3: (109s)

/* 0s since beginning (experimented) */
let withoutData = db.datagroup.aggregate([
    {$project:{_id:1}},       // keep only ids
    {$lookup:{
        from: "data",         // grouping data with datagroups
        localField: "_id",
        foreignField: "vid",
        as: "dgs"
    }},
    {$match:{dgs:[]}}        // keep only datagroups without data
]).map(dg=&gt;dg._id);
/* 1ms49 since beginning (experimented) */


Do you think there is a better way than doing this, or should I use the option 2 because it's just the best way to do this kind of request ? (Or maybe you have ideas in order to speed up my options ?)
There is a better way, but this assume that you have an index on vid field (for example, {"vid": 1} ). If you don't, you can create one using db.data.ensureIndex({"vid": 1}). This can take several minutes to end. 

The fastest solution (option 2) use $group after a $project  Here the $project stage only add overhead and does not speed up the $group stage, so we can get rid of it.

Currently, the $group operator can't take advantage of index, so it can be quite slow for such big dataset. A better solution would be to use distinct() instead


  Index Use
  
  When possible, db.collection.distinct() operations can use indexes.


so we can fetch withData array simply using this: 

let withData = db.data.distinct("vid")


Then, the datagroup collection is filtered using map() and filter(). On really big array, this can be really expensive as javascript is quite slow. Instead, we can let mongodb do this operation with a simple query using $nin:

let withoutData = db.datagroup.distinct("_id", {"_id": {$nin: withData}})


As distinct() returns an array, we get directly the result in proper format, so we can avoid js mapping and filtering. 

so the final script is:

let withData = db.data.distinct("vid")
let withoutData = db.datagroup.distinct("_id", {"_id": {$nin: withData}})




Some results

on my machine (ubuntu16.04) , with a similar dataset ( 100 000 000 docs in data, 20000 in datagroup)  

New script runs in less than 5s :

time mongo test &lt; newOption.js
0,06s user 0,00s system 1% cpu 3,534 total