title: MongoDB - NodeJS - Mongoose Slow Query Array
body: <p>I have a question about Mongo and NodeJS currently in one project that i am working on have some issues with the performance. It does work but has a mix of ideas as far as i see. I would like to see some ideas from a more experience person.</p>
  
  <p>I will share the flow with you and the used libs and tools on the way i try to understand it there days and as far as i see might be a better way to do it but still will be slow and un-efficient.</p>
  
  <p>Used Tools and Lib: lodash, mongoose, async, and more.</p>
  
  <p>This is what is the results from NewRelic:</p>
  
  <ul>
  <li>Database MongoDB preferences toArray 19.8    13.6    423ms</li>
  <li>Database MongoDB spotifies toArray   19.4    12.6    415ms</li>
  <li>Database MongoDB mifi toArray    19.2    12.6    409ms</li>
  <li>Database MongoDB fifi toArray    18.9    12.6    404ms</li>
  <li>Database MongoDB locations toArray   14.3    18.5    305ms</li>
  </ul>
  
  <p>A sample of what is used to add to the model all the time:</p>
  
  <pre><code>const subObjectsOutCards = [{
  name : 'preferencesModel',
  model : 'Preferences'}, {
  name : 'spotifyModel',
  model : 'Spotify'} ,{
  name : 'MifiModel',
  model : 'mifi'}, {
  name : 'fiModel', 
  model : 'Fifi'}];
  </code></pre>
  
  <p>Next One:</p>
  
  <pre><code>exports.fetchUsers = (callback) =&gt; {
  Account.find().populate('userTrips').exec((err, docs) =&gt; {
      if (_.isEmpty(docs)){
          callback(null, null)
      } else {
          async.map(docs, (doc, callback) =&gt; {
              var object = doc.toObject();
              exports.fetchSubObjectsToUserForCards(object, doc._id, (result) =&gt; {
                  callback(null, result);
              })
          }, (err, results1) =&gt; {
              var updatedAccounts = results1; 
              async.map(updatedAccounts, (doc, callback) =&gt; {
                  locationController.getMostRecentLocationById(doc._id, (result) =&gt; {
                      var updatedDoc = doc;
                      updatedDoc.locationModel = result;
                      callback(null, updatedDoc);
                  })
              }, (err, results2) =&gt; {
                  callback(results2);
              })
          })
      }
  })}
  </code></pre>
  
  <p>This is where it goes after:</p>
  
  <pre><code>exports.fetchObjectSub = (account, userId, callback) =&gt; {
  var newAccount = account;
  var itemsProcessed = 0;
  subObjectsOut.forEach(function(object){
      mongoose.model(object.model).find({'accountId' : userId}, function(err, doc){
          if (err){
              callback(false);
          } else {
              if (!_.isNil(doc[0]) &amp;&amp; newAccount !== false){
                  doc = doc[0].toObject();
                  newAccount[object.name] = doc;
                  itemsProcessed++;
                  if (itemsProcessed === subObjectsOut.length){
                      callback(newAccount);
                  }
              } else {
                  itemsProcessed++;
              }
          }
      });
  });}
  </code></pre>
  
  <p>Account Example:</p>
  
  <pre><code>{
  "_id": "59f372389f89d1cb0dbabdbad",
  "residence": "Katowice, Poland",
  "orientation": "Bisexual",
  "lastName": "Kowalczyk",
  "job": "Web Developer @ Freelance",
  "gender": "Female",
  "firstName": "Marina",
  "dob": "7/10/1998",
  "about": "Computer science student",
  "lastConnection": "2017-11-24T19:16:28.780Z",
  "created_at": "2017-10-27T20:55:06.070Z",
  "phone": {
      "number": "7183173136"
  },
  "profilePicture": {
      "url": "https://.JPG",
      "pictureType": ".JPG"
  },
  "interests": [
      "Sports",
      "Food",
      "Cycling",
      "Running",
      "Cooking",
      "Movies",
      "Fashion",
      "Business",
      "Travel",
      "Music",
      "Theatre",
      "Yoga",
      "Party",
      "Dancing",
      "Reading"
  ],
  "__v": 38}
  </code></pre>
  
  <p>Preferences Example:</p>
  
  <pre><code>{
  "_id": "59a83258c7fd5b4ae586c53b",
  "visibilityLocation": true,
  "visibilityGenderPreferences": true,
  "visibilityFb": false,
  "visibilityDistance": false,
  "visibilityAge": false,
  "showMyProfileAs": "Male",
  "showMe": "Males",
  "locationAccuracy": 0,
  "accountId": "59f372389f89d1cb0dbabdbad",
  "created_at": "2017-08-31T13:29:42.462Z",
  "distance": {
      "max": 30,
      "metrics": "K",
      "min": 0
  },
  "ageRange": {
      "max": 32,
      "min": 18
  },
  "__v": 0}
  </code></pre>
  
  <p>EDIT: As requested</p>
  
  <ol>
  <li>Indexes - automatic from the system on the _id</li>
  <li>Index on accountId in each table from the main one</li>
  <li>Operation Performed is on 1000 elements and its really slow: Speeds that i get are 3000ms to 9000ms and all the slow down on average says 5M ms for these documents which is insane....</li>
  <li>Example Account and Preferences can be found above before edit.</li>
  </ol>
  
  <p>When i started with this i thought straight that are the filters that we have but seems its not the case. Since thats pretty fast. Problems comes from here and the tools confirm it.</p>
  
  <p>The idea about the structure of the Schema as far as i see is to be a flat one and after to be used Collections all the time instead of references of the other tables. And for each one since there are a lot filters there need to be a search for each person on each one of their five tables and getting their account on the way and adding for each table a field with it and creating a documents. And all of them wait for each other all the time. So if you have 5 users = 5 new documents and after for each there was a loop and a new documents. Please if anybody can help a bit with this would be great. Thanks you</p>