title: Reducing requests to remote DB with MongoDB
body: <p>I think I need to build a rather complex query in a single batch.</p>
  
  <p><strong>Problem description</strong></p>
  
  <p>A user may ask for <strong>data</strong> resources from each day:</p>
  
  <p><code>/api/data?startDate=2018-03-20&amp;endDate=2018-02-24&amp;mode=day</code></p>
  
  <p>So first, the backend will generate ranges as follow (I am rather using pseudocode):</p>
  
  <pre><code>rangesArray = [
      {start: "2018-03-20 00:00:00", end: "2018-03-20 23:59:59"}
      {start: "2018-03-21 00:00:00", end: "2018-03-21 23:59:59"}
      {start: "2018-03-22 00:00:00", end: "2018-03-22 23:59:59"}
      {start: "2018-03-23 00:00:00", end: "2018-03-23 23:59:59"}
      {start: "2018-03-24 00:00:00", end: "2018-03-24 23:59:59"}
  ]
  </code></pre>
  
  <p>and the query would look like below:</p>
  
  <pre><code>for (var i = 0; i &lt; rangesArray.length; i++) {
  
      let query = {
        'endDate': {'$gte': rangesArray[i].start},
        'startDate': {'$lte': rangesArray[i].end}
      }
  
      // A huge no-no, as each request to remote DB takes some time
      let dataForRange = await Data.find(query)
              .limit(10) // Setting limit here, as we don't
                         // need to present all the data from each day.
                         // The limit is hardcoded for now but should
                         // be dependent on rangesArray.length
  
      // Here, do something useful with dataForRange
  }
  </code></pre>
  
  <p>I need to assume that:</p>
  
  <ul>
  <li>each day might contain say, millions of <strong>data</strong> records (but I need to receive at least one record from each day).</li>
  <li>API will not allow querying for a longer span than 365 days (i.e. <code>/api/data?startDate=2017-03-20&amp;endDate=2018-04-20&amp;mode=day</code> will not be possible)</li>
  <li><strong>data</strong> contains <em>startDate</em>, <em>endDate</em>, and (for example) <em>temperature</em>.</li>
  <li>There shouldn't be duplications. For example: if there is a <em>data</em> object that lasts so long that overlaps 3 days, say:<code>data = {startDate: „2018-03-22 13:20:00”, endDate: „2018-03-24 10:00:00”, temperature: 20.0}</code>, I should receive that particular record just once. I think this is rather hard to satisfy, so this is not an obligatory requirement.</li>
  </ul>
  
  <p>So I am expecting something like this (huge pseudocode):</p>
  
  <pre><code>complexQuery = [
      {'$or': {endDate: {'$gte': '2018-03-20 00:00:00'}, 'startDate': {'$lte': „2018-03-20 23:59:59”}, 'limit': 10}},
      {'$or’: {endDate: {'$gte': '2018-03-21 00:00:00'}, 'startDate': {'$lte': '2018-03-21 23:59:59'}, 'limit': 10}}
      (...)
  ]
  
  // One query to remote DB, hurray! But it doesn't work ...
  let dataForRange = await Data.aggregate( complexQuery )
  </code></pre>