title: Running MongoDB aggregations in parallel
body: <p>Currently I'm running aggregations against a collection containing user and event information. For example:</p>
  
  <pre><code>[
    {
      $match: {
        client: ObjectId('507f1f77bcf86cd799439011'),
        location: 'UK'
      }
    },
    {
      $group: {
        _id: null,
        count: {
          $sum: 1
        }
      }
    }
  ]
  </code></pre>
  
  <p>The above is a big simplification, suffice to say there are around 20 different variables like <code>location</code> that can go into that <code>$match</code> statement. There are also sometimes additional steps between these two, which is why I'm using <code>$group</code> to do a count. (Rather than <code>count</code>)</p>
  
  <p>Currently I have an index on the <code>client</code> field, but haven't created indexes (compound or otherwise) on the other fields. As there are so many other fields, I can't just create indexes on everything - it'll be too expensive.</p>
  
  <p><strong>The problem:</strong> This works great when a client has a small number of documents, but as the number grows, the aggregation has to scan over an ever-increasing number of docs. The index focuses the range down, but it's not enough.</p>
  
  <hr>
  
  <h2>Idea</h2>
  
  <p>Create an additional variable called <code>p</code> (for partition), and create a compound index: <code>{ client: 1, p: 1 }</code>. <code>p</code> can be <code>1</code>-<code>n</code>.</p>
  
  <p>Instead of running the pipeline above, run a similar pipeline <code>n</code> times: (for all possible values of <code>p</code>)</p>
  
  <pre><code>[
    {
      $match: {
        client: ObjectId('507f1f77bcf86cd799439011'),
        p: 1, // or 2, 3, etc
        location: 'UK'
      }
    },
    {
      $group: {
        _id: null,
        count: {
          $sum: 1
        }
      }
    }
  ]
  </code></pre>
  
  <p>The results from all pipelines could then be merged on the application level.</p>
  
  <p>Using this method, I could limit the number of scans that each query has to do, theoretically decreasing query time. </p>
  
  <p>Taking this a step further, this <code>p</code> value could be used as the shard key, so in theory, the analytics queries could be run in parallel across multiple shards.</p>
  
  <p><strong>Has anyone done anything like this before? I've found very little on the subject.</strong> </p>