title: MongoDB cumulatively append
body: <p>MongoDB cumulatively append</p>
  
  <p>I have:</p>
  
  <pre><code>[
      {"d" : 20050511, "u" : 1},
      {"d" : 20050511, "u" : 2},
      {"d" : 20050511, "u" : 3},
      {"d" : 20050511, "u" : 4},
  
      {"d" : 20050512, "u" : 1},
      {"d" : 20050512, "u" : 2},
      {"d" : 20050512, "u" : 3},
      {"d" : 20050512, "u" : 7},
  
      {"d" : 20050513, "u" : 18},
      {"d" : 20050513, "u" : 21},
      {"d" : 20050513, "u" : 23}
  ]
  </code></pre>
  
  <p>I want to convert it to:</p>
  
  <pre><code>[
      {"d" : 20050511, "u" : 1},
      {"d" : 20050511, "u" : 2},
      {"d" : 20050511, "u" : 3},
      {"d" : 20050511, "u" : 4},
  
      {"d" : 20050512, "u" : 1},
      {"d" : 20050512, "u" : 2},
      {"d" : 20050512, "u" : 3},
      {"d" : 20050512, "u" : 7}
      {"d" : 20050512, "u" : 4},
  
      {"d" : 20050513, "u" : 18},
      {"d" : 20050513, "u" : 21},
      {"d" : 20050513, "u" : 23},
      {"d" : 20050513, "u" : 1},
      {"d" : 20050513, "u" : 2},
      {"d" : 20050513, "u" : 3},
      {"d" : 20050513, "u" : 4},
      {"d" : 20050513, "u" : 7}
  ]
  </code></pre>
  
  <p>Basically, I want to cumulatively move forward for each date (I am storing dates as ints); meaning for 20050512 it is distinct union of 20050511 + 20050512; for 20050513 it is distinct union of 20050512 + 20050513</p>
  
  <p>I tried some aggregation pipe lining, but with little success. I can do this in application layer but that will cause huge i/o and long time; since the data is for over 15 years and each date there are over 50K docs.</p>