title: MongoDB aggregation exceeded 16mb why?
body: <p>I'm using pymongo 3.5. The following query throws an exception telling me that the BSON Document exceeds the 16mb limit. 
  But how is it possible? 
  I've used the "$out" option to write the results in a collection instead of returning a value.</p>
  
  <pre><code>pipeline = [
          {"$match": match_dict},
          {"$unwind": "$fields"},
          {"$match": {
                  field_types[field_type]:field
          }},
          { "$group": {
                  "_id": "$fields.value",
                  "count" : {"$sum" : 1},
                  "unique_ids" : {"$addToSet" : {"doc":"$_id","items":"$items"}},
          }},
          {"$unwind": "$unique_ids"},
          { "$group": {
                  "_id": {"value":"$_id","count":"$count"},
                  "items_sum" : {"$sum" : "$unique_ids.items"}
          }},
          { "$project" : {
              "_id": {"value":"$_id.value"},
              "count" : "$_id.count",
              "items_sum" : "$items_sum"
          }},
          {"$out": collection_name}
  ]
  db.records.aggregate(pipeline,allowDiskUse=True)
  </code></pre>
  
  <p><a href="https://docs.mongodb.com/manual/core/aggregation-pipeline-limits/" rel="nofollow noreferrer">Here</a> says that during the pipeline processing, the documents may exceed this size.</p>
  
  <p>An example of the expected result is this:</p>
  
  <pre><code>[{"count": 79293, "_id": {"value": "06101"}, "items_sum": 111700},
  {"count": 47352, "_id": {"value": "06120"},...]
  </code></pre>
  
  <p>Data structure:</p>
  
  <pre><code>{
  "_id": &lt;the_doc_id&gt;
  "items": 30,
  "fields": [
      {
          "marc_field_subfield": "008",
          "value": "150504s2010    gr abe  r     000 0 gre|c"
      },
      {
          "marc_field_subfield": "020a",
          "value": "9789604751235"
      },
      {
          "marc_field_subfield": "020q"
          "value": "(\u03c7\u03b1\u03c1\u03c4\u03cc\u03b4.)"
      },
      {
          "marc_field_subfield": "040a"
          "value": "GR-KeIUL"
      },
      {
          "marc_field_subfield": "040a"
          "value": "EL"
      }
  ]
  }
  </code></pre>
  
  <p>The <code>field_types[field_type]:field</code> matches a specific <code>marc_field_subfield</code> which can exist more than one times in the fields array (e.g. "040a")</p>