title: How to store billions of JSON files and query them
body: <p>I currently have an API which accepts JSON files(which are JSON serialised objects which contains some user transaction data) and stores the same into the server. Every such JSON file has a unique global id and a unique user to which it is associated. There are billions of such files generated every day. The user should then be able to query through all JSON files that are associated to him and produce a bunch of aggregated results calculated on top of those files.</p>
  
  <p>A typical JSON file that needs to be stored looks something like:</p>
  
  <pre><code>[ { "currencyCode" : "INR",
  "receiptNumber" : { "value" : "1E466GDX5X2C" },
  "retailTransaction" : [ { "grandTotal" : 90000.0,
        "lineItem" : [ { "otherAttributes" : {  },
              "sale" : { "description" : "Samsung galaxy S3",
                  "discountAmount" : { "currency" : "INR",
                      "value" : 2500
                    },
                  "itemSubType" : "SmartPhone",
                  "otherAttributes" : {  },
                  "unitCostPrice" : { "quantity" : 1,
                      "value" : 35000
                    }
                },
              "sequenceNumber" : 1000
            },
            { "customerOrderForPickup" : { "description" : "iPhone5",
                  "discountAmount" : { "currency" : "INR",
                      "value" : 5000
                    },
                  "itemSubType" : "SmartPhone",
                  "otherAttributes" : {  },
                  "unitCostPrice" : { "quantity" : 1,
                      "value" : 55000
                    }
                },
              "otherAttributes" : {  },
              "sequenceNumber" : 1000
            }
          ],
        "otherAttributes" : {  },
        "reason" : "Delivery",
        "total" : [ { "otherAttributes" : {  },
              "type" : "TransactionGrossAmount",
              "value" : 35000
            } ]
      },
      null
    ],
  "sequenceNumber" : 125435,
  "vatRegistrationNumber" : "10868758650"
  } ]
  </code></pre>
  
  <p>The above JSON is the serialised version of a complex object containing single or array of Objects of other classes as attributes. So the 'receiptNumber' is the universal id of the JSON file.</p>
  
  <p>I would need to query stuff like quantity and value of the customerOrderForPickup or the grandTotal of the transaction, and in as an aggegate of various such transaction JSONs **</p>
  
  <p>I would like to have some suggestion as to how to go about: 1) Storing these JSON files on the server, the file system ie 2) What kind of a database should I use to query through these JSON files with such a complex structure</p>
  
  <p>My research has resulted in a couple of possibilities: 1) Use a MongoDB database to store the JSON representatives of the object and query through the database. How would the JSON files be stored? What will be the best way to store the transaction JSONs in the MongoDB database? 2) Couple a SQL database containing the unique global id, user id and the address of the JSON file on the server, with an aggregating code on those files. I doubt if this can be scaled</p>
  
  <p>Would be glad if someone has any insights on the problem. Thanks.</p>