title: How to show huge amount of report data client side (mongoDB, JS)
body: <p>Due to the nature of the data that we collect in our application, its necessary to allow user to be able to search through 1,000,000+ rows of data, and receive results client side as fast as possible.</p>
  
  <p>We have played around with various techniques, and currently experimenting with the document storage engine MongoDB. We are essentially storing the report data in Mongo in the way we want to show it on the client side. </p>
  
  <p>But, this is where i wanted to lean on your experience. We have used some report table plugins such as datatables.net, but when a client requests data which may match 100,000+ records, the latency in delivering that data to the client side, can take well over a minute to export the JSON from server to client. </p>
  
  <p>I have tried filtering the resultset by only sending 1000 result at a time to the client, but what if the user decides to sort a column? The entire request has to be performed again, and the 1st 1000 results loaded into the client. </p>
  
  <p>But what if the client chooses to search within a column, the results have to be re-fetched again. All of these problems dont exist when <em>all</em> of the data has been loaded into the client side. </p>
  
  <p>So essentially my question is, how can you efficiently and speedily, allow clients to search and manipulate large datasets, but have the results delivered to the client side in a manner which doesnt require every result to be sent through in one go, therefore keeping the client side as lightweight as possible?</p>
  
  <p>Im sure there must be some front-end datatable type plugins for MongoDB, potentially via node.js for direct JSON communication, but i cant find anything!</p>
  
  <p>Thanks for your help. </p>