title: storing huge amounts of data in mongo
body: <p>I am working on a front end system for a radius server.</p>
  
  <p>The radius server will pass updates to the system every 180 seconds. Which means if I have about 15,000 clients that would be around 7,200,000 entries per day...Which is a lot.</p>
  
  <p>I am trying to understand what the best possible way to store and retrieve this data will be. Obviously as time goes on, this will become substantial. Will MongoDB handle this? Typical document is not much, something this</p>
  
  <pre><code>{
   id: 1
   radiusId: uniqueId
   start: 2017-01-01 14:23:23
   upload: 102323
   download: 1231556
  }
  </code></pre>
  
  <p>However, there will be MANY of these records. I guess this is something similar to the way that SNMP NMS servers handle data which as far as I know they use RRD to do this.</p>
  
  <p>Currently in my testing I just push every document into a single collection. So I am asking, </p>
  
  <p>A) Is Mongo the right tool for the job and
  B) Is there a better/more preferred/more optimal way to store the data</p>
  
  <p>EDIT:</p>
  
  <p>OK, so just incase someone comes across this and needs some help.</p>
  
  <p>I ran it for a while in mongo, I was really not satisfied with performance. We can chalk this up to the hardware I was running on, perhaps my level of knowledge or the framework I was using. However I found a solution that works very well for me. InfluxDB pretty much handles all of this right out of the box, its a time series database which is effectively the data I am trying to store (<a href="https://github.com/influxdata/influxdb" rel="nofollow noreferrer">https://github.com/influxdata/influxdb</a>). Performance for me has been like night &amp; day. Again, could all be my fault, just updating this.</p>
  
  <p>EDIT 2:</p>
  
  <p>So after a while I think I figured out why I never got the performance I was after with Mongo. I am using sailsjs as framework and it was searching by id using regex, which obviously has a huge performance hit. I will eventually try migrate back to Mongo instead of influx and see if its better.</p>