title: Python - Best data structure for incredibly large matrix
body: <p>I need to create about 2 million vectors w/ 1000 slots in each (each slot merely contains an integer).</p>
  
  <p>What would be the best data structure for working with this amount of data? It could be that I'm over-estimating the amount of processing/memory involved.</p>
  
  <p>I need to iterate over a collection of files (about 34.5GB in total) and update the vectors each time one of the the 2-million items (each corresponding to a vector) is encountered on a line.</p>
  
  <p>I could easily write code for this, but I know it wouldn't be optimal enough to handle the volume of the data, which is why I'm asking you experts. :)</p>
  
  <p>Best,
  Georgina</p>