title: Representing a sparse matrix in Python and storing to disk
body: <p>I have a large number of time series (millions) of varying length that I plan to do a clustering analysis on (probably using the sklearn implementation of kmeans).</p>
  
  <p>For my purposes I need to align the time series (such that the maximum value is centered, pad them with zeros (so they are all the same length), and normalize them before I can do the clustering analysis. So as a trivial example, something like:</p>
  
  <pre><code>[5, 0, 7, 10, 6]
  </code></pre>
  
  <p>Would become something like</p>
  
  <pre><code>[0, 0.5, 0, 0.7, 1, 0.6, 0, 0, 0]
  </code></pre>
  
  <p>In the real data, the raw time series are of length 90, and the padded/aligned/normed time series are of length 181. Of course, we have lots of zeros here, so a sparse matrix seems the ideal way of storing the data.</p>
  
  <p>Based on this, I have two related questions:</p>
  
  <p>1 - How best to store these in memory? My current, inefficient method is to calculate the <em>dense</em> normed/aligned/padded matrix for each time series and write to a simple text file for storage purposes, then separately read that data into a scipy sparse lil matrix:</p>
  
  <pre><code>rows, columns = N, 181
  matrix = scipy.sparse.lil_matrix( (rows, columns) )
  
  for i,line in enumerate(open(file_containing_dense_matrix_data)):
      # The first two values in each line are metadata
      line = map(float,line.strip().split(',')[2:])
  
  matrix[i]=line
  </code></pre>
  
  <p>This is both slow and more memory intensive than I had hoped. Is there a preferred method?</p>
  
  <p>2 - Is there a better way to store the time series on disk? I have yet to find an efficient means to write the data to disk directly as a sparse matrix that I can read (relatively) quickly into memory at a later time.</p>
  
  <p><strong>My ideal response here is a method that addresses both questions, i.e. a method to store the dense matrix rows directly into a sparse data structure, and to efficiently read/write the data to/from disk.</strong></p>