title: How do I create a scipy sparse matrix from a pandas dataframe?
body: <p>I'm searching for an better way to create a <a href="http://docs.scipy.org/doc/scipy-0.17.0/reference/sparse.html" rel="nofollow noreferrer">scipy sparse matrix</a> from a <a href="http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html" rel="nofollow noreferrer">pandas dataframe</a>.</p>
  
  <p>Here is the pseudocode for what I currently have</p>
  
  <pre><code>row = []; column = []; values = []
  for each row of the dataframe
      for each column of the row
          add the row_id to row
          add the column_id to column
          add the value to values
  sparse_matrix = sparse.coo_matrix((values, (row, column), shape=(max(row)+1,max(column)+1))
  </code></pre>
  
  <p>But I personally believe there would be a better way to do things. What almost worked was the following</p>
  
  <pre><code>dataframe.unstack().to_sparse().to_coo()
  </code></pre>
  
  <p>However, this returned me a triple of (sparse matrix, column ids, and row ids). The issue is that I need the row ids to actually be part of the sparse matrix.</p>
  
  <p>Here is a full example. I have a dataframe that looks like follows</p>
  
  <pre><code>          instructor_id  primary_department_id
  id
  4109           2093                    129
  6633           2093                    129
  6634           2094                    129
  6635           2095                    129
  </code></pre>
  
  <p>If I do the operation I mentioned above, I get</p>
  
  <pre><code>ipdb&gt; data = dataframe.unstack().to_sparse().to_coo()[0]
  ipdb&gt; data
  &lt;2x4 sparse matrix of type '&lt;type 'numpy.int64'&gt;'
      with 8 stored elements in COOrdinate format&gt;
  ipdb&gt; print data
    (0, 0)    2093
    (0, 1)    2093
    (0, 2)    2094
    (0, 3)    2095
    (1, 0)    129
    (1, 1)    129
    (1, 2)    129
    (1, 3)    129
  </code></pre>
  
  <p>But I need</p>
  
  <pre><code>ipdb&gt; print data
    (4109, 0) 2093
    (6633, 0) 2093
    (6634, 0) 2094
    etc.
  </code></pre>
  
  <p>I am open to using any additional libraries or dependencies.</p>
  
  <p>There seems to be a <a href="https://stackoverflow.com/questions/17818783/populate-a-pandas-sparsedataframe-from-a-scipy-sparse-matrix">question that asks for the reverse operation</a> but I haven't found a solution for this operation.</p>