title: Efficient matricization of a pandas dataframe
body: <p>My first StackOverflow question.</p>
  
  <p>So I have a Pandas DataFrame that looks sort of like this:</p>
  
  <pre><code>String1 String2 String3 value
  word1 word2 word3 5.6
  word4 word5 word6 123.4
  ...
  </code></pre>
  
  <p>This kind of DataFrame comes from a very long processing chain based on a huge amount of text.  (As a side note, I am getting close to memory limits and am considering HDFStores now.)</p>
  
  <p>Now I would like to do linear algebra operations based on a conversion of this table into a (Sparse?)Panel or some other kind of efficient data structure that fills in the blanks with 0s.  That is, I would like to create a table whose rows are String3s and whose columns are String1 x String2 pairs, and then do linear algebra operations on the rows.  However, I would also like to be able to do the same thing with any other column -- ie, take String1 as the rows, and make columns out of String2 x String3.   </p>
  
  <p>I've been experimenting with Panels and pivot tables, but they don't seem to be quite right, and they often overflow the memory.</p>
  
  <p>What's the right way to do this with Pandas or in Python (2.7) in general?</p>
  
  <p>Edited to add this example:</p>
  
  <p>The output table is going to look like this:</p>
  
  <pre><code>String1String2 (word1,word2) (word1,word5) (word4,word2) (word4,word5) ...
  String3
  word3 5.6 0 0 0 ...
  word6 0 0 0 123.4 ...
  </code></pre>
  
  <p>The number of columns is basically going to be |String1| x |String2|.
  Alternatively, String3 as columns and String1String2 as rows would be fine as well, since I can perform the operations on the column series.</p>
  
  <p>Edited further to add the memory issue:</p>
  
  <pre><code>In [1]: import pandas as pd
  
  In [2]: A = pd.load("file.df")
  
  In [3]: A 
  Out[3]: 
  &lt;class 'pandas.core.frame.DataFrame'&gt;
  Int64Index: 18506532 entries, 0 to 18506531
  Columns: 4 entries, 0 to value
  dtypes: float64(1), object(3)
  
  In [4]: B = A[A[1] == 'xyz']
  
  In [5]: C = B.pivot_table('value', [1,2], 0)
  </code></pre>
  
  <p>It crashes with MemoryError at line 160 in reshape.pyc.  This is version 0.11.0 of pandas.</p>