title: Efficient pairwise comparison of rows in pandas DataFrame
body: <p>I am currently working with a smallish dataset (about 9 million rows). Unfortunately, most of the entries are strings, and even with coercion to categories, the frame sits at a few GB in memory.</p>
  
  <p>What I would like to do is compare each row with other rows and do a straight comparison of contents. For example, given</p>
  
  <pre><code>   A   B     C      D
  0 cat blue  old Saturday
  1 dog red   old Saturday
  </code></pre>
  
  <p>I would like to compute</p>
  
  <pre><code>      d_A   d_B   d_C   d_D
  0, 0  True  True  True  True
  0, 1  False False True  True
  1, 0  False False True  True
  1, 1  True  True  True  True
  </code></pre>
  
  <p>Obviously, combinatorial explosion will preclude a comparison of <em>every</em> record with every other record. So we can instead use blocking, by applying groupby, say on column A.</p>
  
  <p>My question is, is there a a way to do this in either pandas or dask, that is faster than the following sequence:</p>
  
  <ol>
  <li>Group by index</li>
  <li>Outer join each group to itself to produce pairs</li>
  <li>dataframe.apply comparison function on each row of pairs</li>
  </ol>
  
  <p>For reference, assume I have access to a good number of cores (hundreds), and about 200G of memory.</p>