title: Most efficient way to calculate pairwise similarity of 250k lists
body: <p>I have 250,000 lists containing an average of 100 strings each, stored across 10 dictionaries.  I need to calculate the pairwise similarity of all lists (the similarity metric isn't relevant here; but, briefly, it involves taking the intersection of the two lists and normalizing the result by some constant).  </p>
  
  <p>The code I've come up with for the pairwise comparisons is quite straightforward. I'm just using itertools.product to compare every list to every other list.  The problem is performing these calculations on  250,000 lists in a time-efficient way.   To anyone who's dealt with a similar problem: Which of the usual options (scipy, PyTables) is best for this in terms of the following criteria:</p>
  
  <ul>
  <li>supports python data types</li>
  <li>smartly stores a very sparse matrix (approx 80% of the values will be 0)</li>
  <li>efficient (can do the calculations in under 10 hours)</li>
  </ul>