title: Comparing numpy array with itself by element efficiently
body: <p>I am performing a large number of these calculations:</p>
  
  <p><code>A == A[np.newaxis].T</code></p>
  
  <p>where A is a dense numpy array which frequently has common values.</p>
  
  <p>For benchmarking purposes we can use:</p>
  
  <pre><code>n = 30000
  A = np.random.randint(0, 1000, n)
  A == A[np.newaxis].T
  </code></pre>
  
  <p>When I perform this calculation, I run into memory issues. I believe this is because the output isn't in more efficient bitarray or np.packedbits format. A secondary concern is we are performing twice as many comparisons as necessary, since the resulting Boolean array is symmetric.</p>
  
  <p>The questions I have are:</p>
  
  <ol>
  <li>Is it possible to produce the Boolean numpy array output in a more memory efficient fashion without sacrificing speed? The options I know about are bitarray and np.packedbits, but I only know how to apply these after the large Boolean array is created.</li>
  <li>Can we utilise the symmetry of our calculation to halve the number of comparisons processed, again without sacrificing speed?</li>
  </ol>
  
  <p>I will need to be able to perform &amp; and | operations on Boolean arrays output. I have tried bitarray, which is super-fast for these bitwise operations. But it is slow to pack np.ndarray -> bitarray and then unpack bitarray -> np.ndarray.</p>
  
  <p>[Edited to provide clarification.]</p>