title: Is this an appropriate use of python's built-in hash function?
body: <p>I need to compare large chunks of data for equality, and I need to compare many pairs per second, <em>fast</em>. Each object is guaranteed to be the same length, it is possible and likely that there may only be slight differences located at unknown positions.</p>
  
  <p>Timings below show that using <code>==</code> operator is very fast if there is a difference near the start of the data, and significantly slower if differences are located towards the end.</p>
  
  <pre><code>&gt;&gt;&gt; import os
  &gt;&gt;&gt; s = os.urandom(1600*1200 - 1)
  &gt;&gt;&gt; Aimg = b"A" + s
  &gt;&gt;&gt; Bimg = b"B" + s
  &gt;&gt;&gt; img1 = s + b"1"
  &gt;&gt;&gt; img2 = s + b"2"
  &gt;&gt;&gt; %timeit Aimg == Bimg
  61.8 ns ± 0.484 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)
  &gt;&gt;&gt; %timeit img1 == img2
  159 µs ± 2.83 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  </code></pre>
  
  <p>In my use case, differences may be located towards the middle or the end of the bytes (context: it is uncompressed image data). I looked for a way to speed things up using a hash or checksum. Using md5 was slower but Python's built-in <code>hash</code> did actually speed things up.</p>
  
  <pre><code>&gt;&gt;&gt; %timeit img1 == img2
  160 µs ± 5.96 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
  &gt;&gt;&gt; %timeit hash(img1) == hash(img2) and img1 == img2
  236 ns ± 5.91 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
  </code></pre>
  
  <p>I'm interested in technical detail of this hash, is it sufficiently hash-like that when <code>hash(a) == hash(b)</code> then <code>a == b</code> is <em>very likely</em>? False positives are acceptable if a hash collision is reasonably rare, the intention is a fast-path to speed up comparisons in the average case.</p>