title: Efficient and accurate way to compact and compare Python lists?
body: <p>I'm trying to a somewhat sophisticated diff between individual rows in two CSV files. I need to ensure that a row from one file does not appear in the other file, but I am given <strong>no guarantee of the order of the rows</strong> in either file. As a starting point, I've been trying to compare the hashes of the string representations of the rows (i.e. Python lists). For example:</p>
  
  <pre><code>import csv
  
  hashes = []
  for row in csv.reader(open('old.csv','rb')):
    hashes.append( hash(str(row)) )
  
  for row in csv.reader(open('new.csv','rb')):
    if hash(str(row)) not in hashes:
      print 'Not found'
  </code></pre>
  
  <p>But this is failing miserably. I am constrained by artificially imposed memory limits that I cannot change, and thusly I went with the hashes instead of storing and comparing the lists directly. Some of the files I am comparing can be <strong>hundreds of megabytes</strong> in size. Any ideas for a way to accurately compress Python lists so that they can be compared in terms of simple equality to other lists? I.e. a hashing system that actually works? <em>Bonus points</em>: why didn't the above method work?</p>
  
  <p><strong>EDIT:</strong></p>
  
  <p>Thanks for all the great suggestions! Let me clarify some things. "Miserable failure" means that two rows that have the exact same data, after being read in by the <code>CSV.reader</code> object are not hashing to the same value after calling <code>str</code> on the list object. I shall try <code>hashlib</code> at some suggestions below. I also cannot do a hash on the raw file, since two lines below contain the same data, but different characters on the line:</p>
  
  <pre><code>1, 2.3, David S, Monday
  1, 2.3, "David S", Monday
  </code></pre>
  
  <p>I am also already doing things like string stripping to make the data more uniform, but it seems to no avail. I'm not looking for an extremely smart diff logic, i.e. that <code>0</code> is the same as <code>0.0</code>. </p>
  
  <p><strong>EDIT 2:</strong></p>
  
  <p>Problem solved. What basically worked is that I needed to a bit more pre-formatting like converting ints and floats, and so forth <em>AND</em> I needed to change my hashing function. Both these changes seemed to do the job for me.</p>