title: Removing duplicates: python results differ from sort -u
body: <p>I have a very long text file (2GB) and I removed duplicates using:</p>
  
  <pre><code>sort -u filename &gt; outfile1
  </code></pre>
  
  <p>and</p>
  
  <pre><code>&gt;&gt;&gt; data = open('filename', 'r').readlines()
  &gt;&gt;&gt; u = list(set(data))
  &gt;&gt;&gt; open('outfile2', 'w').writelines(u)
  </code></pre>
  
  <p>However the two files outfile2 and outfile1 have a different number of entries:</p>
  
  <pre><code>wc -l outfile?
   185866729 filename
   109608242 outfile1
   109611085 outfile2
  </code></pre>
  
  <p>How is this possible?</p>
  
  <p>UPDATE.
  Following up on the request to see the data, I have found that python will remove as duplicates entries like:</p>
  
  <pre><code> låsningernes
   læsningernes
   løsningernes
  </code></pre>
  
  <p>Effectively the second character is ignored in <code>sort -u</code>, and only the first entry is kept. Python instead does a good job of distinguishing the three records.</p>