title: Best way to determine uniqueness and repetition in a text file
body: <p>I have a text file of about 20 million lines.  Each line is 25 characters long.  I estimate that there are probably about 200k-300k unique lines.  What I want to find out is exactly how many unique lines there are, and how many occurrences of each line there are (I expect the result to be power-law-esque).</p>
  
  <p>I could do this:</p>
  
  <pre><code>sort bigfile|uniq -c |sort -nr &gt; uniqcounts
  wc -l uniqcounts
  </code></pre>
  
  <p>but that is horribly inefficient memory and time-wise.</p>
  
  <p>What is your best command line solution to this problem?</p>