title: efficient sort | uniq for the case of a large number of duplicates
body: <p><strong>Summary</strong> : is there a way to get the unique lines from a file and the number of occurrences more efficiently than using a <code>sort | uniq -c | sort -n</code>?</p>
  
  <p><strong>Details</strong>: I often pipe to <code>sort | uniq -c | sort -n</code> when doing log analysis to get a general trending of which log entries show up the most / least etc. This works most of the time - except when I'm dealing with a very large log file that ends up with a very large number of duplicates (in which case <code>sort | uniq -c</code> ends up taking a long time).</p>
  
  <p><strong>Example</strong>: The specific case I'm facing right now is for getting a trend from an 'un-parametrized' mysql bin log to find out which queries are run the most. For a file of a million entries which I pass through a grep/sed combination to remove parameters - resulting in about 150 unique lines  -  I spend about 3 seconds grepping &amp; sedding, and about 15s sorting/uniq'ing.  </p>
  
  <p>Currently, I've settled with a simple c++ program that maintains a map of &lt; line, count > - which does the job in less than a second - but I was wondering if an existing utility already exists.</p>