title: Bash Script: count unique lines in file
body: <h1>Situation:</h1>
  
  <p>I have a large file (millions of lines) containing IP addresses and ports from a several hour network capture, one ip/port per line.  Lines are of this format:</p>
  
  <pre><code>ip.ad.dre.ss[:port]
  </code></pre>
  
  <h1>Desired result:</h1>
  
  <p>There is an entry for each packet I received while logging, so there are a lot of duplicate addresses.  I'd like to be able to run this through a shell script of some sort which will be able to reduce it to lines of the format</p>
  
  <pre><code>ip.ad.dre.ss[:port] count
  </code></pre>
  
  <p>where <code>count</code> is the number of occurrences of that specific address (and port).  No special work has to be done, treat different ports as different addresses.</p>
  
  <p>So far, I'm using this command to scrape all of the ip addresses from the log file:</p>
  
  <pre><code>grep -o -E [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+(:[0-9]+)? ip_traffic-1.log &gt; ips.txt
  </code></pre>
  
  <p>From that, I can use a fairly simple regex to scrape out all of the ip addresses that were sent by my address (which I don't care about)</p>
  
  <p>I can then use the following to extract the unique entries:</p>
  
  <pre><code>sort -u ips.txt &gt; intermediate.txt
  </code></pre>
  
  <p>I don't know how I can aggregate the line counts somehow with sort.</p>