title: Looping through unique lines in a txt file with Bash
body: <p>I am looping through tab-delimited lines in a txt file. This txt file is the output of an xml/xslt process and has duplicates. Below I am looking for a solution with the txt file, but solutions using XSLT are just as appreciated. Please see example txt file.</p>
  
  <p>txtfile.txt: line 3 is a duplicate of line 1</p>
  
  <pre><code>hello@example.com  running  1111
  puppy@kennel.com   running  9876
  hello@example.com  running  1111
  husky@siberia.com  shutdown 1234
  puppy@kennel.com   running  9876
  hello@example.com  running  1111
  </code></pre>
  
  <p><strong>My question is:</strong> Can duplicate lines be skipped in a loop so that the loop only processes unique lines? In this case, how to configure to loop lines 1, 2, 4 and skip lines 3, 5, 6? </p>
  
  <p>My current working loop which reads duplicates:</p>
  
  <pre><code>while read name status num
  do
      echo "&lt;tag1&gt;"
      echo "&lt;tag2&gt;"$name"&lt;/tag2&gt;"
      echo "&lt;tag3&gt;"$status"&lt;/tag3&gt;"
      echo "&lt;tag2&gt;"$num"&lt;/tag2&gt;"
      echo "&lt;/tag1&gt;"
  
  done &lt; txtfile.txt
  </code></pre>
  
  <p>In my txtfile there are hundreds of lines and nearly half are duplicates, so this is a huge problem for me! Any ideas/solutions appreciated. Thanks in Advance.</p>