title: Deleting duplicate text in Applescript
body: <h2>Applescript: Deleting Duplicates in a text file</h2>
  
  <p>I have a .txt file which I've created, and I'm trying to delete certain lines of text which are duplicates. </p>
  
  <p>The name I've given the file is today.txt, located on my desktop, and it contains a list of URLs for the NYtimes Today's Paper. However, through parsing an html file, I receive several urls that are duplicates, like so:</p>
  
  <pre><code>http://www.nytimes.com/2012/07/06/education/no-child-left-behind-whittled-down-under-obama.html
  http://www.nytimes.com/2012/07/06/business/global/markets-look-to-europes-central-bank-for-action.html
  http://www.nytimes.com/2012/07/06/business/global/markets-look-to-europes-central-bank-for-action.html
  http://www.nytimes.com/2012/07/06/nyregion/3-children-killed-in-long-island-boating-accident.html
  http://www.nytimes.com/2012/07/06/nyregion/3-children-killed-in-long-island-boating-accident.html
  http://www.nytimes.com/2012/07/06/world/americas/earthquake-relief-where-haiti-wasnt-broken.html
  http://www.nytimes.com/2012/07/06/world/americas/earthquake-relief-where-haiti-wasnt-broken.html
  http://www.nytimes.com/2012/07/06/us/politics/journal-critique-of-romney-shows-murdoch-doubt-on-candidacy.html
  http://www.nytimes.com/2012/07/06/us/politics/journal-critique-of-romney-shows-murdoch-doubt-on-candidacy.html
  http://www.nytimes.com/2012/07/06/technology/at-hacker-hostels-living-on-the-cheap-and-dreaming-of-digital-glory.html
  http://www.nytimes.com/2012/07/06/technology/at-hacker-hostels-living-on-the-cheap-and-dreaming-of-digital-glory.html
  </code></pre>
  
  <p>I've been trying to remove the duplicates through a <em>do shell script</em> in Applescript, but I haven't been able to make it work. Here is my code:</p>
  
  <pre><code>set delDups to do shell script "sort /Users/paolob/Desktop/today.txt | uniq -u"
  return delDups
  </code></pre>
  
  <p><strong>So my question</strong>:
  How can I delete the duplicates in my <em>today.txt</em> file,
  and then save the result to the same <em>today.txt</em> file</p>
  
  <p>Any help would be very appreciated. Thank you in advance.</p>
  
  <p><strong>Edit</strong><br>
  It would be more economical and faster if in fact the shell script, or whatever duplicate deleter you suggest, reads the text directly in Applescript Editor, and then sets the new text to a *new_text* variable, for example.</p>