title: Remove duplicate url's python
body: <p>I want to remove the duplicate url's from the file having list of url's. My bugun_url_given.txt has "<a href="http://www.bugun.com.tr/ara/Ak%20Parti/1" rel="nofollow">http://www.bugun.com.tr/ara/Ak%20Parti/1</a>" and it fetches all the url's and they are repeating..
  It saves all the unique url's in "bugun_url_collection.tx"
  here is my code:</p>
  
  <pre><code>from cookielib import CookieJar
  import urllib2
  import json
  from bs4 import BeautifulSoup
  cj = CookieJar()
  opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
  try:
      text_file = open('bugun_url_given.txt', 'r')
      for line in text_file:
          print line
          soup = BeautifulSoup(opener.open(line))
          links = soup.select('div.nwslist a')
          for link in links:
              print link
              #unique_url = set(map(lambda url : url.strip("/ "), links))
              with open('bugun_url_collection.txt', 'a') as f:
                  for link in links:
                      f.write(link.get('href') + '\n')
  except ValueError:
      pass               
  </code></pre>