title: Cleaning scraped url in python
body: <p>I am writing a web scraper to scrape links from websites. It works fine but the output links are not clean. It outputs broken html links and also retrieves same html link. This is the code</p>
  
  <pre><code>links = re.findall('&lt;a class=.*?href="?\'?([^"\'&gt;]*)', sourceCode)
              for link in links:  
                  print link      
  </code></pre>
  
  <p>And this is how output looks like</p>
  
  <pre><code>/preferences?hl=en&amp;someting
  /preferences?hl=en&amp;someting
  /history/something
  /history/something
  /support?pr=something
  /support?pr=something
  http://www.web1.com/parameters
  http://www.web1.com/parameters
  http://www.web2.com/parameters
  http://www.web2.com/parameters
  </code></pre>
  
  <p>I tried cleaning links which are not html using this regex</p>
  
  <pre><code>link = re.findall('http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&amp;+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+', link)
                      print link
  </code></pre>
  
  <p>It cleans the url but adds square bracket to it. How to clean this to get without square bracket? How should I prevent printing the same url twice or multiple times</p>
  
  <pre><code>/preferences?hl=en&amp;someting -&gt; []
  http://www.web1.com/parameters -&gt; [http://www.web1.com/parameters]
  </code></pre>