title: simple web crawler i need to eliminate the duplicate URL that present in the array
body: <p>i am using the array to store the URL  and i need to eliminate the URL the are present more than once in array because i don't need to crawl the same URL again:</p>
  
  <pre><code>self.level = []  # array where the URL are present 
  for link in self.soup.find_all('a'):
      self.level.append(link.get('href'))
      print(self.level)
  </code></pre>
  
  <p>i need to eliminate duplicate URL before crawling this URL.</p>