title: Web Scraper - getting duplicates in output
body: <p>I am completely new to Python and just trying my coding skills for developing a few programs</p>
  
  <p>I have coded the following program in Python 2.7 to fetch the profile URLs from the dir - <a href="http://www.uschirodirectory.com/entire-directory/list/alpha/a.html" rel="nofollow">http://www.uschirodirectory.com/entire-directory/list/alpha/a.html</a></p>
  
  <p>However, I am noticing a lot of duplicate entries in the list of URLs fetched. Could someone please review the code and tell me if there's something that I am doing here or is there a way this code could be optimized further.</p>
  
  <p>Many thanks</p>
  
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  def web_crawler(max_pages):
  p = '?site='
  page = 1
  alpha = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']
  while page &lt;= max_pages:
      for i in alpha:
          url = 'http://www.uschirodirectory.com/entire-directory/list/alpha/' + str(i) + '.html' + p + str(page)
          code = requests.get(url)
          text = code.text
          soup = BeautifulSoup(text)
          for link in soup.findAll('a',{'class':'btn'}):
              href = 'http://www.uschirodirectory.com' + link.get('href')
              print(href)
      page += 1
  i += alpha[0 + 1]
  
  #Run the crawler
  web_crawler
  </code></pre>