title: Href extraction by Web crawler using Python
body: <p>I've written a code which extracts all the links from the url specified. I took this idea from an online video tutorial. When I tried with nytimes.com, if worked out. But when I tried with yell.com, I had an error thrown: "Error: HTTP Error 416: Requested Range Not Satisfiable - <a href="http://www.yell.com/" rel="nofollow">http://www.yell.com/</a>". What technique should I adopt in order to bypass this.</p>
  
  <pre><code>import urllib.parse;
  import urllib;
  from bs4 import BeautifulSoup;
  
  ##url = "http://nytimes.com";
  url = "http://www.yell.com/";
  
  urls = [url];   
  visited = [url];
  
  while(len(urls) &gt; 0):
  
      try:
          htmltext = urllib.request.urlopen(urls[0]).read();
  
          soup = BeautifulSoup(htmltext);
  
          urls.pop(0);
          print(len(urls));
  
          for tag in soup.findAll('a',href=True) :
              tag['href'] = urllib.parse.urljoin(url,tag['href']);
              if(url in tag['href'] and tag['href'] not in visited) :
                  urls.append(tag['href']);
                  visited.append(tag['href']);
  
      except urllib.error.HTTPError as e:
          print("Error: " + str(e)
                + " - " + url);
  
  print(visited);
  </code></pre>