title: Run Web crawler in VSCode, got error: Cannot recover from stack overflow
body: <p>I wrote a web crawler, which is supposed to crawl through all hyperlinks under one domain name. I tried it with a very simple web page, it worked, but when I was trying it with a complicated website, it ran for half hour and finally returned this error Cannot recover from stack overflow.</p>
  
  <p>Can anybody help take a look at the main function, see if there is any thing wrong?</p>
  
  <pre><code>visited = set()                                                                     
  storedlinks = set()  
  
  def crawler(url):                                                                   
      'Take an url, crawl all links under that domain name.'
      visited.add(url)                                                                
      if linkchunk in url:                 # Check if under the domain                                                         
          storedlinks.add(url)                                                                
          links = analyze(url)             # analyze is function to get hyperlinks                                                    
  
          counter = 0
          for link in links:               # Check new links                                                   
              if link in visited:          # Check if it's visited
                  counter += 1         
                  if counter == len(links):  # if all link in visited, then stop
                      break
              else:                                                                       
                  try:                                                                    
                      print(link)
                      crawler(link)
                  except:                                                                 
                      pass
  
      return storedlinks           
  </code></pre>