Adding counter to my Python Web Crawler
Question: I've made a web crawler which gives a link and text from link for all sites in
given addres it looks like this:

    
    
    import urllib
    from bs4 import BeautifulSoup
    import urlparse
    import mechanize
    
    url = ["http://adbnews.com/area51"]
    
    
    for u in url:
        br = mechanize.Browser()
        urls = [u]
        visited = [u]
        i = 0
        while i<len(urls):
            try:
                br.open(urls[0])
                urls.pop(0)
    
                for link in br.links():
    
                    levelLinks = []
                    linkText = [] 
    
                    newurl = urlparse.urljoin(link.base_url, link.url)
                    b1 = urlparse.urlparse(newurl).hostname
                    b2 = urlparse.urlparse(newurl).path
                    newurl = "http://"+b1+b2
                    linkTxt = link.text
                    linkText.append(linkTxt)
                    levelLinks.append(newurl)
    
    
                    if newurl not in visited and urlparse.urlparse(u).hostname in newurl:
                        urls.append(newurl)
                        visited.append(newurl)
                        #print newurl
    
                        #get Mechanize Links
                        for l,lt in zip(levelLinks,linkText):
                            print newurl,"\n",lt,"\n"
    
    
            except:
                urls.pop(0)
    

it gets results like that:

    
    
    http://www.adbnews.com/area51/contact.html 
    CONTACT 
    
    http://www.adbnews.com/area51/about.html 
    ABOUT 
    
    http://www.adbnews.com/area51/index.html 
    INDEX 
    
    http://www.adbnews.com/area51/1st/ 
    FIRST LEVEL! 
    
    http://www.adbnews.com/area51/1st/bling.html 
    BLING 
    
    http://www.adbnews.com/area51/1st/index.html 
    INDEX 
    
    http://adbnews.com/area51/2nd/ 
    2ND LEVEL 
    

And I wanna add a counter of somekind which could limit how deep crawler
goes..

I've tried add for example `steps = 3` and change `while i<len(urls)` in
`while i<steps:`

but that would go only to first level even the number says 3...

Any advice is welcome


Answer: If you want to search a certain "depth", consider using a recursive function
instead of just appending a list of URL's.

    
    
    def crawl(url, depth):
      if depth <= 3:
        #Scan page, grab links, title
        for link in links:
          print crawl(link, depth + 1)
      return url +"\n"+ title
    

This allows for easier control of your recursive searching, as well as being
faster and less resource heavy :)

