How to loop through a tags and redirect to retrieve more a tags?
Question: For educational purposes I am trying to write a program that would prompt the
user for "url", "count" and "position". The "url" will be scraped and "a tags"
within the "url" will be retrieved and this would yield a list of "a tags".
The "position" is then used to select a new link from the list of "a tags"
previously retrieved and use it as the new "url" to be scraped. "Count" is the
number of times this process takes place.

    
    
    Code:
    import urllib
    from bs4 import BeautifulSoup as bfs
    
    # Declare global variables
    href_list = []
    no_iterations = 0
    
    # Prompt user for input
    url = raw_input('Enter url - ')
    count = raw_input('Enter count - ')
    position = raw_input('Enter position - ')
    
    # While loop with condition
    while no_iterations != int(count):
        no_iterations += 1
    
        # Scraping the url 
        html = urllib.urlopen(url).read()
        soup = bfs(html)
    
        # Retrieve all of the anchor tags
        tags = soup('a')
        for tag in tags:
            href_list.append(tag.get('href', None))
    
        # Assiginig new url
        url = href_list[int(position)-1]
    
        # Printing info for user
        print 'Retrieving:', href_list[int(position)-1]
    print 'Last Url:', href_list[int(position)-1]
    

When I run the program here is what I get:

    
    
    Enter url - http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Fikret.html 
    Enter count - 4
    Enter position - 3 
    
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Montgomery.html
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Montgomery.html
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Montgomery.html
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Montgomery.html
    Last Url: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Montgomery.html
    

By observing the output, I can see that the URL is not reset as it should, any
advice is appreciated.


Answer: I solved by resetting the list were I stored the retrieved a tags Code:

    
    
    import urllib
    from bs4 import BeautifulSoup as bfs
    
    # Declare global variables
    href_list = []
    no_iterations = 0
    
    # Prompt user for input
    url = raw_input('Enter url - ')
    count = raw_input('Enter count - ')
    position = raw_input('Enter position - ')
    
    # While loop with condition
        while no_iterations != int(count):
        no_iterations += 1
    
        # Scraping the url 
        html = urllib.urlopen(url).read()
        soup = bfs(html)
    
        # Retrieve all of the anchor tags
        tags = soup('a')
        for tag in tags:
            href_list.append(tag.get('href', None))
    
        # Assiginig new url
        url = href_list[int(position)-1]
        href_list = []
        # Printing info for user
        print 'Retrieving:', href_list[int(position)-1]
    print 'Last Url:', url
    

So the new output now is:

    
    
    Enter url - http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Fikret.html 
    Enter count - 4
    Enter position - 3
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Montgomery.html
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Mhairade.html
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Butchi.html
    Retrieving: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Anayah.html
    Last Url: http://pr4e.dr-chuck.com/tsugi/mod/python-data/data/known_by_Anayah.html
    

Thanks for your support

