python urlib in loop
Question: my requirement is to read some page which has so many links available in order
i have to stop at suppose at 4th link and i have to read and connect to the
url at that particular link save the link contents in a list again the
connected link has so many links and i have to connected to the link at 4th
position again repeat this process for suppose 10 times and finally print the
names of the link connected

i am using this code urlllib is working only once

    
    
    import urllib
    from bs4 import *
    url = raw_input('enter url:')
    count = raw_input('enter count:')
    position = raw_input('enter position:')
    count = int(count)
    position = int(position)
    l = list()
    p = 0
    
    for _ in xrange(0,count):
        print 'retrieving:' + url
        html = urllib.urlopen(url).read()
        s = BeautifulSoup(html)
        tags = s.findAll('a')
        for tag in tags:
            w = tag.get('href')
            p = p + 1
            if p == position:
                url = "'" + w + "'"
                l.append(tag.contents[0])
    
    print l
    


Answer: Without knowing the particular site you're talking about this is just a guess,
but could it be that the links in the page you're interested in are relative
and not absolute? If that's the case when you reset url in the for loop then
it would be set to an incomplete link like /link.php instead of
<http://example.com/link.php> and urllib wouldn't know what to do with that.
If you expect all the links you could be interested in to be relative then
you'd need to add the base url before appending the new link for it follow.

