While Loops Problem with Python Twitter Crawler
Question: I'm continuing writing my twitter crawler and am running into more problems.
Take a look at the code below:

    
    
    from BeautifulSoup import BeautifulSoup
    import re
    import urllib2
    
    url = 'http://mobile.twitter.com/NYTimesKrugman'
    
    def gettweets(soup):
        tags = soup.findAll('div', {'class' : "list-tweet"})#to obtain tweet of a follower
        for tag in tags: 
            print tag.renderContents()
            print ('\n\n')
    
    def are_more_tweets(soup):#to check whether there is more than one page on mobile 
        links = soup.findAll('a', {'href': True}, {id: 'more_link'})
        for link in links:
            b = link.renderContents()
            test_b = str(b)
            if test_b.find('more'):
                return True
            else:
                return False
    
    def getnewlink(soup): #to get the link to go to the next page of tweets on twitter 
        links = soup.findAll('a', {'href': True}, {id : 'more_link'})
        for link in links:
            b = link.renderContents()
            if str(b) == 'more':
                c = link['href']
                d = 'http://mobile.twitter.com' +c
                return d
    
    def checkforstamp(soup): # the parser scans a webpage to check if any of the tweets are   
        times = soup.findAll('a', {'href': True}, {'class': 'status_link'})
        for time in times:
            stamp = time.renderContents()
            test_stamp = str(stamp)
            if test_stamp.find('month'): 
                return True
            else:
                return False
    
    
    response = urllib2.urlopen(url)
    html = response.read()
    soup = BeautifulSoup(html)
    gettweets(soup)
    stamp = checkforstamp(soup)
    tweets = are_more_tweets(soup)
    print 'stamp' + str(stamp)
    print 'tweets' +str (tweets)
    while (not stamp) and tweets: 
        b = getnewlink(soup)
        print b
        red = urllib2.urlopen(b)
        html = red.read()
        soup = BeautifulSoup(html)
        gettweets(soup)
        stamp = checkforstamp(soup)
        tweets = are_more_tweets(soup)
    print 'done' 
    

The code works in the following way: For a single user NYTimesKrugman -I
obtain all tweets on a single page(gettweets) -provided more tweets exist(are
more tweets) and that I haven't obtained a month of tweets yet(checkforstamp),
I get the link for the next page of tweets -I go to the next page of tweets
(entering the while loop) and continue the process until one of the above
conditions is violated

However, I have done extensive testing and determined that I am not actually
able to enter the while loop. Rather, the program is not doing so. This is
strange, because my code is written such that tweets are true and stamp should
yield false. However, I'm getting the below results: I am truly baffled!

    
    
    <div>
    <span>
    <strong><a href="http://mobile.twitter.com/nytimeskrugman">NYTimeskrugman</a></strong>
    <span class="status">What Would I Have Done? <a rel="nofollow"   href="http://nyti.ms/nHxb8L" target="_blank" class="twitter_external_link">http://nyti.ms/nHxb8L</a></span>
    </span>
    <div class="list-tweet-status">
    <a href="/nytimeskrugman/status/98046724089716739" class="status_link">3 days ago</a>
    </div>
    <div class="list-tweet-actions">
    </div>
    </div>
    
    
    
    
    stampTrue
    tweetsTrue
    done
    >>> 
    

If someone could help that'd be great. Why can I not get more than 1 page of
tweets? Is my parsing in checkstamp being done incorrectly? Thanx.


Answer: Your `checkforstamp` function returns non-empty, defined strings:

    
    
    return 'True'
    

So `(not stamp)` will always be false.

Change it to return booleans like `are_more_tweets` does:

    
    
    return True
    

and it should be fine.

For reference, see the [boolean
operations](http://docs.python.org/reference/expressions.html#boolean-
operations) documentation:

> In the context of Boolean operations, and also when expressions are used by
> control flow statements, the following values are interpreted as false:
> False, None, numeric zero of all types, and empty strings and containers
> (including strings, tuples, lists, dictionaries, sets and frozensets). **All
> other values are interpreted as true.**
>
> ...
>
> The operator **not** yields True if its argument is false, False otherwise.

Edit:

Same problem with the `if` test in `checkforstamp`. Since `find('substr')`
returns `-1` when the substring is not found, `str.find('substr')` in boolean
context will be `True` if there is no match according to the rules above.

That is not the only place in your code where this problem appears. Please
review all your tests.

