beautifulsoup for subreddits
Question: I have been trying to learn some html parsing with BeautifulSoup and tried to
get it work for reddit. Here is my code,

    
    
    !/usr/bin/python
    
    import BeautifulSoup
    from BeautifulSoup import BeautifulSoup as BSoup
    import os, re, sys, math, os.path, urllib, string, random, time
    
    url = urllib.urlopen(sys.argv[1]).read()
        soup= BSoup(url)
    
    links = []
    for link in soup.findAll('a',attrs={'class':'comments may-blank'}):
        links.append(link.get("href"))
    
    print links
    

I have tested the code successfully for r/gaming and r/worldnews but the code
fails for r/gifs. I have also verified that the same class is used for all the
subreddits. Plus I have tried with just

    
    
    for link in soup.findAll('a'):
    

but still the code fails to find the hyperlink. Any suggestion on why this
happens and how to make the code work with all subreddits.


Answer: If you are doing this too often you will be met with this.

    
    
    As a reminder to developers, we recommend that clients make no
    more than <a href="http://github.com/reddit/reddit/wiki/API">one
    request every two seconds</a> to avoid seeing this message.
    

Reddit does this to prevent abuse by spiders and crawlers.

Either space your requests OR much much more preferably use their Python api
[PRAW](https://praw.readthedocs.org/en/v2.1.16/) :

