Trouble parsing HTML using BeautifulSoup
Question: I'm trying to use BeautifulSoup to parse some HTML in Python. Specifically,
I'm trying to create two arrays of soup objects: one for the dates of postings
on a website, and one for the postings themselves. However, when I use findAll
on the div class that matches the postings, only the initial tag is returned,
not the text inside the tag. On the other hand, my code works just fine for
the dates. What is going on??

    
    
    # store all texts of posts
    texts = soup.findAll("div", {"class":"quote"})
    
    # store all dates of posts
    dates = soup.findAll("div", {"class":"datetab"})
    

The first line above returns only

    
    
    <div class="quote">
    

which is not what I want. The second line returns

    
    
    <div class="datetab">Feb<span>2</span></div>
    

which IS what I want (pre-refining).

I have no idea what I'm doing wrong. [Here](http://harvardfml.com) is the
website I'm trying to parse. This is for homework, and I'm really really
desperate.


Answer: Which version of BeautifulSoup are you using? Version 3.1.0 [performs
significantly
worse](http://www.crummy.com/software/BeautifulSoup/3.1-problems.html) with
real-world HTML (read: invalid HTML) than 3.0.8. This code works with 3.0.8:

    
    
    import urllib2
    from BeautifulSoup import BeautifulSoup
    
    page = urllib2.urlopen("http://harvardfml.com/")
    soup = BeautifulSoup(page)
    for incident in soup.findAll('span', { "class" : "quote" }):
        print incident.contents
    

