Web Scraping a Forum Post in Python Using Beautiful soup and lxml Cannot get all posts
Question: Im having an issue that is driving me absolutely crazy. I am a newbie to web
scraping, and I am practicing web scraping by trying to scrape the contents of
a forum post, namely the actual posts people made. I have isolated the posts
to what i think contains the text which is div id="post message_ 2793649 (see
attached Screenshot_1 for better representation of the
html)[Screenshot_1](http://i.stack.imgur.com/6L3zf.png)

The example above is just one of many posts. Each post has its own unique
identifier number, but the rest is consistent as div id="post_message_.

here is what I am stuck at currently

    
    
    import requests
    from bs4 import BeautifulSoup
    import lxml
    
    r = requests.get('http://www.catforum.com/forum/43-forum-fun/350938-count-one-    billion-2016-a-120.html')
    
    soup = BeautifulSoup(r.content)
    
    data = soup.find_all("td", {"class": "alt1"})
    
    for link in data:
        print(link.find_all('div', {'id': 'post_message'}))
    

the above code just creates a bunch of empty lists that go down the page its
so frustrating. (See Screenshot_2 for the code that I ran with its output next
to it) [Screenshot_2](http://i.stack.imgur.com/nBcOl.png) What am I missing.

The end result that I am looking for is just all the contents of what people
said contained in a long string without any of the html clutter.

I am using Beautiful Soup 4 running the lxml parser


Answer: There's nothing with the id `post_message`, so `link.find_all` returns an
empty list. You'll first want to grab all of the ids within all the `div`s,
and then filter that list of ids with a regex (e.g.) to get only those that
start with `post_message_` and then a number. Then you can do

    
    
    for message_id in message_ids:
        print(link.find_all('div', {'id': message_id}))
    

