Scraping Table With Python/BS4
Question: Im trying to scrape the "Team Stats" table from <http://www.pro-football-
reference.com/boxscores/201602070den.htm> with BS4 and Python 2.7. However Im
unable to get anywhere close to it,

    
    
    url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm'
    page = requests.get(url)
    soup = BeautifulSoup(page.text, "html5lib")
    table=soup.findAll('table', {'id':"team_stats", "class":"stats_table"})  
    print table
    

I thought something like the above code would work but no luck.


Answer: The problem in this case is that the _"Team Stats" table is located inside a
comment_ in the HTML source which you download with `requests`. Locate the
comment and reparse it with `BeautifulSoup` into a "soup" object:

    
    
    import requests
    from bs4 import BeautifulSoup, NavigableString
    
    url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm'
    page = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'})
    
    soup = BeautifulSoup(page.content, "html5lib")
    comment = soup.find(text=lambda x: isinstance(x, NavigableString) and "team_stats" in x)
    
    soup = BeautifulSoup(comment, "html5lib")
    table = soup.find("table", id="team_stats")
    print(table)
    

And/or, you can load the table into, for example, a [`pandas`
dataframe](http://pandas.pydata.org/) which is very convenient to work with:

    
    
    import pandas as pd
    import requests
    from bs4 import BeautifulSoup
    from bs4 import NavigableString
    
    url = 'http://www.pro-football-reference.com/boxscores/201602070den.htm'
    page = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'})
    
    soup = BeautifulSoup(page.content, "html5lib")
    comment = soup.find(text=lambda x: isinstance(x, NavigableString) and "team_stats" in x)
    
    df = pd.read_html(comment)[0]
    print(df)
    

Prints:

    
    
                Unnamed: 0            DEN            CAR
    0          First Downs             11             21
    1         Rush-Yds-TDs        28-90-1       27-118-1
    2    Cmp-Att-Yd-TD-INT  13-23-141-0-1  18-41-265-0-1
    3         Sacked-Yards           5-37           7-68
    4       Net Pass Yards            104            197
    5          Total Yards            194            315
    6         Fumbles-Lost            3-1            4-3
    7            Turnovers              2              4
    8      Penalties-Yards           6-51         12-102
    9     Third Down Conv.           1-14           3-15
    10   Fourth Down Conv.            0-0            0-0
    11  Time of Possession          27:13          32:47
    

