How to parse text in multiple body tags with BeautifulSoup in Python?
Question: I want to parse this website: <https://www.flyingv.cc/project/3724>

and I want fetch the information in the html source code.

Just like this information, 2830:

    
    
    <span class="sharenumber" id="fb_share_span">2830</span>
    

But when I use BeautifulSoup to extract text between span tags, it is totally
failed.

My python codes are:

    
    
    from bs4 import BeautifulSoup
    res = requests.get("https://www.flyingv.cc/project/3724") 
    soup = BeautifulSoup(res.text, "html.parser")
    test = soup.find('span', attrs={'id':'fb_share_span'})
    print test
    print test.txt
    

The output of python is:

    
    
    <span class="sharenumber" id="fb_share_span"></span> 
    None
    

How can I extract the information in the scenario? Furthermore, this website
has multiple body tags. So I suspect this is the reason why I failed, but I
don't know how to handle this kind problem.

Thank for your help, and sorry for that my English is very pool.


Answer: The data/value you want is not rendered by <https://www.flyingv.cc>. It's
received in this XHR (AJAX) call:

[https://api.facebook.com/restserver.php?format=json&method=links.getStats&urls=https://www.flyingv.cc/project/3724](https://api.facebook.com/restserver.php?format=json&method=links.getStats&urls=https://www.flyingv.cc/project/3724)

Which outputs something like this:

    
    
    [{
        "url": "https:\/\/www.flyingv.cc\/project\/3724",
        "normalized_url": "https:\/\/www.flyingv.cc\/project\/3724",
        "share_count": 466,
        "like_count": 1995,
        "comment_count": 369,
        "total_count": 2830,
        "click_count": 0,
        "comments_fbid": "673240602745001",
        "commentsbox_count": 0
    }]
    

So if you want to get every project's **`total_count`** value (which is a sum
of **share_count** \+ **like_count** \+ **comment_count**) just change
**`urls`** variable content in the above link.

* * *

This code could help:

    
    
    #coding:utf-8
    
    import json
    import requests
    
    url_prefix = "https://api.facebook.com/restserver.php?format=json&method=links.getStats&urls="
    project_url = "https://www.flyingv.cc/project/3724"
    response = requests.get("{}{}".format(url_prefix, project_url))
    
    data = json.loads(response.content)[0]
    print data.get('total_count', None)
    

Output:

    
    
    2830
    

