Python:Getting text from html using Beatifulsoup
Question: I am trying to extract the ranking text number from this link [link example:
kaggle user ranking no1](https://www.kaggle.com/titericz). More clear in an
image:

[![enter image description
here](http://i.stack.imgur.com/sClUu.png)](http://i.stack.imgur.com/sClUu.png)

I am using the following code:

    
    
    def get_single_item_data(item_url):
        sourceCode = requests.get(item_url)
        plainText = sourceCode.text
        soup = BeautifulSoup(plainText)
        for item_name in soup.findAll('h4',{'data-bind':"text: rankingText"}):
            print(item_name.string)
    
    item_url = 'https://www.kaggle.com/titericz'   
    get_single_item_data(item_url)
    

The result is `None`. The problem is that `soup.findAll('h4',{'data-
bind':"text: rankingText"})` outputs:

`[<h4 data-bind="text: rankingText"></h4>]`

but in the html of the link when inspecting this is like:

`<h4 data-bind="text: rankingText">1st</h4>`. It can be seen in the image:

[![enter image description
here](http://i.stack.imgur.com/8i76M.png)](http://i.stack.imgur.com/8i76M.png)

Its clear that the text is missing. How can I overpass that?

Edit: Printing the `soup` variable in the terminal I can see that this value
exists: [![enter image description
here](http://i.stack.imgur.com/BFyuz.png)](http://i.stack.imgur.com/BFyuz.png)

So there should be a way to access through `soup`.

Edit 2: I tried unsuccessfully to use the most voted answer from this
[stackoverflow question](http://stackoverflow.com/questions/24118337/fetch-
data-of-variables-inside-script-tag-in-python-or-content-added-from-js). Could
be a solution around there.


Answer: If you aren't going to try browser automation through `selenium` as @Ali
suggested, you would have to _parse the javascript containing the desired
information_. You can do this in different ways. Here is a working code that
locates the `script` by a [regular expression
pattern](http://www.crummy.com/software/BeautifulSoup/bs4/doc/#a-regular-
expression), then extracts the `profile` object, loads it with
[`json`](https://docs.python.org/2/library/json.html) into a Python dictionary
and prints out the desired ranking:

    
    
    import re
    import json
    
    from bs4 import BeautifulSoup
    import requests
    
    
    response = requests.get("https://www.kaggle.com/titericz")
    soup = BeautifulSoup(response.content, "html.parser")
    
    pattern = re.compile(r"profile: ({.*}),", re.MULTILINE | re.DOTALL)
    script = soup.find("script", text=pattern)
    
    profile_text = pattern.search(script.text).group(1)
    profile = json.loads(profile_text)
    
    print profile["ranking"], profile["rankingText"]
    

Prints:

    
    
    1 1st
    

