python web scraping of a dynamically loading page
Question: Let's say i want to scrape this page: <https://twitter.com/nfl>

    
    
    from bs4 import BeautifulSoup
    import requests
    
    page = 'https://twitter.com/nfl'
    r = requests.get(page)
    soup = BeautifulSoup(r.text)
    print soup 
    

The more i scroll down on the page, the more results show up. But this above
request only gives me the initial load. How do i get all the information of
the page as if I were to manually scroll down?


Answer: First parse the `data-max-id="451819302057164799"` value from the html source.

Then using the id `451819302057164799` construct an url like below:

`https://twitter.com/i/profiles/show/nfl/timeline?include_available_features=1&include_entities=1&max_id=451819302057164799`

Now get the html source of the link and parse using `simplejson` or any other
json library.

Remember, the next page load(when you scroll down) is available from the value
`"max_id":"451369755908530175"` in that json.

