title: How to load all entries in an infinite scroll at once to parse the HTML in python
body: <p>I am trying to extract information from <a href="https://medium.com/top-100/december-2013" rel="noreferrer">this page</a>. The page loads 10 items at a time, and I need to scroll to load all entries (for a total of 100). I am able to parse the HTML and get the information that I need for the first 10 entries, but I want to fully load all entries before parsing the HTML.</p>
  
  <p>I am using python, requests, and BeautifulSoup. The way I parse the page when it loads with the first 10 entries is as follows:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  import requests
  s = requests.Session()
  r = s.get('https://medium.com/top-100/december-2013')
  page = BeautifulSoup(r.text)
  </code></pre>
  
  <p>But this only loads the first 10 entries. So I looked at the page and got the AJAX request used to load the subsequent entries and I get a response but it's in the a funky JSON and I'd rather use the HTML parser instead of parsing JSON. Here's the code:</p>
  
  <pre><code>from bs4 import BeautifulSoup
  import requests
  import json
  s = requests.Session()
  url = 'https://medium.com/top-100/december-2013/load-more'
  payload = {"count":100}
  r = s.post(url, data=payload)
  page = json.loads(r.text[16:]) #skip some chars that throw json off
  </code></pre>
  
  <p>This gives me the data but it's in a very long and convoluted JSON, I would much rather load all the data on the page and simply parse the HTML. In addition, the rendered HTML provides more information than the JSON response (i.e. the name of the author instead of obscure userID, etc.) There was a similar question <a href="https://stackoverflow.com/questions/19512250/parse-html-infinite-scroll">here</a> but no relevant answers. Ideally I want to make the POST call and <em>then</em> request the HTML and parse it, but I haven't been able to do that.</p>