title: How do I go about auto expanding an entire webpage for scraping with pandas (python)?
body: <p>I am trying to scrape html tables from <a href="https://coinmarketcap.com/exchanges/binance/" rel="nofollow noreferrer">https://coinmarketcap.com/exchanges/binance/</a> . On the page, there is a &quot;load more&quot; button. When this is clicked, more rows are displayed, but the URL doesn't change. When I pass this URL to pandas using <code>pd.read_html(url(</code>, it pulls the first 100 rows and nothing else. How do I auto load all the tables, either through the URL or through a command? Any help is appreciated.
  Code:</p>
  <pre class="lang-py prettyprint-override"><code>import json
  import requests
  import pandas as pd
  from bs4 import BeautifulSoup
  import lxml
  import html5lib
  
  
  url = &quot;https://coinmarketcap.com/exchanges/binance/&quot;
  
  
  df = pd.read_html(url)
  pd.set_option(&quot;display.max_rows&quot;, None, &quot;display.max_columns&quot;, None)
  print(df)
  </code></pre>