title: How to create many data frames and combine them in one big data frame to avoid creating multiple variables
body: <p>I am scraping a HTML and storing data in a pandas dataframe. I need a loop for because the data in the html is in more than one url. My first idea was to create as many data frames as url I have, creating many variables but I have read that this is a bad idea. The solution I have read is to create a dict but I dont see how I can do this with data frames. I just want a final data frame with the information from the first row of the first data frame to the last row of the last dataframe.</p>
  <p>This is my code so far</p>
  <pre><code># To simulate I am a browser and send request to get the body of the response.
  header = {
    &quot;User-Agent&quot;: &quot;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36&quot;,
    &quot;X-Requested-With&quot;: &quot;XMLHttpRequest&quot;
  }
  # Coingecko containes 100 coins per page
  uncomplete_URL = &quot;https://www.coingecko.com/en?page=&quot;
  for page in  range (1,100): # Because I only want to analise the first 1000 coins
      complete_URL = uncomplete_URL + str(page)
      # To request info and then populate a pandas dataframe
      r = requests.get(complete_URL, headers=header)
      dfs = pd.read_html(r.text)
      df = pd.concat(df) # EDIT I forgot this line
      tmp_df.head()
  </code></pre>
  <p>Most of this code is taking from <a href="https://stackoverflow.com/questions/66609028/scraping-coin-gecko-with-pandas-report-http-error-403">here</a></p>