title: How to concat thousands of pandas dataframes generated by a for loop efficiently?
body: <p>Thousands of dfs of consistent columns are being generated in a for loop reading different files, and I'm trying to merge / concat / append them into a single df, <code>combined</code>:</p>
  <pre><code>combined = pd.DataFrame()
  
  for i in range(1,1000): # demo only
      global combined
      generate_df() # df is created here
      combined = pd.concat([combined, df])
  </code></pre>
  <p>This is initially fast but slows as <code>combined</code> grows, eventually becoming unusably slow. <a href="https://stackoverflow.com/a/57001947/4107349">This answer on how to append rows</a> explains how adding rows to a dict and then creating a df is most efficient but I can't figure out how to do that with <code>to_dict</code>.</p>
  <p>What's a good way to to this? Am I approaching this the wrong way?</p>