title: Dataframes to CSV (on a loop) taking a long time
body: <p>Please be aware that I installed Python 2 days ago and wrote my first two codes copying and pasting other codes I found, changing some things here and there.</p>
  <p>I have a code that:
  1- Creates a Pandas Dataframe
  2- Loop some data, creating new dataframes that are &quot;merged&quot; (not sure this is the right word) to the Dataframe created in step 1</p>
  <p>On my first code, Step 1 created Dataframe df0 and on Step 2, it created a Dataframe df that was merged to df0 using:</p>
  <pre><code>df0=df0.append(df, ignore_index=True)
  </code></pre>
  <p>The final Dataframe was written in a CSV file. But that was taking a long time (around 3 hours) and Python tells me that pandas.append will be suspended soon.</p>
  <p>On my second code, Step 1 creates a Dataframe and writes it directly to a CSV suing:</p>
  <pre><code>df.to_csv('file.csv',mode='w',index=False,header=True)
  </code></pre>
  <p>and Step 2 creates a new dataframe on each step of the loop and writes it to the CSV file created in Step 1 using:</p>
  <pre><code>df.to_csv('file.csv',mode='a',index=False,header=False)
  </code></pre>
  <p>This speeded the proccess of writing all the data in a CSV file (to around 20 minutes).</p>
  <p>But this is still a long time for me, since I have to do this a few times throughout the day.</p>
  <p>There are around 10.000 dataframes written to this CSV. They have only 5 columns and the number of rows varies from 0 to 75.000. The final CSV has around 3 million rows, so not a big deal for Python I guess.</p>
  <p>Is there a smarter and faster way to do what I need to do?</p>
  <p>Thanks!</p>