title: Why does concatenation of DataFrames get exponentially slower?
body: <p>I have a function which processes a DataFrame, largely to process data into buckets create a binary matrix of features in a particular column using <code>pd.get_dummies(df[col])</code>.</p>
  
  <p>To avoid processing all of my data using this function at once (which goes out of memory and causes iPython to crash), I have broken the large DataFrame into chunks using:</p>
  
  <pre><code>chunks = (len(df) / 10000) + 1
  df_list = np.array_split(df, chunks)
  </code></pre>
  
  <p><code>pd.get_dummies(df)</code> will automatically create new columns based on the contents of <code>df[col]</code> and these are likely to differ for each <code>df</code> in <code>df_list</code>.</p>
  
  <p>After processing, I am concatenating the DataFrames back together using:</p>
  
  <pre><code>for i, df_chunk in enumerate(df_list):
      print "chunk", i
      [x, y] = preprocess_data(df_chunk)
      super_x = pd.concat([super_x, x], axis=0)
      super_y = pd.concat([super_y, y], axis=0)
      print datetime.datetime.utcnow()
  </code></pre>
  
  <p>The processing time of the first chunk is perfectly acceptable, however, it grows per chunk! This is not to do with the <code>preprocess_data(df_chunk)</code> as there is no reason for it to increase. Is this increase in time occurring as a result of the call to <code>pd.concat()</code>?</p>
  
  <p>Please see log below:</p>
  
  <pre><code>chunks 6
  chunk 0
  2016-04-08 00:22:17.728849
  chunk 1
  2016-04-08 00:22:42.387693 
  chunk 2
  2016-04-08 00:23:43.124381
  chunk 3
  2016-04-08 00:25:30.249369
  chunk 4
  2016-04-08 00:28:11.922305
  chunk 5
  2016-04-08 00:32:00.357365
  </code></pre>
  
  <p>Is there a workaround to speed this up? I have 2900 chunks to process so any help is appreciated!</p>
  
  <p>Open to any other suggestions in Python!</p>