title: Memory error with large data sets for pandas.concat and numpy.append
body: <p>I am facing a problem where I have to generate large DataFrames in a loop (50 iterations computing every time two 2000 x 800 pandas DataFrames). I would like to keep the results in memory in a bigger DataFrame, or in a dictionary like structure.
  When using pandas.concat, I get a memory error at some point in the loop. The same happens when using numpy.append to store the results in a dictionary of numpy arrays rather than in a DataFrame. In both cases, I still have a lot of available memory (several GB). Is this too much data for pandas or numpy to process? Are there more memory-efficient ways to store my data without saving it on disk?</p>
  
  <p>As an example, the following script fails as soon as <code>nbIds</code> is greater than 376:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  nbIds = 376
  dataids = range(nbIds)
  dataCollection1 = []
  dataCollection2 = []
  for bs in range(50):
      newData1 = pd.DataFrame( np.reshape(np.random.uniform(size = 
                                                            2000 * len(dataids)), 
                                          (2000,len(dataids ))))
      dataCollection1.append( newData1 )
      newData2 = pd.DataFrame( np.reshape(np.random.uniform(size = 
                                                            2000 * len(dataids)), 
                                          (2000,len(dataids ))))
      dataCollection2.append( newData2 )
  dataCollection1 = pd.concat(dataCollection1).reset_index(drop = True)
  dataCollection2 = pd.concat(dataCollection2).reset_index(drop = True)
  </code></pre>
  
  <p>The code below fails when <code>nbIds</code> is 665 or higher</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  nbIds = 665
  dataids = range(nbIds)
  dataCollection1 = dict( (i , np.array([])) for i in dataids )
  dataCollection2 = dict( (i , np.array([])) for i in dataids )
  for bs in range(50):
      newData1 = np.reshape(np.random.uniform(size = 2000 * len(dataids)), 
                           (2000,len(dataids )))
      newData1 = pd.DataFrame(newData1)
      newData2 = np.reshape(np.random.uniform(size = 2000 * len(dataids)), 
                           (2000,len(dataids)))
      newData2 = pd.DataFrame(newData2)
      for i in dataids :
          dataCollection1[i] = np.append(dataCollection1[i] , 
                                         np.array(newData1[i]))
          dataCollection2[i] = np.append(dataCollection2[i] , 
                                         np.array(newData2[i]))
  </code></pre>
  
  <p>I do need to compute both DataFrames everytime, and for each element <code>i</code> of <code>dataids</code> I need to obtain a pandas Series or a numpy array containing the 50 * 2000 numbers generated for <code>i</code>. Ideally, I need to be able to run this with <code>nbIds</code> equal to 800 or more.
  Is there a straightforward way of doing this? </p>
  
  <p>I am using a 32-bit Python with  Python 2.7.5, pandas 0.12.0 and numpy 1.7.1.</p>
  
  <p>Thank you very much for your help!</p>