title: Pandas Combine Multiple CSV's and Output as One Large File
body: <p>So I currently have a directory, we'll call it <em>/mydir</em>, that contains 36 CSV files, each 2.1 GB and with the same dimensions. They are all the same size, and I want to read them into pandas, concatenate them together side-by-side (so the amount of rows stays the same), and then output the resulting dataframe as one large csv. The code I have for this works for combining a few of them but reaches a memory error after a certain point. I was wondering if there is a more efficient way to do this than what I have.</p>
  
  <pre><code>df = pd.DataFrame()
  for file in os.listdir('/mydir'):
      df.concat([df, pd.read_csv('/mydir' + file, dtype = 'float)], axis = 1)
  df.to_csv('mydir/file.csv')
  </code></pre>
  
  <p>It was suggested to me to break it up into smaller pieces, combine the files in groups of 6, then combine these together in turn but I don't know if this is a valid solution that will avoid the memory error problem</p>
  
  <p>EDIT: view of the directory:</p>
  
  <pre><code>-rw-rw---- 1 m2762 2.1G Jul 11 10:35 2010.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:32 2001.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:28 1983.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:21 2009.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:21 1991.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:07 2000.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:06 1982.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:01 1990.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 10:01 2008.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 09:55 1999.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 09:54 1981.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 09:42 2007.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 09:42 1998.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 09:42 1989.csv
  -rw-rw---- 1 m2762 2.1G Jul 11 09:42 1980.csv
  </code></pre>