title: pandas chunksize - concat inside and outside the loops
body: <p>I have to read massive csv files (500 million lines), and I tried to read them with pandas using the chunksize method, in order to reduce memory consumption. But I didn't understand the behaviour of the concat method and the option to read all the file and reduce memory. I'm adding some pseudocode in order to explain what I did so far.</p>
  
  <p>Let's say I'm reading and then concatenate a file with n lines with:</p>
  
  <pre><code>iter_csv = pd.read_csv(file.csv,chunksize=n/2)
  df = pd.concat([chunk for chunk in iter_csv])
  </code></pre>
  
  <p>Then I have to apply a function to the dataframe to create a new column based on some values:</p>
  
  <pre><code>df['newcl'] = df.apply(function)
  </code></pre>
  
  <p>Everything goes fine.
  But now I wonder what's the difference between the above procedure and the following:</p>
  
  <pre><code>iter_csv = pd.read_csv(file.csv,chunksize=n/2)
  for chunk in iter_csv:
      chunk['newcl'] = chunk.apply(function)
      df = pd.concat([chunk])
  </code></pre>
  
  <p>In terms of RAM consumption, I thought that the second method should be better because it applies the function only to the chunk and not to the whole dataframe. But the following issues occur:</p>
  
  <ol>
  <li>putting the <code>df = pd.concat([chunk])</code> inside the loop returns me a dataframe with a size of n/2 (the size of the chunk), and not the full one;</li>
  <li>putting the <code>df = pd.concat([chunk])</code> outside, after the loop returns the same n/2 dataframe length.</li>
  </ol>
  
  <p>So my doubt is whether the first method (concatenate the dataframe just after the <code>read_csv</code> function) is the best one, balancing speed and RAM consumption. And I'm also wondering how may I concat the chunks using the for loop. </p>
  
  <p>Thanks for your support.</p>