title: Read large dataset Pandas
body: <p>I'm trying to read a dataset of 20gb. I've searched for a solution,I've tried:</p>
  
  <pre><code>   data = pd.read_csv('dataset.csv', chunksize=1000,usecols=fields)
     df = pd.concat(data, ignore_index=True)
  </code></pre>
  
  <p>but still getting a memory error when passing to concatenate. (I changed chunksize many time, still the same)</p>
  
  <p>I have 16gb of RAM working at 3000mhz.</p>
  
  <p>Any suggestions?</p>
  
  <p>I am trying to import the data into a dataframe for a Data Analysis and manipulation the export it back. (Data need to be cleaned from nans and noisy data ).</p>