title: How to speed up pandas dataframe creation from huge file?
body: <p>I have a file bigger than 7GB. I am trying to place it into a dataframe using pandas, like this:</p>
  <pre><code>df = pd.read_csv('data.csv') 
  </code></pre>
  <p>But it takes too long. Is there a better way to speed up the dataframe creation? I was considering changing the parameter <code>engine='c'</code>, since it says in the <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html" rel="nofollow noreferrer">documentation</a>:</p>
  <pre><code>&quot;engine{‘c’, ‘python’}, optional
  Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.&quot;
  </code></pre>
  <p>But I dont see much gain in speed</p>