title: Loading very large CSV dataset into Python and R, Pandas struggles
body: <p>I am loading a huge csv (18GB) into memory and noticing very large differences between R and Python. This is on an AWS ec2 <a href="https://aws.amazon.com/blogs/aws/new-next-generation-r4-memory-optimized-ec2-instances/" rel="noreferrer">r4.8xlarge which has 244 Gb of memory</a>. Obviously this is an extreme example, but the principle holds for smaller files on real machines too.</p>
  
  <p>When using <code>pd.read_csv</code> my file took ~30 mins to load and took up 174Gb of memory. Essentially so much that I then can't do anything with it. By contrast, R's <code>fread()</code> from the <code>data.table</code> package took ~7 mins and only ~55Gb of memory.</p>
  
  <p>Why does the pandas object take up so much more memory than the data.table object? Furthermore, why fundamentally is the pandas object almost 10x larger than the text file on disk? It's not like .csv is a particularly efficient way to store data in the first place.</p>