title: Efficiently read in large csv files using pandas or dask in python
body: <p>I want to read in large csv files into python in the fastest way possible. I have a csv file of ~100 million rows. I came across this primer <a href="https://medium.com/casual-inference/the-most-time-efficient-ways-to-import-csv-data-in-python-cc159b44063d" rel="nofollow noreferrer">https://medium.com/casual-inference/the-most-time-efficient-ways-to-import-csv-data-in-python-cc159b44063d</a> and they go through a few packages</p>
  
  <ol>
  <li>csv</li>
  <li>pandas</li>
  <li>dask</li>
  <li>datatable</li>
  <li>paratext</li>
  </ol>
  
  <p>For my purposes, "csv" is too raw and I want to leverage the type inference included in the other packages. I need it to work on both windows and linux machines and have also looked into datatable and paratext, but have had problems installing right package dependencies (neither are on the anaconda package repo). So that leaves pandas and dask. At first glance, dask seems much faster, but I only realized that it only does the computations if you call ".compute"</p>
  
  <p>My specific use case is that even though the raw csv file is 100+ million rows, I only need a subset of it loaded into memory. For example, all rows with date >= T. Is there a more efficient way of doing this than just the below example? Both pandas and dask take similar time.</p>
  
  <p><strong>EDIT: The csv file updates daily and there is no pre-known order of the rows of the file. Ie it is not necessarily the case that the most recent dates are at the end of the file</strong></p>
  
  <pre><code>import pandas as pd
  import dask as dd
  from datetime import datetime
  
  s = datetime.now()
  data1 = pd.read_csv("test.csv", parse_dates=["DATE"])
  data1 = data1[data1.DATE&gt;=datetime(2019,12,24)]
  print(datetime.now()-s)
  
  s = datetime.now()
  data2 = dd.read_csv("test.csv", parse_dates=["DATE"])
  data2 = data2[data2.DATE&gt;=datetime(2019,12,24)].compute()
  print(datetime.now()-s)
  </code></pre>