title: How to read data to dask dataframe and remove bad lines
body: <p>I am attempting to aggregate a large (66gb) database with multiple lines of bad data using dask.</p>
  
  <p>As dask has no remove bad lines functionality I am first reading in all data as a pandas dataframe and removing bad lines. I am then converting this to a dask dataframe. My code is as follows:</p>
  
  <p>import dask.dataframe as dd
  import pandas as pd
  from dask.distributed import Client</p>
  
  <pre><code>#Groups the average Thresholds by NEATGeneration and finds the mean, standard deviation, minimum and maximum of the data
  def group(df):
      res = df.groupby(df["NEATGeneration"]).agg({'averageThreshold': ['mean', 'std','max','min']}).compute()
      return res
  
  
  if __name__ == '__main__':
  
     Client(n_workers=4, threads_per_worker=6,memory_limit='120GB')
  
     #Loads in the data as a pandas datframe inlcuding bad lines
     df = dd.read_csv("agentsvfitness.txt",error_bad_lines=False,usecols=["NEATGeneration","averageThreshold"])
  
     #Replaces elements in  the averageThreshold column that are not numeric with NA
     pd.to_numeric(df['averageThreshold'] , errors ='coerce') 
  
     #Removes rows with NA
     df = df.dropna()
  
     #runs the group() function in parallel
     df = group(df)
  
     #Sets all column names and prepares data for writing to csv
     df.columns = ['mean', 'std','max','min']
  
     #Writes aggregated data to a single csv file
     df.to_csv("averageThreshold.csv")
  </code></pre>
  
  <p>The issue I am running into is when the data has been incorrectly recorded in the following manner (bolded):</p>
  
  <blockquote>
    <p><strong>NEATGeneration,averageFitness,averageResourcesConsumed,averageThreshold</strong></p>
    
    <p>0,8.32,0.8533333333333334,0.4819999999999999</p>
    
    <p><strong>0,8.486666666666666,1.7266666666666666.47333333333333333 #lacking " ,0 "</strong></p>
    
    <p><strong>0,8.0533333333333331.8466666666666667,0.4500000000000001 #lacking a " , "</strong></p>
    
    <p>0,8.306666666666667,1.9466666666666668,0.44933131583851454</p>
  </blockquote>
  
  <p>My current method is unable to remove these lines when reading in the data to a dask dataframe. Is there some way of removing these bad lines from an already existing dataframe? Else is there a way to read in only 'good' data (data with the correct number of datapoints)? I am running on a cluster with 24 CPUs and 120GB of memory.</p>