title: fastest way to load data to a pd.DataFrame using different formats (csv, json, avro)
body: <p>We are loading big amounts of data from <code>google bigquery</code> to <code>pandas dataframe</code> (to be consumed directly as a <code>pandas</code> and also to be consumed as a <code>xgbMatrix</code>). </p>
  
  <p>BQ export formats are <code>CSV</code>, <code>JSON</code> and <code>AVRO</code>, our data has dates, integers, floats and strings, and is usually "wide" (many columns). Our first approach was import the data as CSV, but the parsing time is huge:</p>
  
  <p><code>(32 GB,126 files,CSV) -&gt; 25 min</code></p>
  
  <p>parsing code:</p>
  
  <pre><code>def load_table_files_to_pandas(all_files, 
                             table_ref):
  
  # load files to pandas
  dict_dtype = {}
  date_cols = []
  
  client =  bigquery.Client() # create a bq client
  table = client.get_table(table_ref)
  
  for field in table.schema:
      pd_dtypes = {'string':'object',
                   'date':'object',
                   'float':'float64',
                   'integer':'float64'
                   }
      dict_dtype[field.name] = pd_dtypes[field.field_type.lower()]
      if field.field_type.lower()== 'date':
          date_cols.append(field.name)
  
  print('start reading data')    
  df_from_each_file = []
  for f in all_files:
      # looping over files
      df_from_each_file.append(pd.read_csv(f, 
                                           dtype = dict_dtype, 
                                           parse_dates = date_cols))
  
      print('memory in use = {}'.format(psutil.virtual_memory().percent))
  
  df = pd.concat(df_from_each_file, ignore_index=True)
  print('end reading data')
  return df
  </code></pre>
  
  <p>Which format is a faster for parsing in <code>pandas</code>? <code>[Avro,CSV,JSON]</code>? Is there maybe a third one not considered? </p>
  
  <p><strong>additional</strong> 
  we also tried with <code>dask|csv</code> directly from storage and from local disk, but the parsing time was almost the same.</p>