title: How to read a list of parquet files from S3 as a pandas dataframe using pyarrow?
body: <p>I have a hacky way of achieving this using <code>boto3</code> (1.4.4), <code>pyarrow</code> (0.4.1) and <code>pandas</code> (0.20.3).</p>
  
  <p>First, I can read a single parquet file locally like this:</p>
  
  <pre><code>import pyarrow.parquet as pq
  
  path = 'parquet/part-r-00000-1e638be4-e31f-498a-a359-47d017a0059c.gz.parquet'
  table = pq.read_table(path)
  df = table.to_pandas()
  </code></pre>
  
  <p>I can also read a directory of parquet files locally like this:</p>
  
  <pre><code>import pyarrow.parquet as pq
  
  dataset = pq.ParquetDataset('parquet/')
  table = dataset.read()
  df = table.to_pandas()
  </code></pre>
  
  <p>Both work like a charm. Now I want to achieve the same remotely with files stored in a S3 bucket. I was hoping that something like this would work:</p>
  
  <pre><code>dataset = pq.ParquetDataset('s3n://dsn/to/my/bucket')
  </code></pre>
  
  <p>But it does not:</p>
  
  <p><code>OSError: Passed non-file path: s3n://dsn/to/my/bucket</code></p>
  
  <p>After reading <a href="https://arrow.apache.org/docs/python/" rel="noreferrer">pyarrow's documentation</a> thoroughly, this does not seem possible <a href="http://markmail.org/message/uo4kvzrt6ce6yruw" rel="noreferrer">at the moment</a>. So I came out with the following solution:</p>
  
  <p>Reading a single file from S3 and getting a pandas dataframe:</p>
  
  <pre><code>import io
  import boto3
  import pyarrow.parquet as pq
  
  buffer = io.BytesIO()
  s3 = boto3.resource('s3')
  s3_object = s3.Object('bucket-name', 'key/to/parquet/file.gz.parquet')
  s3_object.download_fileobj(buffer)
  table = pq.read_table(buffer)
  df = table.to_pandas()
  </code></pre>
  
  <p>And here my hacky, not-so-optimized, solution to create a pandas dataframe from a S3 folder path:</p>
  
  <pre><code>import io
  import boto3
  import pandas as pd
  import pyarrow.parquet as pq
  
  bucket_name = 'bucket-name'
  def download_s3_parquet_file(s3, bucket, key):
      buffer = io.BytesIO()
      s3.Object(bucket, key).download_fileobj(buffer)
      return buffer
  
  client = boto3.client('s3')
  s3 = boto3.resource('s3')
  objects_dict = client.list_objects_v2(Bucket=bucket_name, Prefix='my/folder/prefix')
  s3_keys = [item['Key'] for item in objects_dict['Contents'] if item['Key'].endswith('.parquet')]
  buffers = [download_s3_parquet_file(s3, bucket_name, key) for key in s3_keys]
  dfs = [pq.read_table(buffer).to_pandas() for buffer in buffers]
  df = pd.concat(dfs, ignore_index=True)
  </code></pre>
  
  <p>Is there a better way to achieve this? Maybe some kind of connector for pandas using pyarrow? I would like to avoid using <code>pyspark</code>, but if there is no other solution, then I would take it.</p>