title: Read Parquet file stored in S3 with AWS Lambda (Python 3)
body: <p>I am trying to load, process and write Parquet files in S3 with AWS Lambda. My testing / deployment process is:  </p>
  
  <ul>
  <li><a href="https://github.com/lambci/docker-lambda" rel="noreferrer">https://github.com/lambci/docker-lambda</a> as a container to mock the Amazon environment, because of the native libraries that need to be installed (numpy amongst others).</li>
  <li>This procedure to generate a zip file: <a href="http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python" rel="noreferrer">http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example-deployment-pkg.html#with-s3-example-deployment-pkg-python</a></li>
  <li>Add a test python function to the zip, send it to S3, update the lambda and test it</li>
  </ul>
  
  <p>It seems that there are two possible approaches, <strong>which both work locally to the docker container</strong>:  </p>
  
  <ol>
  <li>fastparquet with s3fs: Unfortunately the unzipped size of the package is bigger than 256MB and therefore I can't update the Lambda code with it.</li>
  <li><p>pyarrow with s3fs: I followed <a href="https://github.com/apache/arrow/pull/916" rel="noreferrer">https://github.com/apache/arrow/pull/916</a> and when executed with the lambda function I get either:</p>
  
  <ul>
  <li><em>If I prefix the URI with S3 or S3N</em> (as in the code example): In the Lambda environment <code>OSError: Passed non-file path: s3://mybucket/path/to/myfile</code> in pyarrow/parquet.py, line 848. Locally I get <code>IndexError: list index out of range</code> in pyarrow/parquet.py, line 714</li>
  <li><em>If I don't prefix the URI with S3 or S3N</em>: It works locally (I can read the parquet data). In the Lambda environment, I get the same <code>OSError: Passed non-file path: s3://mybucket/path/to/myfile</code> in pyarrow/parquet.py, line 848.  </li>
  </ul></li>
  </ol>
  
  <p>My questions are  :</p>
  
  <ul>
  <li>why do I get a different result in my docker container than I do in the Lambda environment?</li>
  <li>what is the proper way to give the URI?</li>
  <li>is there an accepted way to read Parquet files in S3 through AWS Lambda?  </li>
  </ul>
  
  <p>Thanks!</p>