title: Jupyter kernel dying when reading parquet files from multiple folders into sagemaker
body: <p>I am trying to read a very large amount of data from s3 parquet files into my SageMaker notebook instance. I am not sure how much data is too much data for the jupyter notebook to handle, so when I try a few files at once by providing a folder with many parquet files at once, the kernel dies. Is there anyway to proceed, given that I need to load even more data to train ML models later?</p>
  
  <p>So far I tried a couple of different methods suggested here in StackOverflow, but since I am extremely new to AWS environment it has been hard to work things out by myself. Right now, the following code is what I am running:</p>
  
  <pre class="lang-py prettyprint-override"><code>
  import s3fs
  import pandas as pd
  import boto3
  import pyarrow.parquet as pq
  
  # Creating an S3 Filesystem (Only required when using S3)
  
  s3 = s3fs.S3FileSystem()
  s3_path = "s3://my_bucket_name"
  directory = 'path/to/folder'
  print(f'{s3_path}/{directory}')
  
  # Loading Files (S3)
  
  data = pq.ParquetDataset(f'{s3_path}/{directory}', filesystem = s3, validate_schema=False).read_pandas().to_pandas()
  
  </code></pre>
  
  <p>I had a bunch of errors trying to read parquet files like these but now it seems to be working when I try to read just one file. How should I proceed?</p>