How to load large multi file parquet files for tensorflow/pytorch
I am trying to load a few parquet files from a directory into Python for tensorflow/pytorch. 

The files are too large to be loaded through the pyarrow.parquet functions

import pyarrow.parquet as pq
dataset = pq.ParquetDataset('dir')
table = dataset.read()


This gives out of memory error. 

I have also tried using petastorm, but that doesn't work for make_reader() because it isn't of the petastorm type. 

with make_batch_reader('dir') as reader:
  dataset = make_petastorm_dataset(reader)


When I used the make_batch_reader() and then the make_petastorm_dataset(reader), it again gave an zip not iterable error or something along those lines. 

I am not sure how to load the file into Python for ML training. 
Some quick help would be greatly appreciated. 

Thanks
Zash

For pyarrow, you can list the directory with Python, iterate over *.parquet files, open each one as pq.ParquetFile, and read it one row group at a time. This will alleviate the memory pressure, but won't be super fast without parallelization.

For petastorm, you are right to use make_batch_reader(). Indeed, the error messages are not always helpful; but you can inspect the stack trace and investigate where in petastorm code it originates from.
