Read only specific rows of .parquet files matching criteria?
As of 0.15.0, pyarrow doesn't have this feature, but we (in the Apache Arrow project) are actively working on this and hope to include it in the next major release.
I'm working against a filesystem filled with .parquet files. One of the columns, 'id', uniquely identifies a machine. I was able to use pyspark to open all .parquet files in a certain directory path, then create a set([]) of the values from the 'id' column. I'd like to open all other rows in all other files, where the 'id' matches one of the values in the previously calculated set.
I was able to do this via pyspark, but it's quite complex and requires me to instantiate a local spark server. I'm trying to find a way to do this via pyarrow, but it seems that it's read_pandas / read methods 'filters' argument can only filter on partition data, and not arbitrary column data.
Is there a way to achieve what I'm looking for here? I can't open the entire dataset and then use Python to filter out rows where the 'id' doesn't match, because it doesn't fit in memory.