AzureML: TabularDataset.to_pandas_dataframe() hangs when parquet file is empty
You can use the on_error='null' parameter to handle the null values.
Your statement will look like this:
TabularDataset.to_pandas_dataframe(on_error='null', out_of_range_datetime='null')
Alternatively, you can check the size of the file before passing it to to_pandas_dataframe method. If the filesize is 0, either write some sample data into it using python open keyword or ignore the file, based on your requirement.
I have created a Tabular Dataset using Azure ML python API. Data under question is a bunch of parquet files (~10K parquet files each of size of 330 KB) residing in Azure Data Lake Gen 2 spread across multiple partitions. When I try to load the dataset using the API TabularDataset.to_pandas_dataframe(), it continues forever (hangs), if there are empty parquet files included in the Dataset. If the tabular dataset doesn't include those empty parquet files, TabularDataset.to_pandas_dataframe() completes within few minutes.
By empty parquet file, I mean that the if I read the individual parquet file using pandas (pd.read_parquet()), it results in an empty DF (df.empty == True).
I discovered the root cause while working on another issue mentioned [here][1].
My question is how can make TabularDataset.to_pandas_dataframe() work even when there are empty parquet files?
Update
The issue has been fixed in the following version:

azureml-dataprep : 3.0.1
azureml-core :  1.40.0