Reading from Hive external table in Pandas or DASK
Dask has the ability to read parquet files, as you have found, so accessing them directly will be the most efficient thing you can do.
There is also the possibility of reading dataframes from SQL queries, which you could push to hive, and get it to do the extraction for you; but in this case the compute is being done by hive, and you will get very poor throughput of the results to dask.
There have been some attempts in the past for dask to directly interact with the hive metastore and achieve exactly the kind of workflow you are after, but I don't think any of those became generally released or usable
We have a Hive metatstore with object store as warehouse setup . External table is created over data present in minio. My requirement is to read data from this table in pandas or dask. Currently I am doing in a crude way by accessing the metadata of the table and extracting location of data and then reading that location to create a dataframe.
Please suggest any other way for it , which will help me support more user given queries .