How to check total allotted space inside a HDFS 'group' Our DBA has created a schema for our team in HDFS/HIVE. Not sure if 'schema' is the right word, they call it a 'group'.
Anyway, we can only write to the data lake inside this schema, whether it is parquet files or hive tables.
Is there a way to check what is the max space allocated to our group , knowing only the schema name?
I dont want to accidentally load too much data.
Thank you.
It is not possible to set space quotas on Hive level only because Hive is quite detached from HDFS storage. In Hive you can create tables not only in the hive.metastore.warehouse.dir but also can have external tables in other directories in HDFS. External tables can be being loaded by not only hive, even Hive-managed tables can be loaded by some other tools without Hive(for example you can put files manually into table directory, does not matter Managed or External). Also you can use HDFS not only for Hive. So, it is not possible to set space quotas in Hive and even if it was possible, it is not practical, Hive is not aware what else you are doing in HDFS.
Read about HDFS space quotas. It is possible to set space quotas for files inside directory. Also it is possible to set permissions for users (groups) allowing them to access some directories. HDFS directory ACL + directory space quota can be combined to restrict users or groups to use only allowed directories (whith space quotas).
You can check directory quota using
hadoop fs -count -q /path/to/directory
To check directory ACL use this command:
hdfs dfs -getfacl [-R] &lt;path&gt;
See more in FS shell commands guide.
Read also about Hive Authorization Options