title: How to read hive table along with metadata in pyspark?
body: <p>I am not able to read a hive table along with its metadata using pyspark</p>
  <p>I am creating the hive table accurately I think</p>
  <p>Setup:</p>
  <pre><code>from pyspark.sql import SparkSession
  from pyspark.sql import functions as F
  spark = SparkSession.builder.enableHiveSupport().getOrCreate()
  
  data1 = [(1,2,3),(3,4,5),(5,6,7)]
  df1=spark.createDataFrame(data1,schema = 'a int,b int,c int')
  parquet_path = './bucket_test_parquet1'
  
  </code></pre>
  <p>Now, checking the tables with <code>DESCRIBE</code></p>
  <pre><code>df1.write.bucketBy(5,&quot;a&quot;).format(&quot;parquet&quot;).saveAsTable('df',path=parquet_path,mode='overwrite')
  spark.sql(&quot;DESCRIBE EXTENDED df&quot;).show(100)
  output:
  +--------------------+--------------------+-------+
  |            col_name|           data_type|comment|
  +--------------------+--------------------+-------+
  |                   a|                 int|   null|
  |                   b|                 int|   null|
  |                   c|                 int|   null|
  |                    |                    |       |
  |# Detailed Table ...|                    |       |
  |            Database|             default|       |
  |               Table|                  df|       |
  |               Owner|               nitin|       |
  |        Created Time|Tue Feb 01 09:05:...|       |
  |         Last Access|             UNKNOWN|       |
  |          Created By|         Spark 3.2.0|       |
  |                Type|            EXTERNAL|       |
  |            Provider|             parquet|       |
  |         Num Buckets|                   5|       |
  |      Bucket Columns|               [`a`]|       |
  |        Sort Columns|                  []|       |
  |            Location|file:/home/nitin/...|       |
  |       Serde Library|org.apache.hadoop...|       |
  |         InputFormat|org.apache.hadoop...|       |
  |        OutputFormat|org.apache.hadoop...|       |
  +--------------------+--------------------+-------+
  
  </code></pre>
  <pre><code>read_parquet1 = spark.read.format(&quot;parquet&quot;).load(parquet_path,header=True)
  read_parquet1.createOrReplaceTempView(&quot;rp1&quot;)
  read_parquet1 = spark.table(&quot;rp1&quot;)
  spark.sql(&quot;DESCRIBE EXTENDED rp1&quot;).show(100)
  output:
  |col_name|data_type|comment|
  +--------+---------+-------+
  |       a|      int|   null|
  |       b|      int|   null|
  |       c|      int|   null|
  +--------+---------+-------+
  </code></pre>
  <p>As you can see the metadata is not getting read in when I am reading in the table from the disk. Can you help me read the table so that I have metadata along with it?</p>