title: Read all partitioned parquet files in PySpark
body: <p>I want to load all parquet files that are stored in a folder structure in S3 AWS.</p>
  
  <p>The folder structure is as follows: <code>S3/bucket_name/folder_1/folder_2/folder_3/year=2019/month/day</code></p>
  
  <p>What I want is to read all parquet files at once, so I want PySpark to read all data from 2019 for all months and days that are available and then store it in one dataframe (so you get a concatenated/unioned dataframe with all days in 2019).</p>
  
  <p>I am told that these are partitioned files (though I am not sure of this).</p>
  
  <p>Is this possible in PySpark and if so how?</p>
  
  <p>When I try  <code>spark.read.parquet('S3/bucket_name/folder_1/folder_2/folder_3/year=2019')</code>
  it works. However, when I want to take a look at the Spark dataframe using <code>spark.read.parquet('S3/bucket_name/folder_1/folder_2/folder_3/year=2019').show()</code> </p>
  
  <p>it says:</p>
  
  <pre><code>An error occurred while calling o934.showString.
  : org.apache.spark.SparkException: 
  Job aborted due to stage failure: Task 0 in stage 36.0 failed 4 times, 
  most recent failure: 
  Lost task 0.3 in stage 36.0 (TID 718, executor 7): 
  java.lang.UnsupportedOperationException: 
  org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary 
  at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:44)
          at org.apache.spark.sql.execution.datasources.parquet.ParquetDictionary.decodeToBinary(ParquetDictionary.java:51)
          at org.apache.spark.sql.execution.vectorized.WritableColumnVector.getUTF8String(WritableColumnVector.java:372)
          at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
          at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
          at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:253)
          at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
          at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:830)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
          at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
          at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
          at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
          at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
          at org.apache.spark.scheduler.Task.run(Task.scala:109)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  
  <p>I want to be able to show the dataframe.</p>