title: How to Ignore Empty Parquet Files in Databricks When Creating Dataframe
body: <p>I am having issues loading multiple files into a dataframe in Databricks.  When I load a parquet file in an individual folder, it is fine, but the following error returns when I try to load multiple files in the dataframe: </p>
  
  <pre><code>DF = spark.read.parquet('S3 path/') 
  </code></pre>
  
  <blockquote>
    <p>"org.apache.spark.sql.AnalysisException: Unable to infer schema for Parquet. It must be specified manually."</p>
  </blockquote>
  
  <p>Per other StackOverflow answers, I added <code>spark.sql.files.ignoreCorruptFiles</code> true to the cluster configuration but it didn't seem to resolve the issue.  Any other ideas?  </p>