title: Databricks Spark Join files with slight mismatch in columns
body: <p>I am trying to use the following type of code to load multiple files into my Dataframe. </p>
  
  <pre><code>myProductDF = spark.read.format("csv").option("header","true").option("delimiter", ",").option("escape", "#").load("/mnt/Product/*")
  </code></pre>
  
  <p>Now it so happens that some of the files have an additional column or two. Ex. <code>ProductTable</code> has <code>ProductTableX.csv</code> with 10 columns and <code>ProductTableY.csv</code> with 11 columns. When I try to load all the files in the Product folder into my dataframe using wildcard * instead of specifying the individual file names, it is skipping the additional column in the files that have this. Since there are 1000's of files to be loaded its not feasible to pick and choose specific flavors. Do let me know if there is a good solution for this scenario. I will be good to go even if it defaults to NULL for the files where the extra column is not available, as long as it loads the extra column into the dataframe with values for the files where it is present.</p>