title: How to merge two parquet files having different schema in spark (java)
body: <p>I am having 2 parquet files with different number of columns and trying to merge them with following code snippet</p>
  <pre class="lang-java prettyprint-override"><code>Dataset&lt;Row&gt; dataSetParquet1 = testSparkSession.read().option(&quot;mergeSchema&quot;,true).parquet(&quot;D:\\ABC\\abc.parquet&quot;);
                
  Dataset&lt;Row&gt; dataSetParquet2 = testSparkSession.read().option(&quot;mergeSchema&quot;,true).parquet(&quot;D:\\EFG\\efg.parquet&quot;);                    
    
  dataSetParquet1.unionByName(dataSetParquet2);
  // dataSetParquet1.union(dataSetParquet2);
  
  
  </code></pre>
  <p>for <code>unionByName()</code> I get the error:</p>
  <pre><code>Caused by: org.apache.spark.sql.AnalysisException: Cannot resolve column name
  </code></pre>
  <p>for <code>union()</code> I get the error:</p>
  <pre><code>Caused by: org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the same number of columns, but the first table has 7 columns and the second table has 6 columns;;
  </code></pre>
  <p>How do I merge these files using spark in java?</p>
  <p><strong>UPDATE :</strong> Example</p>
  <p>dataset 1:</p>
  <pre><code>epochMillis   | one | two | three| four
  --------------------------------------
  1630670242000 | 1   | 2   | 3    | 4
  1630670244000 | 1   | 2   | 3    | 4
  1630670246000 | 1   | 2   | 3    | 4
  </code></pre>
  <p>dataset2 :</p>
  <pre><code>epochMillis   | one | two | three|five
  ---------------------------------------
  1630670242000 | 11  | 22  | 33   | 55
  1630670244000 | 11  | 22  | 33   | 55
  1630670248000 | 11  | 22  | 33   | 55
  </code></pre>
  <p>Final dataset after merging:</p>
  <pre><code>epochMillis   | one | two | three|four |five
  --------------------------------------------
  1630670242000 | 11  | 22  | 33   |4    |55
  1630670244000 | 11  | 22  | 33   |4    |55
  1630670246000 | 1   | 2   | 3    |4    |null
  1630670248000 | 11  | 22  | 33   |null |55
  </code></pre>
  <p>how to obtain this result for merging two <strong>Datasets</strong>?</p>