title: Error incompatible column type when using unionByName
body: <p>I am a newbie to Spark SQL(using Scala) and have some basic questions regarding an error I am facing.
  I am merging 2 data frames (oldData and newData) as follows</p>
  <pre><code>if (!oldData.isEmpty) {
        oldData
          .join(newData, Seq(&quot;internalUUID&quot;),&quot;left_anti&quot;)
          .unionByName(newData)
          .drop(&quot;all&quot;) //Drop records that have null in all fields
      } else {
        newData
      }
  </code></pre>
  <p>The error I see is</p>
  <pre><code>org.apache.spark.sql.AnalysisException: Union can only be performed on tables with the compatible column types. ....
   at the 8th column of the second table;;
  'Union
  :- Project [internalUUID#342, TenantID#339, ObjectName#340, DataSource#341, product#343, plant#344, isMarkedForDeletion#345, distributionProfile#346, productionAspect#347, salesPlant#348, listing#349]
  :  +- Join LeftAnti, (internalUUID#342 = internalUUID#300)
  :     :- Relation[TenantID#339,ObjectName#340,DataSource#341,internalUUID#342,product#343,plant#344,isMarkedForDeletion#345,distributionProfile#346,productionAspect#347,salesPlant#348,listing#349] parquet
  :     +- LogicalRDD [DataSource#296, ObjectName#297, TenantID#298, distributionProfile#299, internalUUID#300, isMarkedForDeletion#301, listing#302, plant#303, product#304, productionAspect#305, salesPlant#306], false
  +- Project [internalUUID#300, TenantID#298, ObjectName#297, DataSource#296, product#304, plant#303, isMarkedForDeletion#301, distributionProfile#299, productionAspect#305, salesPlant#306, listing#302]
     +- LogicalRDD [DataSource#296, ObjectName#297, TenantID#298, distributionProfile#299, internalUUID#300, isMarkedForDeletion#301, listing#302, plant#303, product#304, productionAspect#305, salesPlant#306], false
  </code></pre>
  <p>The schema structure is as follows :
  OldData</p>
  <pre><code>root
   |-- TenantID: string (nullable = true)
   |-- ObjectName: string (nullable = true)
   |-- DataSource: string (nullable = true)
   |-- internalUUID: string (nullable = true)
   |-- product: struct (nullable = true)
   |    |-- id: string (nullable = true)
   |    |-- internalRefUUID: string (nullable = true)
   |-- plant: struct (nullable = true)
   |    |-- id: string (nullable = true)
   |    |-- internalRefUUID: string (nullable = true)
   |-- isMarkedForDeletion: boolean (nullable = true)
   |-- distributionProfile: struct (nullable = true)
   |    |-- code: string (nullable = true)
   |    |-- internalRefUUID: string (nullable = true)
   |-- productionAspect: struct (nullable = true)
   |    |-- productMovementPlants: struct (nullable = true)
   |    |    |-- unitOfIssue: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |-- productPlanningPlants: struct (nullable = true)
   |    |    |-- goodsReceiptProcessDuration: long (nullable = true)
   |    |    |-- goodsIssueProcessDuration: long (nullable = true)
   |    |    |-- mrpType: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |    |-- mrpController: struct (nullable = true)
   |    |    |    |-- id: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |    |-- sourceOfSupplyCategory: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |    |-- abcIndicator: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |-- salesPlant: struct (nullable = true)
   |    |-- loadingGroup: struct (nullable = true)
   |    |    |-- code: string (nullable = true)
   |    |    |-- internalRefUUID: string (nullable = true)
   |-- listing: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- validFrom: string (nullable = true)
   |    |    |-- validTo: string (nullable = true)
   |    |    |-- isListed: boolean (nullable = true)
  </code></pre>
  <p>and NewData</p>
  <pre><code>root
   |-- DataSource: string (nullable = true)
   |-- ObjectName: string (nullable = true)
   |-- TenantID: string (nullable = true)
   |-- distributionProfile: struct (nullable = true)
   |    |-- code: string (nullable = true)
   |    |-- internalRefUUID: string (nullable = true)
   |-- internalUUID: string (nullable = true)
   |-- isMarkedForDeletion: boolean (nullable = true)
   |-- listing: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- isListed: boolean (nullable = true)
   |    |    |-- validFrom: string (nullable = true)
   |    |    |-- validTo: string (nullable = true)
   |-- plant: struct (nullable = true)
   |    |-- id: string (nullable = true)
   |    |-- internalRefUUID: string (nullable = true)
   |-- product: struct (nullable = true)
   |    |-- id: string (nullable = true)
   |    |-- internalRefUUID: string (nullable = true)
   |-- productionAspect: struct (nullable = true)
   |    |-- productMovementPlants: struct (nullable = true)
   |    |    |-- unitOfIssue: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |-- productPlanningPlants: struct (nullable = true)
   |    |    |-- abcIndicator: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |    |-- goodsIssueProcessDuration: long (nullable = true)
   |    |    |-- goodsReceiptProcessDuration: long (nullable = true)
   |    |    |-- mrpController: struct (nullable = true)
   |    |    |    |-- id: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |    |-- mrpType: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |    |    |-- sourceOfSupplyCategory: struct (nullable = true)
   |    |    |    |-- code: string (nullable = true)
   |    |    |    |-- internalRefUUID: string (nullable = true)
   |-- salesPlant: struct (nullable = true)
   |    |-- loadingGroup: struct (nullable = true)
   |    |    |-- code: string (nullable = true)
   |    |    |-- internalRefUUID: string (nullable = true)
  </code></pre>
  <p>However I am not quite sure what does the &quot;<em>8th column of the 2nd table</em>&quot; denote? Moreover the columns are not ordered in the same way in both data frames. Is there any guidance on how to proceed on this?</p>