title: How can two PySpark DataFrames with the same schema raise "AnalysisException: Union can only be performed on tables with the compatible column types"?
body: <p>I have two Spark DataFrames with very large, complex, nested schemas. I want to perform a <code>unionAll</code>, but I get the exception:</p>
  <blockquote>
  <p>AnalysisException: Union can only be performed on tables with the
  compatible column types. struct
  at the second column of the second table;</p>
  </blockquote>
  <p>The second column happens to be the most complex column in the DataFrames. I understand that the schemas need to be identical and the columns must be in the same order to perform this operation (see posts <a href="https://stackoverflow.com/questions/47524325/union-can-only-be-performed-on-tables-with-the-compatible-column-types-spark-dat">here</a>, <a href="https://stackoverflow.com/questions/56942058/spark-union-column-order/56942333">here</a>, and <a href="https://stackoverflow.com/questions/52186226/spark-union-can-only-be-performed-on-tables-with-the-compatible-column-types">here</a>). I tried visual inspection, but I am unable to find a difference between the schemas. I also tried the programatic technique demonstrated in <a href="https://stackoverflow.com/questions/48673247/comparing-schema-of-dataframe-using-pyspark?rq=1">this post</a>, but the function returns zero results (no difference in schema):</p>
  <pre><code>def schema_diff(spark, df_1, df_2):
      s1 = spark.createDataFrame(df_1.dtypes, [&quot;d1_name&quot;, &quot;d1_type&quot;])
      s2 = spark.createDataFrame(df_2.dtypes, [&quot;d2_name&quot;, &quot;d2_type&quot;])
      difference = (
          s1.join(s2, s1.d1_name == s2.d2_name, how=&quot;outer&quot;)
          .where(s1.d1_type.isNull() | s2.d2_type.isNull())
          .select(s1.d1_name, s1.d1_type, s2.d2_name, s2.d2_type)
          .fillna(&quot;&quot;)
      )
      return difference
  </code></pre>
  <pre><code>diff = schema_diff(spark, df_1, df_2)
  diff_count = diff.count()
  diff.show(diff_count, truncate=False)
  </code></pre>
  <pre class="lang-none prettyprint-override"><code>+-------+-------+-------+-------+
  |d1_name|d1_type|d2_name|d2_type|
  +-------+-------+-------+-------+
  +-------+-------+-------+-------+
  </code></pre>
  <pre><code>assert diff_count == 0     # True
  </code></pre>
  <p>What am I missing here?</p>