title: Spark- Ambiguous Column Error
body: <p>I'm trying to build some skills in Spark Sql, mainly in DataFrames. So what I'm trying to do is</p>
  
  <ul>
  <li>join two DataFrames</li>
  <li>apply a udf that takes a tuple of columns</li>
  <li>separate the resulting tuple into two columns</li>
  </ul>
  
  <p>The code I'm using is</p>
  
  <pre><code>import org.apache.spark.sql.functions.udf
  
  val profDF = Seq((1,"James","Detective"),
                   (2,"Harvey","Captain"),
                   (3,"Barbara","Club Owner")).toDF("ID", "Name", "Occ")
  val persDF = Seq((1, 30, "Single"),
                   (2, 35, "Married"),
                   (3, 30, "Single")).toDF("ID", "Age", "Status")
  
  val upperAdd2:(String, Int)=&gt;(String, Int) = (name, age) =&gt; (name.toUpperCase, age + 2)
  
  val upAddUDF = udf(upperAdd2)
  
  profDF.join(persDF, profDF("ID")===persDF("ID"))
      .withColumn("Result", upAdUDF(profDF("Name"),persDF("Age")))
      .withColumn("Caps Name", col("Result._1"))
      .withColumn("More Age", col("Result._2"))
      .drop("Result")
      .show
  </code></pre>
  
  <p>I get the following error after executing the last statement</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Reference 'ID' is ambiguous, could be: ID#4, ID#12.;
  </code></pre>
  
  <p>I think I've properly specified the DataFrames for "ID" because the following statement works just fine</p>
  
  <pre><code>profDF.join(persDF, profDF("ID")===persDF("ID"))
      .withColumn("Result", upAdUDF(profDF("Name"),persDF("Age")))
      .show
  </code></pre>
  
  <p>Is there anything else I need to specify here?</p>
  
  <p>I'm using Spark 1.6.0 and the code is only for learning purposes. Please let me know if it can be improved further.</p>