title: spark array of wrappedArray
body: <p>I have dataframe in spark that is very complex. I'm trying to use a UDF that takes 2 columns and then runs a function on each row of each column at the same time.</p>
  
  <p>each column has the following identical schema:</p>
  
  <pre><code>root
   |-- A: array (nullable = true)
   |    |-- element: double (containsNull = true)
  </code></pre>
  
  <p>In some cases, the array will be empty and in other cases, it will have elements, the count will vary.</p>
  
  <p>when I do .dtypes on a column I get:</p>
  
  <pre><code>test: Array[(String, String)] = Array((A,ArrayType(DoubleType,true)))
  </code></pre>
  
  <p>When I do a take(1) on one of the columns I get a </p>
  
  <pre><code>Array[org.apache.spark.sql.Row] = Array([WrappedArray(1234, 4567, 789, 1346)])
  </code></pre>
  
  <p>When I simply run a select on a column I get:</p>
  
  <pre><code>org.apache.spark.sql.DataFrame = [A: array&lt;double&gt;]
  </code></pre>
  
  <p>My goal is to run a function that takes each column's same element.</p>
  
  <pre><code>def inRange = udf((A: ???, B: ??? ) =&gt; {
     //iterate over the array and run coolFunction(A(0),B(0))
   })
  </code></pre>
  
  <p>I'm running the udf in this </p>
  
  <pre><code>df.withColumn("coolFunction", coolFunction(df("A"), df("B"))) 
  </code></pre>