title: SparkR dataframe with R lists as elements
body: <p>I have an R data.frame where one of the columns contains integer <strong>lists</strong> -- i.e., each of its elements embeds a whole list of integers. I'd like to convert this data.frame to a SparkR DataFrame, though SparkR is returning an error.</p>
  
  <p>Here's how to create an example R data.frame that gives the issue:</p>
  
  <pre><code>indices &lt;- 1:4
  myDf &lt;- data.frame(indices)
  myDf$data &lt;- list(rep(0, 20))
  </code></pre>
  
  <p>Looking at <code>myDf</code> gives the following (as I'd expect):</p>
  
  <pre><code>&gt; str(myDf) 
  'data.frame':   4 obs. of  2 variables:  
   $ indices: int  1 2 3 4  
   $ data   :List of 4
     ..$ : num  0 0 0 0 0 0 0 0 0 0 ...
     ..$ : num  0 0 0 0 0 0 0 0 0 0 ...
     ..$ : num  0 0 0 0 0 0 0 0 0 0 ...
     ..$ : num  0 0 0 0 0 0 0 0 0 0 ...
  
  &gt; head(myDf)   
    indices                                                       data 
  1       1 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 
  2       2 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 
  3       3 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 
  4       4 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
  </code></pre>
  
  <p>And creating a SparkR DataFrame from it actually does execute without error:</p>
  
  <pre><code>library(SparkR, lib.loc=paste0(Sys.getenv("SPARK_HOME"),"/R/lib"))
  sparkR.session(master = "local[*]")
  
  mySparkDf &lt;- as.DataFrame(myDf)
  </code></pre>
  
  <p>However, subsequent operations on <code>mySparkDf</code> don't seem to work:</p>
  
  <pre><code>&gt; collect(mySparkDf)
  17/07/13 17:23:00 ERROR executor.Executor: Exception in task 0.0 in stage 1.0 (TID 1)
  java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: 
  java.lang.Double is not a valid external type for schema of array&lt;double&gt;
  if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null 
  else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, indices), IntegerType) AS indices#0
  ... long stack trace ...
  </code></pre>
  
  <p>According to the Spark <a href="https://spark.apache.org/docs/2.1.0/sql-programming-guide.html#data-types" rel="nofollow noreferrer">documentation</a>, lists should in fact be allowed? Even looking at the schema for the Spark DataFrame shows that the list is (successfully) stored as an ArrayType:</p>
  
  <pre><code>&gt; schema(mySparkDf)
  StructType
  |-name = "indices", type = "IntegerType", nullable = TRUE
  |-name = "data", type = "ArrayType(DoubleType,true)", nullable = TRUE
  </code></pre>
  
  <p>Using Spark 2.2.0, R 3.4.0, Java 1.8.0_131.</p>
  
  <p>Any ideas? Thanks in advance for the help!</p>