title: Sparklyr's spark_apply function seems to run on single executor and fails on moderately-large dataset
body: <p>I am trying to use <code>spark_apply</code> to run the R function below on a Spark table. This works fine if my input table is small (e.g. 5,000 rows), but after ~30 mins throws an error when the table is moderately large (e.g. 5,000,000 rows):
  <code>sparklyr worker rscript failure, check worker logs for details</code></p>
  
  <p>Looking at the Spark UI shows that there is only ever a single task being created, and a single executor being applied to this task.</p>
  
  <p><strong>Can anyone give advice on why this function is failing for 5 million row dataset?</strong>  Could the problem be that a single executor is being made to do all the work, and failing?</p>
  
  <pre><code># Create data and copy to Spark
  testdf &lt;- data.frame(string_id=rep(letters[1:5], times=1000), # 5000 row table
                   string_categories=rep(c("", "1", "2 3", "4 5 6", "7"), times=1000))
  testtbl &lt;- sdf_copy_to(sc, testdf, overwrite=TRUE, repartition=100L, memory=TRUE)
  
  # Write function to return dataframe with strings split out
  myFunction &lt;- function(inputdf){
    inputdf$string_categories &lt;- as.character(inputdf$string_categories)
    inputdf$string_categories=with(inputdf, ifelse(string_categories=="", "blank", string_categories))
    stringCategoriesList &lt;- strsplit(inputdf$string_categories, ' ')
    outDF &lt;- data.frame(string_id=rep(inputdf$string_id, times=unlist(lapply(stringCategoriesList, length))),
                    string_categories=unlist(stringCategoriesList))
   return(outDF)
  }
  
  # Use spark_apply to run function in Spark
  outtbl &lt;- testtbl %&gt;%
    spark_apply(myFunction,
            names=c('string_id', 'string_categories'))
  outtbl
  </code></pre>