title: Process table by row in sparklyr or sparkR
body: <p>I loaded a gene expression matrix (33000 rows x 180 samples) in sparklyr and I want to process the probes (rows) instead of the samples (columns).</p>
  
  <pre><code>library(sparklyr) 
  sc &lt;- spark_connect(master = "local")
  library(dplyr)
  ge &lt;- read.delim("plier-gcbg-sketch.summary.txt", 
      sep="\t", comment.char="#")
  ge_tbl &lt;- copy_to(sc, ge)
  </code></pre>
  
  <p>My original idea was to apply:</p>
  
  <pre><code>mean_by_gene &lt;- ge %&gt;% select(-probeset_id) %&gt;% 
     rowwise() %&gt;% do( data.frame(M=mean(as.numeric(.))) )
  </code></pre>
  
  <p>But it seems that rowwise is not available in sparklyr. So, after some googling I applied:</p>
  
  <pre><code>ge_tbl2 %&gt;% select(-probeset_id) %&gt;% 
      spark_apply(function(df) { 
          data.frame(apply(df, 1, function(x) mean(as.numeric(x)))) 
      } )
  </code></pre>
  
  <p>This works fine since it generates the data.frame with the mean per row. But I have some doubts about it.</p>
  
  <p><strong>Questions:</strong></p>
  
  <ol>
  <li>When I test for time (using system.time) the sparklyr is slower than the standard R version (sparklyr: 0.747s, 0.702s, 0.731s; standard R: 0.009, 0.008, 0.008). Why? (For the test I only used 10 probes (rows)).</li>
  <li>When I try to compute the full matrix (33000 rows x 180 columns) instead of a small number of columns (10~100) spark crashes. I assume that I am not processing properly the table, so how I can do this to take advantage of the spark capabilities?</li>
  </ol>
  
  <p>This is a fragment of the error I obtain:</p>
  
  <pre><code>[...]
  18/06/13 13:44:21 INFO sparklyr: RScript (4377) found 33297 rows
  18/06/13 13:46:08 INFO sparklyr: RScript (4377) retrieved 33297 rows
  18/06/13 13:46:19 INFO sparklyr: RScript (4377) computing closure
  18/06/13 13:46:51 ERROR sparklyr: RScript (4377) terminated unexpectedly: invalid subscript type 'list'
  18/06/13 13:46:51 ERROR sparklyr: Worker (4377) failed to complete R process
  [...]
  </code></pre>