title: Creating dummy variables in sparklyr?
body: <p>I'm trying to scale up some of my ML pipelines and i love the combination of rstudio, spark, and h2o that sparklyr opens up (<a href="http://spark.rstudio.com/" rel="nofollow noreferrer">http://spark.rstudio.com/</a>)</p>
  
  <p>One thing i'm trying to figure out is how to use sparklyr to create dummy cols from a character vector in a data frame. </p>
  
  <p>I've tried the below but i think it might be mixing some functions that sparklyr has not yet implemented. </p>
  
  <pre><code>library(sparklyr)
  library(dplyr)
  
  sc &lt;- spark_connect(master = "local")
  
  flights_tbl &lt;- copy_to(sc, nycflights13::flights, "flights")
  
  flights_tbl %&gt;% bind_cols(as_data_frame(setNames(lapply(unique(flights_tbl$carrier), 
                                                 function(x){as.integer(flights_tbl$carrier == x)}), 
                                          paste0('var2_', levels(flights_tbl$carrier)))))
  </code></pre>
  
  <p>In this example i'd like to create indicator variables for each carrier. </p>
  
  <p>To be honest i'm not really familiar with dplyr so not sure how i'd do this. </p>
  
  <p>Is there maybe a spark function that already exits that i might be able to call via a sparklyr extension or is there a way to do this with just dplyr functions?</p>
  
  <p><strong>Update</strong></p>
  
  <p>I think i have a way to do this in pyspark on a dataframe (in case useful for anyone). </p>
  
  <p>So if data looks like:</p>
  
  <pre><code>&gt;&gt;&gt; df.show(20)
  +------------+----------+------+-----------+
  |        word|word_count|corpus|corpus_date|
  +------------+----------+------+-----------+
  |  profession|         1|hamlet|       1600|
  |      augury|         1|hamlet|       1600|
  |undertakings|         1|hamlet|       1600|
  |     surmise|         1|hamlet|       1600|
  |    religion|         1|hamlet|       1600|
  |    advanced|         1|hamlet|       1600|
  |    Wormwood|         1|hamlet|       1600|
  |   parchment|         1|hamlet|       1600|
  |     villany|         1|hamlet|       1600|
  |        digs|         1|hamlet|       1600|
  | authorities|         1|hamlet|       1600|
  |      Popp'd|         1|hamlet|       1600|
  |  retrograde|         1|hamlet|       1600|
  |       tax'd|         1|hamlet|       1600|
  |        tomb|         1|hamlet|       1600|
  |       moral|         1|hamlet|       1600|
  | foreknowing|         1|hamlet|       1600|
  |  dreadfully|         1|hamlet|       1600|
  |      parted|         1|hamlet|       1600|
  |      willow|         1|hamlet|       1600|
  +------------+----------+------+-----------+
  </code></pre>
  
  <p>Then the following seems to work for making dummies on the [word] field:</p>
  
  <pre><code>categories = df.select("word").distinct().rdd.flatMap(lambda x: x).collect()
  
  exprs = [F.when(F.col("word") == category, 1).otherwise(0).alias(category) for category in categories]
  
  df_dummies = df.select("word", *exprs).limit(1000)
  </code></pre>
  
  <p>taken from (<a href="https://stackoverflow.com/questions/35879372/pyspark-matrix-with-dummy-variables">pyspark matrix with dummy variables</a>)</p>