title: What's the best way to subset a spark dataframe (in sparklyr) based on the column data type
body: <p>I'm converting a bunch of columns into dummy variables. I want to remove the original categorical variable from the dataframe. I'm struggling to figure out how to do it in sparklyr. It's straightforward in dplyr, but the dplyr functionality isn't working in sparklyr.</p>
  
  <p>For example:</p>
  
  <p>First create a spark dataframe:</p>
  
  <pre><code>    ###create dummy data to figure out how model matrix formulas work in sparklyr
  v1 &lt;- sample( LETTERS[1:4], 50000, replace=TRUE, prob=c(0.1, 0.2, 0.65, 0.05))
  v2 &lt;- sample( LETTERS[5:6], 50000, replace=TRUE, prob=c(0.7,0.3))
  v3 &lt;- sample( LETTERS[7:10], 50000, replace=TRUE, prob=c(0.3, 0.2, 0.4, 0.1))
  v4 &lt;- sample( LETTERS[11:15], 50000, replace=TRUE, prob=c(0.1, 0.1, 0.3, 0.05,.45))
  v5 &lt;- sample( LETTERS[16:17], 50000, replace=TRUE, prob=c(0.4,0.6))
  v6 &lt;- sample( LETTERS[18:21], 50000, replace=TRUE, prob=c(0.1, 0.1, 0.65, 0.15))
  v7 &lt;- sample( LETTERS[22:26], 50000, replace=TRUE, prob=c(0.1, 0.2, 0.65, 0.03,.02))
  v8 &lt;- rnorm(n=50000,mean=.5,sd=.1)
  v9 &lt;- rnorm(n=50000,mean=5,sd=3)
  v10 &lt;- rnorm(n=50000,mean=3,sd=.5)
  response &lt;- rnorm(n=50000,mean=10,sd=2)
  
  dat &lt;- data.frame(v1,v2,v3,v4,v5,v6,v7,v8,v9,v10,response)
  write.csv(dat,file='fake_dat.csv',row.names = FALSE)
  
  #push "fake_dat" to the hdfs
  
  library(dplyr)
  library(sparklyr)
  #configure the spark session and connect
  config &lt;- spark_config()
  config$`sparklyr.shell.driver-memory` &lt;- "2G" #change depending on the size of the data
  config$`sparklyr.shell.executor-memory` &lt;- "2G"
  
  # sc &lt;-  spark_connect(master='local', spark_home='/usr/hdp/2.5.0.0-1245/spark',config = config)
  # sc
  
  sc &lt;-  spark_connect(master='yarn-client', spark_home='/usr/hdp/2.5.0.0-1245/spark',config = config)
  sc
  
  #can also set spark_home as '/usr/hdp/current/spark-client'
  
  #read in the data from the hdfs
  df &lt;- spark_read_csv(sc,name='fdat',path='hdfs://pnhadoop/user/stc004/fake_dat.csv')
  
  #create spark table
  dat &lt;- tbl(sc,'fdat')
  </code></pre>
  
  <p>Now create dummy variables:</p>
  
  <pre><code>for(i in 1:7){
    dat &lt;- ml_create_dummy_variables(x=dat,colnames(dat)[i], reference = NULL)
  }
  </code></pre>
  
  <p>I could simply drop the original categorical variables using </p>
  
  <pre><code>drop.cols &lt;- colnames(dat)[1:7]
  dat1 &lt;-
    dat %&gt;%
    select(-one_of(drop.cols))
  </code></pre>
  
  <p>However, the data that I'm actually working with has 300 categorical variables. I need a quick way to identify which columns are character/factor. After converting those columns to dummy variables - I can then remove the original categorical variables.  I've tried the following:</p>
  
  <pre><code>test &lt;-
    dat %&gt;%
    select_if(is.character)
  </code></pre>
  
  <p>I then get the following error:</p>
  
  <pre><code>Error: Selection with predicate currently require local sources
  </code></pre>
  
  <p>I've also tried:</p>
  
  <pre><code>cls &lt;- sapply(dat, class)
  cls
  </code></pre>
  
  <p>But I get:</p>
  
  <pre><code>&gt; cls
  
           src         ops
      [1,] "src_spark" "op_base_remote"
      [2,] "src_sql"   "op_base"
      [3,] "src"       "op"
  </code></pre>
  
  <p>Any ideas on how to do this?</p>