title: Calling any Spark MLlib function from R?
body: <p>I found this example of calling spark.mllib functions directly from Scala library. I don't get all things here, but anyway is it possible to call any MLlib function (which is not present via, let's say, spaklyr) this way? In particular I am interested in </p>
  
  <pre><code>org.apache.spark.mllib.stat.kolmogorovSmirnovTest
  </code></pre>
  
  <p>The code I found:</p>
  
  <pre><code>envir &lt;- new.env(parent = emptyenv())
  df    &lt;- spark_dataframe(x)
  sc    &lt;- spark_connection(df)
  df    &lt;- ml_prepare_features(df, features)
  tdf   &lt;- ml_prepare_dataframe(df, features, ml.options = ml.options, envir = envir)
  </code></pre>
  
  <blockquote>
    <p>sparklyr ensures that you have proper connection to spark data frame
    and prepares features in convenient form and naming convention. At the
    end it prepares a Spark DataFrame for Spark ML routines.</p>
    
    <p>You can construct a simple model calling a Spark ML class like this</p>
  </blockquote>
  
  <pre><code>envir$model &lt;- "org.apache.spark.ml.clustering.KMeans"
  kmeans      &lt;- invoke_new(sc, envir$model)
  
  model &lt;- kmeans %&gt;%
      invoke("setK", centers) %&gt;%
      invoke("setMaxIter", iter.max) %&gt;%
      invoke("setTol", tolerance) %&gt;%
      invoke("setFeaturesCol", envir$features)
  # features where set in ml_prepare_dataframe
  
  fit &lt;- model %&gt;% invoke("fit", tdf)
  # reminder: 
  # tdf   &lt;- ml_prepare_dataframe(df, features, ml.options = ml.options, envir = envir)
  </code></pre>
  
  <p><strong>Source:</strong> <a href="http://r-addict.com/DataScienceWarsaw25/show/#/preparing-spark-ml-algorithm" rel="nofollow noreferrer">http://r-addict.com/DataScienceWarsaw25/show/#/preparing-spark-ml-algorithm</a></p>