title: How to convert a sklearn pipeline into a pyspark pipeline?
body: <p>We have a machine learning classifier model that we have trained with a pandas dataframe and a standard sklearn pipeline (StandardScaler, RandomForestClassifier, GridSearchCV etc). We are working on Databricks and would like to scale up this pipeline to a large dataset using the parallel computation spark offers.</p>
  <p>What is the quickest way to <strong>convert our sklearn pipeline into something that computes in parallel</strong>? (We can easily switch between pandas and spark DFs as required.)</p>
  <p>For context, our options seem to be:</p>
  <ol>
  <li>Rewrite the pipeline using MLLib (time-consuming)</li>
  <li>Use a sklearn-spark bridging library</li>
  </ol>
  <p>On option 2, Spark-Sklearn seems to be <a href="https://github.com/databricks/spark-sklearn" rel="noreferrer">deprecated</a>, but Databricks instead <a href="https://github.com/databricks/spark-sklearn" rel="noreferrer">recommends</a> that we use joblibspark. However, this raises an exception on Databricks:</p>
  <pre><code>from sklearn import svm, datasets
  from sklearn.model_selection import GridSearchCV
  from joblibspark import register_spark
  from sklearn.utils import parallel_backend
  register_spark() # register spark backend
  
  iris = datasets.load_iris()
  parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
  svr = svm.SVC(gamma='auto')
  
  clf = GridSearchCV(svr, parameters, cv=5)
  with parallel_backend('spark', n_jobs=3):
      clf.fit(iris.data, iris.target)
  </code></pre>
  <p>raises</p>
  <pre><code>py4j.security.Py4JSecurityException: Method public int org.apache.spark.SparkContext.maxNumConcurrentTasks() is not whitelisted on class class org.apache.spark.SparkContext
  </code></pre>